Global Statistics

All countries
222,003,590
Confirmed
Updated on 07/09/2021 7:39 am
All countries
196,892,170
Recovered
Updated on 07/09/2021 7:39 am
All countries
4,589,259
Deaths
Updated on 07/09/2021 7:39 am

Global Statistics

All countries
222,003,590
Confirmed
Updated on 07/09/2021 7:39 am
All countries
196,892,170
Recovered
Updated on 07/09/2021 7:39 am
All countries
4,589,259
Deaths
Updated on 07/09/2021 7:39 am

How to use multi-threaded processing in bash scripts – CloudSavvy IT

Aleksey Mnogosmyslov / Shutterstock.com

Multithreaded programming has always been in the interest of developers to increase application performance and optimize resource use. This guide will introduce you to the basics of Bash multithreaded encoding.

What is it multithreaded programming?

A picture is worth a thousand words, and this is valid when it comes to showing the difference between single (1) thread scheduling and multi-threaded (> 1) scheduling in Bash:

sleep 1
sleep 1 & sleep 1

(Last Command) A simple two-threaded one-line command that will run two suspend processes in parallel, one in the background

Our first single line mini script or multi-threaded scheduling setup couldn’t have been simpler; in the first line, we sleep a second using the sleep 1 I send. As far as the user is concerned, a single thread was executing a single one-second dream.

In the second line, we have two one-second suspend commands. We join them using a & separator, which not only acts as a separator between the two sleep commands, but also as a Bash prompt to start the first command in a background thread.

Normally, one would end a command using a semicolon (;). Doing so will execute the command and only then proceed to the next command that appears after the semicolon. For example, running sleep 1; sleep 1 it would take just over two seconds: exactly one second for the first command, one second for the second, and a small amount of system overhead for each of the two commands.

However, instead of ending a command with a semicolon, you can use other command terminators that Bash recognizes as &, && other ||. The && the syntax has nothing to do with multithreaded programming, it just does this; proceed with the execution of the second command only if the first command was successful. The || It is the opposite of && and it will execute the second command only if the first command failed.

Going back to multithreaded programming, using & since our command terminator will start a background process by executing the command that precedes it. It then immediately proceeds to execute the following command in the current shell while letting the background process (thread) run on its own.

In the output of the command we can see that a process is starting in the background (as indicated by [1] 445317 where 445317 is the process ID or PID of the newly started background process and [1] it is indicated that this is our first background process) and later it will be terminated (as indicated [1]+ Done sleep 1).

For an additional example of background process handling, see our article Bash Automation and Scripting Basics (Part 3). Also, the termination tricks of the bash process may be of interest.

Now let’s show that we are actually running two sleep processes at the same time:

time sleep 1; echo 'done'
time $(sleep 1 & sleep 1); echo 'done'

Run two suspend threads in parallel, with one in the background, using a sublayer

Here we start our sleep low process time and we can see how our single threaded command ran for exactly 1,003 seconds before the command line prompt was returned.

However, in the second example, it took roughly the same time (1.005 seconds) even though we were running two sleep periods (and processes), though not consecutively. Again, we used a background process for the first suspend command, which led to a (semi) parallel execution, that is, multithreaded.

We also use a sublayer container ($(...)) around our two sleep commands to combine them under time. How can we see our done the output is displayed in 1,005 seconds and therefore the two sleep 1 the commands must have been executed simultaneously. Interesting is the very small increase in total processing time (0.002 seconds) which can easily be explained by the time required to start a sublayer and the time required to start a background process.

Managing multi-threaded (and background) processes

In Bash, multi-threaded encoding will typically involve background threads from a main one-line script or a full Bash script. In essence, one can think of multi-threaded encoding in Bash as starting multiple background threads. When one starts coding using multiple threads, it quickly becomes clear that such threads will generally require some handling. For example, take the fictitious example where we start five concurrent sleep periods (and processes) in a bash script;

#!/bin/bash

sleep 10 & 
sleep 600 & 
sleep 1200 & 
sleep 1800 & 
sleep 3600 &

Running five parallel suspend threads in the background from a script

When we start the script (after making it executable using chmod +x rest.sh), We don’t see any way out! Even if we run jobs (the command that shows the background jobs in progress), there is no output. Why?

The reason is that the shell that was used to start this script (i.e. the current shell) is not the same shell (nor the same thread; to start thinking in terms of sublayers as threads in and of themselves) that you ran the real dream. commands or placed them in the background. It was rather the (sub) shell that started when ./rest.sh what he executed.

Let’s change our script by adding jobs within the script. This will ensure that jobs runs from within the (sub) shell where it is relevant, the same one in which the sleep periods (and processes) were started.

Add jobs command in script

This time we can see the list of background processes that are starting thanks to jobs command at the end of the script. We can also see their PIDs (process identifiers). These PIDs are very important when it comes to handling and managing background processes.

Another way to get the Background Process Identifier is to query it immediately after placing a background process / program:

#!/bin/bash

sleep 10 & 
echo ${!}
sleep 600 & 
echo ${!}
sleep 1200 & 
echo ${!}
sleep 1800 & 
echo ${!}
sleep 3600 &
echo ${!}

Query the PID (Process ID) of the last background process to start with $ {!}

Similar to our jobs command (with new PIDs now when we restart our rest.sh script), thanks to the Bash ${!} When the variable is repeated, we will now see that all five PIDs are displayed almost immediately after the script was started – the various suspend processes were put into background threads one after another.

The wait Command

Once we have started our processes in the background, we have nothing else to do but wait for them to finish. However, when each background process is executing a complex subtask, and we need the main script (which started the background processes) to resume execution when one or more of the background processes finish, we need additional code to handle this.

Let’s expand our script now with the wait command to handle our background threads:

#!/bin/bash

sleep 10 & 
T1=${!}
sleep 600 & 
T2=${!}
sleep 1200 & 
T3=${!}
sleep 1800 & 
T4=${!}
sleep 3600 &
T5=${!}

echo "This script started 5 background threads which are currently executing with PID's ${T1}, ${T2}, ${T3}, ${T4}, ${T5}."
wait ${T1}
echo "Thread 1 (sleep 10) with PID ${T1} has finished!"
wait ${T2}
echo "Thread 2 (sleep 600) with PID ${T2} has finished!"

Here we extend our script with two wait commands that wait for the PID attached to the first and second threads to finish. After 10 seconds, our first thread exists and we are notified of it. Step by step, this script will do the following: start five threads at almost the same time (although the start of the threads itself is still sequential and not parallel) where each of the five sleepit will run in parallel.

The main script then reports (sequentially) the created thread and then waits for the process ID of the first thread to finish. When that happens it will report sequentially on the end of the first thread and start waiting for the second thread to finish etc.

Using bash idioms &, ${!} and the wait The command gives us great flexibility when it comes to running multiple threads in parallel (as background threads) in Bash.

Ending

In this article, we explore the basics of Bash multithreaded scripting. We introduced the background process operator (&) using some easy to follow examples showing both single threaded and multi threaded sleep commands. Next, we discuss how to handle background processes through commonly used bash idioms. ${!} other wait. We also explore the jobs command to view background threads / processes running.

If you enjoyed reading this article, take a look at our Bash Process Termination Hacks article. Enjoy!

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Hot Topics

Related Articles