Python3 Multithreading

Multithreading is similar to executing multiple different programs at the same time. Running multithreading has the following advantages:

  • Using threads can put tasks that occupy a long time in a program into the background for processing.
  • The user interface can be more attractive. For example, when a user clicks a button to trigger the handling of certain events, a progress bar can pop up to show the progress of the processing.
  • The running speed of the program may increase.
  • Threads are more useful for implementing some waiting tasks, such as user input, file reading and writing, and network data sending and receiving. In such cases, we can release some precious resources such as memory usage.

Each independent thread has an entry point for program execution, a sequential execution sequence, and an exit point for the program. However, threads cannot execute independently; they must depend on the application, which provides control for multiple thread execution.

Each thread has its own set of CPU registers, called the thread's context, which reflects the state of the CPU registers when the thread last ran.

The instruction pointer and stack pointer registers are the two most important registers in the thread context. Threads always run in the context obtained by the process, and these addresses are used to mark memory in the address space of the process that owns the thread.

  • Threads can be preempted (interrupted).
  • While other threads are running, a thread can be temporarily shelved (also called sleeping) -- this is thread yielding.

Threads can be divided into:

  • Kernel threads:Created and revoked by the operating system kernel.
  • User threads:Threads implemented in user programs without requiring kernel support.

Two commonly used modules for threads in Python3 are:

  • _thread
  • threading (recommended)

The thread module has been deprecated. Users can use the threading module instead. Therefore, in Python3, the "thread" module can no longer be used. For compatibility, Python3 renamed thread to "_thread".


Start learning Python threads

There are two ways to use threads in Python: functions or using classes to wrap thread objects.

Functional style: call the start_new_thread() function in the _thread module to generate a new thread. The syntax is as follows:

_thread.start_new_thread ( function, args[, kwargs] )

Parameter description:

  • function - The thread function.
  • args - The parameters passed to the thread function; it must be a tuple type.
  • kwargs - Optional parameters.

Example

#!/usr/bin/python3

import _thread
import time

# Define a function for the thread
def print_time( threadName, delay):
   count = 0
   while count < 5:
      time.sleep(delay)
      count += 1
      print ("%s: %s" % ( threadName, time.ctime(time.time()) ))

# Create two threads
try:
   _thread.start_new_thread( print_time, ("Thread-1", 2, ) )
   _thread.start_new_thread( print_time, ("Thread-2", 4, ) )
except:
   print ("Error: unable to start thread")

while 1:
   pass

The output of executing the above program is as follows:

Thread-1: Wed Jan  5 17:38:08 2022
Thread-2: Wed Jan  5 17:38:10 2022
Thread-1: Wed Jan  5 17:38:10 2022
Thread-1: Wed Jan  5 17:38:12 2022
Thread-2: Wed Jan  5 17:38:14 2022
Thread-1: Wed Jan  5 17:38:14 2022
Thread-1: Wed Jan  5 17:38:16 2022
Thread-2: Wed Jan  5 17:38:18 2022
Thread-2: Wed Jan  5 17:38:22 2022
Thread-2: Wed Jan  5 17:38:26 2022

After executing the above program, you can press ctrl-c to exit.

Thread module

Python3 provides support for threads through two standard libraries: _thread and threading.

_thread provides low-level, primitive threads and a simple lock. Its functionality is still relatively limited compared to the threading module.

In addition to containing all the methods in the _thread module, the threading module also provides the following methods:

  • threading.current_thread(): Returns the current thread variable.
  • threading.enumerate(): Returns a list containing all running threads. "Running" means after the thread has started and before it has finished, not including threads before startup or after termination.
  • threading.active_count(): Returns the number of running threads, with the same result as len(threading.enumerate()).
  • threading.Thread(target, args=(), kwargs={}, daemon=None)：
    • CreateThreadinstance of the class.
    • target: The target function that the thread is to execute.
    • args: The parameters of the target function, passed in tuple form.
    • kwargs: The keyword arguments of the target function, passed in dictionary form.
    • daemon: Specifies whether the thread is a daemon thread.

The threading.Thread class provides the following methods and attributes:

  1. __init__(self, group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)：

    • InitializationThreadobject.
    • group: Thread group, currently unused, reserved for future expansion.
    • target: The target function that the thread is to execute.
    • name: The name of the thread.
    • args: The parameters of the target function, passed in tuple form.
    • kwargs: The keyword arguments of the target function, passed in dictionary form.
    • daemon: Specifies whether the thread is a daemon thread.

  2. start(self)：

    • Start the thread. It will call the thread'srun()method.

  3. run(self)：

    • The thread defines the code to be executed in this method.

  4. join(self, timeout=None)：

    • Wait for the thread to terminate. By default,join()it will block until the called thread terminates. If thetimeoutparameter is specified, it waits at mosttimeoutseconds.

  5. is_alive(self)：

    • Returns whether the thread is running. If the thread has started and has not yet terminated, it returnsTrue, otherwise it returnsFalse。

  6. getName(self)：

    • Returns the name of the thread.

  7. setName(self, name)：

    • Sets the name of the thread.

  8. identAttributes:

    • The unique identifier of the thread.

  9. daemonAttributes:

    • The daemon flag of the thread, used to indicate whether it is a daemon thread.

  10. isDaemon()Methods:

A simple thread example:

Example

import threading
import time

def print_numbers():
    for i in range(5):
        time.sleep(1)
        print(i)

# Create thread
thread = threading.Thread(target=print_numbers)

# Start thread
thread.start()

# Wait for thread to finish
thread.join()

The output result is:

0
1
2
3
4

Creating threads using the threading module

We can create a new subclass by directly inheriting from threading.Thread, and after instantiation, call the start() method to start the new thread. That is, it calls the thread's run() method:

Example

#!/usr/bin/python3

import threading
import time

exitFlag = 0

class myThread (threading.Thread):
    def __init__(self, threadID, name, delay):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.delay = delay
    def run(self):
        print ("Start thread:" + self.name)
        print_time(self.name, self.delay, 5)
        print ("Exit thread:" + self.name)

def print_time(threadName, delay, counter):
    while counter:
        if exitFlag:
            threadName.exit()
        time.sleep(delay)
        print ("%s: %s" % (threadName, time.ctime(time.time())))
        counter -= 1

# Create a new thread
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)

# Start the new thread
thread1.start()
thread2.start()
thread1.join()
thread2.join()
print ("Exit main thread")

The execution result of the above program is as follows;

开始线程：Thread-1
开始线程：Thread-2
Thread-1: Wed Jan  5 17:34:54 2022
Thread-2: Wed Jan  5 17:34:55 2022
Thread-1: Wed Jan  5 17:34:55 2022
Thread-1: Wed Jan  5 17:34:56 2022
Thread-2: Wed Jan  5 17:34:57 2022
Thread-1: Wed Jan  5 17:34:57 2022
Thread-1: Wed Jan  5 17:34:58 2022
退出线程：Thread-1
Thread-2: Wed Jan  5 17:34:59 2022
Thread-2: Wed Jan  5 17:35:01 2022
Thread-2: Wed Jan  5 17:35:03 2022
退出线程：Thread-2
退出主线程

Thread synchronization

If multiple threads modify a certain data item together, unexpected results may occur. In order to ensure the correctness of the data, multiple threads need to be synchronized.

Using the Lock and Rlock of Thread objects can achieve simple thread synchronization. Both objects have acquire and release methods. For data that only one thread is allowed to operate on at a time, the operations can be placed between the acquire and release methods. As follows:

The advantage of multithreading is that it can run multiple tasks at the same time (at least it feels that way). But when threads need to share data, there may be a problem of data unsynchronization.

Consider a situation: all elements in a list are 0. The "set" thread changes all elements from back to front to 1, while the "print" thread reads the list from front to back and prints it.

Then, it is possible that when the "set" thread starts modifying, the "print" thread comes to print the list, and the output becomes half 0s and half 1s. This is data unsynchronization. To avoid this situation, the concept of locks is introduced.

A lock has two states - locked and unlocked. Whenever a thread, such as "set", wants to access shared data, it must first acquire the lock; if another thread, such as "print", has already acquired the lock, then the "set" thread is paused, which is synchronous blocking; after the "print" thread finishes accessing and releases the lock, the "set" thread continues.

With such processing, when printing the list, either all 0s or all 1s are output, and the awkward scene of half 0s and half 1s will no longer appear.

Example

#!/usr/bin/python3

import threading
import time

class myThread (threading.Thread):
    def __init__(self, threadID, name, delay):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.delay = delay
    def run(self):
        print ("Start thread: " + self.name)
        # Acquire the lock for thread synchronization
        threadLock.acquire()
        print_time(self.name, self.delay, 3)
        # Release the lock to start the next thread
        threadLock.release()

def print_time(threadName, delay, counter):
    while counter:
        time.sleep(delay)
        print ("%s: %s" % (threadName, time.ctime(time.time())))
        counter -= 1

threadLock = threading.Lock()
threads = []

# Create a new thread
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)

# Start the new thread
thread1.start()
thread2.start()

# Add threads to the thread list
threads.append(thread1)
threads.append(thread2)

# Wait for all threads to complete
for t in threads:
    t.join()
print ("Exit main thread")

Execute the above program, and the output result is:

开启线程： Thread-1
开启线程： Thread-2
Thread-1: Wed Jan  5 17:36:50 2022
Thread-1: Wed Jan  5 17:36:51 2022
Thread-1: Wed Jan  5 17:36:52 2022
Thread-2: Wed Jan  5 17:36:54 2022
Thread-2: Wed Jan  5 17:36:56 2022
Thread-2: Wed Jan  5 17:36:58 2022
退出主线程

Thread priority queue (Queue)

Python's Queue module provides synchronized, thread-safe queue classes, including FIFO (first-in, first-out) queue Queue, LIFO (last-in, first-out) queue LifoQueue, and priority queue PriorityQueue.

These queues all implement lock primitives and can be used directly in multithreading. Queues can be used to achieve synchronization between threads.

Common methods in the Queue module:

  • Queue.qsize() returns the size of the queue.
  • Queue.empty() If the queue is empty, returns True; otherwise False
  • Queue.full() If the queue is full, returns True; otherwise False
  • Queue.full corresponds to the size of maxsize
  • Queue.get([block[, timeout]]) Gets a queue; timeout is the wait time
  • Queue.get_nowait() Equivalent to Queue.get(False)
  • Queue.put(item) Writes to the queue; timeout is the wait time
  • Queue.put_nowait(item) Equivalent to Queue.put(item, False)
  • Queue.task_done() After completing a task, the Queue.task_done() function sends a signal to the queue that the task has been completed
  • Queue.join() Actually means wait until the queue is empty, then perform other operations

Example

#!/usr/bin/python3

import queue
import threading
import time

exitFlag = 0

class myThread (threading.Thread):
    def __init__(self, threadID, name, q):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.q = q
    def run(self):
        print ("Starting thread:" + self.name)
        process_data(self.name, self.q)
        print ("Exiting thread:" + self.name)

def process_data(threadName, q):
    while not exitFlag:
        queueLock.acquire()
        if not workQueue.empty():
            data = q.get()
            queueLock.release()
            print ("%s processing %s" % (threadName, data))
        else:
            queueLock.release()
        time.sleep(1)

threadList = ["Thread-1", "Thread-2", "Thread-3"]
nameList = ["One", "Two", "Three", "Four", "Five"]
queueLock = threading.Lock()
workQueue = queue.Queue(10)
threads = []
threadID = 1

# Creating a new thread
for tName in threadList:
    thread = myThread(threadID, tName, workQueue)
    thread.start()
    threads.append(thread)
    threadID += 1

# Filling the queue
queueLock.acquire()
for word in nameList:
    workQueue.put(word)
queueLock.release()

# Waiting for the queue to be emptied
while not workQueue.empty():
    pass

# Notifying the thread it's time to exit
exitFlag = 1

# Waiting for all threads to complete
for t in threads:
    t.join()
print ("Exiting main thread")

The result of executing the above program:

开启线程：Thread-1
开启线程：Thread-2
开启线程：Thread-3
Thread-3 processing One
Thread-1 processing Two
Thread-2 processing Three
Thread-3 processing Four
Thread-1 processing Five
退出线程：Thread-3
退出线程：Thread-2
退出线程：Thread-1
退出主线程
Other extensions