By default, Python’s runtime executes in a single thread, traffic-directed by its Global Interpreter Lock (GIL). Most of the time this isn’t a significant bottleneck, but it becomes one when you want to run many jobs in parallel.
Python provides two ways to work around this: threading and multiprocessing. Each allows you to take long-running jobs, break them into parallel batches, and work on them side-by-side.
Depending on the job in question, you can sometimes speed up operations tremendously. At the very least, you can treat tasks in such a way that they don’t block other work while they wait to be completed.
In this article we’ll look at one of the fastest ways to make use of threading and subprocesses in Python, the thread and process pool.
