ArjanCodes

Overview of Python Concurrency Modern applications spend a significant amount of time waiting. Whether it is a database query, an API call, or a file read, your CPU often sits idle while I/O operations complete. Asyncio solves this by allowing a single thread to handle multiple tasks concurrently. Unlike parallelism, which runs tasks on separate cores, concurrency switches between tasks during their waiting periods. This makes your software feel faster and handle more data without needing massive hardware upgrades. Prerequisites and Core Tools To follow this guide, you should have a solid grasp of Python 3.10 or later. Understanding the difference between blocking and non-blocking operations is essential. Key libraries include: - **Asyncio**: The built-in library for managing event loops and concurrent tasks. - **Aiohttp**: A popular third-party library for making asynchronous HTTP requests, superior to the standard Requests library for high-concurrency needs. - **Time**: Used here for performance benchmarking using `perf_counter`. Moving from Synchronous to Concurrent Code Converting a linear script into a concurrent one requires changing function definitions and the execution entry point. By marking a function with `async def`, you turn it into a coroutine. To execute it, you must use `await`. When dealing with multiple independent tasks, such as fetching 20 different data points, using a simple loop with `await` is still slow because it waits for each task to finish before starting the next. Instead, use `asyncio.gather` to fire all requests at once. Here is how you implement it with a list comprehension: ```python import asyncio async def fetch_data(id): # Simulate API call return f"Data {id}" async def main(): tasks = [fetch_data(i) for i in range(20)] results = await asyncio.gather(*tasks) print(results) asyncio.run(main()) ``` This approach can result in a 10x speed increase for I/O-bound tasks. Handling Blocking Code with Threads Sometimes you must use a library that doesn't support `async`. In these cases, the `asyncio.to_thread` function is a lifesaver. It allows you to run blocking functions in a separate thread without halting the main event loop. This effectively turns legacy or synchronous code into something that plays nicely with your concurrent architecture. ```python import asyncio import requests def blocking_get(url): return requests.get(url).status_code async def main(): # Run the blocking function in a separate thread status = await asyncio.to_thread(blocking_get, "https://google.com") print(status) asyncio.run(main()) ``` Syntax Notes and Practical Tips Python's integration of `async` extends to generators and list comprehensions. An `async for` loop allows you to iterate over data as it becomes available from a stream. However, remember that `asyncio.gather` is the tool for speed, while `async for` is for sequential processing of asynchronous data. **Common Gotchas:** - **The GIL**: Python’s Global Interpreter Lock means you won't get true multi-core speedups for CPU-heavy tasks with threads; use `multiprocessing` for that. - **Forgotten Awaits**: If you call an `async` function without `await`, it returns a coroutine object instead of the result. - **Nested Context Managers**: While `aiohttp` uses `async with` for sessions, it can lead to deeply nested code. Use `to_thread` for simpler tasks if the complexity of session management isn't required.