Overview: The Monorepo Challenge Managing multiple Python applications within a single repository often leads to a "dependency hell" of local virtual environments and duplicated logic. uv workspaces solve this by allowing developers to manage several apps—like a FastAPI web service and a Typer CLI tool—using a shared lockfile and a unified virtual environment. This structure maintains a "DRY" (Don't Repeat Yourself) codebase where internal logic lives in private packages used across the entire project. Prerequisites To follow this guide, you should have a solid grasp of Python development, including experience with virtual environments and `pyproject.toml` configuration. You should also have uv installed, as it serves as the primary engine for workspace orchestration. Key Libraries & Tools - **uv**: An extremely fast Python package and project manager. - **FastAPI**: A modern web framework for building APIs. - **Typer**: A library for creating CLI applications. - **OpenAI**: Used here for summarizing text via GPT models. - **httpx** & **Beautiful%20Soup**: Tools for web requests and HTML parsing. Code Walkthrough: Configuring the Workspace 1. Root Configuration In the project root, create a `pyproject.toml` that defines the workspace members. This tells uv which directories to treat as part of the collective environment. ```toml [project] name = "my-monorepo" version = "0.1.0" dependencies = ["python-dotenv"] [tool.uv.workspace] members = ["packages/*"] ``` 2. Defining Local Sources To import an internal package (e.g., `core`) without publishing it to PyPI, use the `tool.uv.sources` table. This points uv to the local file path. ```toml [tool.uv.sources] core = { workspace = true } ``` 3. Syncing the Environment Run the following command at the root to create a single `.venv` that encompasses all dependencies for every package in the workspace: ```bash uv sync ``` Syntax Notes: The Workspace Source Pattern The `workspace = true` flag is a specific uv convention. It ensures that when you run `uv sync` or `uv run`, the tool looks internally for the dependency rather than searching the public registry. This allows for seamless cross-package imports like `from core.news import fetch_headlines` across different applications. Practical Examples Consider an automation repo containing dozens of small scripts. Instead of each script having its own `httpx` version, they all draw from the workspace root. When you update a shared accounting layer in a `core` package, every web-hook listener and data-extraction tool in the repository immediately benefits from the update without manual re-installs. Tips & Gotchas - **VS Code Support**: Sometimes the IDE struggles with workspace-level imports. Selecting the root virtual environment as your Python interpreter usually resolves type-checking issues. - **When to Avoid**: If your apps are completely unrelated or you intend to release a package independently on PyPI, separate repositories remain the better choice. Workspaces thrive on high overlap and shared context.
Httpx
Products
ArjanCodes (6 mentions) highlights Httpx as a modern HTTP client, useful for network requests and as a test dependency, as seen in "How to Write Great Unit Tests in Python" and "Why Are Not More People Using These Python Libraries?".
- Oct 24, 2025
- Aug 1, 2025
- Jul 11, 2025
- Jun 13, 2025
- Jan 24, 2025
Overview of the Requests Library The Requests library stands as a monument in the Python ecosystem. It revolutionized how developers interact with HTTP by providing a human-readable interface over the complex and often clunky urllib3. For years, its motto, 'HTTP for Humans,' has guided its design, making it the de facto standard for sending API calls, scraping web content, and managing sessions. However, being an industry standard does not make a codebase immune to technical debt or questionable design patterns. By examining the internals of Requests, we gain insight into how a widely-used library manages cross-version compatibility, abstraction layers, and low-level networking. This walkthrough explores the core components—adapters, sessions, and models—while critiquing the architectural decisions through the lens of modern software engineering best practices. We will see how legacy requirements often conflict with clean code principles like the Single Responsibility Principle and Composition over Inheritance. Prerequisites To get the most out of this deep dive, you should have a solid grasp of the following: - **Python Proficiency**: Familiarity with classes, inheritance, and keyword arguments (`**kwargs`). - **HTTP Basics**: Understanding of methods (GET, POST), status codes, headers, and SSL/TLS verification. - **Design Patterns**: Awareness of the Adapter pattern and the concept of 'Mixins.' - **Testing Tools**: Basic knowledge of Pytest and the concept of mocking network requests. Key Libraries & Tools - Requests: The primary HTTP library for Python being reviewed. - urllib3: The low-level dependency that Requests wraps to handle connection pooling and thread safety. - Pytest: The testing framework used to validate the library's behavior. - charset-normalizer: A dependency used for character encoding detection. - Docker: A suggested tool for improving local and CI testing environments through containerization. Code Walkthrough: Adapters and Type Handling One of the most critical parts of the Requests architecture is the Transport Adapter. This layer allows the library to define how it communicates with different protocols. By default, Requests uses the `HTTPAdapter`, which relies on urllib3 to manage the actual socket connections. The Problem with Mixed Type Arguments In the `adapters.py` file, we encounter a pattern that often complicates maintenance: arguments that accept multiple types to perform different logical tasks. A prime example is the `verify` parameter. It can be a `bool` (to toggle SSL verification) or a `str` (providing a path to a CA bundle). ```python Current implementation pattern in Requests adapters def cert_verify(self, conn, url, verify, cert): if verify is False: # Disable SSL verification logic pass elif isinstance(verify, str): # Logic to load certificate from path pass ``` This design forces the method to perform 'type-switching' using `isinstance()` checks. While flexible for the user, it creates a brittle internal structure. A cleaner approach would involve splitting these into distinct parameters or using a more robust configuration object. This would allow the type system to catch errors at compile-time (or via static analysis) rather than relying on runtime checks. Refining Type Logic with Guard Clauses A better way to handle these scenarios is to separate the boolean toggle from the path configuration. By using guard clauses, we can flatten the nested logic and make the code more readable. For instance, if `verify` is false, we can exit the logic early, reducing the cognitive load for anyone reading the method. Architecture Critique: Mixins vs. Composition Requests makes heavy use of 'Mixins,' specifically the `SessionRedirectMixin`. In Python, a Mixin is a class that provides methods to other classes through multiple inheritance but is not intended to stand on its own. While popular in older Python frameworks, Mixins often lead to confusing 'spaghetti' inheritance where a superclass calls a method that is only defined in its subclass. The Session and Redirect Relationship The `Session` class inherits from `SessionRedirectMixin`. Looking at the source, the `SessionRedirectMixin` calls `self.send()`, yet the `send()` method is defined in the `Session` class itself. This circular dependency makes the code difficult to trace. It's nearly impossible to unit test the Mixin in isolation because it lacks the context of the class it is mixed into. Moving Toward Composition Modern software design favors composition over inheritance. Instead of making `Session` a child of a redirect class, we should treat 'redirect logic' as a tool that `Session` uses. By creating a standalone `RedirectHandler` and passing it to the session, we decouple the components. ```python class RedirectHandler: def resolve(self, response, session): # Logic lives here independently pass class Session: def __init__(self, redirect_handler=None): self.redirect_handler = redirect_handler or RedirectHandler() ``` This makes the code more modular. If you need to change how redirects work, you only touch the handler. If you want to test redirect logic, you don't need to instantiate a heavy `Session` object. Syntax Notes: Type Annotations and Compatibility You might notice that Requests often uses string literals for type hints, such as `"Response"` instead of just `Response`. This is a common practice in libraries that support older versions of Python or deal with circular imports. String annotations tell the interpreter to treat the type as a forward reference, preventing 'NameError' exceptions when a class hasn't been fully defined yet at the time of the type check. Furthermore, the library avoids modern features like `dataclasses` to maintain compatibility with legacy environments. While this makes the library incredibly stable and portable, it results in more boilerplate code in the `__init__` methods where every attribute must be manually assigned to `self`. Practical Examples: Custom Adapters The power of the Adapter design pattern is that you can extend Requests to support non-standard protocols. For example, if you wanted to add support for a 'mock' protocol for testing without hitting the network, you could subclass the `BaseAdapter`. ```python from requests.adapters import BaseAdapter from requests.models import Response class LocalFileAdapter(BaseAdapter): def send(self, request, **kwargs): response = Response() response.status_code = 200 # Logic to read a local file based on the URL response._content = b"Local content" return response Usage import requests s = requests.Session() s.mount('file://', LocalFileAdapter()) resp = s.get('file:///path/to/data.txt') ``` This demonstrates why the `BaseAdapter` exists, even if the current implementation of `HTTPAdapter` is a bit bloated. It provides the hook for developers to customize the transport layer entirely. Tips & Gotchas - **The 'is' vs '==' Trap**: In the Requests source, you'll see comparisons like `verify is False`. This is used because `True` and `False` are singleton objects in Python. Using `is` checks for identity, which is slightly faster than the equality check `==`, but it should be used carefully, as it won't work for generic values like strings or custom objects. - **Test Structure**: Always try to make your `tests/` directory mirror your `src/` directory. In Requests, some tests (like `test_requests.py`) have grown too large, covering multiple modules. Keeping a 1:1 mapping between source files and test files makes it significantly easier for new contributors to find where a specific feature is validated. - **CI/CD Automation**: For complex networking libraries, using Docker in your CI pipeline is a best practice. It allows you to spin up actual mock servers (like the `test_server` used in Requests) in a controlled environment, ensuring that your tests aren't failing due to local network flakes. - **Hierarchy of Exceptions**: When designing a library, create a base exception (e.g., `RequestException`) that all other custom errors inherit from. This allows users to write a single `except RequestException:` block to catch any error generated by your package.
Aug 16, 2024Overview Python’s dynamic type system is incredibly user-friendly, but it can lead to catastrophic failures when handling external data. Pydantic solves this by enforcing strict data validation and serialization using Python's type annotation mechanism. It ensures your data follows specific rules and structures before it ever touches your core logic. This "fail-fast" approach prevents corrupted data from migrating through your pipeline, making it an essential tool for robust software design. Prerequisites To get the most out of this tutorial, you should have a solid grasp of Python 3.10+ and type hints. Familiarity with JSON structures and basic object-oriented programming concepts like class inheritance will help you understand how Pydantic models are structured. Key Libraries & Tools * Pydantic: The core library for data validation and settings management. * FastAPI: A modern web framework that natively integrates with Pydantic for request validation. * HTTPX: A next-generation HTTP client for testing API endpoints. Code Walkthrough Defining a model starts with inheriting from `BaseModel`. This provides the foundation for Pydantic's validation engine. ```python from pydantic import BaseModel, EmailStr, Field, SecretStr class User(BaseModel): name: str = Field(..., examples=["Ariel"], description="The user's full name") email: EmailStr password: SecretStr role: str ``` In this snippet, `EmailStr` automatically validates the format of an email address. The `Field` function adds metadata like descriptions, which FastAPI uses to generate documentation. `SecretStr` is a security feature; when you print the object, the password shows as asterisks rather than plain text. For complex logic, use the `@field_validator` decorator: ```python from pydantic import field_validator @field_validator("name") @classmethod def validate_name(cls, value: str): if len(value) < 2: raise ValueError("Name must be at least 2 characters") return value ``` Syntax Notes Notice the use of `classmethod` for field validators. When using `mode='before'`, the validator runs before Pydantic's internal coercion, allowing you to handle raw input like strings that need conversion to Enums. For cross-field validation, use `@model_validator(mode='after')`, which gives you access to the entire instance data as a dictionary. Practical Examples Beyond simple scripts, Pydantic is the backbone of FastAPI. When you define a route, FastAPI uses your model to validate incoming JSON request bodies automatically. If the client sends a malformed email, the API returns a 422 Unprocessable Entity error before your function even executes. Tips & Gotchas Avoid over-complicating your models. If you don't need validation, standard Python dataclasses are more lightweight. However, always use `model_dump()` or `model_dump_json()` for serialization rather than manually converting objects to dictionaries to ensure your `exclude` and `alias` settings are respected.
Mar 8, 2024Overview of Python HTTP Clients Communicating with external APIs is a cornerstone of modern software development. While Python offers several ways to handle these interactions, choosing the right tool impacts both developer productivity and application performance. We are looking at three heavyweights: Requests, aiohttp, and HTTPX. While Requests remains the industry standard for simplicity, newer alternatives provide the asynchronous capabilities required for high-concurrency environments. Prerequisites To follow this tutorial, you should have a solid grasp of Python 3.x. Familiarity with basic HTTP verbs (GET, POST, PUT, DELETE) is essential. For the advanced sections, a working knowledge of `asyncio` and the `async/await` syntax will help you understand how concurrency improves network-bound operations. Key Libraries & Tools - **Requests**: The gold standard for synchronous HTTP; optimized for ease of use. - **aiohttp**: A performance-focused library built specifically for asynchronous operations. - **HTTPX**: A next-generation client that supports both sync and async interfaces while maintaining compatibility with the Requests API. - **HTTPBin**: A specialized web service used for testing and echoing HTTP requests. Code Walkthrough: Moving Beyond Basic Requests Most developers start with basic synchronous calls. However, creating a new connection for every request is inefficient. Reusing connections via a session—or a Client in HTTPX—is the first step toward optimization. ```python import httpx def fetch_sync(): # Using a Client manages a connection pool automatically with httpx.Client() as client: response = client.get("https://httpbin.org/get") return response.json() ``` To truly unlock performance, we switch to an `AsyncClient`. This allows the program to initiate multiple requests without waiting for the previous ones to finish. ```python import asyncio import httpx async def fetch_async(client): # Non-blocking request response = await client.get("https://httpbin.org/get") return response.json() async def main(): async with httpx.AsyncClient() as client: # Fire multiple requests concurrently tasks = [fetch_async(client) for _ in range(4)] results = await asyncio.gather(*tasks) print(f"Retrieved {len(results)} responses") asyncio.run(main()) ``` In the async example, `asyncio.gather` acts as the orchestrator, executing the task list concurrently. This approach typically cuts execution time by half or more compared to sequential loops. Syntax Notes HTTPX stands out because it mirrors the Requests API almost exactly. This minimizes the learning curve. Pay attention to the use of context managers (`with` and `async with`). These ensure that connection pools are properly closed, preventing resource leaks in long-running applications. Practical Examples Concurrency is a lifesaver when building dashboards that aggregate data from multiple microservices. Instead of your user waiting for three separate 1-second API calls to finish sequentially, they wait only for the slowest single call. Tips & Gotchas Avoid the temptation to fire thousands of concurrent requests at once. You will likely hit rate limits or trigger 429 status codes from the server. Always implement error handling and consider using a semaphore to limit the number of active tasks. Furthermore, while aiohttp is incredibly fast, its syntax is more verbose than HTTPX, requiring nested context managers for simple tasks. For most modern projects, HTTPX provides the best balance of speed and developer experience.
Dec 15, 2023Python's true strength lies not just in its syntax, but in the massive ecosystem that surrounds it. For developers looking to write cleaner, more efficient code, choosing the right tool for the job is the difference between a project that scales and one that becomes a maintenance nightmare. These fifteen libraries represent the cutting edge of productivity and performance. Refined Debugging and Display Tools Traditional print debugging is a mess. It clutters the terminal and lacks context. IceCream changes this by inspecting its own arguments, outputting not just the value but the function and variables involved with full syntax highlighting. When you need to move beyond simple output to professional terminal interfaces, Rich provides the ability to render markdown, complex tables, and progress bars directly in the console. For developers still fighting the built-in logging module, Loguru removes the need for complex logger objects, allowing for instant, color-coded tracking of application behavior. Data Management and High Performance When Pandas hits a performance ceiling with massive datasets, Polars steps in. Written in Rust, it utilizes a blazingly fast engine that handles multi-threading by default. For those dealing with multi-dimensional labeled data, Xarray provides a more intuitive way to handle complex scientific computing than standard arrays. Visualizing this data becomes significantly easier with Seaborn, which builds on Matplotlib to create beautiful statistical charts with minimal configuration. The Modern Web Stack Building APIs has shifted toward FastAPI. It prioritizes modern features like concurrency and async/await while leveraging Pydantic for robust data validation. This pair ensures that errors are caught before they reach production. To bridge the gap between Python objects and your database, SQLModel combines the best of SQLAlchemy and Pydantic into a single, intuitive interface. Finally, for making web requests, HTTPX is the successor to the classic requests library, offering full async support for high-performance network calls. Handling Logic and Environments Errors shouldn't always be catastrophic. Result introduces "railroad oriented programming," allowing developers to handle success and failure paths without messy try-except blocks. For project configuration, python-dotenv keeps sensitive credentials out of the source code by loading variables from a simple .env file. These tools, along with specialized utilities like Pendulum for painless timezone management and PyPDF for document automation, create a professional toolkit that elevates any Python project.
Sep 15, 2023