Hydra

Overview Managing configuration settings is often the messiest part of a data science project. Whether you are adjusting hyperparameters for a PyTorch model or switching between local and cloud data paths, hardcoding these values directly into your script creates a brittle architecture. In this tutorial, we move beyond the "constants at the top of the file" approach. You will learn how to decouple your logic from your settings using Hydra, a powerful framework that allows you to manage complex configurations through YAML files and Python data classes. Separating config from code is not just about cleanliness; it's about scalability. By moving settings into external files, you allow non-programmers to tweak parameters without touching the source code and enable automated scripts to run experiments with randomized values on the fly. Prerequisites To get the most out of this guide, you should be comfortable with basic Python syntax and understand the concept of decorators. Familiarity with Data Classes is helpful, as we will use them to add type safety to our configuration objects. You should also have a working Python environment where you can install third-party packages. Key Libraries & Tools - **Hydra**: A framework for elegantly configuring complex applications. It handles YAML loading, command-line overrides, and configuration composition. - **OmegaConf**: The underlying library Hydra uses to manage configuration objects, providing a flexible, dictionary-like interface. - **Python Data Classes**: Used here to define the structure and types of our configuration, enabling autocompletion and error checking in your IDE. Code Walkthrough: Implementing Hydra First, we move our parameters from the script into a `config.yaml` file. We group them logically into sections like `params` and `paths` to maintain order. ```yaml conf/config.yaml defaults: - _self_ - files: mnist params: epoch_count: 10 lr: 0.01 batch_size: 64 paths: log: "runs/" data: "${hydra:runtime.cwd}/data/" ``` Next, we define the structure using data classes in a separate `config.py`. This ensures our code knows exactly what to expect from the YAML file. ```python from dataclasses import dataclass @dataclass class Paths: log: str data: str @dataclass class Params: epoch_count: int lr: float batch_size: int @dataclass class Config: paths: Paths params: Params ``` Finally, we integrate these into the main entry point. We use the Hydra `ConfigStore` to bridge the gap between our raw data and our typed classes. ```python import hydra from hydra.core.config_store import ConfigStore from .config import Config cs = ConfigStore.instance() cs.store(name="base_config", node=Config) @hydra.main(config_path="conf", config_name="config") def main(cfg: Config): print(f"Training for {cfg.params.epoch_count} epochs") print(f"Learning rate set to: {cfg.params.lr}") if __name__ == "__main__": main() ``` By using the `@hydra.main` decorator, Hydra automatically instantiates the `cfg` object before the function runs. It looks in the `conf` directory, finds `config.yaml`, and maps the values to our `Config` data class. Syntax Notes One standout feature of Hydra is its use of variable interpolation. In the YAML example, the syntax `${hydra:runtime.cwd}` is a special resolver. Because Hydra changes the working directory to a unique output folder for every run, this resolver ensures you can still find your data folder relative to where you launched the script. Note the use of the `_self_` keyword in the defaults list. This tells Hydra how to prioritize values when merging multiple files. If you want values in the main `config.yaml` to take precedence over sub-configs, you place `_self_` at the bottom of the list. Practical Examples In a real-world machine learning pipeline, you might have different configuration sets for "Development" and "Production." Instead of changing code, you create a `dev.yaml` and a `prod.yaml`. At runtime, you simply pass an argument: `python main.py files=prod`. Hydra swaps the entire file set without you touching a single line of logic. This is also indispensable for hyperparameter sweeps, where a shell script can trigger dozens of runs with different learning rates by overriding values via the command line. Tips & Gotchas **Working Directories**: Remember that Hydra creates a new folder for every run (usually under `outputs/`). If your code tries to save a file to a relative path like `./results.csv`, it will end up inside that timestamped Hydra folder. Use absolute paths or the runtime resolvers if you need files saved elsewhere. **Type Matching**: If your data class expects an `int` for `batch_size` but your YAML contains a string, Hydra will throw a validation error. This is a feature, not a bug! It catches configuration errors before your heavy training loop even starts, saving you hours of wasted compute time.

Overview Writing code that works is only half the battle. In software engineering, the real challenge lies in making that code maintainable, testable, and flexible. When dealing with complex tasks like web scraping or PDF analysis, scripts often start as a single, monolithic file where configuration, logic, and external dependencies are tightly coupled. This tutorial focuses on high-level Python refactoring techniques. We will dismantle "god classes" that instantiate their own subclasses—a major anti-pattern—and replace them with clean functions and Protocols. Furthermore, we will explore how to move hardcoded strings and settings into external JSON configuration files, allowing the application to change behavior without a single line of code being rewritten. Prerequisites To get the most out of this guide, you should be comfortable with: * Intermediate Python syntax (classes, functions, and decorators). * The concept of Object-Oriented Programming (OOP) and composition. * Type hinting and why it matters for modern development. * Basic understanding of Data Classes. Key Libraries & Tools * **Python Protocols**: Part of the `typing` module, used for structural subtyping (duck typing). * **Pandas**: Used for data manipulation, specifically handling data frames in the scraper. * **tqdm**: A library for displaying smart progress bars during long-running loops. * **JSON**: The standard format for our external configuration files. * **Hydra** (Mentioned): A framework for elegantly configuring complex applications. Code Walkthrough: From Classes to Functions One of the biggest issues in the original code was a `ScrapeRequest` class that was responsible for creating its own subclasses. This creates a circular dependency and makes the code difficult to extend. We solve this by using composition and Protocols. 1. Defining the Scraper Protocol Instead of a rigid class hierarchy, we define what a "scraper" looks like using a Protocol. Any class that has a `scrape` method matching this signature is now a valid scraper. ```python from typing import Protocol from dataclasses import dataclass @dataclass class ScrapeResult: keywords: list[str] word_frequencies: dict[str, int] class Scraper(Protocol): def scrape(self, search_text: str) -> ScrapeResult: ... ``` 2. Refactoring Requests into Functions We don't need a class for every type of request. By converting them into functions, we simplify the flow. These functions now accept a `Scraper` instance as a dependency. ```python def fetch_terms_from_doi(target: str, scraper: Scraper) -> ScrapeResult: # Logic to process target and call the scraper result = scraper.scrape(target) return result ``` 3. Centralizing Logging Duplicate logging logic is a maintenance nightmare. We create a dedicated `log.py` to handle both file logging and console printing in one place. ```python import logging def log_message(message: str): logging.info(message) print(message) ``` The Power of External Configuration Hardcoding paths, URLs, and word lists directly into your logic makes your script brittle. If you want to share your tool with a non-programmer, they shouldn't have to touch Python code to change the input folder. We use Data Classes to map JSON data into a typed object. ```python @dataclass class ScrapeConfig: export_dir: str paper_folder: str target_words_file: str def read_config(config_file: str) -> ScrapeConfig: with open(config_file, "r") as f: data = json.load(f) return ScrapeConfig(**data) ``` By passing this `ScrapeConfig` object down the call stack, we ensure that every component has access to the settings it needs without relying on global variables. Syntax Notes * **Protocol**: This is a powerful feature of Python's typing system. Unlike traditional inheritance, a class doesn't need to explicitly inherit from `Scraper` to be considered a `Scraper`. It just needs the right method. * **Unpacking Operators (`**data`)**: We use the double asterisk to unpack a dictionary directly into the initializer of a Data Class. This only works if the keys in the JSON exactly match the field names in the class. * **Context Managers**: Always use `with open(...)` for file operations and directory changes to ensure resources are cleaned up even if an error occurs. Practical Examples This refactoring approach is essential for: * **Data Science Pipelines**: Where file paths and filtering parameters change with every experiment. * **CI/CD Environments**: Where different configurations are needed for testing, staging, and production. * **User-Facing Tools**: Allowing users to modify a simple `config.json` instead of editing source code. Tips & Gotchas * **Avoid Instance Variable Bloat**: Don't store temporary data as `self.variable` in a class if it's only used within a single method. Use local variables to keep the object state clean. * **Type Checking Gaps**: Libraries like tqdm and Pandas don't always have perfect type hints. You might encounter "Unknown" types; use `typing.Any` or `# type: ignore` sparingly when these external tools fail the linter. * **Configuration Trickle**: High-level objects should receive the whole `ScrapeConfig`, but low-level helpers should only receive the specific strings or sets they need. This keeps the low-level code reusable in other projects that don't use your specific config structure.