Convex

Overview: The Analytic Power of DuckDB DuckDB represents a shift in how we handle local data analysis. While SQLite dominates transactional workloads, it often struggles with the heavy aggregation and scanning required for big data analytics. DuckDB fills this gap as a relational database management system (RDBMS) designed specifically for analytical workloads. It operates as an embedded database, meaning it runs directly inside your application process without the overhead of a separate server. This architecture allows for lightning-fast querying of Pandas DataFrames, CSVs, and Parquet files using standard SQL. Prerequisites To follow this guide, you should have a basic understanding of Python and SQL syntax. Familiarity with Pandas DataFrames is helpful, as DuckDB's primary advantage is its ability to interface with these objects. Ensure you have a Python environment ready (version 3.8+ recommended). Key Libraries & Tools - **DuckDB**: The core engine for analytical SQL queries. - **Pandas**: The industry-standard library for data manipulation in Python. - **uv**: A high-performance Python package and project manager used for dependency installation. - **Jupyter Notebook**: An interactive computing environment for testing queries. Code Walkthrough: Querying DataFrames Directly One of the most impressive features of DuckDB is its "Python magic"—the ability to recognize local variables within a SQL string. ```python import pandas as pd import duckdb Create a sample DataFrame df = pd.DataFrame({"name": ["Alice", "Bob"], "salary": [150000, 90000]}) Query the DataFrame variable 'df' directly using SQL result = duckdb.query("SELECT * FROM df WHERE salary > 100000").to_df() print(result) ``` DuckDB inspects the calling scope to find the variable name used in the `FROM` clause. While this is convenient, it can confuse IDEs like PyLance, which may flag the variable as unused. For cleaner code, I recommend explicit registration: ```python con = duckdb.connect() con.register("employees", df) filtered_df = con.execute("SELECT * FROM employees").df() ``` Persistent vs. In-Memory Storage By default, `duckdb.connect()` creates an in-memory database. This is perfect for unit tests where you want a clean state for every run. However, once the connection closes, the data vanishes. To save your work, specify a file path: ```python This creates a persistent database file on disk con = duckdb.connect("company_data.duckdb") con.execute("CREATE TABLE IF NOT EXISTS staff AS SELECT * FROM 'data.csv'") ``` Advanced SQL Extensions DuckDB includes powerful diagnostic tools that usually require heavy enterprise databases. Use `DESCRIBE` to see schema details, or `SUMMARIZE` to get instant statistics like percentiles and null counts. If a query is running slowly, prepend it with `EXPLAIN` to see the physical execution plan, including filters and projections. Tips & Gotchas - **Explicit is Better**: Always use `con.register()` to avoid IDE errors and make data lineage clear. - **Thread Safety**: DuckDB supports multithreading, but ensure you manage connections properly when using the `threading` or `multiprocessing` modules. - **CSV Performance**: While DuckDB reads CSVs quickly, repeatedly scanning massive files in an in-memory database will slow down your scripts. Use persistent storage for large datasets.