Kaggle

Overview When you process large datasets, memory becomes your most expensive resource. Pandas is built on top of Python and NumPy, providing a high-level interface for data manipulation. However, if you rely solely on default settings, your memory usage can balloon by over 90% unnecessarily. This tutorial explores how to control data types to build efficient, scalable data pipelines. Prerequisites To follow along, you should be comfortable with Python basics. You will need Pandas installed in your environment. Familiarity with tabular data concepts like rows and columns is essential. Key Libraries & Tools * Pandas: The primary library for data structures (DataFrames and Series). * NumPy: The numerical engine that provides the underlying C-based data types. * Pip: The package manager used to install these tools. Code Walkthrough Type Inference and Metadata Issues When reading a CSV, Pandas often struggles with files containing metadata rows. This results in every column defaulting to the expensive `object` type. ```python import pandas as pd Skipping metadata rows to help Pandas infer types correctly df = pd.read_csv("airports.csv", skiprows=2) print(df.dtypes) ``` By skipping the first two rows, Pandas correctly identifies integers and floats rather than treating everything as generic objects. Manual Type Casting You can force specific types using the `astype` method or specialized conversion functions like `to_numeric` and `to_datetime`. ```python Mapping multiple columns at once type_map = { "name": "string", "is_active": "bool" } df = df.astype(type_map) Converting to datetime df["last_updated"] = pd.to_datetime(df["last_updated"]) ``` The Power of Categorical Types For columns with many repeated strings (like 'State' or 'City'), the `category` type stores data as integers internally, mapped to a unique set of strings. This can reduce memory footprints by up to 98%. ```python df["state"] = df["state"].astype("category") ``` Syntax Notes * **Object Type**: The fallback for any data Pandas doesn't recognize; it is highly memory-inefficient. * **astype()**: A versatile method that accepts a single type or a dictionary for bulk conversion. * **memory_usage(deep=True)**: Essential for seeing the true cost of string data stored in object columns. Practical Examples In a Brazilian e-commerce dataset with 100,000 records, switching a "State" column from `object` to `category` slashed memory usage significantly because there are only 26 unique states. This optimization allows you to process millions of rows on standard hardware. Tips & Gotchas Avoid using the categorical type if the column has high cardinality—meaning almost every value is unique (like a Zip Code). In these cases, the overhead of maintaining the category map actually increases memory consumption.