Neo4j

The Context Gap in Modern Data Lakehouses AI agents routinely fail because they lack structural context. Traditional data architectures split information into structured data warehouses and unstructured data lakes. When developers construct retrieval-augmented generation (RAG) pipelines over these "lakehouses," they typically rely on two tools: vector search for unstructured documents and text-to-SQL engines for relational tables. This approach has fundamental blind spots. Vector search slices documents into isolated chunks, stripping away hierarchical structure and document relationships. Text-to-SQL engines struggle when schema sizes balloon, losing track of how massive relational tables connect across a system. The result? The agent retrieves individual pieces of data but remains blind to how they relate, leading to confident, incorrect hallucinations. To bridge this gap, we must model metadata structurally rather than relying entirely on raw queries. Context is not just raw text or a SQL result; context comes in distinct structural shapes. By building lightweight graph representations on top of existing lakehouse structures, we give agents the ability to navigate, reason about, and traverse the entire data estate. We will construct three reusable graph shapes using Neo4j on top of lakehouse data: * **Connections (Semantic Layer):** A metadata graph representing relational schemas and join paths to guide text-to-SQL tasks. * **Trees (Table of Contents):** A hierarchical containment graph allowing agents to navigate unstructured documents and cross-links logically. * **Communities (Themes):** A cluster-based graph structure to surface global, unnamed patterns across documents. Using a fictional nationwide auto repair chain, "Autofix Group," we will walk through setting up your workspace, configuring Claude as an autonomous coding agent, and implementing these three powerful structural context shapes. --- Prerequisites Before diving in, make sure you are comfortable with these core concepts and languages: * **Python 3.10+** (specifically writing simple scripting wrappers and interacting with database drivers) * **SQL Basics** (understanding primary/foreign keys and basic table joins) * **Graph Concepts** (understanding nodes, relationships/edges, and properties) * **Neo4j and Cypher** (familiarity with basic graph operations is helpful, though our coding agent will handle complex queries) * **Command Line Interfaces (CLI)** (comfortable executing bash scripts and running CLI utilities) --- Key Libraries & Tools * **Neo4j Graph Database:** Our core graph database used to store schema metadata and document hierarchies. * **Claude Code:** Anthropic's agentic terminal client used to perform agentic coding tasks and run graph queries. * **NeoCarta:** An open-source Neo4j Labs project that extracts database metadata and writes it into a graph representation. * **Model Context Protocol (MCP) Server:** An open standard protocol used to expose Neo4j and NeoCarta tools directly to Claude. * **Neo4j CLI:** A command-line client providing Claude with dedicated Cypher execution and schema analysis skills. * **Google BigQuery:** The source relational data warehouse containing our structured repair logs. --- Environment Setup and Configuration First, let's configure your development environment. This guide uses GitHub Codespaces for instant environment provisioning, but you can clone the repository and run the scripts locally. Step 1: Open Your Workspace Enroll in the official workshop through GraphAcademy and launch the provided GitHub Codespaces environment. This will provision an online IDE with all necessary CLI tools, sample data, and Python libraries pre-installed. Step 2: Configure Environment Variables Locate the `.env` file in the root directory of your project. If it has not populated yet, copy `.env.example` to `.env`. Fill in your API keys and your personal Neo4j sandbox credentials (retrieved from your GraphAcademy dashboard): ```bash ANTHROPIC_API_KEY="your_anthropic_api_key_here" BIGQUERY_KEY="your_bigquery_read_only_key" NEO4J_URI="bolt://your-sandbox-uri:7687" NEO4J_USERNAME="neo4j" NEO4J_PASSWORD="your-sandbox-password" ``` Step 3: Initialize Claude and Enable MCP In your Codespaces terminal, run the starting command to initialize Claude Code. Claude will check your directory, read your environment variables, and establish connections: ```bash claude ``` During initialization, you will be prompted to enable the **Model Context Protocol (MCP) Server**. Select **Yes**. This allows Claude to automatically call the local Python scripts and database tools we are about to build. To verify Claude can reach your graph database, ask a simple test question inside the Claude prompt: ```text Can you run a test query to confirm you can connect to my Neo4j instance? ``` If successful, Claude will execute a quick Cypher statement and return node counts (which will initially be zero in a fresh sandbox). --- Shape 1: The Connections Semantic Layer Our first graph shape maps structured schema metadata. Instead of copying massive tables into our graph, we keep our operational data in Google BigQuery and use Neo4j purely as a **semantic layer**. This metadata graph maps how tables join, resolving terms and guiding our text-to-SQL engine. If our agent knows the join paths beforehand, it does not have to guess foreign key relationships in massive schemas. Building the Connections Graph We will use **NeoCarta** to scan our BigQuery schema and automatically write the structural layout into Neo4j. Run the following initialization script in your terminal: ```bash python load/build_connections.py ``` This script reads our relational database schema and populates Neo4j. It represents our relational database as a localized graph containing tables, columns, and verified join paths: ``` (:Database) -> [:HAS_SCHEMA] -> (:Schema) -> [:HAS_TABLE] -> (:Table) (:Table) -> [:HAS_COLUMN] -> (:Column) (:Column) -> [:JOINS_TO] -> (:Column) ``` Querying the Connections Layer Now that our metadata graph is populated, we can prompt Claude to translate natural language into accurate SQL queries. Claude will query Neo4j's metadata first to understand the table schema, and then run a clean, joined SQL query against BigQuery. ```text Which vehicle received part IC 2042? Use the MCP server to check the metadata graph for join paths first, then write and run the SQL query. ``` Claude queries the Neo4j semantic layer to find the join path between the `vehicles` table and the `work_order_parts` table, constructs the SQL query, and executes it directly against BigQuery: ```python Inside the agent's execution loop, it runs a query like this against Neo4j first: """ MATCH (t1:Table {name: 'vehicles'})-[:HAS_COLUMN]->(c1:Column) MATCH (t2:Table {name: 'work_order_parts'})-[:HAS_COLUMN]->(c2:Column) MATCH (c1)-[:JOINS_TO]-(c2) RETURN t1.name, c1.name, t2.name, c2.name """ ``` With the verified relationship mapped out, the agent generates and executes a precise BigQuery SQL join without hallucinating column names. --- Shape 2: Trees (Document Table of Contents) Unstructured documentation is usually parsed into isolated text blocks and vector-indexed. This approach completely breaks document hierarchies. In complex vehicle manuals, safety bulletins, and recalls, context is highly structural. A safety bulletin section might link directly to an external repair manual segment. We will build a deterministic **containment tree** (or document outline graph) to model document hierarchies and cross-document links. ``` (:Library) -> [:HAS_FOLDER] -> (:Folder) -> [:HAS_DOCUMENT] -> (:Document) (:Document) -> [:HAS_SECTION] -> (:Section) (:Section) -> [:NEXT_SECTION] -> (:Section) (:Section) -> [:LINKS_TO] -> (:Section) ``` By representing our technical library this way, we generate hierarchical, web-like paths. This allows our agent to navigate our documents logically, reading folders, sections, and cross-references. Loading the Document Graph Execute the Python document loader in your terminal. This script uses standard regular expressions to parse Markdown source files, preserving headers and reference links: ```bash python load/load_documents.py ``` Open your Neo4j browser and view the structure. You will see folder hierarchies and `LINKS_TO` relationships mapping real hyperlinks between sections across different documents. Implementing the `outline.py` Tool To make this tree structure usable by Claude, we need to implement a Python wrapper script (`outline.py`). This script will run a parameterized Cypher query to retrieve sections, their parent documents, and all outgoing links starting at any given Uniform Resource Identifier (URI). Let's use Claude Code to write the query logic inside `outline.py`. Prompt Claude in the terminal: ```text Use the neo4j cipher skill to build the query inside outline.py based on the specifications in docs/specs/outline_format_spec.md. ``` Claude will read the specifications file and fill in the missing Cypher query inside `outline.py`. Let's inspect the generated Cypher structure: ```cypher MATCH (start:Section {uri: $uri}) // Traverse down the containment tree up to a specified depth MATCH path = (start)-[:HAS_SECTION*0..5]->(subSection:Section) OPTIONAL MATCH (subSection)-[r:LINKS_TO]->(target:Section) RETURN subSection.uri AS uri, subSection.title AS title, subSection.content AS content, collect(target.uri) AS outbound_links ORDER BY subSection.order ``` This Cypher query leverages a variable-length path pattern (`-[:HAS_SECTION*0..5]->`) to dynamically scale the retrieved section hierarchy based on a specified depth limit. To verify the tool works, run the script from your terminal to view the containment outline of a specific manual section: ```bash python outline.py --uri "technical_library/manuals/abs_system/diagnostic_troubleshooting" ``` This outputs the structural breakdown of that diagnostic section, complete with all outbound hyperlinks leading to other manual segments. Adding Lucene Full-Text Search To complement our hierarchical navigation, we want our agent to be able to jump directly to specific nodes using full-text search. We will implement `search.py` using Neo4j's built-in Apache Lucene index. Instruct Claude to write this script: ```text Implement the search query inside search.py using our content_search full-text index as specified in docs/specs/search_spec.md. ``` Claude writes the following parameterized Cypher query, which combines a full-text search query with a hierarchical URI prefix filter: ```cypher CALL db.index.fulltext.queryNodes("content_search", $query) YIELD node, score WHERE node.uri STARTS WITH $uri_prefix RETURN node.uri AS uri, node.title AS title, score LIMIT 10 ``` This combination is incredibly powerful. The agent can search globally across all documents, or constrain its search query to a specific folder sub-tree (e.g., `technical_library/bulletins`) using the `STARTS WITH` post-filter. --- Shape 3: Communities (Surfacing Unseen Themes) Our third shape uncovers hidden document patterns. When documents link to each other, they naturally form thematic clusters. By analyzing these structural relationships, we can organize documents into groups without relying on manual taggers or expensive AI classification pipelines. We will use the **Leiden community detection algorithm**, which is highly efficient at detecting tightly knit clusters within large networks. We will run these calculations using Neo4j Graph Data Science (GDS). ``` In-Memory Projection -> Run Leiden Algorithm -> Write Community IDs to Graph ``` Creating the GDS Projection To run GDS algorithms, we first project our database graph into a highly concurrent in-memory structure. This projection collapses our section-level links to represent connections at the document level. Prompt Claude to write the projection logic in `theme.py`: ```text Help me write the GDS projection and Leiden algorithm run inside theme.py according to docs/specs/theme_format_spec.md. ``` Claude writes the following sequence inside `theme.py` using our GDS Python client: ```python 1. Project the in-memory graph g, projection_result = gds.graph.project( "document_network", "Document", { "LINKS_TO": { "type": "LINKS_TO", "orientation": "UNDIRECTED", "aggregation": "COUNT" } } ) 2. Run Leiden community detection result = gds.leiden.write( g, writeProperty="communityId", includeIntermediateCommunities=False ) ``` Running Theme Analysis Execute the script in your terminal to calculate communities and output the detected themes: ```bash python theme.py ``` The algorithm clusters your files into distinct, thematic groups based purely on their structural connections. For example: * **Community 1 (Brakes):** Groups files covering hydraulic lines, calipers, and pads. * **Community 2 (Electronics):** Clusters manuals covering the body control module, battery diagnostics, and wiring diagrams. These clusters provide our agent with global, estate-wide context. By observing these structural groupings, our agent can analyze whole clusters of documentation to identify patterns, track topics, and flag systemic issues. --- Syntax Notes 1. Variable-Length Relationships in Cypher In our outline search queries, we used the `*` modifier to represent variable-length paths: ```cypher MATCH (doc:Document)-[:HAS_SECTION*0..5]->(sec:Section) ``` This instructs the query engine to match paths starting at `Document` and traversing down `HAS_SECTION` relationships between 0 and 5 times. This syntax pattern is ideal for traversing nested XML structures, file directories, or document sub-sections of unknown depth. 2. Full-Text Index Query Syntax Neo4j leverages Apache Lucene under the hood, enabling standard search modifiers directly within your Cypher statements: ```cypher CALL db.index.fulltext.queryNodes("content_search", "misfire OR 'rough idle'") ``` This supports boolean logic, wildcards, and fuzzy string searches, making it a powerful fallback search pattern for keyword matching. --- Practical Examples Now that all three shapes are deployed, let's explore real-world use cases where traditional RAG pipelines struggle. Scenario 1: Diagnostics and Guided Root-Cause Analysis Our technician has a vehicle in their repair bay displaying diagnostic trouble code (DTC) `P0300` (engine misfire). They want to know the verified fix for this specific engine family. ```text For VIN 1FT8W21Y... displaying DTC code P0300, find the correct repair procedure. Show your step-by-step logic and the exact tools used. ``` **Claude's Step-by-Step Logic:** 1. **Grounding:** Runs `search.py` to search for `P0300` inside our technical library. It matches a specific safety bulletin explaining a known ignition coil issue. 2. **Navigation:** Runs `outline.py` to pull the containment outline of that bulletin. The tree reveals that this bulletin links directly to an external, step-by-step repair procedure. 3. **Relational Analysis:** Queries the semantic layer to find the join path between our BigQuery `work_orders` and `work_order_parts` tables. 4. **SQL Execution:** Joins the tables to verify how floor technicians successfully resolved this code in the past. It confirms that replacing old ignition coils with the updated model (`IC-2042`) eliminated repeat visits. By traversing these structural links, the agent provides a grounded, verified repair path instead of guessing. Scenario 2: Proving a Negative (Gap Analysis) Imagine an operations director asking: *"Are we seeing trouble codes in our service history that aren't actually covered in our technical manuals?"* Traditional vector search cannot answer this because it can only search for documents that exist. By contrasting our structured repair logs against our document tree nodes, our agent can easily spot these gaps. ```text Run a gap analysis to find any DTC codes in our BigQuery repair history that have no matching documentation in our technical library. ``` Claude runs a database query to list all active diagnostic trouble codes, and then queries Neo4j's document tree nodes. It quickly flags that code `C0021` (brake booster sensor failure) is appearing in structured repair logs, but has no corresponding section in our technical manual library. This tells our operations team exactly where they need to write new documentation. --- Tips & Gotchas 1. Avoid Complex Custom Schemes Keep relationship names clean and uniform. If you use overly granular types like `HAS_FOLDER`, `HAS_FILE`, and `HAS_SUBSECTION`, your Cypher queries become verbose and hard to maintain. Sticking to a generic `HAS_SECTION` relationship type allows you to run elegant, variable-length path queries like `[:HAS_SECTION*]` to traverse the entire tree. 2. Guard Against Graph Schema Drift When you use agentic coding tools (like Claude Code) to build Cypher statements, always provide a clear schema definition file (`schema.md` or a dedicated system prompt). If you do not explicitly define your schema constraints, agents may write queries containing non-existent relationships or outdated node labels. 3. Handle Community Drift on Large Datasets Remember that Leiden community IDs are generated dynamically. If you add new documents to your library and rerun the Leiden algorithm, the community IDs will change. If you need to track thematic groups over time, save stable metadata snapshots or run an LLM-driven classifier over each detected cluster to assign permanent names and descriptions.