JP Morgan Chase

Traditional API monitoring systems miss silent, structural failures because they rely on aggregate metrics. If a critical validation step is skipped, overall latency might actually drop, signaling a "healthy" API to standard dashboards while leaving security vulnerabilities wide open. Ritvik Pandya, payments team lead at JP Morgan Chase, introduced a system designed to catch these invisible failures using learned execution graphs. Shifting From Static Trees to Short-Lived Graphs Unlike persistent property databases, an execution graph is short-lived. It exists only to model a single request as a Directed Acyclic Graph (DAG). Every hop—from edge gateways and authentication to parallel service calls and client notifications—becomes a node. Representing request processing as a DAG structures the execution order and locks in the context passed between steps. It also handles edge cases like loops and retries. By modeling each loop iteration as a distinct, tracked entity, the system isolates execution anomalies without breaking the overall graph structure. Moving Beyond Single-Threshold Alerting Standard monitoring relies on broad, static thresholds that trigger noisy alerts. The execution graph architecture uses a tiered inspection strategy to minimize resource overhead. Think of it as a basic check at airport security. If the end-to-end trace matches the learned baseline, the system skips deeper inspection. The architecture only triggers tier-two diagnostics when it detects anomalies, applying client-specific baselines to prevent false positives. For example, local clients and international API consumers have naturally distinct latency signatures. Evaluating them against the same threshold causes alert fatigue; partitioning them keeps checks highly targeted. How to Separate Temporary Anomalies From Systemic Drift System failures fall into two categories: sudden anomalies and gradual drift. An anomaly is a sudden, one-off latency spike, like getting stuck in traffic. Systemic drift is a slow, structural shift, like a population increase that permanently increases your daily commute time by twenty minutes. Ritvik Pandya breaks drift down into three categories: * **Structural Drift**: A node is added or removed. A cafe introducing a mandatory loyalty check at checkout is a structural drift; your baseline execution path has permanently changed. * **Volume Drift**: Traffic surges overload downstream dependencies, requiring horizontal scaling or asynchronous processing changes. * **Covariate Drift**: The incoming request mix changes. A sudden surge in international transactions shifting the baseline currency-exchange ratio represents a covariate change. The system is healthy, but the aggregate baseline must adapt. Building a Zero-Impact Telemetry Pipeline To run this under heavy production loads without degrading payment processing latency, JP Morgan Chase routes trace data asynchronously. The architecture feeds data into OpenTelemetry and stream processors, splitting the evaluation into a fast "hot path" for immediate, automated mitigation and a slower "reconciliation path" for precise, historical baseline training. Tuning this pipeline requires handling delayed telemetry events. If a single node delays sending its trace data, the engine might misinterpret the missing node as a structural change. The system implements tail-based sampling and cold-start rules for new endpoints to ensure it only alerts on real issues, dropping the mean time to discovery down to seconds.