Ultra-Fast Anomaly Detection using Apache Spark Real-Time Mode

Ultra-Fast Anomaly Detection using Apache Spark Real-Time Mode | Databricks Blog Skip to main content

This post establishes a reusable pattern for operational workloads that genuinely move the needle: fraud detection, IoT sensor monitoring, real-time personalization, security signal processing—any scenario where immediate response to events is critical for business outcomes. The core objective: When an event appears suspicious or invalid, flag it immediately and route it for appropriate downstream action. In this blog, we demonstrate anomaly detection on Ethereum blockchain transactions. We analyze Ethereum blockchain data and flag transactions with invalid patterns in real-time. Specifically, we detect: Data quality violations: Blocks where gas_used > gas_limit are physically impossible under the Ethereum protocol, indicating data corruption, producer bugs, or schema parsing failures Payload hygiene violations: The extra_data field containing recognizable PII or credential patterns (email addresses, JWT tokens, AWS access keys) signals data leakage or misconfigured producers

While we use Ethereum data for demonstration, this "suspicious or invalid" classification applies across numerous high-value use cases: Fraud detection: A transaction exhibits anomalous patterns → trigger downstream investigation workflows IoT monitoring: A sensor reading falls outside physically possible parameters → initiate automated response Security operations: Payload contains secrets or PII patterns → quarantine in real time with unified governance Personalization engines: Respond to specific behavioral events with immediate, contextual offers

The same detection logic maps directly onto any domain: financial transactions carrying unexpected PII, IoT payloads with out-of-range sensor readings, or API event logs containing secrets that should have been redacted. The Ethereum stream provides a clean, reproducible dataset to demonstrate the pattern at scale. About Spark Real-Time Mode Real-Time Mode (RTM) is a new trigger type for Apache Spark™ Structured Streaming that delivers millisecond-level latency to Spark APIs — without a separate, specialized engine like Apache Flink . While Structured Streaming's default microbatch mode operates like an airport shuttle bus that waits to fill up before departing, RTM operates like a high-speed moving walkway, processing each event as it arrives. It achieves this through three architectural innovations: continuous data flow (events are processed as they arrive, not in discrete chunks),  pipeline scheduling  (all query stages run simultaneously, with no blocking), and  streaming shuffle  (data is passed between tasks immediately in memory, bypassing disk). RTM is purpose-built for operational workloads where latency directly impacts business outcomes — fraud detection, real-time personalization, ML feature computation, and IoT monitoring. For workloads that can tolerate 1–2 seconds of latency, traditional microbatch remains the more cost-effective choice.

For this anomaly detection pipeline, Real-Time Mode enables immediate flagging and routing of suspicious events—exactly the response time these use cases demand.

Why RTM is a Game Changer 1. Ultra-Fast: Sub-Second Latency Now Achievable Real-Time Mode fundamentally changes what's possible with Apache Spark. End-to-end latencies  ranging from ~5ms to ~300ms, depending on workload complexity,  bring Spark into the territory previously dominated by specialized stream processing engines. Traditional micro-batch delivers 1–2 second latency; Real-Time Mode achieves ~5ms to ~300ms. The architecture achieves this through pre-allocated execution pipelines and asynchronous checkpointing, eliminating the scheduling overhead that traditionally constrained micro-batch processing. For operational workloads where milliseconds matter—fraud detection, IoT monitoring, real-time offers—this level of performance is transformative. 2. Simplified Stack: No Separate Technology Required Organizations frequently encounter a costly misconception: "Spark isn't performant enough for real-time use cases, so we need an entirely separate stack for this one requirement." For workloads tolerating 1–2 seconds of latency, Spark micro-batch reliably delivers data into Delta Lake with strong price/performance characteristics. For operational workloads demanding sub-second response times, Real-Time Mode eliminates the need for separate technologies entirely — as validated by teams at Coinbase , DraftKings , and MakeMyTrip who consolidated onto a single Spark-based stack for both analytical and operational workloads. With Real-Time Mode, Spark handles both analytical (second-range) and operational (millisecond-range) workloads within a single, unified platform. This reduces: Operational complexity: One technology stack to manage, monitor, and troubleshoot Training overhead: Existing Spark expertise transfers directly to real-time use cases Integration friction: No complex handoffs between separate streaming engines Total cost of ownership: Consolidation reduces infrastructure, licensing, and operational costs

3. Developer-Friendly: Simple Trigger Change, No Code Rewrites Perhaps the most compelling aspect of Real-Time Mode is its remarkable simplicity for developers already familiar with Structured Streaming. Enabling this powerful capability requires no complex migration or fundamental code restructuring. Organizations can unlock millisecond-level latency by simply modifying the trigger configuration:

That's it. The same familiar Structured Streaming API. The same checkpoint management. The same at-least-once delivery semantics. Just one configuration change to enable sub-second operational intelligence. Note on Delivery Guarantees: RTM with Kafka sink provides at-least-once delivery guarantees. Downstream consumers should handle potential duplicates via idempotent writes or deduplication logic. This seamless integration represents a critical advantage. Teams can prototype and productionize operational workloads without the substantial overhead of learning, deploying, and managing entirely separate technology stacks. This approach dramatically accelerates innovation while reducing the risk traditionally associated with adopting new real-time capabilities. Having established why Real-Time Mode matters, let's examine how to implement this pattern in practice. The following sections demonstrate a production-ready...