Filter, reshape, and redact logs before export with Transformers

Filter, reshape, and redact logs before export with Transformers | Cloudflare Blog

Skip to content

Logpush delivers logs in batches as quickly as possible, enabling Cloudflare to provide information almost in real time. During our recent Birthday Week , we made Logpush available on all self-serve plans and Transformers  generally available. Those changes were among eight major updates to Cloudflare Observability  and reflect our commitment to make every Cloudflare feature available to everyone . This post takes a deeper look at how Transformers work inside Logpush's export pipeline. Observability teams use Logpush  to export Cloudflare logs to security information and event management systems (SIEMs), analytics platforms, and object storage. Before those logs reach their destination, teams may need to remove health checks, omit sensitive fields, or reshape records for dashboards and investigations. Today, this often requires a separate processing service that teams must deploy, scale, monitor, and maintain. For example, a team that collects logs from multiple CDNs can add a static provider field to normalize records before they reach its analytics platform. Teams also pay to export, ingest, and store records they discard as soon as they arrive, so filtering them before delivery can reduce downstream data volume and cost. With Transformers, that processing moves into the Logpush export workflow.Teams define a SQL query to filter records, redact sensitive data, and reshape the output by adding, renaming, removing, or nesting fields. Logpush applies the query before export, so each destination receives data in the form it needs. We made this workflow simple for customers by embedding a SQL transformation engine in Logpush's high-throughput export pipeline while keeping performance and resource use predictable. How Logpush Transformers work A Transformer is a SQL query written for a specific Logpush dataset. The query references fields from that dataset and defines which records and fields Logpush should export. Logpush applies the query to each batch before export. Records that do not match the WHERE clause are removed. For the remaining records, the output of the SELECT statement becomes the exported record. Transformers can change the structure and contents of each record, and the transformed results are exported in NDJSON format. Customers can create, preview, and manage Transformers in the Cloudflare dashboard through Transformer Studio or programmatically through the API . After saving a Transformer, they can attach it to a compatible NDJSON Logpush job that uses the same dataset. Each update creates a new version, so customers can review how a Transformer has changed over time. The following example reshapes a Cloudflare HTTP request log  to match a team-defined schema used across CDN providers. It does not convert between serialization formats: both the input and output use NDJSON. { "ClientIP" : "203.0.113.42" , "ClientRequestHost" : "example.com" , "ClientRequestMethod" : "POST" , "ClientRequestPath" : "/api/checkout" , "ClientRequestUserAgent" : "Mozilla/5.0" , "EdgeResponseStatus" : 502 , "EdgeStartTimestamp" : "2026-05-21T15:00:00Z" , "RayID" : "8e2a1c60ef9e1c9a" } The Transformer renames and nests fields, redacts part of the client IP address, adds provider and error information, removes unused fields, and filters out internal and health-check traffic: SELECT extract(epoch FROM to_timestamp(EdgeStartTimestamp)) AS unix_ts, regexp_replace(ClientIP, '\.\d+\.\d+$' , '.XXX.XXX' ) AS redacted_ip, UPPER (ClientRequestMethod) AS method, EdgeResponseStatus AS status , RayID AS ray_id, 'cloudflare' AS provider , named_struct( 'host' , ClientRequestHost, 'path' , ClientRequestPath ) AS request, EdgeResponseStatus >= 500 AS is_server_error FROM http_requests WHERE ClientRequestHost NOT IN ( 'internal.example.com' , 'health.example.com' ) AND ClientRequestPath NOT LIKE '/healthz%' AND ClientRequestPath NOT LIKE '/metrics%' The resulting NDJSON record uses the normalized schema: { "unix_ts" : 1779375600 , "redacted_ip" : "203.0.XXX.XXX" , "method" : "POST" , "status" : 502 , "ray_id" : "8e2a1c60ef9e1c9a" , "provider" : "cloudflare" , "request" : { "host" : "example.com" , "path" : "/api/checkout" }, "is_server_error" : true } How Transformers executes SQL with Apache DataFusion and Arrow Cloudflare's data pipeline ingests more than one billion events per second. Its Logpush delivery service, which we call Bufstream, delivers more than 110 million Logpush records per second to customer destinations. It runs across hundreds of Kubernetes pods, each claiming a time slice of data from Buftee , Cloudflare's distributed log-based in-house message broker. This model distributes work horizontally across Logpush jobs. Job volume follows a Pareto-like distribution: most jobs operate at lower volumes, while a small number of high-volume jobs process millions of logs each second. Transformers needed to meet the same operational requirements as the rest of Logpush: reliable delivery, high throughput, and predictable resource use without introducing additional failure scenarios into the export hot path. Our initial prototype ran customer-defined WebAssembly Transformers in Cloudflare Workers, which provides globally distributed, isolated compute. However, invoking a Worker would insert a network call into the Logpush export path. That remote dependency could introduce transient timeouts and retries, reducing delivery throughput and increasing delivery latency. We decided to keep Transformer execution in-process within our delivery service to prioritize reliability and delivery throughput. We evaluated several SQL execution engines and selected Apache DataFusion , which already powers Cloudflare Basin . Apache DataFusion is a Rust query engine built on Apache Arrow, with a SQL planner and vectorized, columnar execution engine. In a benchmark, executing a prepared Transformer against a synthetic 5,000-row http_requests batch from a single Go goroutine took about 3 ms, equivalent to roughly 1.6 million rows per second. Attaching a Transformer to a job adds CPU work to its processing path. Bufstream uses per-job transform parallelism so high-volume jobs can execute transform batches concurrently. Embedding DataFusion in Bufstream lets those batches use the service's existing delivery infrastructure, including its scheduling, batching, and destination controls, without introducing a separate...