Nobody sets out to build a point-to-point telemetry estate. It arrives one integration at a time: an agent for a new source, a forwarder for a new destination, each reasonable on the day someone configured it. What accumulates is dozens of them, every one with its own configuration and its own way of failing.
What the layer is
One place where telemetry is collected, processed and routed, sitting between the systems that produce it and the systems that consume it.
Centralizing that logic buys two things the scattered version cannot. A transformation is written once instead of per agent. And there is somewhere to stand and see what is actually flowing, which is the precondition for changing any of it.
Building it
Architecture and migration. We start from what is running: which sources exist, what volume they produce, where it goes today. The target architecture follows from that. So does the order of migration, which is a question about risk rather than tidiness.
Collection across a mixed estate. Application logs, infrastructure metrics, distributed traces, security events. Collection is designed to sit lightly on the systems being observed, because an agent that costs production performance gets removed by the team running production, and then the data stops.
Processing. Parsing, enrichment, filtering and normalization, applied in the pipeline rather than at either end. Downstream platforms receive data already in the shape they expect.
Routing. Data goes where it is worth what it costs. Shaping data before it lands is where that gets decided in detail.
What it has to survive
A layer everything flows through is a layer everything depends on, so the failure modes are the design.
Telemetry does not wait. A destination that goes down does not pause the systems producing data, and the pipeline buffers rather than drops while that destination is unavailable. Backpressure handles the opposite case, where a source produces faster than the pipeline can accept.
Nothing runs as a single instance. Processing nodes are redundant and horizontally scalable, so an instance can be lost or taken out for upgrade without an interruption, and capacity is added by adding nodes rather than by moving to a bigger one. Where regulation requires it, deployments are regional, and data stays in the region that produced it.
Also worth knowing
- Millions of events per second, sustained
- Logs, metrics, traces and structured events
- Kubernetes, containers or serverless, depending on what you already run
- Isolated pipelines per team or business unit
- Teams onboard their own sources without waiting on us