NF
FAHAD.SYSTEMS
Backend & Systems
Project #02

Hyperion: 2.4M Event/Sec Distributed Telemetry & Analytics Engine

Architected high-throughput telemetry pipeline slashing query latency by 78% and infrastructure spend by 45%.

Specification & Technical Metadata
Topic / DomainBackend & Systems
RolePrincipal Infrastructure Architect
Duration9 months
Year2024
StatusProduction
Stack6 Core Techs
Tech Stack:Go 1.22Apache KafkaClickHouseRedis ClusterKubernetesGrafana
Hyperion real-time telemetry network topology and latency monitors
EXPAND

Hyperion real-time telemetry network topology and latency monitors

01 / The Problem: Crushing Ingestion Spikes

During major global game season launches, concurrent player spikes generated over 2,400,000 telemetry events every second. The legacy Elasticsearch and MongoDB cluster suffered from catastrophic write bottlenecks, resulting in backpressured ingestion queues, 45-minute analytics delay, and month-over-month cloud infrastructure bills exceeding $80,000.

Distributed Pipeline Topology

SYSTEM TOPOLOGY
Rendering architectural SVG topology...

ClickHouse & Postgres Schema Architecture

DATABASE ER SCHEMA
Rendering architectural SVG topology...

Performance Transformation

2.4M ev/s
Sustained Peak
Zero dropped frames
3.2 ms
P99 Ingestion Latency
Edge to Kafka broker
78% Faster
Query Speedup
Aggregate scans < 400ms
-45%
Monthly Infra Cost
Optimized compaction

High-Throughput Ring Buffer Dispatcher in Go

go
package ingestion

import (
	"sync/atomic"
	"time"
)

type EventBatcher struct {
	bufSize   int
	flushRate time.Duration
	queue     chan TelemetryPayload
	dropped   uint64
}

func (eb *EventBatcher) IngestNonBlocking(event TelemetryPayload) bool {
	select {
	case eb.queue <- event:
		return true
	default:
		// Atomic telemetry dropped counter for monitoring under backpressure
		atomic.AddUint64(&eb.dropped, 1)
		return false
	}
}

func (eb *EventBatcher) FlushWorker(flusher func([]TelemetryPayload) error) {
	batch := make([]TelemetryPayload, 0, eb.bufSize)
	ticker := time.NewTicker(eb.flushRate)
	defer ticker.Stop()

	for {
		select {
		case ev := <-eb.queue:
			batch = append(batch, ev)
			if len(batch) >= eb.bufSize {
				_ = flusher(batch)
				batch = batch[:0]
			}
		case <-ticker.C:
			if len(batch) > 0 {
				_ = flusher(batch)
				batch = batch[:0]
			}
		}
	}
}

02 / Why ClickHouse Replaced Document Stores

By structuring telemetry as columnar rows with LZ4 block compression, disk usage collapsed from 14 Terabytes to 1.8 Terabytes for the same 90-day retention window. Analytical aggregations (e.g. median player kill-death ratio by weapon type and map quadrant) that previously took 4 minutes now execute in 180 milliseconds.