Field notes on log intelligence and incident management at scale.
Architecture decisions, load-test findings, and hard-won lessons from shipping a multi-tenant incident platform solo, on one laptop. Each post drills into one box on the diagram.
Overview: log0's multi-tenant incident pipeline on Kafka
Series map for log0: HTTP log ingest, fingerprint deduplication, windowed clustering, incident lifecycle, and Slack notification across seven Spring Boot services-with load-test figures from a single-laptop deployment.
When /actuator/health stays UP while log ingestion blocks on Kafka
log0's first real POST to the ingestion gateway hung for about 60 seconds while every liveness check returned 200. The failure was a Kafka producer blocking on max.block.ms against an unreachable broker address.
Deterministic SHA-256 log fingerprinting for incident deduplication
log0 normalizes each error line and hashes a stable signature so repeated occurrences share one fingerprint-no ML, O(1) per event. Includes regex ordering pitfalls and the under-dedup behavior used on ambiguous messages.
Windowed fingerprint counting for incident detection in Kafka
log0's clustering consumer increments per-fingerprint counts in 5-minute tumbling event-time windows and publishes one incident-events record when a window reaches ten occurrences-benchmark p50 403ms from log to incident row.
Returning HTTP 202 after Kafka enqueue in the log ingestion gateway
log0's ingestion gateway responds with 202 Accepted once the log event is submitted to the Kafka producer, without waiting for broker acknowledgement, clustering, Postgres, or notification-measured with k6 from 50 to 800 virtual users.
Single-row INSERTs in ClickHouse: why normalization became the bottleneck
After accept-fast ingestion, log0's normalization consumer capped out at about 40 rows per second while Kafka consumer lag peaked at 67,000 messages. This post explains MergeTree part churn and why one-row INSERTs do not fit a columnar write path.
Batching ClickHouse INSERTs in a Kafka normalization pipeline
log0 buffers normalized log events and flushes with JDBC batch INSERTs (500 rows or 1 second). On the same hardware, write throughput rose from 40 to 4,185 rows per second and consumer lag went to zero.
Using Kafka topic lag as backpressure in an ingestion pipeline
When producers outrun consumers, durable topic lag can absorb bursts without blocking clients, as long as the broker has headroom. Measurements from log0 before and after fixing the ClickHouse writer, and under broker memory pressure.
Designing tenant isolation across Kafka, ClickHouse, and Postgres
log0 derives tenant identity from validated API keys, keys Kafka records by tenantId, and scopes analytical and relational queries. The post also covers gaps in per-tenant fairness on a shared topic.
Manual Kafka acknowledgements and a dead-letter path for poison logs
log0's normalization consumer commits offsets only after a successful write or a publish to raw-logs-dlq. The post covers the consume loop, fault injection, and remaining gaps for DLQ replay and alerts.
Modeling incident lifecycle with an enforced state machine
Incidents in log0 move through NEW, ASSIGNED, ACKNOWLEDGED, and RESOLVED under server-side rules, with immutable history rows and idempotent upserts for at-least-once Kafka delivery.
Decoupling incident creation from LLM summary generation
log0 persists incidents as soon as detection completes and requests summaries on a separate async path with an HTTP callback. The post explains the boundary, provider rate limits, and why HTTP 202 means accepted, not summarized.
Resource limits and health checks when Redpanda runs on one machine
A single-host resource profile of log0 under ingest load: which containers consume CPU and memory, why Redpanda hit OOM first, and why Spring liveness did not reflect broker availability.
Packaging shared UI as a copy-in registry (charfield)
log0's ASCII canvas animations were extracted into charfield, an npm CLI that copies field source into your repository so each app owns the code without a shared runtime dependency.
Building log0's AI Service: Chains Before Agents
Why log0 uses a LangChain LCEL pipeline for auto incident summaries and LangGraph only for Investigate with AI. Python migration, structured output boundaries, LangSmith tracing, and chains vs agents as an engineering choice.