log0
Architecture

Architecture Decisions

Architecture Decision Records (ADRs) for every significant design choice in log0 - why Kafka, why ClickHouse, why deterministic fingerprinting, why manual ACK, and more.

About These Records

Architecture Decision Records (ADRs) document the why behind design choices. Code shows you what was built. ADRs show you why it was built that way - including the alternatives that were considered and rejected.

Each record follows this format: Context (what problem forced a decision), Decision (what was chosen), Alternatives considered (what was rejected), and Consequences (trade-offs accepted).


ADR-001: Kafka-API Event Bus (Redpanda)

Status: Accepted · updated 2026-06 (Apache Kafka → Redpanda)

Context

log0 needs to move log events through a multi-stage processing pipeline: ingestion → normalization → clustering → incident creation → notification. The pipeline stages have different throughput requirements, can fail independently, and need to scale independently. The ingestion gateway must return immediately to the client - it cannot wait for downstream processing to complete.

A message queue or event bus is required to decouple the stages.

Decision

Use a Kafka-API event bus between all services. The deployed broker is Redpanda v24.2.7 - a single Kafka-compatible binary with no Zookeeper. The original local stack used Apache Kafka (Confluent 7.5.0); the migration to Redpanda changed only the bootstrap-servers address. The Kafka wire protocol, topics, consumer groups, and all producer/consumer code are identical across both, so everything below applies unchanged.

Alternatives Considered

AlternativeWhy Rejected
RabbitMQAMQP-based push model makes it harder to replay events. No native consumer group semantics. Log replay (for debugging or DLQ recovery) requires additional tooling.
AWS SQS/SNSVendor lock-in. No consumer group semantics - SQS requires one queue per consumer. Fan-out to multiple consumers requires SNS → multiple SQS queues. Replay is not native.
Redis StreamsSuitable for low-throughput use cases. At 10,000 logs/second, Redis becomes a bottleneck. No partition-level ordering guarantee keyed by tenantId.
Direct HTTPSynchronous coupling. If the normalization service is down, ingestion fails. Does not support the at-least-once reliability model.

Consequences

Accepted trade-offs:

  • An event bus is still infrastructure to run locally (Docker Compose) - though Redpanda makes this a single container
  • Message ordering is guaranteed per partition (per tenantId), not globally

Benefits realized:

  • Each service can scale independently based on its consumer lag
  • Consumer group semantics allow multiple normalization instances to share load automatically
  • Message replay is native - reprocessing from a given offset requires no extra tooling
  • The DLQ pattern is a natural extension of the topic model
  • Producer idempotence (enable.idempotence=true) prevents duplicate messages under retries
  • Redpanda specifically: no Zookeeper, one binary, lower memory (~1 GB saved vs a JVM broker + Zookeeper), and faster startup - ideal for the single-host deployment. See Deployment.

ADR-002: ClickHouse for Log Event Storage

Status: Accepted

Context

log0 stores every normalized log event for historical queries and AI summary generation. The access patterns are:

  1. High-volume writes: up to 10,000 events/second
  2. Aggregation queries: GROUP BY fingerprint ORDER BY COUNT DESC over millions of rows
  3. Time-range scans: "show all ERROR logs for tenant X in the last 24 hours"
  4. Filtered reads: "show logs matching fingerprint Y for incident Z"

These are analytical access patterns, not transactional ones.

Decision

Use ClickHouse as the log event store.

Alternatives Considered

AlternativeWhy Rejected
ElasticsearchExcellent full-text search, but expensive to operate at scale. Storage is 3–5x ClickHouse for the same data. Aggregation queries (GROUP BY, COUNT) are significantly slower on Elasticsearch than ClickHouse.
PostgreSQLAlready used for incident state. Not suitable for log storage - OLTP database, poor compression for time-series data, slow on large aggregations. GROUP BY fingerprint over millions of rows requires full table scans.
DynamoDBNo native aggregation support. Running GROUP BY queries requires exporting to Athena. Operational complexity disproportionate to the use case.
Loki (Grafana)Opinionated log label model. Limited query language. Better suited for infrastructure logs than application event analytics.

Consequences

Accepted trade-offs:

  • ClickHouse has a different query dialect (not standard SQL in all aspects)
  • ACID transactions are not supported - not appropriate for transactional writes
  • Learning curve for engineers familiar with PostgreSQL but not ClickHouse

Benefits realized:

  • Columnar storage means GROUP BY fingerprint reads only the fingerprint column - dramatically faster than row-based alternatives
  • Native compression reduces storage cost 5–10x compared to Elasticsearch for log data
  • DateTime64 type provides nanosecond precision for accurate log correlation
  • Map(String, String) type for attributes allows schema-free structured data without migrations

ADR-003: Deterministic SHA-256 Fingerprinting

Status: Accepted

Context

Incident deduplication requires grouping "similar" log events together. Two errors that differ only in dynamic values (user ID, IP address, timeout duration) should be treated as the same error pattern. The question is: how do you define "similar"?

Decision

Use deterministic SHA-256 fingerprinting based on a normalized message template.

The fingerprint formula:

fingerprint = SHA-256(
    serviceName + "|" +
    messageTemplate + "|" +    ← dynamic values stripped
    exceptionType + "|" +      ← nullable
    firstStackFrame            ← line numbers stripped
)

Message templating strips dynamic values before hashing:

  • Numbers → <number>
  • IP addresses → <ip>
  • UUIDs → <uuid>

Alternatives Considered

AlternativeWhy Rejected
ML-based semantic clusteringRequires training data, model maintenance, and introduces non-determinism. The same error may be clustered differently after a model update, breaking existing incidents. No reproducibility guarantee.
Edit distance / LevenshteinO(n²) complexity. At 10,000 events/second, pairwise comparison is computationally infeasible.
Exact message matchToo strict. Two instances of timeout after 30000ms and timeout after 28514ms would create separate incidents.
Exception class onlyToo loose. NullPointerException appears everywhere. Grouping by exception type only would aggregate unrelated errors.

Consequences

Accepted trade-offs:

  • Novel log formats with irregular structure may not template cleanly, producing noisy fingerprints
  • The regex patterns for stripping dynamic values must be maintained as log formats evolve
  • Edge cases: logs with no stack trace, logs in non-English languages, logs with custom dynamic patterns

Benefits realized:

  • Fully deterministic - the same log always produces the same fingerprint, regardless of time or deployment
  • No infrastructure required beyond a SHA-256 hash function
  • Fingerprints are stable across deployments - an incident created today and a reoccurrence next week will share the same fingerprint and be correctly linked
  • Zero false negatives for structurally identical errors (unlike fuzzy matching)

ADR-004: Always-ACK DLQ Pattern

Status: Accepted

Context

Kafka consumers in log0 use manual offset acknowledgment. When a consumer fails to process a message (normalization error, serialization failure, downstream Kafka publish failure), a decision is needed: what should happen to the unprocessed message?

The naive answer is to not ACK - let Kafka redeliver the message. But if the message itself is the problem (malformed payload, unexpected schema), it will fail on every redelivery, permanently stalling the partition.

Decision

On any processing failure:

  1. Wrap the original event in a DlqEvent (capturing originalEvent, errorMessage, failedAt, failedAtTs)
  2. Publish the DlqEvent to raw-logs-dlq
  3. Always ACK the original offset - even if the DLQ publish fails

Alternatives Considered

AlternativeWhy Rejected
Retry without DLQA bad message is retried indefinitely. One malformed event from a misconfigured service stalls the entire partition for all tenants sharing that partition.
Discard on failureData loss. No ability to replay or investigate failures.
DLQ only if retries exhaustedAdds retry delay before DLQ capture. For a known-bad message, retries add latency without benefit.
Block partition on failureGuarantees ordering but makes the system brittle. One bad actor (a service sending malformed logs) can halt the entire ingestion pipeline.

Consequences

Accepted trade-offs:

  • Messages that fail DLQ publish are lost (the ACK has already been sent). This is an acknowledged edge case - a failure to write to the DLQ is treated as fatal and triggers an alert.
  • The DLQ must be monitored. A growing DLQ is a silent signal that something is broken upstream.

Benefits realized:

  • The ingestion and normalization pipelines are never stalled by individual bad messages
  • Failed events are preserved with full context (originalEvent, error, timestamp, service) for post-mortem analysis
  • Replay is possible: fix the bug, re-publish originalEvent to raw-logs, the pipeline processes it correctly

ADR-005: Manual Kafka Offset Acknowledgment

Status: Accepted

Context

Kafka consumers can manage offset commits in two ways: automatically (Kafka commits periodically, regardless of processing state) or manually (the application commits only when it decides to).

Decision

Use manual offset acknowledgment for all Kafka consumers in log0.

Configuration:

kafka:
  consumer:
    enable-auto-commit: false
  listener:
    ack-mode: manual

Consumer code:

@KafkaListener(topics = "raw-logs")
public void consume(RawLogEvent event, Acknowledgment ack) {
    try {
        // process...
        producer.publish(normalized);
        ack.acknowledge();   // commit only on success
    } catch (Exception e) {
        dlqProducer.publish(dlqEvent);
        ack.acknowledge();   // commit after DLQ (ADR-004)
    }
}

Alternatives Considered

AlternativeWhy Rejected
Auto-commitKafka commits the offset after a configurable interval, regardless of whether processing succeeded. If the service crashes after committing but before completing processing, messages are silently lost. At-most-once semantics - not acceptable for incident data.

Consequences

Accepted trade-offs:

  • If the service crashes after processing but before ACKing, the message is reprocessed. Consumers must be idempotent.
  • Slightly more complex consumer code

Benefits realized:

  • At-least-once delivery guarantee - no message is ever silently lost
  • Combined with the DLQ pattern, every message either succeeds or is captured for replay

ADR-006: Custom Kafka Serializers

Status: Accepted

Context

Spring Kafka provides a JsonSerializer and JsonDeserializer that use Jackson to convert Java objects to/from bytes. They work automatically but rely on Spring's managed Jackson ObjectMapper, which introduces coupling to Spring's auto-configuration and Jackson version management.

Decision

Write custom serializers and deserializers for every Kafka event type.

Each serializer is a straightforward wrapper:

public class RawLogEventSerializer implements Serializer<RawLogEvent> {
    private final ObjectMapper objectMapper = new ObjectMapper()
        .registerModule(new JavaTimeModule());

    @Override
    public byte[] serialize(String topic, RawLogEvent data) {
        return objectMapper.writeValueAsBytes(data);
    }
}

Consequences

Accepted trade-offs:

  • More classes to maintain (one serializer + one deserializer per event type)
  • Jackson version must be explicitly managed per serializer

Benefits realized:

  • Full control over serialization behavior - no Spring magic
  • Custom serializers can be unit tested independently
  • Jackson version can be chosen per serializer (tools.jackson 3.x for ingestion, com.fasterxml.jackson 2.x for normalization - intentional due to Spring Boot 4 transitional period)
  • Schema evolution is explicit: changing a serializer is a deliberate code change, not an implicit configuration change

ADR-007: tenantId as the Kafka Partition Key

Status: Accepted

Context

Kafka assigns messages to partitions based on the message key. The partition a message lands on determines which consumer instance processes it (within a consumer group). The key design determines two things: ordering guarantees and load distribution.

Decision

Use tenantId as the Kafka message key for all topics in the data pipeline (raw-logs, normalized-logs). Use incidentId as the key for notification-events.

Reasoning

All messages with the same key land on the same partition, and Kafka guarantees ordering within a partition. Setting tenantId as the key means:

  1. All logs from the same tenant are processed in order. This is critical for the Clustering Service - if two logs from the same tenant are processed out of order, the clustering time window may be computed incorrectly.
  2. Tenant data is isolated to specific partitions. While this is not a security boundary (all consumers read all partitions), it simplifies reasoning about which consumer instance is handling which tenant's data.

notification-events uses incidentId as the key because multiple notifications about the same incident (created, assigned, resolved) must be processed in order by the Notification Service to correctly update the Slack message.

Consequences

Accepted trade-offs:

  • Uneven partition distribution if tenant log volume is highly skewed (one large tenant, many small ones). Mitigated at scale by sub-tenant keying (e.g., tenantId:serviceId).
  • A tenant cannot be distributed across multiple consumer instances within the same group - all their logs are on one partition. Horizontal scaling applies across tenants, not within a single tenant.

Benefits realized:

  • Per-tenant ordering guarantee - essential for correct clustering
  • Simple, predictable routing - given a tenantId, you know which partition its events are on
  • Natural foundation for per-tenant rate limiting at the partition level

ADR-008: Strategy Pattern for LLM Provider

Status: Accepted (Java implementation, March 2026). Updated September 2026: ai-service is Python; provider selection uses LangChain and LLM_* / API-key env vars. The goal-swap models without changing incident-service- is unchanged.

Context

The AI Summary Service must call an external LLM to generate incident summaries. The LLM landscape is fragmented: OpenAI, Azure OpenAI, Google Gemini, Anthropic, Groq, and self-hosted models all have different base URLs, authentication schemes, and request/response shapes. Hardcoding any single provider creates lock-in and makes development difficult (not everyone has a paid OpenAI key).

Decision

Use the Strategy Pattern to abstract LLM provider selection behind a single interface.

public interface LlmProvider {
    String generateSummary(String prompt);
}

Each provider implements this interface independently. The active implementation is selected at startup via @ConditionalOnProperty:

# application.yml
ai:
  provider: groq   # swap to: openai | gemini | anthropic | azure

Default provider: Groq. Groq offers a free API tier with an OpenAI-compatible endpoint (/openai/v1/chat/completions), making GroqProvider nearly identical in code to OpenAiProvider. This means the system works out-of-the-box with a free API key during development.

Alternatives Considered

AlternativeWhy Rejected
Hardcode OpenAIRequires a paid API key. Breaks local dev for contributors without one. Any provider change requires code surgery across the service.
LangChain4j or Spring AIRejected for the original Java service. The Python rewrite uses LangChain for summaries and planned LangGraph investigations.
Kafka event for summary requestAdds a new topic and consumer just to trigger an LLM call. The AI Service is already a dedicated service - a synchronous REST call from Incident Service is simpler and sufficient.

Consequences

Accepted trade-offs:

  • Each LlmProvider implementation must handle its own HTTP client, error handling, and retry logic. Some duplication across providers.
  • Adding a new provider requires a new class and a new @ConditionalOnProperty config block.

Benefits realized:

  • Swap providers by changing one config line - no code changes, no redeployment of other services
  • GroqProvider works with a free API key, unblocking local development
  • Each provider is independently unit-testable
  • Same pattern already used in the codebase: OccurrenceStore interface → InMemoryOccurrenceStore in the Clustering Service

Implementation update (September 2026)

The Java ai-service was replaced by a FastAPI service under services/ai-service/:

  • HTTP contract unchanged: POST /api/v1/summaries (202, background work), callback PATCH /api/v1/incidents/{id}/ai-summary with { "aiSummary": "..." }.
  • Providers: app/llm/ resolves LLM_SUMMARY, LLM_TRIAGE, etc. (provider:model) via LangChain; platform keys in .env (GROQ_API_KEY, …).
  • Observability: configure_tracing() mirrors LangSmith settings from pydantic-settings into os.environ.
  • Incident caller: AiSummarizer in incident-service uses RestTemplate + JSON body.

The Java LlmProvider / ai.provider YAML described above is historical reference for the first implementation.


ADR-009: Stateless JWT Validation in Each Service

Status: Accepted

Context

log0 has a dedicated auth-service (port 8086) that issues JWT access tokens on login. When a protected service (e.g. incident-service) receives a request carrying a JWT, it must verify the token is valid. There are two architectural options:

  1. Remote validation - each service calls auth-service on every request: GET /api/v1/auth/validate?token=...
  2. Local validation - each service validates the JWT locally using the shared JWT_SECRET signing key

Decision

Use local (stateless) JWT validation in every protected service via a JwtAuthFilter (OncePerRequestFilter). Each service holds the JWT_SECRET as an environment variable and verifies the HS256 signature and exp claim directly - no network call to auth-service.

API key validation (ingestion path) remains remote - the Ingestion Gateway calls POST /api/v1/auth/validate-key on auth-service because API keys require a database lookup (hash comparison against the API_KEY table) that cannot be done statelessly.

Alternatives Considered

AlternativeWhy Rejected
Remote JWT validation on every requestAdds network latency on every authenticated API call. Makes auth-service a synchronous dependency - if it is slow or down, all API calls fail. Unnecessary given JWT is cryptographically self-verifiable.
API Gateway handling all authWould require introducing Spring Cloud Gateway or a similar proxy. Adds infrastructure complexity that is not justified for the current number of services. Can be added later without changing the JWT design.
Opaque tokens (session IDs)Would require a stateful session store (Redis) shared across services. Adds infrastructure dependency and per-request DB lookup. JWT's stateless verifiability is a key advantage for a distributed system.

Consequences

Accepted trade-offs:

  • JWT_SECRET must be shared across all services that validate tokens. Secret rotation requires redeployment of all protected services simultaneously.
  • Issued tokens cannot be revoked before expiry (short 1-hour TTL mitigates this). Logout only revokes the refresh token - the access token remains valid until it expires.
  • Each protected service must include JwtAuthFilter and JwtUtil - small duplication, mitigated by a shared library (Phase 0.4).

Benefits:

  • Zero latency overhead on the hot API path - no auth-service call per request
  • auth-service is not a runtime dependency of incident-service for reads
  • Horizontally scalable - each service instance validates independently with no shared state

ADR-010: Python and LangGraph for the AI Service

Supersedes the "no framework" part of ADR-008. The summary path keeps its HTTP contract.

Context

ADR-008 gave the AI service a Strategy-pattern provider layer in Java, enough for one prompt in and one summary out. Investigating an incident is a different shape of work: gather evidence from several places in parallel, loop while the model wants more, stop for a human to review, resume after a restart, and never lose the run. That needs a stateful graph with checkpoints and a way to pause, which the Java ecosystem did not offer at the time.

Decision

Rewrite the AI service in Python 3.13 with FastAPI and LangGraph 1.2, with a Postgres checkpointer. The POST /api/v1/summaries contract is byte-compatible, so incident-service changed nothing. Nodes stay thin: the decisions between them live in plain, tested functions (graph/routing.py), and the graph builder only wires them.

Alternatives Considered

AlternativeWhy Rejected
Stay in Java (Spring AI, LangChain4j)No equivalent of interrupt-and-resume with durable checkpoints; weaker tooling for evaluation.
A hand-written state machineRe-implements checkpointing, parallel fan-out and interrupts, the parts that are hardest to get right.
A multi-agent "swarm"Hard to bound in cost and time, and hard to evaluate. One graph with explicit budgets is easier to reason about.

Consequences

  • The repository now has two languages; the AI service has its own Docker image and CI job (uv, ruff, pytest).
  • Every run has hard limits (tool calls, rounds, time) enforced in code, not requested from the model.
  • A run survives a restart: it resumes from its last checkpoint instead of starting again.

ADR-011: Structured Changes, Deterministic Diff

Context

When asked for a unified diff, language models often produce one that looks right and does not apply: wrong line numbers, mangled context, a missing newline marker. A fix the engineer cannot apply is worse than no fix.

Decision

The model proposes edits: file, the exact old text, the new text. Code does the rest. It checks that the old text occurs exactly once (overlapping matches count), refuses a match that differs only in whitespace, builds the unified diff itself with difflib, and runs git apply --check in a private worktree of the pinned commit. Only a diff that passes is stored. A failed validation goes back to the model once or twice with the exact reasons, then the run gives up honestly.

Alternatives Considered

AlternativeWhy Rejected
Trust the model's unified diffFails often enough to make the feature unusable.
Ask for whole filesCostly, and silently rewrites code the model did not mean to touch.

Consequences

  • A stored diff always applies cleanly to the commit the investigation read; a bad proposal never reaches the user.
  • Whitespace mismatches cost a retry. Diffs over 500 KB are refused.
  • The demo repository's own tests can be run against the diff in the evaluation (see ADR-017).

ADR-012: A Local Clone and ripgrep, Not Code-Search APIs or Embeddings

Context

The investigation has to read a customer's private code: find the file named in a stack trace, the commits that touched it, the line that fails.

Decision

Keep a disposable partial clone per repository, pin the commit at the start of the investigation, and use git and ripgrep through bounded, read-only tools (list, search, read, recent commits, blame, show commit). The token is injected for the duration of one git command and never written to .git/config. Secret-looking files (.env, keys) and paths that climb out of the checkout are refused.

Alternatives Considered

AlternativeWhy Rejected
GitHub code-search APIDefault branch only, rate limited, no blame or history.
Embeddings over the repositoryStale after every push, costly to build, and worse than exact search for debugging: a symbol name is a better key than a vector.

Consequences

  • Deterministic, citable results: every cited line was really returned by a tool.
  • Disk use per connected repository; the cache is safe to delete.
  • Only GitHub is supported, with a personal access token (read-only is enough).

ADR-013: pgvector in the Existing Postgres, with Plain SQL

Context

Runbook search and long-term memory need nearest-neighbour search over embeddings, per tenant, on a host with no room for another service.

Decision

Enable pgvector in the Postgres log0 already runs (the pgvector/pgvector:pg16 image keeps the same data volume). Use plain SQL: a vector(384) column, an HNSW index with cosine distance, and the tenant filter written out in the query. MMR selection, relevance grading and query rewriting are code on top. The default embedder is a built-in hashing embedder that needs no model or network; a Hugging Face sentence-transformer is opt-in because runbook text then leaves the service.

Alternatives Considered

AlternativeWhy Rejected
A vector database (Chroma, Qdrant, Pinecone)Another service to run and secure, with its own tenancy model.
langchain-postgres or LangGraph PostgresStoreHides the parts that matter most here (the tenant filter, the model filter, the transaction that replaces a runbook) and adds table layouts we do not control.

Consequences

  • One database to back up; tenant isolation is a WHERE clause with tests, and a mutation check proves the tests notice its removal.
  • The embedding size is part of the column type; another model size needs a migration.
  • With the default embedder, matching is by shared words, not meaning.

ADR-014: A Persisted Event Log Behind the Stream

Context

An investigation takes minutes. The console must show progress, survive a dropped connection, and show the same history to someone who opens the page later.

Decision

Every step writes an event (sequence number, type, small payload) to ai.investigation_event. The stream endpoint reads that log and sends Server-Sent Events with the sequence number as the event id; a reconnecting client sends Last-Event-ID and receives exactly what it missed. The console reads the stream with fetch, not EventSource (which cannot send an Authorization header), and falls back to polling /events?after= when streaming does not work. Events carry facts and one-line summaries, never prompts or raw log content.

Alternatives Considered

AlternativeWhy Rejected
WebSocketsBidirectional is not needed, and it is harder through the Vercel proxy.
In-memory progressLost on restart, invisible to a second viewer.

Consequences

  • The history of a finished run is the same log the live view used.
  • Waiting for a human review is not an end of stream: the run stays open across disconnects.
  • Streaming through a proxy that buffers degrades to polling instead of failing.

ADR-015: The Model Proposes, Code Decides

Context

Language models are fluent and sometimes wrong, and an incident report that cites evidence that does not exist, or states a confidence nobody earned, is worse than none.

Decision

Every claim is checked by code before it counts. A root cause may cite only evidence ids that exist in this run. A hypothesis is "supported" only when the cited evidence exists, and only a supported one can raise the confidence band, which comes from a deterministic rule (for example, two independent kinds of evidence), not from the model's own number. A file or commit may be cited only if a tool returned it. Every language-model step has a deterministic fallback: if triage fails the plan searches every allowed source; if a step cannot conclude, the run records a gap ("what we could not find out") instead of guessing. The model's own confidence is logged for calibration and never shown.

Alternatives Considered

AlternativeWhy Rejected
Show the model's confidenceNot calibrated, and it reads as authority.
Let the model self-verify onlyThe same failure mode checks itself.

Consequences

  • Some correct answers are downgraded or dropped when the evidence behind them cannot be shown. That is intended.
  • Gaps are part of the product: the report says what was not examined.

Context

Logs, stack traces, code and runbooks contain things nobody meant to send to a model vendor: keys, passwords in connection strings, customers' e-mail addresses. A trace sent to a monitoring service is a second copy.

Decision

Redaction happens where text enters the investigation, not where it is printed. Every piece of evidence is redacted when it is created (one validator on the Evidence type, so no node or tool can skip it); the incident's messages are redacted where they are read; tool output a model reads is redacted while the exact result is kept for grounding; the question, stored memories and everything returned over MCP are redacted too. Matches become [REDACTED:kind], and an event says how many values of which kinds were hidden, never what they were. Tracing (LangSmith) follows consent: tenants that brought their own LLM key are traced only if they opted in; tenants on the platform's keys are traced when tracing is on.

Alternatives Considered

AlternativeWhy Rejected
Redact in the prompt buildersOne forgotten builder leaks.
Trust the provider's retention termsNot a control we own.

Consequences

  • The model sees markers, so it knows something was there.
  • Redaction is pattern based: a password written in prose is not caught. It is one layer beside tenant scoping, read-only tools and refusing secret files, not a guarantee.
  • A secret already in the repository can still appear in a diff's context lines; the evaluation judges only what the system adds.

ADR-017: Evaluate on Recorded Incidents, Planted Bugs and a Scripted Self-Test

Context

"It seems to work" is not evidence, and an evaluation that needs a running stack cannot gate a pull request. Real customer incidents cannot be shared or reproduced.

Decision

Build the evidence ourselves. A demo shop in three languages (Node, Go, Rust) has eight planted bugs, one commit each, plus an isolated branch per bug so a case has exactly one cause; its answer key (bugs.yml) is checked against git by a script. Each case's incident and the exact rows the tools read are recorded from a real run of log0's own pipeline, then replayed through fixture backends, so the production nodes run unchanged and offline. Evaluators are deterministic where possible (localization, grounding, fix pattern, diff validity, safety, tests passing with the fix applied) with an optional model judge for the root cause. A scripted model answers perfectly, wrongly or hostilely so the harness itself can be tested: it proves the evaluators score a perfect run as perfect and a bad one as bad. A committed baseline gates regressions; real-model runs are manual and nightly, never on pull requests.

Alternatives Considered

AlternativeWhy Rejected
Hand-written fixturesThey encode what we expect rather than what the pipeline produces.
Judge only with a modelSlow, costly, and itself unmeasured.
Evaluate only on a live stackCannot run in CI.

Consequences

  • The harness found real defects before any model was involved (Rust dependency frames counted as application code, a redaction hole for tokens with underscores, no fix ever proposed by default). Those are fixed and tested.
  • Numbers from the scripted model test the harness, not log0's quality. Quality numbers need a real-model run, which sends the synthetic logs and the demo code to the provider.
  • Eight bugs and one canary is a small set: it is a regression gate and a comparison tool, not a benchmark.

ADR-018: MCP Is an Adapter Over the Same Tools

Context

Engineers already work in Claude Desktop and Cursor. log0's tools are useful there, but a second implementation would drift from the one the investigation uses.

Decision

An MCP server exposes ten read-only tools that call the same functions as the graph. The tenant is never a tool argument: it comes from the verified token (per request over HTTP, once for a stdio session, with the token's expiry checked on every call). The HTTP endpoint is stateless, so one client's identity cannot leak into another's, and is off unless enabled. Expected failures become short safe messages; unexpected ones become a generic message with the real cause only in the server log. Results are redacted like everything else.

Alternatives Considered

AlternativeWhy Rejected
MCP as the internal architectureCouples the graph to a protocol it does not need.
Tenant as an argumentA client could ask about another tenant.

Consequences

  • A fix to a tool benefits the investigation and MCP at once.
  • A stdio token is a secret with the reach of its tenant; it expires and is not refreshed.

ADR-019: With the Review Gate Off, a Root Cause Is Approved

Found by the evaluation harness; recorded because it changed a default.

Context

The review gate pauses an investigation after the root cause so an engineer can approve, refine or stop. It is off by default. With the gate off the graph ended at the root cause: the review decision is what routes to the step that proposes a fix, so a default deployment never produced a diff, and the demo guide said it did.

Decision

Gate off means "no human in the loop", which means approved: a root cause is approved automatically when the server can read code, and a fix is proposed. A server with no code access still ends at the root cause, silently, as before. Turning the gate on restores the pause.

Alternatives Considered

AlternativeWhy Rejected
Turn the gate on by defaultA waiting investigation blocks a new one for the same incident until someone answers; not a safe default.
Document that diffs need the gateThe feature would look broken out of the box.

Consequences

  • Every investigation with a mapped repository now spends the extra tool calls and model turns of a fix proposal.
  • A proposal is never applied: it is a validated diff to read, copy or download.

How is this guide?

On this page

About These Records
ADR-001: Kafka-API Event Bus (Redpanda)
Context
Decision
Alternatives Considered
Consequences
ADR-002: ClickHouse for Log Event Storage
Context
Decision
Alternatives Considered
Consequences
ADR-003: Deterministic SHA-256 Fingerprinting
Context
Decision
Alternatives Considered
Consequences
ADR-004: Always-ACK DLQ Pattern
Context
Decision
Alternatives Considered
Consequences
ADR-005: Manual Kafka Offset Acknowledgment
Context
Decision
Alternatives Considered
Consequences
ADR-006: Custom Kafka Serializers
Context
Decision
Consequences
ADR-007: tenantId as the Kafka Partition Key
Context
Decision
Reasoning
Consequences
ADR-008: Strategy Pattern for LLM Provider
Context
Decision
Alternatives Considered
Consequences
Implementation update (September 2026)
ADR-009: Stateless JWT Validation in Each Service
Context
Decision
Alternatives Considered
Consequences
ADR-010: Python and LangGraph for the AI Service
Context
Decision
Alternatives Considered
Consequences
ADR-011: Structured Changes, Deterministic Diff
Context
Decision
Alternatives Considered
Consequences
ADR-012: A Local Clone and ripgrep, Not Code-Search APIs or Embeddings
Context
Decision
Alternatives Considered
Consequences
ADR-013: pgvector in the Existing Postgres, with Plain SQL
Context
Decision
Alternatives Considered
Consequences
ADR-014: A Persisted Event Log Behind the Stream
Context
Decision
Alternatives Considered
Consequences
ADR-015: The Model Proposes, Code Decides
Context
Decision
Alternatives Considered
Consequences
ADR-016: Redact at the Door, and Trace Only With Consent
Context
Decision
Alternatives Considered
Consequences
ADR-017: Evaluate on Recorded Incidents, Planted Bugs and a Scripted Self-Test
Context
Decision
Alternatives Considered
Consequences
ADR-018: MCP Is an Adapter Over the Same Tools
Context
Decision
Alternatives Considered
Consequences
ADR-019: With the Review Gate Off, a Root Cause Is Approved
Context
Decision
Alternatives Considered
Consequences