Cloud-Native Observability Stack Part 3 - Structured Logging with Correlation IDs

Series Introduction

1. Part 1: OpenTelemetry Instrumentation
2. Part 2: Distributed Tracing Across Microservices
3. Part 3: Structured Logging with Correlation IDs (Current)
4. Part 4: Metrics and Alerting with Prometheus/Grafana
5. Part 5: Debugging Production Issues with Observability Data

Problems with Traditional Logging

Limitations of traditional logging approaches:

2026-01-27 10:30:15 INFO OrderService - Processing order
2026-01-27 10:30:15 ERROR PaymentService - Payment failed
2026-01-27 10:30:15 INFO OrderService - Order completed

• Cannot identify which request a log belongs to
• No correlation between service logs
• Difficult to search and filter

Structured Logging

JSON Log Format

{
  "timestamp": "2026-01-27T10:30:15.123Z",
  "level": "INFO",
  "service": "order-service",
  "traceId": "abc123",
  "spanId": "def456",
  "correlationId": "req-789",
  "message": "Processing order",
  "order.id": "order-123",
  "customer.id": "cust-456",
  "thread": "http-nio-8080-exec-1"
}

Logback JSON Configuration

<!-- logback-spring.xml -->
<configuration>
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>spanId</includeMdcKeyName>
            <includeMdcKeyName>correlationId</includeMdcKeyName>
            <customFields>{"service":"order-service"}</customFields>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="JSON"/>
    </root>
</configuration>

MDC (Mapped Diagnostic Context)

MDC Filter Configuration

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
class CorrelationIdFilter : OncePerRequestFilter() {

    override fun doFilterInternal(
        request: HttpServletRequest,
        response: HttpServletResponse,
        filterChain: FilterChain
    ) {
        try {
            val correlationId = request.getHeader("X-Correlation-ID")
                ?: UUID.randomUUID().toString()

            MDC.put("correlationId", correlationId)
            MDC.put("requestPath", request.requestURI)
            MDC.put("requestMethod", request.method)

            response.setHeader("X-Correlation-ID", correlationId)

            filterChain.doFilter(request, response)
        } finally {
            MDC.clear()
        }
    }
}

OpenTelemetry Trace ID Integration

@Component
class TraceIdMdcFilter : OncePerRequestFilter() {

    override fun doFilterInternal(
        request: HttpServletRequest,
        response: HttpServletResponse,
        filterChain: FilterChain
    ) {
        try {
            val span = Span.current()
            if (span.spanContext.isValid) {
                MDC.put("traceId", span.spanContext.traceId)
                MDC.put("spanId", span.spanContext.spanId)
            }
            filterChain.doFilter(request, response)
        } finally {
            MDC.remove("traceId")
            MDC.remove("spanId")
        }
    }
}

Logging Best Practices

Meaningful Log Messages

// Bad example
logger.info("Processing")
logger.error("Error occurred")

// Good example
logger.info("Starting order processing", kv("orderId", orderId), kv("customerId", customerId))
logger.error("Payment processing failed", kv("orderId", orderId), kv("errorCode", e.errorCode), e)

Structured Logging Utility

import net.logstash.logback.argument.StructuredArguments.kv

@Service
class OrderService(private val logger: Logger = LoggerFactory.getLogger(OrderService::class.java)) {

    fun processOrder(order: Order) {
        logger.info("Order processing started",
            kv("orderId", order.id),
            kv("customerId", order.customerId),
            kv("itemCount", order.items.size),
            kv("totalAmount", order.totalAmount)
        )

        try {
            // Processing logic
            logger.info("Order processing completed",
                kv("orderId", order.id),
                kv("processingTimeMs", processingTime)
            )
        } catch (e: Exception) {
            logger.error("Order processing failed",
                kv("orderId", order.id),
                kv("errorType", e.javaClass.simpleName),
                kv("errorMessage", e.message),
                e
            )
            throw e
        }
    }
}

Propagating Correlation ID in Kafka Messages

@Component
class KafkaCorrelationInterceptor : ProducerInterceptor<String, String> {

    override fun onSend(record: ProducerRecord<String, String>): ProducerRecord<String, String> {
        MDC.get("correlationId")?.let { correlationId ->
            record.headers().add("X-Correlation-ID", correlationId.toByteArray())
        }
        MDC.get("traceId")?.let { traceId ->
            record.headers().add("X-Trace-ID", traceId.toByteArray())
        }
        return record
    }
}

@Component
class OrderEventConsumer {

    @KafkaListener(topics = ["order-events"])
    fun handleOrderEvent(
        @Payload payload: String,
        @Header("X-Correlation-ID") correlationId: String?,
        @Header("X-Trace-ID") traceId: String?
    ) {
        try {
            correlationId?.let { MDC.put("correlationId", it) }
            traceId?.let { MDC.put("traceId", it) }

            logger.info("Processing order event", kv("eventType", "OrderCreated"))
            // Event processing
        } finally {
            MDC.clear()
        }
    }
}

Log Level Guidelines

Level	When to Use	Examples
ERROR	Immediate action required	Payment failure, DB connection failure
WARN	Potential issues	Retry occurred, performance degradation
INFO	Business events	Order created, user login
DEBUG	Development/debugging	Method entry/exit, variable values
TRACE	Detailed debugging	Values inside loops

Log Aggregation

Loki Configuration (Grafana Stack)

# docker-compose.yml
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yaml

LogQL Query Examples

# Query all service logs for a specific traceId
{service=~".+"} |= "traceId=abc123"

# Filter only error logs
{service="order-service"} | json | level="ERROR"

# Logs related to a specific orderId
{service=~".+"} | json | orderId="order-123"

Summary

Key aspects of structured logging:

Item	Description
JSON Format	Easy parsing and searching
Correlation ID	Track requests across services
MDC	Thread-level context management
Trace ID Integration	Unify distributed tracing and logs

In the next post, we’ll cover metrics and alerting with Prometheus/Grafana.