울이

Love Code. Work. Sports.

3 분 소요

시리즈 소개

  1. Part 1: OpenTelemetry Instrumentation
  2. Part 2: 마이크로서비스 분산 추적
  3. Part 3: 구조화된 로깅과 Correlation ID (현재 글)
  4. Part 4: Prometheus/Grafana로 메트릭과 알림
  5. Part 5: Observability 데이터로 프로덕션 이슈 디버깅

로그의 문제점

기존 로그 방식의 한계:

2026-01-27 10:30:15 INFO OrderService - Processing order
2026-01-27 10:30:15 ERROR PaymentService - Payment failed
2026-01-27 10:30:15 INFO OrderService - Order completed
  • 어떤 요청의 로그인지 알 수 없음
  • 서비스 간 로그 연관성 부재
  • 검색과 필터링이 어려움

구조화된 로깅 (Structured Logging)

JSON 로그 포맷

{
  "timestamp": "2026-01-27T10:30:15.123Z",
  "level": "INFO",
  "service": "order-service",
  "traceId": "abc123",
  "spanId": "def456",
  "correlationId": "req-789",
  "message": "Processing order",
  "order.id": "order-123",
  "customer.id": "cust-456",
  "thread": "http-nio-8080-exec-1"
}

Logback JSON 설정

<!-- logback-spring.xml -->
<configuration>
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>spanId</includeMdcKeyName>
            <includeMdcKeyName>correlationId</includeMdcKeyName>
            <customFields>{"service":"order-service"}</customFields>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="JSON"/>
    </root>
</configuration>

MDC (Mapped Diagnostic Context)

MDC 필터 설정

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
class CorrelationIdFilter : OncePerRequestFilter() {

    override fun doFilterInternal(
        request: HttpServletRequest,
        response: HttpServletResponse,
        filterChain: FilterChain
    ) {
        try {
            val correlationId = request.getHeader("X-Correlation-ID")
                ?: UUID.randomUUID().toString()

            MDC.put("correlationId", correlationId)
            MDC.put("requestPath", request.requestURI)
            MDC.put("requestMethod", request.method)

            response.setHeader("X-Correlation-ID", correlationId)

            filterChain.doFilter(request, response)
        } finally {
            MDC.clear()
        }
    }
}

OpenTelemetry Trace ID 연동

@Component
class TraceIdMdcFilter : OncePerRequestFilter() {

    override fun doFilterInternal(
        request: HttpServletRequest,
        response: HttpServletResponse,
        filterChain: FilterChain
    ) {
        try {
            val span = Span.current()
            if (span.spanContext.isValid) {
                MDC.put("traceId", span.spanContext.traceId)
                MDC.put("spanId", span.spanContext.spanId)
            }
            filterChain.doFilter(request, response)
        } finally {
            MDC.remove("traceId")
            MDC.remove("spanId")
        }
    }
}

로깅 모범 사례

의미 있는 로그 메시지

// 나쁜 예
logger.info("Processing")
logger.error("Error occurred")

// 좋은 예
logger.info("Starting order processing", kv("orderId", orderId), kv("customerId", customerId))
logger.error("Payment processing failed", kv("orderId", orderId), kv("errorCode", e.errorCode), e)

구조화된 로깅 유틸리티

import net.logstash.logback.argument.StructuredArguments.kv

@Service
class OrderService(private val logger: Logger = LoggerFactory.getLogger(OrderService::class.java)) {

    fun processOrder(order: Order) {
        logger.info("Order processing started",
            kv("orderId", order.id),
            kv("customerId", order.customerId),
            kv("itemCount", order.items.size),
            kv("totalAmount", order.totalAmount)
        )

        try {
            // 처리 로직
            logger.info("Order processing completed",
                kv("orderId", order.id),
                kv("processingTimeMs", processingTime)
            )
        } catch (e: Exception) {
            logger.error("Order processing failed",
                kv("orderId", order.id),
                kv("errorType", e.javaClass.simpleName),
                kv("errorMessage", e.message),
                e
            )
            throw e
        }
    }
}

Kafka 메시지에서 Correlation ID 전파

@Component
class KafkaCorrelationInterceptor : ProducerInterceptor<String, String> {

    override fun onSend(record: ProducerRecord<String, String>): ProducerRecord<String, String> {
        MDC.get("correlationId")?.let { correlationId ->
            record.headers().add("X-Correlation-ID", correlationId.toByteArray())
        }
        MDC.get("traceId")?.let { traceId ->
            record.headers().add("X-Trace-ID", traceId.toByteArray())
        }
        return record
    }
}

@Component
class OrderEventConsumer {

    @KafkaListener(topics = ["order-events"])
    fun handleOrderEvent(
        @Payload payload: String,
        @Header("X-Correlation-ID") correlationId: String?,
        @Header("X-Trace-ID") traceId: String?
    ) {
        try {
            correlationId?.let { MDC.put("correlationId", it) }
            traceId?.let { MDC.put("traceId", it) }

            logger.info("Processing order event", kv("eventType", "OrderCreated"))
            // 이벤트 처리
        } finally {
            MDC.clear()
        }
    }
}

로그 레벨 가이드라인

레벨 사용 시점 예시
ERROR 즉각적인 조치 필요 결제 실패, DB 연결 실패
WARN 잠재적 문제 재시도 발생, 성능 저하
INFO 비즈니스 이벤트 주문 생성, 사용자 로그인
DEBUG 개발/디버깅용 메서드 진입/종료, 변수 값
TRACE 상세 디버깅 루프 내부 값

로그 집계 (Log Aggregation)

Loki 설정 (Grafana Stack)

# docker-compose.yml
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yaml

LogQL 쿼리 예시

# 특정 traceId로 모든 서비스 로그 조회
{service=~".+"} |= "traceId=abc123"

# 에러 로그만 필터링
{service="order-service"} | json | level="ERROR"

# 특정 orderId 관련 로그
{service=~".+"} | json | orderId="order-123"

정리

구조화된 로깅의 핵심:

항목 설명
JSON 포맷 파싱과 검색이 용이
Correlation ID 서비스 간 요청 추적
MDC 스레드별 컨텍스트 관리
Trace ID 연동 분산 추적과 로그 통합

다음 글에서는 Prometheus/Grafana를 활용한 메트릭과 알림을 다루겠습니다.

Series Introduction

  1. Part 1: OpenTelemetry Instrumentation
  2. Part 2: Distributed Tracing Across Microservices
  3. Part 3: Structured Logging with Correlation IDs (Current)
  4. Part 4: Metrics and Alerting with Prometheus/Grafana
  5. Part 5: Debugging Production Issues with Observability Data

Problems with Traditional Logging

Limitations of traditional logging:

  • Cannot identify which request a log belongs to
  • No correlation between service logs
  • Difficult to search and filter

Structured Logging

JSON Log Format

{
  "timestamp": "2026-01-27T10:30:15.123Z",
  "level": "INFO",
  "service": "order-service",
  "traceId": "abc123",
  "spanId": "def456",
  "correlationId": "req-789",
  "message": "Processing order"
}

MDC (Mapped Diagnostic Context)

MDC allows you to add contextual information (like correlation IDs, trace IDs) to all log messages within a request thread.

Logging Best Practices

Meaningful Log Messages

// Bad
logger.info("Processing")

// Good
logger.info("Starting order processing",
    kv("orderId", orderId),
    kv("customerId", customerId)
)

Summary

Key aspects of structured logging:

Item Description
JSON Format Easy parsing and searching
Correlation ID Track requests across services
MDC Thread-level context management
Trace ID Integration Unify distributed tracing and logs

In the next post, we’ll cover metrics and alerting with Prometheus/Grafana.

댓글남기기

참고