> ## Documentation Index
> Fetch the complete documentation index at: https://285e39fd5e337e58f16290.sightscreen.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Structured logging with pino, error tracking with Sentry, and distributed tracing for the LA pipeline.

## Overview

Sightscreen's observability stack covers three layers:

1. **Structured logging** -- pino on the backend, `os.Logger` + `AppLogger` on iOS
2. **Error tracking** -- Sentry on both platforms, with a pino-to-Sentry bridge on the backend
3. **Distributed tracing** -- Sentry transactions on iOS (`LATracer`) and `component` tags on the backend for CloudWatch filtering

All backend logs flow to **stdout as JSON** and are captured by **AWS CloudWatch Logs** via App Runner.

***

## Pino structured logging

The root logger is created in `src/utils/logger.ts` and exported as a singleton. Every other module imports it.

### Environment variables

| Variable    | Default | Purpose                                                                     |
| ----------- | ------- | --------------------------------------------------------------------------- |
| `LOG_LEVEL` | `info`  | Minimum pino log level (`trace`, `debug`, `info`, `warn`, `error`, `fatal`) |
| `NODE_ENV`  | --      | When `development`, pino-pretty is used for human-readable output           |

### Output modes

<CodeGroup>
  ```json Production (JSON to stdout) theme={null}
  {"level":30,"time":1711612345678,"msg":"request completed","method":"GET","path":"/v1/matches","statusCode":200,"responseTime":12,"requestId":"a1b2c3d4"}
  ```

  ```text Development (pino-pretty) theme={null}
  [14:32:05.678] INFO: request completed
      method: "GET"
      path: "/v1/matches"
      statusCode: 200
      responseTime: 12
      requestId: "a1b2c3d4"
  ```
</CodeGroup>

<Note>
  `pino-pretty` is an optional dev dependency. If it is not installed, the logger silently falls back to JSON output even in development.
</Note>

### Child logger conventions

Child loggers carry contextual fields that appear in every log line from that scope.

```ts theme={null}
// Request-scoped (created by requestLogger middleware)
req.log = logger.child({ requestId });

// Job-scoped
const jobLog = logger.child({ job: 'MatchOrchestrator' });

// Provider-scoped
const providerLog = logger.child({ provider: 'SportMonks' });

// Component-scoped (for CloudWatch filtering)
const laLog = logger.child({ matchId, component: 'live-activity' });
```

### Error logging pattern

Always pass errors under the `err` key so pino serializes the full stack trace:

```ts theme={null}
// Correct -- pino extracts message + stack
log.error({ err: error }, 'failed to send push notification');

// Wrong -- loses stack trace
log.error(`failed: ${error.message}`);
```

***

## Request logging middleware

The `requestLogger` middleware in `src/middleware/logging.ts` runs on every request:

1. Generates a `requestId` (UUIDv4) and attaches it to `req.requestId`
2. Creates a child logger at `req.log` scoped to that request
3. On response finish, logs the request with method, path, status code, and response time
4. Routes the log to the appropriate level based on status code:
   * `>= 500` -- `error`
   * `>= 400` -- `warn`
   * otherwise -- `info`

The companion `errorLogger` middleware catches unhandled errors and logs them with `{ err }` before passing to the next error handler.

***

## Sentry (backend)

Sentry is **opt-in** -- it only initializes when `SENTRY_DSN` is set.

### Initialization

`src/instrument.ts` is imported before anything else in the entry point. It calls `Sentry.init()` with the DSN and a 100% trace sample rate.

```ts theme={null}
if (SENTRY_DSN) {
  Sentry.init({
    dsn: SENTRY_DSN,
    sendDefaultPii: true,
    tracesSampleRate: 1.0,
  });
}
```

### Express error handler

After all routes are registered, `Sentry.setupExpressErrorHandler(app)` is called. This captures unhandled Express errors as Sentry exceptions automatically.

### Pino-to-Sentry stream

The logger uses `pino.multistream` to tee output to both stdout and a custom Sentry writable stream:

* The Sentry stream only processes logs at level **>= 50** (error and fatal)
* If the log contains an `err` object with `message` and `stack`, it calls `Sentry.captureException`
* Otherwise it calls `Sentry.captureMessage` with the appropriate severity
* The stream is wrapped in a try/catch so Sentry failures never break logging

### Graceful shutdown

On `SIGTERM` or `SIGINT`, the shutdown handler calls `Sentry.close(2000)` with a 2-second timeout to flush any pending events before the process exits.

<Warning>
  `sendDefaultPii: true` is currently set without a `beforeSend` scrubber. This means Sentry may capture IP addresses, cookies, and other PII from request headers. See [Known issues](#known-issues) below.
</Warning>

***

## Sentry (iOS)

The iOS app uses **sentry-cocoa** SDK, configured in `SightscreenApp.configureSentry()`.

### Configuration

| Setting                   | Value                                  | Notes                                                         |
| ------------------------- | -------------------------------------- | ------------------------------------------------------------- |
| `tracesSampleRate`        | `1.0`                                  | 100% of transactions are sampled                              |
| `tracePropagationTargets` | `["api.sightscreen.app", "localhost"]` | Auto-injects `sentry-trace` header on requests to these hosts |
| `attachScreenshot`        | `true`                                 | Captures a screenshot on every error event                    |
| `enableAppHangTracking`   | `true`                                 | Detects and reports main-thread hangs                         |
| `enableMetricKit`         | `true`                                 | Forwards MetricKit diagnostics to Sentry                      |

### AppLogger

`AppLogger` mirrors pino's child logger pattern on iOS. It is the single entry point for all logging.

```swift theme={null}
let log = AppLogger.root.child("FollowManager")
log.info("follow match", extra: ["matchId": matchId])
log.error("failed", error: error, extra: ["matchId": matchId])

// Nested child loggers
let childLog = log.child("TokenSync")  // category: "FollowManager.TokenSync"
```

**Backends:**

* **os.Logger** -- always active. Visible in Xcode console and Console.app
* **SentryBackend** -- device builds only (disabled on simulator). Routes errors to `SentrySDK.capture` and all other levels to breadcrumbs
* **DebugLogService** -- admin overlay, enabled via `DebugLogSettings`

### SentryBackend

`SentryLogger.swift` isolates the Sentry import behind a `SentryBackend` enum:

* **`captureError`** -- calls `SentrySDK.capture(error:)` or `SentrySDK.capture(message:)`, attaching the logger category as a tag and any extras
* **`addBreadcrumb`** -- logs non-error events as Sentry breadcrumbs with category and level mapping
* **`setGlobalContext`** -- attaches device context (OS version, device model, app version, build number, vendor device ID, locale) under the `"app"` scope key
* **`setLiveActivityStatus`** -- records LA authorization and frequent-push capability under the `"live_activity"` scope key

<Note>
  `SentryBackend.isAvailable` returns `false` on the simulator via `#if targetEnvironment(simulator)`. This avoids noisy local development events.
</Note>

***

## Distributed tracing

### LATracer (iOS)

`LATracer` creates Sentry transactions for the follow-to-Live-Activity lifecycle:

```swift theme={null}
let tracer = LATracer(matchId: matchId)

// Root transaction
tracer.startFollow(isLive: true)  // transaction: "la.follow", operation: "la.lifecycle"

// Child spans for each step
tracer.startSpan(LATracer.opAPICall, description: "POST /v1/follow")
// ... API call completes
tracer.finishSpan(LATracer.opAPICall)

tracer.startSpan(LATracer.opLAStart, description: "ActivityKit request")
// ... LA starts
tracer.finishSpan(LATracer.opLAStart)

// Finish the root transaction
tracer.finish()
```

**Span operation constants:**

| Constant         | Value               | Description               |
| ---------------- | ------------------- | ------------------------- |
| `opAPICall`      | `la.api.follow`     | Follow API request        |
| `opAPIUnfollow`  | `la.api.unfollow`   | Unfollow API request      |
| `opLAStart`      | `la.activity.start` | ActivityKit start request |
| `opLAAdopt`      | `la.activity.adopt` | Adopt existing activity   |
| `opTokenRefresh` | `la.token.refresh`  | Push token refresh cycle  |

### Trace propagation

The iOS SDK auto-injects `sentry-trace` and `baggage` headers on outgoing requests to hosts matching `tracePropagationTargets`:

```
api.sightscreen.app
localhost
```

This links iOS transactions to backend spans in the Sentry trace view.

### Backend component tags

The backend tags Live Activity-related logs with `component: 'live-activity'` for CloudWatch filtering:

```ts theme={null}
logger.child({ matchId, component: 'live-activity' }).info('sending push-to-start');
```

***

## CloudWatch integration

App Runner captures all **stdout** output and forwards it to **CloudWatch Logs**. Since pino outputs JSON in production, you can use CloudWatch Insights filter patterns:

```
# All live-activity logs
{ $.component = "live-activity" }

# Errors only
{ $.level >= 50 }

# Specific match
{ $.matchId = "12345" }

# Slow requests (> 1 second)
{ $.responseTime > 1000 }
```

<Note>
  No custom CloudWatch agent or log driver is required. App Runner's built-in log capture handles everything as long as logs go to stdout.
</Note>

***

## Known issues

<Accordion title="sendDefaultPii:true without beforeSend scrubber">
  `instrument.ts` sets `sendDefaultPii: true` which tells Sentry to capture request headers, cookies, and IP addresses. There is currently **no `beforeSend` hook** to strip sensitive fields.

  **Risk:** User IP addresses and session cookies may appear in Sentry events.

  **Fix:** Add a `beforeSend` callback that strips or redacts sensitive headers, or disable `sendDefaultPii` entirely.
</Accordion>

<Accordion title="Raw emails in CognitoAuthService logs">
  The Cognito auth middleware logs the decoded JWT claims at `debug` level, which can include the user's email address. If `LOG_LEVEL` is set to `debug` in production, these will appear in CloudWatch and potentially in Sentry via the pino-to-Sentry stream.

  **Risk:** PII in log storage.

  **Fix:** Redact email from claims before logging, or only log the `sub` claim.
</Accordion>

<Accordion title="Duplicate error reporting in AppConfigService">
  Some error paths in the iOS `AppConfigService` call both `AppLogger.error()` (which routes to Sentry) **and** directly call `SentrySDK.capture`. This results in duplicate Sentry events for the same error.

  **Fix:** Remove direct `SentrySDK.capture` calls and rely solely on `AppLogger.error()` for Sentry routing.
</Accordion>