> ## Documentation Index
> Fetch the complete documentation index at: https://285e39fd5e337e58f16290.sightscreen.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Background jobs

> The background jobs that power the real-time scoring and Live Activity system.

Sightscreen runs three background jobs inside the main server process. They execute on independent timers and coordinate through DynamoDB state and an internal EventBus.

## Job overview

| Job                 | Purpose                                              | Interval                | Status |
| ------------------- | ---------------------------------------------------- | ----------------------- | ------ |
| `MatchOrchestrator` | Detect live transitions, create LA records, recovery | 30s                     | Active |
| `ScorePoller`       | Poll for live score data with cockatiel resilience   | 5s (idle) / 2.5s (live) | Active |
| `FixtureSyncJob`    | Sync fixture metadata from SportMonks                | 60s                     | Active |

<Note>
  `LiveScoreJob` and `ReplayScoreJob` still exist as thin wrappers around `ScorePoller`. LiveScoreJob points at SportMonks; ReplayScoreJob points at mock-sportmonks.
</Note>

***

## System diagram

```mermaid theme={null}
flowchart TD
    MO[MatchOrchestrator] -->|detect live transitions| AM[(ActiveMatches table)]
    MO -->|create LA records| LA[(LiveActivities table)]
    MO -->|startup recovery & stale takeover| AM

    SM[SportMonks API] -->|poll scores| SP[ScorePoller]
    SP -->|distributed lock| AM
    SP -->|publish updates| EB[EventBus]

    EB -->|score updates| APNS[ApnsDeliveryService]
    EB -->|score updates| LAS[LiveActivityService]

    APNS -->|push updates| APNs[APNs]
    LAS -->|update/end LAs| APNs

    FSJ[FixtureSyncJob] -->|sync fixtures| SM
    FSJ -->|write fixtures| DDB[(DynamoDB)]

    MSM[mock-sportmonks] -->|replay data| SP2[ScorePoller - replay]
    SP2 -->|publish updates| EB

    subgraph Background Jobs
        MO
        SP
        SP2
        FSJ
    end

    subgraph Services
        EB
        APNS
        LAS
    end
```

***

## MatchOrchestrator

Runs every **30 seconds**. Responsible for detecting when matches go live and managing Live Activity lifecycle.

### What it does

1. Queries the matches table for matches that have transitioned to `live` status.
2. For each newly live match, creates pending Live Activity records for all users following that match.
3. Sends push-to-start tokens via APNs to boot Live Activities on user devices.
4. Writes the match into the ActiveMatches table so ScorePoller picks it up.

### Startup recovery

On process startup, MatchOrchestrator scans the ActiveMatches table and reconciles with current match state. If matches are live but have no active orchestration (e.g., after a deploy or crash), it re-creates the necessary records and resumes polling.

### Stale instance takeover

If a match record in ActiveMatches has not been updated within a configurable threshold, MatchOrchestrator assumes the owning instance is dead and takes over. This prevents matches from going "dark" during rolling deploys or instance failures.

### Tables read/written

| Table                | Operation                         |
| -------------------- | --------------------------------- |
| Matches table        | Read (detect live transitions)    |
| ActiveMatches table  | Read/Write (track active matches) |
| LiveActivities table | Write (create pending LA records) |
| Subscriptions table  | Read (users following the match)  |
| Devices table        | Read (push-to-start tokens)       |

### Relevant env vars

| Variable                | Purpose                                          |
| ----------------------- | ------------------------------------------------ |
| `ACTIVE_MATCHES_TABLE`  | Table for tracking actively orchestrated matches |
| `LIVE_ACTIVITIES_TABLE` | Table for Live Activity records                  |
| `MARQUEE_LEAGUE_ID`     | League ID for marquee match detection            |

### Failure modes

* **DynamoDB read failure**: Cycle skipped, retried next interval.
* **Push-to-start failure**: LA record remains pending. Next cycle retries delivery.
* **Stale takeover race condition**: Two instances may briefly both claim a match. The distributed lock in ScorePoller prevents duplicate score pushes.

***

## ScorePoller

Reusable polling component used by both `LiveScoreJob` (real matches) and `ReplayScoreJob` (mock-sportmonks replay). Polls a score source at a configurable interval and publishes updates to the EventBus.

### What it does

1. Acquires a distributed lock for the target match (prevents duplicate polling across instances).
2. Fetches score data from the configured source (SportMonks or mock-sportmonks).
3. Compares with the last known state. If changed, publishes a score update event to the EventBus.
4. The EventBus fans out to downstream services: `ApnsDeliveryService`, `LiveActivityService`, `HubDataService`.

### Polling interval

* **5 seconds** in idle mode (waiting for ball-by-ball data).
* **2.5 seconds** when a match is actively live with ball data flowing.

### Cockatiel resilience

ScorePoller uses [cockatiel](https://github.com/connor-peet/cockatiel) for resilience policies:

* **Retry**: Exponential backoff on transient failures (network timeouts, 5xx responses).
* **Circuit breaker**: Opens after consecutive failures to avoid hammering a down upstream. Half-open probe after cooldown.
* **Timeout**: Per-request timeout to prevent hung connections from blocking the poll loop.
* **Bulkhead**: Limits concurrent outbound requests to prevent resource exhaustion.

<Note>
  During a circuit breaker open state, the last known scores remain in DynamoDB. Clients continue to see the most recent data, just not real-time updates.
</Note>

### Distributed lock

Each ScorePoller instance acquires a DynamoDB-based lock for the match it is polling. The lock has a TTL and is renewed on each successful cycle. If an instance dies, the lock expires and another instance (via MatchOrchestrator stale takeover) picks up the match.

### Tables read/written

| Table               | Operation                                      |
| ------------------- | ---------------------------------------------- |
| ActiveMatches table | Read/Write (distributed lock, last poll state) |
| Scores table        | Write (updated score data)                     |

### Dependencies

* **SportMonks API** (LiveScoreJob) or **mock-sportmonks** (ReplayScoreJob) as the data source.
* **EventBus** for downstream fan-out.

### Relevant env vars

| Variable                | Purpose                              |
| ----------------------- | ------------------------------------ |
| `SPORTMONKS_API_TOKEN`  | API token for SportMonks             |
| `SPORTMONKS_SEASON_ID`  | Season to query                      |
| `MOCK_SPORTMONKS_URL`   | URL of mock-sportmonks (replay mode) |
| `MOCK_SPORTMONKS_TOKEN` | Token for mock-sportmonks            |

### Failure modes

* **SportMonks down**: Circuit breaker opens. Scores freeze at last known state. Auto-recovers when SportMonks returns.
* **DynamoDB write failure**: Retried on next cycle. No data loss since the upstream API is the source of truth.
* **Lock contention**: If two instances race, only one acquires the lock. The other backs off.
* **Malformed response**: Logged and skipped. Existing score record preserved.

<Warning>
  ReplayScoreJob (ScorePoller pointed at mock-sportmonks) should never be enabled in production. It is gated behind mock mode configuration.
</Warning>

***

## FixtureSyncJob

Syncs fixture metadata (start times, venues, team assignments) from SportMonks.

### What it does

1. Fetches the fixture list for the configured season from SportMonks.
2. Compares with existing fixture records in DynamoDB.
3. Updates any changed fields (start time, venue, status).
4. Creates records for newly discovered fixtures.

### Polling interval

Runs every **60 seconds**.

### Tables read/written

| Table         | Operation                           |
| ------------- | ----------------------------------- |
| Matches table | Read/Write (fixture metadata)       |
| Leagues table | Read (to map SportMonks league IDs) |

### Dependencies

* **SportMonks API** -- fixture list endpoint.

### Relevant env vars

| Variable                   | Purpose                    |
| -------------------------- | -------------------------- |
| `SPORTMONKS_API_TOKEN`     | API token for SportMonks   |
| `SPORTMONKS_SEASON_ID`     | Season to sync             |
| `FIXTURE_SYNC_INTERVAL_MS` | Override the sync interval |

### Failure modes

* **SportMonks down**: Sync skipped. Existing fixture data remains unchanged. Retries on next cycle.
* **Partial response**: The job processes whatever fixtures are returned. Missing fixtures remain unchanged.