jarvis/docs/backend/BACKEND_HANDOFF.md

# Jarvis Backend — Build & Deploy Handoff (for Codex on Proxmox)

You are building the **server side** of Jarvis: a self-hosted RSS news-correlation
platform. A finished SwiftUI iOS client already exists and will connect to you
**unmodified** — so the API shapes below are not suggestions, they are a contract.
The canonical spec is `API/contract.md` in this repo; this document adds the
implementation plan, the algorithms, the client-compat gotchas, and the Proxmox
deployment.

**Stack (already decided):** Python 3.11+ · FastAPI · Uvicorn · SQLite (via SQLModel) ·
scikit-learn TF-IDF for correlation · rule-based scoring. No external API keys.
No auth (local network / self-hosted only).

---

## 0. TL;DR of what to build

A single FastAPI app that does four jobs:

1. **Ingest** — poll RSS feeds on a schedule, parse articles, track feed health.
2. **Correlate** — cluster articles from different sources into *Stories* (TF-IDF + cosine).
3. **Score** — compute a server-side `signalScore` (5 components) + topic + consensus/conflict.
4. **Serve** — the REST API under `/api/v1` and a WebSocket at `/ws`, broadcasting live events.

Then deploy it as a systemd service inside a Proxmox **LXC container**, bound to
`0.0.0.0:8080`, reachable from the iPhone on the LAN.

---

## 1. NON-NEGOTIABLE client-compatibility rules

The iOS client decodes JSON with `keyDecodingStrategy = .convertFromSnakeCase`
and `dateDecodingStrategy = .iso8601`, and it **silently drops any object that
fails to decode**. So:

1. **Dates must be exactly `YYYY-MM-DDTHH:MM:SSZ`** — UTC, the literal `Z`, and
   **no fractional seconds**. Pydantic's default (`...+00:00` with microseconds)
   will FAIL to decode and stories will vanish. Force the format with a serializer
   (see §4).
2. **JSON keys: camelCase** (`signalScore`, `sourceCount`, `pollIntervalSeconds`,
   `publishedAt`, `nextCursor`, `hasMore`, …). FastAPI: set field aliases via
   `alias_generator=to_camel`, serialize with `response_model_by_alias=True` (default).
3. **Exact query-param names** the client sends to `GET /stories`:
   `limit`, `after` (the cursor), `topic`, `min_signal`. Read those names verbatim.
4. **`DELETE /feeds/:id` must return HTTP 204** (no body). The client checks for 204.
5. **`POST /feeds` returns 201** with the created feed object. The client sends
   `{"url": ..., "name": ..., "pollIntervalSeconds": ...}` — accept camelCase.
6. **Base URL is `http://<host>/api/v1`** and **WebSocket is `ws://<host>/ws`** —
   the client builds these strings literally, so REST lives under `/api/v1` and the
   socket at the root path `/ws`. Plain `http`/`ws` (not TLS) is expected.
7. **No CORS needed** (native app, not a browser), but harmless to enable `*`.
8. Error responses use `{"error": {"code", "message", "status"}}` (see contract §Error).

---

## 2. Endpoints (all under `/api/v1` unless noted)

Full example payloads are in `API/contract.md`. Summary:

| Method | Path | Returns | Notes |
|---|---|---|---|
| GET | `/health` | `{status, version, storiesCount, feedsCount, uptime}` | client calls on launch to validate connection |
| GET | `/stories` | `{data[], nextCursor, hasMore, total}` | sorted by `signalScore` desc; params `limit`(≤100, def 20), `after`, `topic`, `min_signal` |
| GET | `/stories/{id}` | `StoryDetail` (adds `timeline[]`) | 404 `story_not_found` if missing |
| GET | `/articles/{id}` | `Article` (full body for offline cache) | 404 if missing |
| GET | `/feeds` | `{data: [Feed]}` | all feeds + health |
| POST | `/feeds` | `Feed` (201) | body `{url, name, pollIntervalSeconds?}`; 400 `invalid_url` if not a real feed |
| DELETE | `/feeds/{id}` | 204 | 404 `feed_not_found` if missing |
| WS | `/ws` (root, not `/api/v1`) | event stream | see §6 |

### Object shapes (camelCase, dates as `...Z`)

- **StorySummary**: `id, headline, summary, topic, signalScore, scoreBreakdown,
  sourceCount, sources[], consensus, conflict, updatedAt, createdAt`
- **scoreBreakdown**: `{sourceAuthority, freshness, localRelevance,
  crossSourceConfirmation, topicImportance}` (ints, sum == `signalScore`)
- **StorySource**: `{id, name, url, publishedAt, isBreaking}`
- **StoryDetail**: StorySummary fields **minus `sources`**, **plus** `timeline[]`
- **TimelineEntry**: `{articleId, source, headline, publishedAt, isBreaking}`
- **Article**: `{id, storyId, source, sourceUrl, headline, body, imageUrl, author, publishedAt}`
- **Feed**: `{id, name, url, health, pollIntervalSeconds, failureCount, lastFetchedAt, articleCountToday}`
  - `health ∈ {"active","failing","dead"}`
- **topic** slugs used by the app's filter pills: `finance`, `tech`, `politics`, `africa`
  (classify everything into one of these; fall back to the best match).

---

## 3. Suggested project layout

```
server/
  app/
    main.py          # FastAPI app, lifespan: init DB, seed, start poller
    config.py        # tunables: thresholds, lexicons, source-authority map
    db.py            # SQLModel engine/session, init_db()
    models.py        # Feed, Article, Story (SQLModel tables)
    schemas.py       # Pydantic response models (camelCase aliases + Z dates)
    ingest.py        # RSS poll + parse + feed-health state machine
    correlate.py     # TF-IDF clustering: assign articles -> stories
    scoring.py       # signal-score components, topic classify, consensus/conflict
    events.py        # WebSocket ConnectionManager + broadcast + 30s ping loop
    scheduler.py     # asyncio loop: poll due feeds, correlate, score, sweep stale
    routers/
      stories.py  articles.py  feeds.py  health.py  ws.py
  seed_feeds.py      # default feed list inserted on first run
  requirements.txt   # already in this folder
  jarvis.service     # systemd unit (see §8)
  README.md
```

---

## 4. Data model (SQLite via SQLModel)

- **Feed**: `id (pk, "feed_"+short uuid)`, `name`, `url`, `health`,
  `poll_interval_seconds`, `failure_count`, `last_fetched_at (nullable)`, `created_at`.
  `articleCountToday` is **computed** at query time (count of this feed's articles
  with `created_at >= today 00:00 UTC`).
- **Article**: `id ("art_"+uuid)`, `feed_id (fk)`, `story_id (fk, nullable)`,
  `source` (feed name), `source_url`, `headline`, `body`, `image_url (nullable)`,
  `author (nullable)`, `published_at`, `guid (unique — dedup key)`, `created_at`,
  `match_text` (lowercased headline + body, for TF-IDF).
- **Story**: `id ("story_"+uuid)`, `headline`, `summary`, `topic`, `signal_score`,
  the 5 breakdown ints, `consensus (nullable)`, `conflict (nullable)`,
  `source_count`, `is_stale (bool)`, `created_at`, `updated_at`.

Store **all datetimes as naive UTC**. Serialize with this exact helper so the
client's `.iso8601` accepts them:

```python
from datetime import datetime, timezone
from pydantic import field_serializer

def iso_z(dt: datetime) -> str:
    if dt.tzinfo is not None:
        dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
    return dt.strftime("%Y-%m-%dT%H:%M:%SZ")   # NO microseconds, literal Z
```

Apply via `@field_serializer("publishedAt","updatedAt","createdAt","lastFetchedAt")`
returning `iso_z(value)` (return `None` for null `lastFetchedAt`).

Camel aliases:

```python
from pydantic import BaseModel, ConfigDict
from pydantic.alias_generators import to_camel

class Schema(BaseModel):
    model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True)
```

---

## 5. The algorithms

### 5a. Ingest (`ingest.py`)
- Fetch each due feed with `httpx.AsyncClient` (timeout ~10s), parse bytes with
  `feedparser` (run in `asyncio.to_thread`, it's blocking).
- For each entry: dedup by `guid` (entry.id or link). Extract `headline` (title),
  `body` (richest of `content[].value` / `summary`, HTML-stripped with BeautifulSoup),
  `image_url` (media:content / enclosure / first `<img>`), `author`, `published_at`
  (`entry.published_parsed` → UTC; fall back to now).
- **Feed-health state machine** on each poll:
  - success → `health="active"`, `failure_count=0`, `last_fetched_at=now`.
  - failure → `failure_count += 1`; `failing` at ≥1, `dead` at ≥5 (tunable).
  - On any health **change**, broadcast a `feed.health` WS event.

### 5b. Correlate (`correlate.py`)
Goal: same real-world event across sources → one Story; stable Story `id`s.
- Consider only articles from a rolling window (e.g. last **72h**).
- Single `TfidfVectorizer` fit per cycle over (existing recent stories' aggregate
  text + the new unclustered articles' `match_text`). English stopwords, `ngram_range=(1,2)`.
- **Greedy online assignment** per new article (process oldest→newest):
  - cosine-compare against current cluster representatives;
  - if `max_sim >= 0.22` (tunable) **and** within the time window → join that story;
  - else → create a **new** Story seeded by this article (append its vector as a new rep).
- After assignment, recompute each touched story's aggregates: `source_count`
  (distinct feeds), canonical `headline`/`summary` (take the highest-authority or
  earliest article), then re-score (§5c). Track which stories were **created** vs
  **updated** to emit the right WS event.

### 5c. Score (`scoring.py`) — rule-based, must sum to `signalScore`
Component caps chosen so totals land like the contract example (91 = 28+22+15+16+10):

| Component | Range | Rule |
|---|---|---|
| `sourceAuthority` | 0–30 | sum of per-source authority weights (default 6; majors higher via config map), capped 30 |
| `freshness` | 0–25 | `round(25 * max(0, 1 - age_hours_of_newest / 48))` |
| `localRelevance` | 0–15 | hits against a local lexicon (place names, currency, institutions) scaled, capped 15 |
| `crossSourceConfirmation` | 0–20 | `min(20, (distinct_sources - 1) * 5)` |
| `topicImportance` | 0–10 | per-topic map: politics/finance=10, africa=8, tech=7, else 5 |

`signalScore = clamp(sum, 0, 100)`.

- **Topic classification**: keyword lexicons per `{finance, tech, politics, africa}`;
  pick the highest-scoring; tie-break to `africa`. Keep lexicons in `config.py`.
- **Consensus** (set when ≥2 sources): e.g. `"All N sources confirm the report.
  {SourceA} and {SourceB} agree."` — template is fine for v1.
- **Conflict** (nullable, default null): v1 heuristic — set a generic line only if a
  headline contains dispute markers (`deny`, `dispute`, `contradict`, `reject`),
  else `null`. (Leave a TODO to upgrade with NLP/LLM later.)

### 5d. Staleness sweep (`scheduler.py`)
- Periodically (e.g. every 5 min), any non-stale Story with **no new article in 24h**
  → set `is_stale=True`, broadcast `story.stale`. **Exclude stale stories** from
  `GET /stories` by default.

### 5e. Pagination
Keyset cursor over `(signal_score DESC, id DESC)`. `nextCursor` = base64 of
`"{signal_score}:{id}"` of the last row; on `after`, decode and filter
`(signal_score < s) OR (signal_score == s AND id < id)`. `hasMore` = more rows exist;
`total` = total non-stale stories (optionally filtered by topic/min_signal).

---

## 6. WebSocket `/ws`

On connect: `accept()`, register the socket. The server emits these (client refetches
full objects via REST as needed):

```jsonc
{"type":"story.updated","storyId":"story_x","signalScore":94,"sourceCount":9,"updatedAt":"...Z"}
{"type":"story.created","storyId":"story_y","headline":"...","signalScore":42,"topic":"politics","createdAt":"...Z"}
{"type":"story.stale","storyId":"story_z","updatedAt":"...Z"}
{"type":"feed.health","feedId":"feed_a","health":"failing","failureCount":3,"updatedAt":"...Z"}
```

- Send `{"type":"ping"}` every **30s**; the client replies `{"type":"pong"}` (you can
  ignore it). The client ALSO sends WebSocket-protocol pings — Uvicorn/websockets
  auto-answers those, no action needed.
- Keep a `ConnectionManager` with `broadcast(dict)`; the ingest/correlate/scoring code
  calls it. Drop dead sockets on send failure.
- Reconnect/backoff is handled entirely client-side — just accept new connections.

---

## 7. Proxmox deployment (recommended: unprivileged LXC)

A Python web service is a perfect fit for an **LXC container** (lighter than a VM).

1. **Create the container** (Proxmox shell or UI):
   ```bash
   # Debian 12 template, on the Proxmox host:
   pveam update && pveam available | grep debian-12
   pveam download local debian-12-standard_*_amd64.tar.zst
   pct create 140 local:vztmpl/debian-12-standard_*_amd64.tar.zst \
     --hostname jarvis-api --cores 2 --memory 1024 --swap 512 \
     --net0 name=eth0,bridge=vmbr0,ip=dhcp \
     --rootfs local-lvm:8 --unprivileged 1 --features nesting=1
   pct start 140 && pct enter 140
   ```
   Note the container's DHCP IP (`ip a`) — that's the host the iPhone points at. A
   **static IP / DHCP reservation** is recommended so the app's saved host stays valid.

2. **Inside the container:**
   ```bash
   apt update && apt install -y python3 python3-venv python3-pip git
   adduser --system --group jarvis
   mkdir -p /opt/jarvis && chown jarvis:jarvis /opt/jarvis
   # copy the server/ folder here (git clone, scp, or pct push)
   cd /opt/jarvis
   python3 -m venv .venv && . .venv/bin/activate
   pip install -r requirements.txt
   ```

3. **systemd service** (`/etc/systemd/system/jarvis.service`):
   ```ini
   [Unit]
   Description=Jarvis news-correlation API
   After=network-online.target
   Wants=network-online.target

   [Service]
   User=jarvis
   WorkingDirectory=/opt/jarvis
   ExecStart=/opt/jarvis/.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8080
   Restart=always
   RestartSec=3
   Environment=JARVIS_DB=/opt/jarvis/jarvis.db

   [Install]
   WantedBy=multi-user.target
   ```
   ```bash
   systemctl daemon-reload && systemctl enable --now jarvis
   systemctl status jarvis
   ```

4. **Verify from the Proxmox host or your Mac (same LAN):**
   ```bash
   curl http://<container-ip>:8080/api/v1/health
   curl http://<container-ip>:8080/api/v1/stories | head
   ```

5. **Point the app at it:** launch Jarvis on the iPhone/sim → onboarding → enter
   `http://<container-ip>:8080` **as `<container-ip>:8080`** (the app prepends
   `http://` and `ws://` itself — enter just `host:port`, e.g. `192.168.30.50:8080`).

> Networking notes: bind `0.0.0.0` (not `127.0.0.1`) so the LAN can reach it. If the
> Proxmox host or container runs a firewall, allow TCP **8080**. No TLS/auth by design —
> keep this on a trusted VLAN or behind the homelab firewall/VPN, per the contract.

---

## 8. Acceptance checklist (done = the app works end-to-end)

- [ ] `GET /api/v1/health` returns ok JSON; onboarding "Connect" succeeds.
- [ ] `GET /api/v1/stories` returns camelCase, `...Z` dates, sorted by `signalScore` desc.
- [ ] Stories actually cluster multi-source articles (not 1 story per article).
- [ ] `scoreBreakdown` ints sum exactly to `signalScore`.
- [ ] `GET /stories/{id}` includes a chronological `timeline[]`.
- [ ] `GET /articles/{id}` returns full `body` (offline reader works).
- [ ] `GET /feeds` shows health states; `POST /feeds` (201) and `DELETE` (204) work.
- [ ] WebSocket `/ws` connects; the app's header shows **LIVE**; a score change pushes
      `story.updated` and the feed reorders.
- [ ] Runs as a systemd service in the Proxmox LXC and survives a container reboot.

---

## 9. Hand-back

The iOS client is in `../Jarvis` (SwiftUI, builds with `xcodegen` + Xcode 26 / iOS 17).
While wiring the frontend I had to fix three bugs in the provided client layer that
also document real expectations of your server — worth knowing:
- `StoryStore` patches stories in-place on `story.updated` using `signalScore` +
  `sourceCount` from the event (so those two fields are required on that event).
- The client treats **HTTP `http://`** and **`ws://`** (no TLS).
- On relaunch it reconnects REST+WS from the saved host, then immediately calls
  `GET /stories` — so the server should be ready to serve quickly after boot.

Questions for whoever runs this: which RSS feeds to seed by default, and the local
lexicon (which country/region is "local" for `localRelevance`)? The contract examples
are Uganda/East-Africa centric (Daily Monitor, NilePost, BoU) — default `seed_feeds.py`
and the local lexicon to that unless told otherwise.
```