Initial commit: Jarvis iOS app
SwiftUI client for a self-hosted RSS news-correlation platform: signal feed, story detail, article reader, feed manager, and LAN⇄Tailscale connectivity. Project generated from project.yml via XcodeGen. Includes CI build matrix (macOS 14/15 × Debug/Release), issue templates, backlog, and API/backend handoff docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Robin Kutesa
336
docs/backend/BACKEND_HANDOFF.md
Normal file
336
docs/backend/BACKEND_HANDOFF.md
Normal file
@@ -0,0 +1,336 @@
|
||||
# Jarvis Backend — Build & Deploy Handoff (for Codex on Proxmox)
|
||||
|
||||
You are building the **server side** of Jarvis: a self-hosted RSS news-correlation
|
||||
platform. A finished SwiftUI iOS client already exists and will connect to you
|
||||
**unmodified** — so the API shapes below are not suggestions, they are a contract.
|
||||
The canonical spec is `API/contract.md` in this repo; this document adds the
|
||||
implementation plan, the algorithms, the client-compat gotchas, and the Proxmox
|
||||
deployment.
|
||||
|
||||
**Stack (already decided):** Python 3.11+ · FastAPI · Uvicorn · SQLite (via SQLModel) ·
|
||||
scikit-learn TF-IDF for correlation · rule-based scoring. No external API keys.
|
||||
No auth (local network / self-hosted only).
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR of what to build
|
||||
|
||||
A single FastAPI app that does four jobs:
|
||||
|
||||
1. **Ingest** — poll RSS feeds on a schedule, parse articles, track feed health.
|
||||
2. **Correlate** — cluster articles from different sources into *Stories* (TF-IDF + cosine).
|
||||
3. **Score** — compute a server-side `signalScore` (5 components) + topic + consensus/conflict.
|
||||
4. **Serve** — the REST API under `/api/v1` and a WebSocket at `/ws`, broadcasting live events.
|
||||
|
||||
Then deploy it as a systemd service inside a Proxmox **LXC container**, bound to
|
||||
`0.0.0.0:8080`, reachable from the iPhone on the LAN.
|
||||
|
||||
---
|
||||
|
||||
## 1. NON-NEGOTIABLE client-compatibility rules
|
||||
|
||||
The iOS client decodes JSON with `keyDecodingStrategy = .convertFromSnakeCase`
|
||||
and `dateDecodingStrategy = .iso8601`, and it **silently drops any object that
|
||||
fails to decode**. So:
|
||||
|
||||
1. **Dates must be exactly `YYYY-MM-DDTHH:MM:SSZ`** — UTC, the literal `Z`, and
|
||||
**no fractional seconds**. Pydantic's default (`...+00:00` with microseconds)
|
||||
will FAIL to decode and stories will vanish. Force the format with a serializer
|
||||
(see §4).
|
||||
2. **JSON keys: camelCase** (`signalScore`, `sourceCount`, `pollIntervalSeconds`,
|
||||
`publishedAt`, `nextCursor`, `hasMore`, …). FastAPI: set field aliases via
|
||||
`alias_generator=to_camel`, serialize with `response_model_by_alias=True` (default).
|
||||
3. **Exact query-param names** the client sends to `GET /stories`:
|
||||
`limit`, `after` (the cursor), `topic`, `min_signal`. Read those names verbatim.
|
||||
4. **`DELETE /feeds/:id` must return HTTP 204** (no body). The client checks for 204.
|
||||
5. **`POST /feeds` returns 201** with the created feed object. The client sends
|
||||
`{"url": ..., "name": ..., "pollIntervalSeconds": ...}` — accept camelCase.
|
||||
6. **Base URL is `http://<host>/api/v1`** and **WebSocket is `ws://<host>/ws`** —
|
||||
the client builds these strings literally, so REST lives under `/api/v1` and the
|
||||
socket at the root path `/ws`. Plain `http`/`ws` (not TLS) is expected.
|
||||
7. **No CORS needed** (native app, not a browser), but harmless to enable `*`.
|
||||
8. Error responses use `{"error": {"code", "message", "status"}}` (see contract §Error).
|
||||
|
||||
---
|
||||
|
||||
## 2. Endpoints (all under `/api/v1` unless noted)
|
||||
|
||||
Full example payloads are in `API/contract.md`. Summary:
|
||||
|
||||
| Method | Path | Returns | Notes |
|
||||
|---|---|---|---|
|
||||
| GET | `/health` | `{status, version, storiesCount, feedsCount, uptime}` | client calls on launch to validate connection |
|
||||
| GET | `/stories` | `{data[], nextCursor, hasMore, total}` | sorted by `signalScore` desc; params `limit`(≤100, def 20), `after`, `topic`, `min_signal` |
|
||||
| GET | `/stories/{id}` | `StoryDetail` (adds `timeline[]`) | 404 `story_not_found` if missing |
|
||||
| GET | `/articles/{id}` | `Article` (full body for offline cache) | 404 if missing |
|
||||
| GET | `/feeds` | `{data: [Feed]}` | all feeds + health |
|
||||
| POST | `/feeds` | `Feed` (201) | body `{url, name, pollIntervalSeconds?}`; 400 `invalid_url` if not a real feed |
|
||||
| DELETE | `/feeds/{id}` | 204 | 404 `feed_not_found` if missing |
|
||||
| WS | `/ws` (root, not `/api/v1`) | event stream | see §6 |
|
||||
|
||||
### Object shapes (camelCase, dates as `...Z`)
|
||||
|
||||
- **StorySummary**: `id, headline, summary, topic, signalScore, scoreBreakdown,
|
||||
sourceCount, sources[], consensus, conflict, updatedAt, createdAt`
|
||||
- **scoreBreakdown**: `{sourceAuthority, freshness, localRelevance,
|
||||
crossSourceConfirmation, topicImportance}` (ints, sum == `signalScore`)
|
||||
- **StorySource**: `{id, name, url, publishedAt, isBreaking}`
|
||||
- **StoryDetail**: StorySummary fields **minus `sources`**, **plus** `timeline[]`
|
||||
- **TimelineEntry**: `{articleId, source, headline, publishedAt, isBreaking}`
|
||||
- **Article**: `{id, storyId, source, sourceUrl, headline, body, imageUrl, author, publishedAt}`
|
||||
- **Feed**: `{id, name, url, health, pollIntervalSeconds, failureCount, lastFetchedAt, articleCountToday}`
|
||||
- `health ∈ {"active","failing","dead"}`
|
||||
- **topic** slugs used by the app's filter pills: `finance`, `tech`, `politics`, `africa`
|
||||
(classify everything into one of these; fall back to the best match).
|
||||
|
||||
---
|
||||
|
||||
## 3. Suggested project layout
|
||||
|
||||
```
|
||||
server/
|
||||
app/
|
||||
main.py # FastAPI app, lifespan: init DB, seed, start poller
|
||||
config.py # tunables: thresholds, lexicons, source-authority map
|
||||
db.py # SQLModel engine/session, init_db()
|
||||
models.py # Feed, Article, Story (SQLModel tables)
|
||||
schemas.py # Pydantic response models (camelCase aliases + Z dates)
|
||||
ingest.py # RSS poll + parse + feed-health state machine
|
||||
correlate.py # TF-IDF clustering: assign articles -> stories
|
||||
scoring.py # signal-score components, topic classify, consensus/conflict
|
||||
events.py # WebSocket ConnectionManager + broadcast + 30s ping loop
|
||||
scheduler.py # asyncio loop: poll due feeds, correlate, score, sweep stale
|
||||
routers/
|
||||
stories.py articles.py feeds.py health.py ws.py
|
||||
seed_feeds.py # default feed list inserted on first run
|
||||
requirements.txt # already in this folder
|
||||
jarvis.service # systemd unit (see §8)
|
||||
README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Data model (SQLite via SQLModel)
|
||||
|
||||
- **Feed**: `id (pk, "feed_"+short uuid)`, `name`, `url`, `health`,
|
||||
`poll_interval_seconds`, `failure_count`, `last_fetched_at (nullable)`, `created_at`.
|
||||
`articleCountToday` is **computed** at query time (count of this feed's articles
|
||||
with `created_at >= today 00:00 UTC`).
|
||||
- **Article**: `id ("art_"+uuid)`, `feed_id (fk)`, `story_id (fk, nullable)`,
|
||||
`source` (feed name), `source_url`, `headline`, `body`, `image_url (nullable)`,
|
||||
`author (nullable)`, `published_at`, `guid (unique — dedup key)`, `created_at`,
|
||||
`match_text` (lowercased headline + body, for TF-IDF).
|
||||
- **Story**: `id ("story_"+uuid)`, `headline`, `summary`, `topic`, `signal_score`,
|
||||
the 5 breakdown ints, `consensus (nullable)`, `conflict (nullable)`,
|
||||
`source_count`, `is_stale (bool)`, `created_at`, `updated_at`.
|
||||
|
||||
Store **all datetimes as naive UTC**. Serialize with this exact helper so the
|
||||
client's `.iso8601` accepts them:
|
||||
|
||||
```python
|
||||
from datetime import datetime, timezone
|
||||
from pydantic import field_serializer
|
||||
|
||||
def iso_z(dt: datetime) -> str:
|
||||
if dt.tzinfo is not None:
|
||||
dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%SZ") # NO microseconds, literal Z
|
||||
```
|
||||
|
||||
Apply via `@field_serializer("publishedAt","updatedAt","createdAt","lastFetchedAt")`
|
||||
returning `iso_z(value)` (return `None` for null `lastFetchedAt`).
|
||||
|
||||
Camel aliases:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
from pydantic.alias_generators import to_camel
|
||||
|
||||
class Schema(BaseModel):
|
||||
model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. The algorithms
|
||||
|
||||
### 5a. Ingest (`ingest.py`)
|
||||
- Fetch each due feed with `httpx.AsyncClient` (timeout ~10s), parse bytes with
|
||||
`feedparser` (run in `asyncio.to_thread`, it's blocking).
|
||||
- For each entry: dedup by `guid` (entry.id or link). Extract `headline` (title),
|
||||
`body` (richest of `content[].value` / `summary`, HTML-stripped with BeautifulSoup),
|
||||
`image_url` (media:content / enclosure / first `<img>`), `author`, `published_at`
|
||||
(`entry.published_parsed` → UTC; fall back to now).
|
||||
- **Feed-health state machine** on each poll:
|
||||
- success → `health="active"`, `failure_count=0`, `last_fetched_at=now`.
|
||||
- failure → `failure_count += 1`; `failing` at ≥1, `dead` at ≥5 (tunable).
|
||||
- On any health **change**, broadcast a `feed.health` WS event.
|
||||
|
||||
### 5b. Correlate (`correlate.py`)
|
||||
Goal: same real-world event across sources → one Story; stable Story `id`s.
|
||||
- Consider only articles from a rolling window (e.g. last **72h**).
|
||||
- Single `TfidfVectorizer` fit per cycle over (existing recent stories' aggregate
|
||||
text + the new unclustered articles' `match_text`). English stopwords, `ngram_range=(1,2)`.
|
||||
- **Greedy online assignment** per new article (process oldest→newest):
|
||||
- cosine-compare against current cluster representatives;
|
||||
- if `max_sim >= 0.22` (tunable) **and** within the time window → join that story;
|
||||
- else → create a **new** Story seeded by this article (append its vector as a new rep).
|
||||
- After assignment, recompute each touched story's aggregates: `source_count`
|
||||
(distinct feeds), canonical `headline`/`summary` (take the highest-authority or
|
||||
earliest article), then re-score (§5c). Track which stories were **created** vs
|
||||
**updated** to emit the right WS event.
|
||||
|
||||
### 5c. Score (`scoring.py`) — rule-based, must sum to `signalScore`
|
||||
Component caps chosen so totals land like the contract example (91 = 28+22+15+16+10):
|
||||
|
||||
| Component | Range | Rule |
|
||||
|---|---|---|
|
||||
| `sourceAuthority` | 0–30 | sum of per-source authority weights (default 6; majors higher via config map), capped 30 |
|
||||
| `freshness` | 0–25 | `round(25 * max(0, 1 - age_hours_of_newest / 48))` |
|
||||
| `localRelevance` | 0–15 | hits against a local lexicon (place names, currency, institutions) scaled, capped 15 |
|
||||
| `crossSourceConfirmation` | 0–20 | `min(20, (distinct_sources - 1) * 5)` |
|
||||
| `topicImportance` | 0–10 | per-topic map: politics/finance=10, africa=8, tech=7, else 5 |
|
||||
|
||||
`signalScore = clamp(sum, 0, 100)`.
|
||||
|
||||
- **Topic classification**: keyword lexicons per `{finance, tech, politics, africa}`;
|
||||
pick the highest-scoring; tie-break to `africa`. Keep lexicons in `config.py`.
|
||||
- **Consensus** (set when ≥2 sources): e.g. `"All N sources confirm the report.
|
||||
{SourceA} and {SourceB} agree."` — template is fine for v1.
|
||||
- **Conflict** (nullable, default null): v1 heuristic — set a generic line only if a
|
||||
headline contains dispute markers (`deny`, `dispute`, `contradict`, `reject`),
|
||||
else `null`. (Leave a TODO to upgrade with NLP/LLM later.)
|
||||
|
||||
### 5d. Staleness sweep (`scheduler.py`)
|
||||
- Periodically (e.g. every 5 min), any non-stale Story with **no new article in 24h**
|
||||
→ set `is_stale=True`, broadcast `story.stale`. **Exclude stale stories** from
|
||||
`GET /stories` by default.
|
||||
|
||||
### 5e. Pagination
|
||||
Keyset cursor over `(signal_score DESC, id DESC)`. `nextCursor` = base64 of
|
||||
`"{signal_score}:{id}"` of the last row; on `after`, decode and filter
|
||||
`(signal_score < s) OR (signal_score == s AND id < id)`. `hasMore` = more rows exist;
|
||||
`total` = total non-stale stories (optionally filtered by topic/min_signal).
|
||||
|
||||
---
|
||||
|
||||
## 6. WebSocket `/ws`
|
||||
|
||||
On connect: `accept()`, register the socket. The server emits these (client refetches
|
||||
full objects via REST as needed):
|
||||
|
||||
```jsonc
|
||||
{"type":"story.updated","storyId":"story_x","signalScore":94,"sourceCount":9,"updatedAt":"...Z"}
|
||||
{"type":"story.created","storyId":"story_y","headline":"...","signalScore":42,"topic":"politics","createdAt":"...Z"}
|
||||
{"type":"story.stale","storyId":"story_z","updatedAt":"...Z"}
|
||||
{"type":"feed.health","feedId":"feed_a","health":"failing","failureCount":3,"updatedAt":"...Z"}
|
||||
```
|
||||
|
||||
- Send `{"type":"ping"}` every **30s**; the client replies `{"type":"pong"}` (you can
|
||||
ignore it). The client ALSO sends WebSocket-protocol pings — Uvicorn/websockets
|
||||
auto-answers those, no action needed.
|
||||
- Keep a `ConnectionManager` with `broadcast(dict)`; the ingest/correlate/scoring code
|
||||
calls it. Drop dead sockets on send failure.
|
||||
- Reconnect/backoff is handled entirely client-side — just accept new connections.
|
||||
|
||||
---
|
||||
|
||||
## 7. Proxmox deployment (recommended: unprivileged LXC)
|
||||
|
||||
A Python web service is a perfect fit for an **LXC container** (lighter than a VM).
|
||||
|
||||
1. **Create the container** (Proxmox shell or UI):
|
||||
```bash
|
||||
# Debian 12 template, on the Proxmox host:
|
||||
pveam update && pveam available | grep debian-12
|
||||
pveam download local debian-12-standard_*_amd64.tar.zst
|
||||
pct create 140 local:vztmpl/debian-12-standard_*_amd64.tar.zst \
|
||||
--hostname jarvis-api --cores 2 --memory 1024 --swap 512 \
|
||||
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
|
||||
--rootfs local-lvm:8 --unprivileged 1 --features nesting=1
|
||||
pct start 140 && pct enter 140
|
||||
```
|
||||
Note the container's DHCP IP (`ip a`) — that's the host the iPhone points at. A
|
||||
**static IP / DHCP reservation** is recommended so the app's saved host stays valid.
|
||||
|
||||
2. **Inside the container:**
|
||||
```bash
|
||||
apt update && apt install -y python3 python3-venv python3-pip git
|
||||
adduser --system --group jarvis
|
||||
mkdir -p /opt/jarvis && chown jarvis:jarvis /opt/jarvis
|
||||
# copy the server/ folder here (git clone, scp, or pct push)
|
||||
cd /opt/jarvis
|
||||
python3 -m venv .venv && . .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. **systemd service** (`/etc/systemd/system/jarvis.service`):
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Jarvis news-correlation API
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=jarvis
|
||||
WorkingDirectory=/opt/jarvis
|
||||
ExecStart=/opt/jarvis/.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
Environment=JARVIS_DB=/opt/jarvis/jarvis.db
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
```bash
|
||||
systemctl daemon-reload && systemctl enable --now jarvis
|
||||
systemctl status jarvis
|
||||
```
|
||||
|
||||
4. **Verify from the Proxmox host or your Mac (same LAN):**
|
||||
```bash
|
||||
curl http://<container-ip>:8080/api/v1/health
|
||||
curl http://<container-ip>:8080/api/v1/stories | head
|
||||
```
|
||||
|
||||
5. **Point the app at it:** launch Jarvis on the iPhone/sim → onboarding → enter
|
||||
`http://<container-ip>:8080` **as `<container-ip>:8080`** (the app prepends
|
||||
`http://` and `ws://` itself — enter just `host:port`, e.g. `192.168.30.50:8080`).
|
||||
|
||||
> Networking notes: bind `0.0.0.0` (not `127.0.0.1`) so the LAN can reach it. If the
|
||||
> Proxmox host or container runs a firewall, allow TCP **8080**. No TLS/auth by design —
|
||||
> keep this on a trusted VLAN or behind the homelab firewall/VPN, per the contract.
|
||||
|
||||
---
|
||||
|
||||
## 8. Acceptance checklist (done = the app works end-to-end)
|
||||
|
||||
- [ ] `GET /api/v1/health` returns ok JSON; onboarding "Connect" succeeds.
|
||||
- [ ] `GET /api/v1/stories` returns camelCase, `...Z` dates, sorted by `signalScore` desc.
|
||||
- [ ] Stories actually cluster multi-source articles (not 1 story per article).
|
||||
- [ ] `scoreBreakdown` ints sum exactly to `signalScore`.
|
||||
- [ ] `GET /stories/{id}` includes a chronological `timeline[]`.
|
||||
- [ ] `GET /articles/{id}` returns full `body` (offline reader works).
|
||||
- [ ] `GET /feeds` shows health states; `POST /feeds` (201) and `DELETE` (204) work.
|
||||
- [ ] WebSocket `/ws` connects; the app's header shows **LIVE**; a score change pushes
|
||||
`story.updated` and the feed reorders.
|
||||
- [ ] Runs as a systemd service in the Proxmox LXC and survives a container reboot.
|
||||
|
||||
---
|
||||
|
||||
## 9. Hand-back
|
||||
|
||||
The iOS client is in `../Jarvis` (SwiftUI, builds with `xcodegen` + Xcode 26 / iOS 17).
|
||||
While wiring the frontend I had to fix three bugs in the provided client layer that
|
||||
also document real expectations of your server — worth knowing:
|
||||
- `StoryStore` patches stories in-place on `story.updated` using `signalScore` +
|
||||
`sourceCount` from the event (so those two fields are required on that event).
|
||||
- The client treats **HTTP `http://`** and **`ws://`** (no TLS).
|
||||
- On relaunch it reconnects REST+WS from the saved host, then immediately calls
|
||||
`GET /stories` — so the server should be ready to serve quickly after boot.
|
||||
|
||||
Questions for whoever runs this: which RSS feeds to seed by default, and the local
|
||||
lexicon (which country/region is "local" for `localRelevance`)? The contract examples
|
||||
are Uganda/East-Africa centric (Daily Monitor, NilePost, BoU) — default `seed_feeds.py`
|
||||
and the local lexicon to that unless told otherwise.
|
||||
```
|
||||
Reference in New Issue
Block a user