Alpha/README.md

# ALPHA_PROJECT — System Context for GitHub Copilot

## Who I am

I am a Cloud Architect with 6 years of Big4 consulting experience (currently at manager level at EY), working daily on Azure, Dynamics 365 / Dataverse, .NET, and enterprise integration patterns. I run a production-grade homelab on Kubernetes at home, and I am building ALPHA_PROJECT as a personal initiative in my spare time (evenings and weekends).

---

## What ALPHA_PROJECT is

ALPHA_PROJECT is a **proactive, multi-agent personal AI assistant** built entirely on self-hosted infrastructure. It is not a chatbot. It is an autonomous system that:

- Monitors my digital life (email, calendar, home automation, finances, infrastructure)
- Maintains a persistent, structured memory of facts, habits, and preferences
- Takes initiative to notify me of relevant events, correlations, and pending actions
- Interacts with me via voice (Amazon Echo / Alexa custom skill named **"Pompeo"**) and Telegram
- Runs local LLMs on dedicated hardware — no cloud AI inference (except GitHub Copilot completions, available via EY license at zero cost)

The assistant is named **Pompeo** (the Alexa skill wake word).

---

## Infrastructure

### LLM Server (new, dedicated node — outside the Kubernetes cluster)

- **CPU**: AMD Ryzen 5 4500
- **RAM**: 16 GB DDR4
- **GPU**: NVIDIA GeForce RTX 3060 (16 GB VRAM)
- **Runtime**: Ollama (API-compatible with OpenAI)
- **Primary model**: Qwen2.5-14B-Instruct Q4_K_M (fits entirely in VRAM, no offload)
- **Secondary model**: Qwen2.5-Coder-14B-Instruct Q4_K_M (for code-related tasks)
- **Embedding model**: TBD — to be served via Ollama (e.g. `nomic-embed-text`)
- **Constraint**: zero RAM offload — all models must fit entirely in 16 GB VRAM

### Kubernetes Homelab Cluster

Production-grade self-hosted stack. Key components relevant to ALPHA_PROJECT:

| Component | Role |
|---|---|
| **n8n** | Primary orchestrator and workflow engine for all agents |
| **Node-RED** | Event-driven automation, Home Assistant bridge |
| **Patroni / PostgreSQL** | Persistent structured memory store — `postgres.persistence.svc.cluster.local:5432/pompeo` |
| **Qdrant** | Vector store for semantic/episodic memory — `qdrant.persistence.svc.cluster.local:6333` |
| **Home Assistant** | IoT hub — device tracking, automations, sensors, Google Calendar proxy |
| **MikroTik** | Network — VLANs, firewall rules, device presence detection |
| **Paperless-ngx** | Document archive (`docs.mt-home.uk`) |
| **Actual Budget** | Personal finance |
| **Mealie** | Meal planning / recipes |
| **Immich** | Photo library |
| **Outline** | Internal wiki / knowledge base |
| **Radarr / Sonarr** | Media management |
| **Jenkins** | CI/CD |
| **AdGuard** | DNS filtering |
| **WireGuard** | VPN |
| **Minio** | S3-compatible object storage |
| **Longhorn** | Distributed block storage |
| **Velero** | Disaster recovery / backup |

### External Services (in use)

- **Gmail** — primary email
- **Google Calendar** — calendar (multiple calendars: Work, Family, Formula 1, WEC, Inter, Birthdays, Tasks, Pulizie, Spazzatura, Festività Italia, Varie)
- **Amazon Echo** — voice interface for Pompeo
- **AWS Lambda** — bridge between Alexa skill and n8n webhook
- **Telegram** — notifications, logging, manual document upload
- **GitHub Copilot** (GPT-4.1 via `api.githubcopilot.com`) — LLM completions at zero cost (EY license)

### Internal Services / Custom

- `orchestrator.mt-home.uk` — n8n instance
- `docs.mt-home.uk` — Paperless-ngx
- `filewizard.home.svc.cluster.local:8000` — custom OCR microservice (async, job-based API)

---

## Architecture Overview

### Multi-Agent Design

ALPHA_PROJECT uses specialized agents, each responsible for a specific data domain. All agents are implemented as **n8n workflows**.

| Agent | Trigger | Responsibility |
|---|---|---|
| **Mail Agent** | Cron every 15-30 min | Read Gmail, classify emails, extract facts, detect invoices/bills |
| **Finance Agent** | Triggered by Mail Agent or Telegram | Process PDF invoices/bills, archive to Paperless, persist to memory |
| **Calendar Agent** | Cron + on-demand | Read Google Calendar, detect upcoming events, cross-reference with other agents |
| **Infrastructure Agent** | Cron + alert webhooks | Monitor Kubernetes cluster health, disk usage, failed jobs |
| **IoT Agent** | Event-driven (Home Assistant webhooks) | Monitor device presence, home state, learn behavioral patterns |
| **Newsletter Agent** | Cron morning | Digest newsletters, extract relevant articles |
| **Proactive Arbiter** | Cron (adaptive frequency) + high-priority queue messages | Consume agent outputs, correlate, decide what to notify |

### Message Broker (Blackboard Pattern)

Agents do not call each other directly. They write observations to the **`agent_messages` table** in PostgreSQL (blackboard pattern). The **Proactive Arbiter** polls this table, batches low-priority messages, and immediately processes high-priority ones. High-urgency events trigger a direct n8n webhook call bypassing the queue.

**ADR: No dedicated message broker** — Postgres is sufficient for the expected message volume and avoids operational overhead. Revisit if throughput exceeds 1k messages/day.

### Memory Architecture

Three layers of persistence:

**1. Structured memory — PostgreSQL (Patroni)**

Episodic facts, finance records, reminders, behavioral observations. Fast, queryable, expirable.

```sql
-- Generic episodic facts
CREATE TABLE memory_facts (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  source TEXT NOT NULL,         -- 'email', 'calendar', 'iot', 'paperless', ...
  category TEXT,                -- 'finance', 'personal', 'work', 'health', ...
  subject TEXT,
  detail JSONB,                 -- flexible per-source payload
  action_required BOOLEAN DEFAULT false,
  action_text TEXT,
  created_at TIMESTAMP DEFAULT now(),
  expires_at TIMESTAMP,         -- facts have a TTL
  qdrant_id UUID                -- FK to vector store
);

-- Finance documents (frequent structured queries)
CREATE TABLE finance_documents (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  paperless_doc_id INT,
  correspondent TEXT,
  amount NUMERIC(10,2),
  currency TEXT DEFAULT 'EUR',
  doc_date DATE,
  doc_type TEXT,
  tags TEXT[],
  created_at TIMESTAMP DEFAULT now()
);

-- Behavioral context (used by IoT agent and Arbiter)
CREATE TABLE behavioral_context (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  event_type TEXT,              -- 'sport_event', 'dog_walk', 'work_session', ...
  start_at TIMESTAMP,
  end_at TIMESTAMP,
  do_not_disturb BOOLEAN DEFAULT false,
  home_presence_expected BOOLEAN,
  notes TEXT
);
```

**2. Semantic memory — Qdrant** — `qdrant.persistence.svc.cluster.local:6333`

Vector embeddings for similarity search. Three collections with **multi-tenant design**: isolation via `user_id` payload field (`"martin"`, `"shared"`, future users).

| Collection | Content |
|---|---|
| `episodes` | Conversations, episodic facts with timestamp |
| `knowledge` | Documents, Outline notes, newsletters, knowledge base |
| `preferences` | Preferences, habits, behavioral patterns |

Each Qdrant point includes a metadata payload for pre-filtering (`user_id`, `source`, `date`, `category`, `action_required`) to avoid full-scan similarity searches.

**3. Profile memory — PostgreSQL (static table)**

User preferences, fixed facts, communication style. Updated manually or via explicit agent action.

---

## IoT Agent — Design Notes

### Data Source: Home Assistant

Home Assistant (`http://10.30.20.100:8123`, HA OS 2026.3.2, Alzano Lombardo BG) is the primary hub for physical-world context. It aggregates Google Pixel 10, Pixel Watch 4, smart home devices, and 25 Google Calendars.

**Person allowlist** (permanent by design — `person.ajada_tahiraj` is explicitly excluded):

| Person | Entity | Notes |
|---|---|---|
| Martin Tahiraj | `person.martin_tahiraj` | ✅ Tracked |
| Ajada Tahiraj | `person.ajada_tahiraj` | ❌ Excluded (sister — privacy) |

**Key sensors for Martin:**

| Sensor | Entity ID | Signal |
|---|---|---|
| Activity (Google) | `sensor.pixel_10_detected_activity` | still / walking / running / in_vehicle |
| Geocoded location | `sensor.pixel_10_geocoded_location` | Human-readable street address |
| EY laptop | `device_tracker.ey_hp` | Router tracker — online = laptop on home WiFi |
| Spotify | `media_player.spotify_martin` | Current track, playing/paused |
| Sleep duration | `sensor.pixel_10_sleep_duration` | Pixel Watch 4 |
| Next alarm | `sensor.pixel_10_next_alarm` | Scheduled wake-up |
| Work Profile | `binary_sensor.pixel_10_work_profile` | Android Work Profile active |
| Screen on | `binary_sensor.pixel_10_interactive` | Phone screen on/off |
| Do Not Disturb | `binary_sensor.pixel_10_do_not_disturb` | DND mode |
| Daily steps | `sensor.pixel_10_daily_steps` | Pixel Watch 4 |
| Heart rate | `sensor.pixel_10_heart_rate` | Pixel Watch 4 |
| GPS Zone | `person.martin_tahiraj` | home / not_home / zone name |

Room presence sensors (PIR-based) are considered **unreliable** — excluded for now.

### Sensor Allowlist — `ha_sensor_config`

Instead of hardcoded rules, the IoT Agent uses a dynamic allowlist stored in Postgres. Sensors are matched by **regex pattern**, allowing glob-style additions:

```sql
CREATE TABLE ha_sensor_config (
  id          SERIAL PRIMARY KEY,
  pattern     TEXT NOT NULL,          -- regex pattern, e.g. 'sensor\.pixel_10_.*'
  user_id     TEXT NOT NULL,
  group_name  TEXT NOT NULL,          -- 'mobile_device' | 'work_presence' | 'entertainment' | ...
  description TEXT,
  active      BOOLEAN NOT NULL DEFAULT true
);

-- Seed entries
INSERT INTO ha_sensor_config (pattern, user_id, group_name, description) VALUES
  ('sensor\.pixel_10_.*',           'martin', 'mobile_device', 'All Pixel 10 sensors'),
  ('device_tracker\.ey_hp',         'martin', 'work_presence', 'EY Laptop router tracker'),
  ('media_player\.spotify_martin',  'martin', 'entertainment', 'Spotify'),
  ('binary_sensor\.pixel_10_.*',    'martin', 'mobile_device', 'Pixel 10 binary sensors'),
  ('person\.martin_tahiraj',        'martin', 'presence',      'Martin GPS zone state');
```

This allows adding new sensors (e.g. `sensor.pixel_watch_.*`) without workflow changes.

### Activity State Machine (LLM-based — no fixed rules)

The IoT Agent sends a snapshot of all allowlisted sensor values to GPT-4.1 and asks it to infer the current activity label and confidence. **No if/else rules are coded** — the LLM performs inference.

Example LLM output:
```json
{
  "activity": "home_working",
  "confidence": 0.92,
  "do_not_disturb": true,
  "location": "home",
  "notes": "Laptop EY online, work profile attivo, orario lavorativo 09-18"
}
```

Activity labels: `sleeping`, `home_relaxing`, `home_working`, `commuting`, `at_office`, `out_errands`, `out_with_dog`, `exercising`, `traveling`, `unknown`.

### Three-Layer Data Flow

| Layer | Trigger | Frequency | Output |
|---|---|---|---|
| Webhook | HA automation (zone change, motion) | Event-driven | Immediate `agent_messages` entry |
| Polling | n8n cron | Every 20 min | Sensor snapshot → LLM → `behavioral_context` |
| Daily cron | n8n cron midnight | Once/day | Day summary → Qdrant `episodes` embedding |

### Historical Bootstrap

One-time job: last 12 months of HA sensor history → daily LLM summaries → Qdrant `episodes`.
- Source: HA History API (`/api/history/period/{start}?filter_entity_id=...`)
- Output: one Qdrant point per day per user, with full behavioral context

### Confidence-Gated Clarification

When activity inference confidence < 0.6, or when Pompeo detects a potential life change (new employer from emails, travel pattern, etc.), it asks Martin directly via Telegram:

> "Ciao Martin, sto notando email di Avanade — lavori ancora per EY o sei passato lì? 🤔"

Pompeo updates `user_profile` or `memory_facts` with the confirmed fact and adjusts its confidence threshold.

---

## Calendar Agent — Design Notes

### Design Decisions

- **Data source**: Google Calendar events fetched via **Home Assistant REST API** (`/api/calendars/{entity_id}?start=&end=`) — HA proxies all 25 calendars and removes the need for a direct Google OAuth credential in n8n.
- **Dedup**: `memory_facts.source_ref` stores the HA event UID; `ON CONFLICT (user_id, source, source_ref) WHERE source_ref IS NOT NULL DO NOTHING` prevents duplicates.
- **LLM enrichment**: GPT-4.1 classifies each event in batch (category, action_required, do_not_disturb, priority, behavioral_context, pompeo_note).
- **No Qdrant embedding yet** (Phase 2): individual events go to Postgres only; a weekly aggregated embedding will be added later.

### Calendars Tracked

| Calendar | Entity ID | Category | User |
|---|---|---|---|
| Lavoro | `calendar.calendar` | work | martin |
| Famiglia | `calendar.famiglia` | personal | martin |
| Spazzatura | `calendar.spazzatura` | chores | martin |
| Pulizie | `calendar.pulizie` | chores | martin |
| Formula 1 | `calendar.formula_1` | leisure | martin |
| WEC | `calendar.lm_wec_fia_world_endurance_championship` | leisure | martin |
| Inter | `calendar.inter_calendar` | leisure | martin |
| Compleanni | `calendar.birthdays` | social | martin |
| Varie | `calendar.varie` | misc | martin |
| Festività Italia | `calendar.festivita_in_italia` | holiday | shared |
| Films (Radarr) | `calendar.films` | leisure | martin |
| Serie TV (Sonarr) | `calendar.serie_tv` | leisure | martin |

### n8n Workflow

**`📅 Pompeo — Calendar Agent [Schedule]`** — ID `4ZIEGck9n4l5qaDt`

```
⏰ Schedule (06:30) → 📅 Imposta Range → 🔑 Token Copilot
  → 📋 Prepara Calendari (12 items)
  → 📡 HA Fetch (×12, one per calendar)
  → 🏷️ Estrai ed Etichetta (tagged events, flat)
  → 📝 Prepara Prompt (dedup + LLM prompt)
  → 🤖 GPT-4.1 (batch classify all events)
  → 📋 Parse Risposta
  → 💾 Postgres Upsert (memory_facts, per event, ON CONFLICT DO NOTHING)
  → 📦 Aggrega → ✍️ Prepara Messaggio → 📱 Telegram Briefing
```

### Embedding Strategy

- Embeddings are generated via Ollama (`nomic-embed-text` or equivalent) once the LLM server is online
- During bootstrap phase: embeddings generated via GitHub Copilot (`text-embedding-3-small` at `api.githubcopilot.com/embeddings`) — same token acquisition pattern already in use
- Never embed raw content — always embed **LLM-generated summaries + extracted entities**

### Proactive Notification Logic

The Arbiter runs on an **adaptive schedule**:

| Time slot | Frequency | Behavior |
|---|---|---|
| 23:00–07:00 | Never | Silence |
| 07:00–09:00 | Once | Morning briefing (calendar, reminders, pending actions) |
| 09:00–19:00 | Every 2-3h | Only high-priority or correlated events |
| 19:00–22:00 | Once | Evening recap + next day preview |

High-priority queue messages bypass the schedule and trigger immediate notification.

Notification is sent via **Amazon Echo / Pompeo** (TTS) for voice, and **Telegram** for logging. Every Arbiter decision (notify / discard / defer) is logged to a dedicated Telegram audit channel.

### Voice Interface (Pompeo)

- Amazon Echo → **Alexa Custom Skill** → **AWS Lambda** (bridge) → **n8n webhook** → Ollama (Qwen2.5-14B) → TTS response back to Echo
- Wake phrase: "Pompeo"
- Lambda is intentionally thin — it only translates the Alexa request format to the n8n webhook payload and returns the TTS response

---

## Existing n8n Workflows (already in production)

### 📬 Gmail — Daily Digest [Schedule] (`1lIKvVJQIcva30YM`)

- Runs every 3 hours (+ test webhook)
- Fetches unread emails from the last 3 hours
- Calls GPT-4.1 (via Copilot) to classify each email: category, sentiment, labels, action_required, whether it has a Paperless-relevant PDF attachment
- Applies Gmail labels, marks as read, trashes spam
- If a bill/invoice PDF is detected → triggers the **Upload Bolletta** webhook
- Sends a digest report to Telegram

### 📄 Paperless — Upload Documento [Multi] (`GBPFFq8rmbdFrNn9`) ✅ Active

Replaces the two retired workflows below. Single core pipeline with two entry points:

- **Trigger 1 — Telegram**: PDF sent to bot with caption starting with "Documento" → downloads file
- **Trigger 2 — Webhook** (`POST /webhook/paperless-upload`): called by Daily Digest with `{email_id, attachment_id, filename, hint, from}` → downloads attachment from Gmail API

Both paths converge:
- FileWizard OCR (async, polls job) → GPT-4.1 metadata inference → Paperless upload → PATCH metadata → Telegram confirmation → FileWizard cleanup
- Dedup: if Paperless returns "duplicate", patches the existing document's metadata instead

> 🔴 **Retired** (deactivated):
> - `vbzQ3fgUalOPdcOq` — Paperless — Upload Bolletta [Email]
> - `ZX5rLSETg6Xcymps` — Paperless — Upload Documento [Telegram]

---

### 💰 Actual — Import Estratto Conto [Telegram] (`qtvB3r0cgejyCxUp`) ✅ Active

Imports bank CSV statements (Banca Sella format) into Actual Budget via Telegram upload.

- **Trigger**: Telegram bot, send CSV file with caption starting with "Estratto"
- Fetches token Copilot → downloads CSV binary from Telegram
- Parses CSV: skips SALDO rows, extracts transactions with `Id. XXXXXXXXX` as dedup key
- Calls Actual HTTP API to fetch existing payees, categories, and transactions since `min_date`
- Deduplicates: skips transactions already present (by `imported_id = banca-sella-{Id}`)
- Splits remaining transactions into batches of 30
- **Per batch**: GPT-4.1 classifies each transaction (payee, category, notes) → creates missing payees/categories on Actual → imports transactions
- Final Telegram report: imported count, skipped, new payees/categories created
- After report: marks Google Task "Actual - Estratto conto" in list "Finanze" as completed (non-blocking)

**CSV format** (Banca Sella): `;` separator, `dd/mm/yyyy` dates, `.` decimal, Row 1 = header, Row 2 = SALDO FINALE (skip), Last row = SALDO INIZIALE (skip).

### ⏰ Actual — Reminder Estratto Conto [Schedule] (`w0oJ1i6sESvaB5W1`) ✅ Active

- Daily cron at 09:00 → checks Google Tasks for "Actual - Estratto conto" in "Finanze" list
- If task exists and is not completed (due date ≤ today or no due date) → sends Telegram reminder
- Reminder instructs to upload CSV via Telegram with caption `Estratto [mese]`

**Common pattern across Paperless + Actual workflows**: GitHub Copilot token is obtained fresh at each run (`GET https://api.github.com/copilot_internal/v2/token`), then used for `POST https://api.githubcopilot.com/chat/completions` with model `gpt-4.1`.

### 🎬 Pompeo — Jellyfin Playback Agent [Webhook] (`AyrKWvboPldzZPsM`) ✅ Active

Webhook-based — triggered in real time by the Jellyfin Webhook plugin on every `PlaybackStart` / `PlaybackStop` event.

- Webhook path: `jellyfin-playback` (via `https://orchestrator.mt-home.uk/webhook/jellyfin-playback`)
- Normalizes the Jellyfin payload (body is a JSON-encoded string → `JSON.parse` required)
- On **PlaybackStart**: INSERT into `behavioral_context` (`event_type=watching_media`, `do_not_disturb=true`, metadata in `notes` JSONB: `item`, `device`, `item_type`)
- On **PlaybackStop**: UPDATE `behavioral_context` → set `end_at=now()`, `do_not_disturb=false`
- Both events write to `agent_messages` (subject: `▶️ {title} ({device})` or `⏹️ …`)
- **User filter**: only processes events for `UserId = 42369255a7c64917a28fc26d4c7f8265` (Martin)
- **Jellyfin stop behavior**: `PlaybackStop` fires only on player close, NOT on pause

Known quirks fixed:
- Jellyfin webhook plugin sends body as `$json.body` (JSON string) — must `JSON.parse()` before reading fields
- Real Jellyfin field names: `Name` (not `ItemName`), `DeviceName` / `Client`, `UserId` (no dashes)

---

### 🎬 Pompeo — Media Library Sync [Schedule] (`o3uM1xDLTAKw4D6E`) ✅ Active

Weekly cron — every **Sunday at 03:00**.

- Fetches all movies from **Radarr** (`/radarr/api/v3/movie`) and all series from **Sonarr** (`/sonarr/api/v3/series`)
- Merges into unified list: `{type, title, year, genres, status: available|missing|monitored|unmonitored, source, source_id}`
- **GPT-4.1 analysis**: extracts `top_genres`, `preferred_types`, `library_stats`, `taste_summary`, `notable_patterns`
- **Postgres upsert**: `memory_facts` (source=`media_library`, source_ref=`media_preferences_summary`, expires +7d)
- **Per-item loop**: for each movie/series → Ollama `nomic-embed-text` (768-dim) → Qdrant upsert into `media_preferences` collection
  - Qdrant payload: `{title, year, type, genres, status, source, source_id, expires_at (+6 months)}`

Internal endpoints used:
- `http://radarr.media.svc.cluster.local:7878/radarr/api/v3/movie?apikey=922d1405ab1147019d98a2997d941765`
- `http://sonarr.media.svc.cluster.local:8989/sonarr/api/v3/series?apikey=22140655993a4ff6bf12314813ec6982`
- `http://ollama.ai.svc.cluster.local:11434/api/embeddings` (model: `nomic-embed-text`)
- `http://qdrant.persistence.svc.cluster.local:6333` (api-key: sealed secret `qdrant-api-secret`)

---

### 🎞️ Pompeo — Jellyfin Watch History Sync [Schedule] (`K07e4PPANXDkmQsr`) ✅ Active

Daily cron — every day at **04:00**.

- Fetches last 100 played/partially-played items for Martin from Jellyfin API
  - Endpoint: `/Users/42369255a7c64917a28fc26d4c7f8265/Items?Recursive=true&IncludeItemTypes=Movie,Episode&SortBy=DatePlayed&SortOrder=Descending&Limit=100`
  - Auth: `Authorization: MediaBrowser Token="d153606c1ca54574a20d2b40fcf1b02e"` (Pompeo API key)
- Filters items with `PlayCount > 0` and `LastPlayedDate` within 90 days
- **GPT-4.1 analysis**: extracts `recent_favorites`, `preferred_genres`, `watch_patterns`, `completion_rate`, `notes`
- **Postgres upsert**: `memory_facts` (source=`jellyfin`, source_ref=`watch_history_summary`, expires +30d)
- Skips silently if no played items found (IF node guards the LLM call)

Jellyfin credentials:
- API Token (app: Pompeo): `d153606c1ca54574a20d2b40fcf1b02e` — created via `POST /Auth/Keys?app=Pompeo` with admin session
- Martin UserId: `42369255a7c64917a28fc26d4c7f8265` (from Jellyfin SQLite DB / webhook payload)
- Admin user `admin` with password `__Montecarlo00!` (local auth, SSO via Authentik is separate)

---

### 📅 Pompeo — Calendar Agent [Schedule] (`4ZIEGck9n4l5qaDt`) ✅ Active

Runs every morning at 06:30 (and on-demand via manual trigger).

- Fetches events for the next 7 days from 12 Google Calendars via **Home Assistant REST API** (calendar proxy — no Google OAuth needed in n8n)
- Tags each event with calendar name, category, user_id
- **GPT-4.1 batch classification**: category, action_required, do_not_disturb, priority, behavioral_context, pompeo_note
- **Postgres upsert** → `memory_facts` (source=calendar, source_ref=HA event UID, dedup ON CONFLICT DO NOTHING)
- **Telegram briefing**: daily grouped summary sent to the notification channel

Calendars: Lavoro, Famiglia, Spazzatura, Pulizie, Formula 1, WEC, Inter, Compleanni, Varie, Festività Italia, Films (Radarr), Serie TV (Sonarr).

### n8n Credentials (IDs)

| ID | Name | Type |
|---|---|---|
| `qvOikS6IF0H5khr8` | Gmail OAuth2 | OAuth2 |
| `uTXHLqcCJxbOvqN3` | Telegram account | Telegram API |
| `vBwUxlzKrX3oDHyN` | GitHub Copilot OAuth Token | HTTP Header Auth |
| `uvGjLbrN5yQTQIzv` | Paperless-NGX API | HTTP Header Auth |
| `ZIVFNgI3esCKuYXc` | Google Calendar account | Google Calendar OAuth2 (also used for Tasks API) |
| `u0JCseXGnDG5hS9F` | Home Assistant API | HTTP Header Auth (long-lived HA token) |
| `mRqzxhSboGscolqI` | Pompeo — PostgreSQL | Postgres (database: `pompeo`, user: `martin`) |
| `u0JCseXGnDG5hS9F` | Home Assistant API | HTTP Header Auth (long-lived HA token) |

### Qdrant Collections

| Collection | Dimensions | Distance | Model | Used by |
|---|---|---|---|---|
| `media_preferences` | 768 | Cosine | `nomic-embed-text` (Ollama) | Media Library Sync (B1) |

Qdrant API key: sealed secret `qdrant-api-secret` in namespace `persistence` → `__Montecarlo00!`

---

## Coding Conventions

- **n8n workflows**: nodes named in Italian, descriptive emoji prefixes on trigger nodes
- **Workflow naming**: `{icon} {App} — {Azione} {Tipo} [{Sorgente}]` (e.g. `📄 Paperless — Upload Documento [Telegram]`)
- **HTTP nodes**: always use `predefinedCredentialType` for authenticated services already configured in n8n credentials
- **GPT body**: use `contentType: "raw"` + `rawContentType: "application/json"` + `JSON.stringify({...})` inline expression — never `specifyBody: string`
- **LLM output parsing**: always defensive — handle missing `choices`, malformed JSON, empty responses gracefully
- **Copilot token**: always fetched fresh per workflow run, never cached across executions
- **Binary fields**: Telegram node `file.get` with `download: true` stores binary in field named `data` (not `attachment`)
- **Postgres**: use UUID primary keys with `gen_random_uuid()`, JSONB for flexible payloads, always include `created_at`
- **Qdrant upsert**: always include full metadata payload for filtering; use `message_id` / `thread_id` / `doc_id` as logical dedup keys

---

## TO-DO

### Phase 0 — Infrastructure Bootstrap *(prerequisite for everything)*

- [x] ~~Deploy **Qdrant** on the Kubernetes cluster~~ ✅ 2026-03-21
  - Collections: `episodes`, `knowledge`, `preferences` (multi-tenant via `user_id` payload field)
  - Payload indexes: `user_id`, `source`, `category`, `date`, `action_required`
  - Endpoint: `qdrant.persistence.svc.cluster.local:6333`
- [x] ~~Run **PostgreSQL migrations** on Patroni~~ ✅ 2026-03-21
  - Database `pompeo` creato (Zalando Operator)
  - Tabelle: `user_profile`, `memory_facts` (+ `source_ref` + dedup index), `finance_documents`, `behavioral_context`, `agent_messages`
  - Multi-tenancy: campo `user_id` su tutte le tabelle, seed `martin` + `shared`
  - Script DDL: `alpha/db/postgres.sql`
- [x] Verify embedding endpoint via Copilot (`text-embedding-3-small`) as bootstrap fallback
- [x] ~~Plan migration to local Ollama embedding model once LLM server is online~~ ✅ Active — `nomic-embed-text` via `http://ollama.ai.svc.cluster.local:11434` (768-dim, multilingual)
- [ ] Create `ha_sensor_config` table in Postgres and seed initial sensor patterns

---

### Phase 1 — Memory Integration into Existing Workflows

- [ ] **Daily Digest**: after `Parse risposta GPT-4.1`, add:
  - Postgres INSERT into `memory_facts` (source=email, category, subject, detail JSONB, action_required, expires_at)
  - Embedding generation (Copilot endpoint) → Qdrant upsert into `episodes` (user_id=martin)
  - Thread dedup: use `thread_id` as logical key, update existing Qdrant point if thread already exists

- [ ] **Upload Bolletta** + **Upload Documento (Telegram)**: after `Paperless - Patch Metadati`, add:
  - Postgres INSERT into `finance_documents` (correspondent, amount, doc_date, doc_type, tags, paperless_doc_id)
  - Postgres INSERT into `memory_facts` (source=paperless, category=finance, cross-reference)
  - Embedding of OCR text chunks → Qdrant upsert into `knowledge` (user_id=martin)

---

### Phase 2 — New Agents

- [x] ~~**Calendar Agent**~~ ✅ 2026-03-20 — `4ZIEGck9n4l5qaDt`
  - 12 calendari Google via HA proxy, fetch next 7 days
  - GPT-4.1 batch classification → `memory_facts` (dedup by HA event UID)
  - Telegram daily briefing at 06:30
  - **Phase 2**: add weekly Qdrant embedding for semantic retrieval

- [x] ~~**Jellyfin Playback Agent**~~ ✅ 2026-03-21 — `AyrKWvboPldzZPsM`
  - Webhook: PlaybackStart → `behavioral_context` INSERT (`do_not_disturb=true`), PlaybackStop → UPDATE (`end_at`, `do_not_disturb=false`)
  - `agent_messages` populated with `▶️`/`⏹️` + title + device
  - User filter: Martin only (UserId `42369255…`)

- [x] ~~**Media Library Sync (B1)**~~ ✅ 2026-03-21 — `o3uM1xDLTAKw4D6E`
  - Weekly (Sunday 03:00): Radarr + Sonarr → GPT-4.1 taste analysis → `memory_facts` + Qdrant `media_preferences` (768-dim, nomic-embed-text)

- [x] ~~**Jellyfin Watch History Sync (B2)**~~ ✅ 2026-03-21 — `K07e4PPANXDkmQsr`
  - Daily (04:00): Jellyfin play history (90d window) → GPT-4.1 pattern analysis → `memory_facts`

- [ ] **Finance Agent** (extend beyond Paperless)
  - Read Actual Budget export or API
  - Persist transactions, monthly summaries to `finance_documents`
  - Trend analysis prompt for periodic financial summary

- [ ] **Infrastructure Agent**
  - Webhook receiver for Kubernetes/Longhorn/Minio alerts
  - Cron-based cluster health check (disk, pod status, backup freshness)
  - Publishes to message broker with `priority: high` for critical alerts

- [ ] **IoT Agent** — *design complete, implementation pending*
  - Sensor allowlist via `ha_sensor_config` Postgres table (regex-based)
  - No fixed rules: GPT-4.1 infers activity label + confidence from sensor snapshot
  - Three layers: webhook (events) + polling 20min (behavioral_context) + daily cron (Qdrant episodes)
  - Historical bootstrap: 12 months HA history → daily LLM summaries → Qdrant `episodes`
  - Confidence-gated clarification: ask Martin via Telegram if confidence < 0.6

- [ ] **Newsletter Agent**
  - Separate Gmail label for newsletters (excluded from Daily Digest main flow)
  - Morning cron: summarize + extract relevant articles → `knowledge`

---

### Phase 3 — Message Broker + Proactive Arbiter

- [ ] Implement **Proactive Arbiter** n8n workflow:
  - Adaptive schedule (morning briefing, midday, evening recap)
  - Consume `agent_messages` batch → LLM correlation prompt → structured `notify/defer/discard` output
  - High-priority bypass path (direct webhook)
  - All decisions logged to Telegram audit channel
- [ ] Implement **correlation logic**: detect when 2+ agents report related events (e.g. IoT presence + calendar event + open reminder)

---

### Phase 4 — Voice Interface (Pompeo)

- [ ] Create Alexa Custom Skill ("Pompeo")
- [ ] AWS Lambda bridge (thin translator: Alexa request → n8n webhook → TTS response)
- [ ] n8n webhook handler: receive transcribed text → prepend memory context → Ollama inference → return TTS string
- [ ] TTS response pipeline back to Echo
- [ ] Proactive push: Arbiter → Lambda → Echo notification (Alexa proactive events API)

---

### Phase 5 — Generalization and Backlog

- [ ] **OCR on email attachments in Daily Digest**: generalize the ingest pipeline to extract text from any PDF attachment (not just bills), using FileWizard OCR — produce richer embeddings and enable full-text retrieval on any emailed document
- [ ] **Flusso Cedolino** (payslip pipeline):
  - Trigger: Gmail label `Lavoro/Cedolino` or Telegram upload
  - PDF → FileWizard OCR → GPT-4.1 metadata extraction (month, gross, net, deductions)
  - Paperless upload with tag `Cedolino`
  - Persist structured data to `finance_documents` (custom fields for payslip)
  - Trend embedding in `knowledge` for finance agent queries
- [ ] Behavioral habit modeling: aggregate `behavioral_context` records over time, generate periodic "habit summary" embeddings in `preferences`
- [ ] Outline → Qdrant pipeline: sync selected Outline documents into `knowledge` on edit/publish event
- [ ] Chrome browsing history ingestion (privacy-filtered): evaluate browser extension or local export → embedding pipeline for interest/preference modeling
- [ ] "Posti e persone" graph: structured contact/location model in Postgres, populated from email senders, calendar attendees, Home Assistant presence data
- [ ] Local embedding model: migrate from Copilot `text-embedding-3-small` to Ollama-served model (e.g. `nomic-embed-text`) once LLM server is stable