martin/Alpha
1
0
Fork 0
Code Issues 15 Pull Requests Actions Packages Projects 1 Releases Wiki Activity
18 Commits 1 Branch 0 Tags
734a1a94408b414906c46df11af5dbdcd58eed75
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE

README.md

ALPHA_PROJECT — System Context for GitHub Copilot

Who I am

I am a Cloud Architect with 6 years of Big4 consulting experience (currently at manager level at EY), working daily on Azure, Dynamics 365 / Dataverse, .NET, and enterprise integration patterns. I run a production-grade homelab on Kubernetes at home, and I am building ALPHA_PROJECT as a personal initiative in my spare time (evenings and weekends).

What ALPHA_PROJECT is

ALPHA_PROJECT is a proactive, multi-agent personal AI assistant built entirely on self-hosted infrastructure. It is not a chatbot. It is an autonomous system that:

  • Monitors my digital life (email, calendar, home automation, finances, infrastructure)
  • Maintains a persistent, structured memory of facts, habits, and preferences
  • Takes initiative to notify me of relevant events, correlations, and pending actions
  • Interacts with me via voice (Amazon Echo / Alexa custom skill named "Pompeo") and Telegram
  • Runs local LLMs on dedicated hardware — no cloud AI inference (except GitHub Copilot completions, available via EY license at zero cost)

The assistant is named Pompeo (the Alexa skill wake word).

Infrastructure

LLM Server (new, dedicated node — outside the Kubernetes cluster)

  • CPU: AMD Ryzen 5 4500
  • RAM: 16 GB DDR4
  • GPU: NVIDIA GeForce RTX 3060 (16 GB VRAM)
  • Runtime: Ollama (API-compatible with OpenAI)
  • Primary model: Qwen2.5-14B-Instruct Q4_K_M (fits entirely in VRAM, no offload)
  • Secondary model: Qwen2.5-Coder-14B-Instruct Q4_K_M (for code-related tasks)
  • Embedding model: TBD — to be served via Ollama (e.g. nomic-embed-text)
  • Constraint: zero RAM offload — all models must fit entirely in 16 GB VRAM

Kubernetes Homelab Cluster

Production-grade self-hosted stack. Key components relevant to ALPHA_PROJECT:

Component Role
n8n Primary orchestrator and workflow engine for all agents
Node-RED Event-driven automation, Home Assistant bridge
Patroni / PostgreSQL Persistent structured memory store — postgres.persistence.svc.cluster.local:5432/pompeo
Qdrant Vector store for semantic/episodic memory — qdrant.persistence.svc.cluster.local:6333
Home Assistant IoT hub — device tracking, automations, sensors, Google Calendar proxy
MikroTik Network — VLANs, firewall rules, device presence detection
Paperless-ngx Document archive (docs.mt-home.uk)
Actual Budget Personal finance
Mealie Meal planning / recipes
Immich Photo library
Outline Internal wiki / knowledge base
Radarr / Sonarr Media management
Jenkins CI/CD
AdGuard DNS filtering
WireGuard VPN
Minio S3-compatible object storage
Longhorn Distributed block storage
Velero Disaster recovery / backup

External Services (in use)

  • Gmail — primary email
  • Google Calendar — calendar (multiple calendars: Work, Family, Formula 1, WEC, Inter, Birthdays, Tasks, Pulizie, Spazzatura, Festività Italia, Varie)
  • Amazon Echo — voice interface for Pompeo
  • AWS Lambda — bridge between Alexa skill and n8n webhook
  • Telegram — notifications, logging, manual document upload
  • GitHub Copilot (GPT-4.1 via api.githubcopilot.com) — LLM completions at zero cost (EY license)

Internal Services / Custom

  • orchestrator.mt-home.uk — n8n instance
  • docs.mt-home.uk — Paperless-ngx
  • filewizard.home.svc.cluster.local:8000 — custom OCR microservice (async, job-based API)

Architecture Overview

Multi-Agent Design

ALPHA_PROJECT uses specialized agents, each responsible for a specific data domain. All agents are implemented as n8n workflows.

Agent Trigger Responsibility
Mail Agent Cron every 15-30 min Read Gmail, classify emails, extract facts, detect invoices/bills
Finance Agent Triggered by Mail Agent or Telegram Process PDF invoices/bills, archive to Paperless, persist to memory
Calendar Agent Cron + on-demand Read Google Calendar, detect upcoming events, cross-reference with other agents
Infrastructure Agent Cron + alert webhooks Monitor Kubernetes cluster health, disk usage, failed jobs
IoT Agent Event-driven (Home Assistant webhooks) Monitor device presence, home state, learn behavioral patterns
Newsletter Agent Cron morning Digest newsletters, extract relevant articles
Proactive Arbiter Cron (adaptive frequency) + high-priority queue messages Consume agent outputs, correlate, decide what to notify

Message Broker (Blackboard Pattern)

Agents do not call each other directly. They write observations to the agent_messages table in PostgreSQL (blackboard pattern). The Proactive Arbiter polls this table, batches low-priority messages, and immediately processes high-priority ones. High-urgency events trigger a direct n8n webhook call bypassing the queue.

ADR: No dedicated message broker — Postgres is sufficient for the expected message volume and avoids operational overhead. Revisit if throughput exceeds 1k messages/day.

Memory Architecture

Three layers of persistence:

1. Structured memory — PostgreSQL (Patroni)

Episodic facts, finance records, reminders, behavioral observations. Fast, queryable, expirable.

-- Generic episodic facts
CREATE TABLE memory_facts (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  source TEXT NOT NULL,         -- 'email', 'calendar', 'iot', 'paperless', ...
  category TEXT,                -- 'finance', 'personal', 'work', 'health', ...
  subject TEXT,
  detail JSONB,                 -- flexible per-source payload
  action_required BOOLEAN DEFAULT false,
  action_text TEXT,
  created_at TIMESTAMP DEFAULT now(),
  expires_at TIMESTAMP,         -- facts have a TTL
  qdrant_id UUID                -- FK to vector store
);

-- Finance documents (frequent structured queries)
CREATE TABLE finance_documents (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  paperless_doc_id INT,
  correspondent TEXT,
  amount NUMERIC(10,2),
  currency TEXT DEFAULT 'EUR',
  doc_date DATE,
  doc_type TEXT,
  tags TEXT[],
  created_at TIMESTAMP DEFAULT now()
);

-- Behavioral context (used by IoT agent and Arbiter)
CREATE TABLE behavioral_context (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  event_type TEXT,              -- 'sport_event', 'dog_walk', 'work_session', ...
  start_at TIMESTAMP,
  end_at TIMESTAMP,
  do_not_disturb BOOLEAN DEFAULT false,
  home_presence_expected BOOLEAN,
  notes TEXT
);

2. Semantic memory — Qdrant — qdrant.persistence.svc.cluster.local:6333

Vector embeddings for similarity search. Three collections with multi-tenant design: isolation via user_id payload field ("martin", "shared", future users).

Collection Content
episodes Conversations, episodic facts with timestamp
knowledge Documents, Outline notes, newsletters, knowledge base
preferences Preferences, habits, behavioral patterns

Each Qdrant point includes a metadata payload for pre-filtering (user_id, source, date, category, action_required) to avoid full-scan similarity searches.

3. Profile memory — PostgreSQL (static table)

User preferences, fixed facts, communication style. Updated manually or via explicit agent action.

IoT Agent — Design Notes

Data Source: Home Assistant

Home Assistant (http://10.30.20.100:8123, HA OS 2026.3.2, Alzano Lombardo BG) is the primary hub for physical-world context. It aggregates Google Pixel 10, Pixel Watch 4, smart home devices, and 25 Google Calendars.

Person allowlist (permanent by design — person.ajada_tahiraj is explicitly excluded):

Person Entity Notes
Martin Tahiraj person.martin_tahiraj ✅ Tracked
Ajada Tahiraj person.ajada_tahiraj ❌ Excluded (sister — privacy)

Key sensors for Martin:

Sensor Entity ID Signal
Activity (Google) sensor.pixel_10_detected_activity still / walking / running / in_vehicle
Geocoded location sensor.pixel_10_geocoded_location Human-readable street address
EY laptop device_tracker.ey_hp Router tracker — online = laptop on home WiFi
Spotify media_player.spotify_martin Current track, playing/paused
Sleep duration sensor.pixel_10_sleep_duration Pixel Watch 4
Next alarm sensor.pixel_10_next_alarm Scheduled wake-up
Work Profile binary_sensor.pixel_10_work_profile Android Work Profile active
Screen on binary_sensor.pixel_10_interactive Phone screen on/off
Do Not Disturb binary_sensor.pixel_10_do_not_disturb DND mode
Daily steps sensor.pixel_10_daily_steps Pixel Watch 4
Heart rate sensor.pixel_10_heart_rate Pixel Watch 4
GPS Zone person.martin_tahiraj home / not_home / zone name

Room presence sensors (PIR-based) are considered unreliable — excluded for now.

Sensor Allowlist — ha_sensor_config

Instead of hardcoded rules, the IoT Agent uses a dynamic allowlist stored in Postgres. Sensors are matched by regex pattern, allowing glob-style additions:

CREATE TABLE ha_sensor_config (
  id          SERIAL PRIMARY KEY,
  pattern     TEXT NOT NULL,          -- regex pattern, e.g. 'sensor\.pixel_10_.*'
  user_id     TEXT NOT NULL,
  group_name  TEXT NOT NULL,          -- 'mobile_device' | 'work_presence' | 'entertainment' | ...
  description TEXT,
  active      BOOLEAN NOT NULL DEFAULT true
);

-- Seed entries
INSERT INTO ha_sensor_config (pattern, user_id, group_name, description) VALUES
  ('sensor\.pixel_10_.*',           'martin', 'mobile_device', 'All Pixel 10 sensors'),
  ('device_tracker\.ey_hp',         'martin', 'work_presence', 'EY Laptop router tracker'),
  ('media_player\.spotify_martin',  'martin', 'entertainment', 'Spotify'),
  ('binary_sensor\.pixel_10_.*',    'martin', 'mobile_device', 'Pixel 10 binary sensors'),
  ('person\.martin_tahiraj',        'martin', 'presence',      'Martin GPS zone state');

This allows adding new sensors (e.g. sensor.pixel_watch_.*) without workflow changes.

Activity State Machine (LLM-based — no fixed rules)

The IoT Agent sends a snapshot of all allowlisted sensor values to GPT-4.1 and asks it to infer the current activity label and confidence. No if/else rules are coded — the LLM performs inference.

Example LLM output:

{
  "activity": "home_working",
  "confidence": 0.92,
  "do_not_disturb": true,
  "location": "home",
  "notes": "Laptop EY online, work profile attivo, orario lavorativo 09-18"
}

Activity labels: sleeping, home_relaxing, home_working, commuting, at_office, out_errands, out_with_dog, exercising, traveling, unknown.

Three-Layer Data Flow

Layer Trigger Frequency Output
Webhook HA automation (zone change, motion) Event-driven Immediate agent_messages entry
Polling n8n cron Every 20 min Sensor snapshot → LLM → behavioral_context
Daily cron n8n cron midnight Once/day Day summary → Qdrant episodes embedding

Historical Bootstrap

One-time job: last 12 months of HA sensor history → daily LLM summaries → Qdrant episodes.

  • Source: HA History API (/api/history/period/{start}?filter_entity_id=...)
  • Output: one Qdrant point per day per user, with full behavioral context

Confidence-Gated Clarification

When activity inference confidence < 0.6, or when Pompeo detects a potential life change (new employer from emails, travel pattern, etc.), it asks Martin directly via Telegram:

"Ciao Martin, sto notando email di Avanade — lavori ancora per EY o sei passato lì? 🤔"

Pompeo updates user_profile or memory_facts with the confirmed fact and adjusts its confidence threshold.

Calendar Agent — Design Notes

Design Decisions

  • Data source: Google Calendar events fetched via Home Assistant REST API (/api/calendars/{entity_id}?start=&end=) — HA proxies all 25 calendars and removes the need for a direct Google OAuth credential in n8n.
  • Dedup: memory_facts.source_ref stores the HA event UID; ON CONFLICT (user_id, source, source_ref) WHERE source_ref IS NOT NULL DO NOTHING prevents duplicates.
  • LLM enrichment: GPT-4.1 classifies each event in batch (category, action_required, do_not_disturb, priority, behavioral_context, pompeo_note).
  • No Qdrant embedding yet (Phase 2): individual events go to Postgres only; a weekly aggregated embedding will be added later.

Calendars Tracked

Calendar Entity ID Category User
Lavoro calendar.calendar work martin
Famiglia calendar.famiglia personal martin
Spazzatura calendar.spazzatura chores martin
Pulizie calendar.pulizie chores martin
Formula 1 calendar.formula_1 leisure martin
WEC calendar.lm_wec_fia_world_endurance_championship leisure martin
Inter calendar.inter_calendar leisure martin
Compleanni calendar.birthdays social martin
Varie calendar.varie misc martin
Festività Italia calendar.festivita_in_italia holiday shared
Films (Radarr) calendar.films leisure martin
Serie TV (Sonarr) calendar.serie_tv leisure martin

n8n Workflow

📅 Pompeo — Calendar Agent [Schedule] — ID 4ZIEGck9n4l5qaDt

⏰ Schedule (06:30) → 📅 Imposta Range → 🔑 Token Copilot
  → 📋 Prepara Calendari (12 items)
  → 📡 HA Fetch (×12, one per calendar)
  → 🏷️ Estrai ed Etichetta (tagged events, flat)
  → 📝 Prepara Prompt (dedup + LLM prompt)
  → 🤖 GPT-4.1 (batch classify all events)
  → 📋 Parse Risposta
  → 💾 Postgres Upsert (memory_facts, per event, ON CONFLICT DO NOTHING)
  → 📦 Aggrega → ✍️ Prepara Messaggio → 📱 Telegram Briefing

Embedding Strategy

  • Embeddings are generated via Ollama (nomic-embed-text or equivalent) once the LLM server is online
  • During bootstrap phase: embeddings generated via GitHub Copilot (text-embedding-3-small at api.githubcopilot.com/embeddings) — same token acquisition pattern already in use
  • Never embed raw content — always embed LLM-generated summaries + extracted entities

Proactive Notification Logic

The Arbiter runs on an adaptive schedule:

Time slot Frequency Behavior
23:00–07:00 Never Silence
07:00–09:00 Once Morning briefing (calendar, reminders, pending actions)
09:00–19:00 Every 2-3h Only high-priority or correlated events
19:00–22:00 Once Evening recap + next day preview

High-priority queue messages bypass the schedule and trigger immediate notification.

Notification is sent via Amazon Echo / Pompeo (TTS) for voice, and Telegram for logging. Every Arbiter decision (notify / discard / defer) is logged to a dedicated Telegram audit channel.

Voice Interface (Pompeo)

  • Amazon Echo → Alexa Custom Skill → AWS Lambda (bridge) → n8n webhook → Ollama (Qwen2.5-14B) → TTS response back to Echo
  • Wake phrase: "Pompeo"
  • Lambda is intentionally thin — it only translates the Alexa request format to the n8n webhook payload and returns the TTS response

Existing n8n Workflows (already in production)

📬 Gmail — Daily Digest [Schedule] (1lIKvVJQIcva30YM)

  • Runs every 3 hours (+ test webhook)
  • Fetches unread emails from the last 3 hours
  • Calls GPT-4.1 (via Copilot) to classify each email: category, sentiment, labels, action_required, whether it has a Paperless-relevant PDF attachment
  • Applies Gmail labels, marks as read, trashes spam
  • If a bill/invoice PDF is detected → triggers the Upload Bolletta webhook
  • Sends a digest report to Telegram

📄 Paperless — Upload Documento [Multi] (GBPFFq8rmbdFrNn9) ✅ Active

Replaces the two retired workflows below. Single core pipeline with two entry points:

  • Trigger 1 — Telegram: PDF sent to bot with caption starting with "Documento" → downloads file
  • Trigger 2 — Webhook (POST /webhook/paperless-upload): called by Daily Digest with {email_id, attachment_id, filename, hint, from} → downloads attachment from Gmail API

Both paths converge:

  • FileWizard OCR (async, polls job) → GPT-4.1 metadata inference → Paperless upload → PATCH metadata → Telegram confirmation → FileWizard cleanup
  • Dedup: if Paperless returns "duplicate", patches the existing document's metadata instead

🔴 Retired (deactivated):

  • vbzQ3fgUalOPdcOq — Paperless — Upload Bolletta [Email]
  • ZX5rLSETg6Xcymps — Paperless — Upload Documento [Telegram]

💰 Actual — Import Estratto Conto [Telegram] (qtvB3r0cgejyCxUp) ✅ Active

Imports bank CSV statements (Banca Sella format) into Actual Budget via Telegram upload.

  • Trigger: Telegram bot, send CSV file with caption starting with "Estratto"
  • Fetches token Copilot → downloads CSV binary from Telegram
  • Parses CSV: skips SALDO rows, extracts transactions with Id. XXXXXXXXX as dedup key
  • Calls Actual HTTP API to fetch existing payees, categories, and transactions since min_date
  • Deduplicates: skips transactions already present (by imported_id = banca-sella-{Id})
  • Splits remaining transactions into batches of 30
  • Per batch: GPT-4.1 classifies each transaction (payee, category, notes) → creates missing payees/categories on Actual → imports transactions
  • Final Telegram report: imported count, skipped, new payees/categories created
  • After report: marks Google Task "Actual - Estratto conto" in list "Finanze" as completed (non-blocking)

CSV format (Banca Sella): ; separator, dd/mm/yyyy dates, . decimal, Row 1 = header, Row 2 = SALDO FINALE (skip), Last row = SALDO INIZIALE (skip).

⏰ Actual — Reminder Estratto Conto [Schedule] (w0oJ1i6sESvaB5W1) ✅ Active

  • Daily cron at 09:00 → checks Google Tasks for "Actual - Estratto conto" in "Finanze" list
  • If task exists and is not completed (due date ≤ today or no due date) → sends Telegram reminder
  • Reminder instructs to upload CSV via Telegram with caption Estratto [mese]

Common pattern across Paperless + Actual workflows: GitHub Copilot token is obtained fresh at each run (GET https://api.github.com/copilot_internal/v2/token), then used for POST https://api.githubcopilot.com/chat/completions with model gpt-4.1.

🎬 Pompeo — Jellyfin Playback Agent [Webhook] (AyrKWvboPldzZPsM) ✅ Active

Webhook-based — triggered in real time by the Jellyfin Webhook plugin on every PlaybackStart / PlaybackStop event.

  • Webhook path: jellyfin-playback (via https://orchestrator.mt-home.uk/webhook/jellyfin-playback)
  • Normalizes the Jellyfin payload (body is a JSON-encoded string → JSON.parse required)
  • On PlaybackStart: INSERT into behavioral_context (event_type=watching_media, do_not_disturb=true, metadata in notes JSONB: item, device, item_type)
  • On PlaybackStop: UPDATE behavioral_context → set end_at=now(), do_not_disturb=false
  • Both events write to agent_messages (subject: ▶️ {title} ({device}) or ⏹️ …)
  • User filter: only processes events for UserId = 42369255a7c64917a28fc26d4c7f8265 (Martin)
  • Jellyfin stop behavior: PlaybackStop fires only on player close, NOT on pause

Known quirks fixed:

  • Jellyfin webhook plugin sends body as $json.body (JSON string) — must JSON.parse() before reading fields
  • Real Jellyfin field names: Name (not ItemName), DeviceName / Client, UserId (no dashes)

🎬 Pompeo — Media Library Sync [Schedule] (o3uM1xDLTAKw4D6E) ✅ Active

Weekly cron — every Sunday at 03:00.

  • Fetches all movies from Radarr (/radarr/api/v3/movie) and all series from Sonarr (/sonarr/api/v3/series)
  • Merges into unified list: {type, title, year, genres, status: available|missing|monitored|unmonitored, source, source_id}
  • GPT-4.1 analysis: extracts top_genres, preferred_types, library_stats, taste_summary, notable_patterns
  • Postgres upsert: memory_facts (source=media_library, source_ref=media_preferences_summary, expires +7d)
  • Per-item loop: for each movie/series → Ollama nomic-embed-text (768-dim) → Qdrant upsert into media_preferences collection
    • Qdrant payload: {title, year, type, genres, status, source, source_id, expires_at (+6 months)}

Internal endpoints used:

  • http://radarr.media.svc.cluster.local:7878/radarr/api/v3/movie?apikey=922d1405ab1147019d98a2997d941765
  • http://sonarr.media.svc.cluster.local:8989/sonarr/api/v3/series?apikey=22140655993a4ff6bf12314813ec6982
  • http://ollama.ai.svc.cluster.local:11434/api/embeddings (model: nomic-embed-text)
  • http://qdrant.persistence.svc.cluster.local:6333 (api-key: sealed secret qdrant-api-secret)

🎞️ Pompeo — Jellyfin Watch History Sync [Schedule] (K07e4PPANXDkmQsr) ✅ Active

Daily cron — every day at 04:00.

  • Fetches last 100 played/partially-played items for Martin from Jellyfin API
    • Endpoint: /Users/42369255a7c64917a28fc26d4c7f8265/Items?Recursive=true&IncludeItemTypes=Movie,Episode&SortBy=DatePlayed&SortOrder=Descending&Limit=100
    • Auth: Authorization: MediaBrowser Token="d153606c1ca54574a20d2b40fcf1b02e" (Pompeo API key)
  • Filters items with PlayCount > 0 and LastPlayedDate within 90 days
  • GPT-4.1 analysis: extracts recent_favorites, preferred_genres, watch_patterns, completion_rate, notes
  • Postgres upsert: memory_facts (source=jellyfin, source_ref=watch_history_summary, expires +30d)
  • Skips silently if no played items found (IF node guards the LLM call)

Jellyfin credentials:

  • API Token (app: Pompeo): d153606c1ca54574a20d2b40fcf1b02e — created via POST /Auth/Keys?app=Pompeo with admin session
  • Martin UserId: 42369255a7c64917a28fc26d4c7f8265 (from Jellyfin SQLite DB / webhook payload)
  • Admin user admin with password __Montecarlo00! (local auth, SSO via Authentik is separate)

📅 Pompeo — Calendar Agent [Schedule] (4ZIEGck9n4l5qaDt) ✅ Active

Runs every morning at 06:30 (and on-demand via manual trigger).

  • Fetches events for the next 7 days from 12 Google Calendars via Home Assistant REST API (calendar proxy — no Google OAuth needed in n8n)
  • Tags each event with calendar name, category, user_id
  • GPT-4.1 batch classification: category, action_required, do_not_disturb, priority, behavioral_context, pompeo_note
  • Postgres upsert → memory_facts (source=calendar, source_ref=HA event UID, dedup ON CONFLICT DO NOTHING)
  • Telegram briefing: daily grouped summary sent to the notification channel

Calendars: Lavoro, Famiglia, Spazzatura, Pulizie, Formula 1, WEC, Inter, Compleanni, Varie, Festività Italia, Films (Radarr), Serie TV (Sonarr).

n8n Credentials (IDs)

ID Name Type
qvOikS6IF0H5khr8 Gmail OAuth2 OAuth2
uTXHLqcCJxbOvqN3 Telegram account Telegram API
vBwUxlzKrX3oDHyN GitHub Copilot OAuth Token HTTP Header Auth
uvGjLbrN5yQTQIzv Paperless-NGX API HTTP Header Auth
ZIVFNgI3esCKuYXc Google Calendar account Google Calendar OAuth2 (also used for Tasks API)
u0JCseXGnDG5hS9F Home Assistant API HTTP Header Auth (long-lived HA token)
mRqzxhSboGscolqI Pompeo — PostgreSQL Postgres (database: pompeo, user: martin)
u0JCseXGnDG5hS9F Home Assistant API HTTP Header Auth (long-lived HA token)

Qdrant Collections

Collection Dimensions Distance Model Used by
media_preferences 768 Cosine nomic-embed-text (Ollama) Media Library Sync (B1)

Qdrant API key: sealed secret qdrant-api-secret in namespace persistence → __Montecarlo00!

Coding Conventions

  • n8n workflows: nodes named in Italian, descriptive emoji prefixes on trigger nodes
  • Workflow naming: {icon} {App} — {Azione} {Tipo} [{Sorgente}] (e.g. 📄 Paperless — Upload Documento [Telegram])
  • HTTP nodes: always use predefinedCredentialType for authenticated services already configured in n8n credentials
  • GPT body: use contentType: "raw" + rawContentType: "application/json" + JSON.stringify({...}) inline expression — never specifyBody: string
  • LLM output parsing: always defensive — handle missing choices, malformed JSON, empty responses gracefully
  • Copilot token: always fetched fresh per workflow run, never cached across executions
  • Binary fields: Telegram node file.get with download: true stores binary in field named data (not attachment)
  • Postgres: use UUID primary keys with gen_random_uuid(), JSONB for flexible payloads, always include created_at
  • Qdrant upsert: always include full metadata payload for filtering; use message_id / thread_id / doc_id as logical dedup keys

TO-DO

Phase 0 — Infrastructure Bootstrap (prerequisite for everything)

  • Deploy Qdrant on the Kubernetes cluster ✅ 2026-03-21
    • Collections: episodes, knowledge, preferences (multi-tenant via user_id payload field)
    • Payload indexes: user_id, source, category, date, action_required
    • Endpoint: qdrant.persistence.svc.cluster.local:6333
  • Run PostgreSQL migrations on Patroni ✅ 2026-03-21
    • Database pompeo creato (Zalando Operator)
    • Tabelle: user_profile, memory_facts (+ source_ref + dedup index), finance_documents, behavioral_context, agent_messages
    • Multi-tenancy: campo user_id su tutte le tabelle, seed martin + shared
    • Script DDL: alpha/db/postgres.sql
  • Verify embedding endpoint via Copilot (text-embedding-3-small) as bootstrap fallback
  • Plan migration to local Ollama embedding model once LLM server is online ✅ Active — nomic-embed-text via http://ollama.ai.svc.cluster.local:11434 (768-dim, multilingual)
  • Create ha_sensor_config table in Postgres and seed initial sensor patterns

Phase 1 — Memory Integration into Existing Workflows

  • Daily Digest: after Parse risposta GPT-4.1, add:

    • Postgres INSERT into memory_facts (source=email, category, subject, detail JSONB, action_required, expires_at)
    • Embedding generation (Copilot endpoint) → Qdrant upsert into episodes (user_id=martin)
    • Thread dedup: use thread_id as logical key, update existing Qdrant point if thread already exists

  • Upload Bolletta + Upload Documento (Telegram): after Paperless - Patch Metadati, add:

    • Postgres INSERT into finance_documents (correspondent, amount, doc_date, doc_type, tags, paperless_doc_id)
    • Postgres INSERT into memory_facts (source=paperless, category=finance, cross-reference)
    • Embedding of OCR text chunks → Qdrant upsert into knowledge (user_id=martin)

Phase 2 — New Agents

  • Calendar Agent ✅ 2026-03-20 — 4ZIEGck9n4l5qaDt

    • 12 calendari Google via HA proxy, fetch next 7 days
    • GPT-4.1 batch classification → memory_facts (dedup by HA event UID)
    • Telegram daily briefing at 06:30
    • Phase 2: add weekly Qdrant embedding for semantic retrieval

  • Jellyfin Playback Agent ✅ 2026-03-21 — AyrKWvboPldzZPsM

    • Webhook: PlaybackStart → behavioral_context INSERT (do_not_disturb=true), PlaybackStop → UPDATE (end_at, do_not_disturb=false)
    • agent_messages populated with ▶️/⏹️ + title + device
    • User filter: Martin only (UserId 42369255…)

  • Media Library Sync (B1) ✅ 2026-03-21 — o3uM1xDLTAKw4D6E

    • Weekly (Sunday 03:00): Radarr + Sonarr → GPT-4.1 taste analysis → memory_facts + Qdrant media_preferences (768-dim, nomic-embed-text)

  • Jellyfin Watch History Sync (B2) ✅ 2026-03-21 — K07e4PPANXDkmQsr

    • Daily (04:00): Jellyfin play history (90d window) → GPT-4.1 pattern analysis → memory_facts

  • Finance Agent (extend beyond Paperless)

    • Read Actual Budget export or API
    • Persist transactions, monthly summaries to finance_documents
    • Trend analysis prompt for periodic financial summary

  • Infrastructure Agent

    • Webhook receiver for Kubernetes/Longhorn/Minio alerts
    • Cron-based cluster health check (disk, pod status, backup freshness)
    • Publishes to message broker with priority: high for critical alerts

  • IoT Agent — design complete, implementation pending

    • Sensor allowlist via ha_sensor_config Postgres table (regex-based)
    • No fixed rules: GPT-4.1 infers activity label + confidence from sensor snapshot
    • Three layers: webhook (events) + polling 20min (behavioral_context) + daily cron (Qdrant episodes)
    • Historical bootstrap: 12 months HA history → daily LLM summaries → Qdrant episodes
    • Confidence-gated clarification: ask Martin via Telegram if confidence < 0.6

  • Newsletter Agent

    • Separate Gmail label for newsletters (excluded from Daily Digest main flow)
    • Morning cron: summarize + extract relevant articles → knowledge

Phase 3 — Message Broker + Proactive Arbiter

  • Implement Proactive Arbiter n8n workflow:
    • Adaptive schedule (morning briefing, midday, evening recap)
    • Consume agent_messages batch → LLM correlation prompt → structured notify/defer/discard output
    • High-priority bypass path (direct webhook)
    • All decisions logged to Telegram audit channel
  • Implement correlation logic: detect when 2+ agents report related events (e.g. IoT presence + calendar event + open reminder)

Phase 4 — Voice Interface (Pompeo)

  • Create Alexa Custom Skill ("Pompeo")
  • AWS Lambda bridge (thin translator: Alexa request → n8n webhook → TTS response)
  • n8n webhook handler: receive transcribed text → prepend memory context → Ollama inference → return TTS string
  • TTS response pipeline back to Echo
  • Proactive push: Arbiter → Lambda → Echo notification (Alexa proactive events API)

Phase 5 — Generalization and Backlog

  • OCR on email attachments in Daily Digest: generalize the ingest pipeline to extract text from any PDF attachment (not just bills), using FileWizard OCR — produce richer embeddings and enable full-text retrieval on any emailed document
  • Flusso Cedolino (payslip pipeline):
    • Trigger: Gmail label Lavoro/Cedolino or Telegram upload
    • PDF → FileWizard OCR → GPT-4.1 metadata extraction (month, gross, net, deductions)
    • Paperless upload with tag Cedolino
    • Persist structured data to finance_documents (custom fields for payslip)
    • Trend embedding in knowledge for finance agent queries
  • Behavioral habit modeling: aggregate behavioral_context records over time, generate periodic "habit summary" embeddings in preferences
  • Outline → Qdrant pipeline: sync selected Outline documents into knowledge on edit/publish event
  • Chrome browsing history ingestion (privacy-filtered): evaluate browser extension or local export → embedding pipeline for interest/preference modeling
  • "Posti e persone" graph: structured contact/location model in Postgres, populated from email senders, calendar attendees, Home Assistant presence data
  • Local embedding model: migrate from Copilot text-embedding-3-small to Ollama-served model (e.g. nomic-embed-text) once LLM server is stable
Reference in New Issue View Git Blame Copy Permalink
Description
No description provided
Readme 2.4 MiB
Languages
Python 99.4%
JavaScript 0.3%
Shell 0.3%