📄 raw markdown for agents/LLMs: /FUNNEL.md · all docs

Bluebut Funnel — the self-owned acquisition pipeline (ZERO fb-bot)

✅ STATUS 2026-07-08 — Layers 1-2 SHIPPED + committed (4eebae3), 3-5 next

• Layer 1 Harvest — DONE (wire scrapers, pre-existing).
• Layer 2 Judge — SHIPPED. classify-posts (browserless Strategy.PUBLIC job) + _lib/llm.js (one llm.hostbun.cc client, claude-haiku-4-5) + _lib/judge.js (ONE combined classify+extract call). NAMING CHANGED from this doc's draft: instead of fb_judgment + housing_extract, the output lands in the two standard rental tables post_extract_listings (offers) + post_extract_seekers (demand) — the operator rejected the fb_judgment/append-only shape as over-engineered (domain nouns, not clever names). A post's type = which table it landed in; fb_post.classified_at marks it judged; fb_person.intent is the derived cache.
• Identity spine (prereq for dedup + outreach) — DONE: fb_person canonical on fb_id (74 split humans merged); the 3 duplicate User hubs (fb_person/fb_user/fb_profile) collapsed to ONE. fb_group_member member is an ACTOR (User or Page → no FK). See CLAUDE.md + reference_fb_identity_model memory.
• Layer 4 Act — dedup HALF-SHIPPED: the "have we messaged this person? block" guard (messenger/lib/dm-dedup.js alreadyMessaged/stampMessaged) is wired into send-message. The outreach ORCHESTRATION (candidate query → route sender → drain) is still to build.
• Layers 3 (Decide/match-listings) + 5 (Measure) — not started.

Design goal: full cutover. Bluebut owns the ENTIRE housing-lead funnel end to end — harvest → judge → decide → outreach → measure — inside its own data plane (bluebut-db), with no dependency on fb-bot and no read/write to bofrid.dev. This document is the target architecture. fb-bot (fb_scraped_posts, fb_sent_messages, fb_broadcast_posts, the realtime classifier, the approval manager) is retired — see §9 Cutover.

0. The one idea

fb-bot models a pipeline with a materialized queue: scrape rows into fb_scraped_posts → a manager mutates a status column → outreach drains it. That coupling (a status machine some daemon owns, a realtime trigger, an approval service) is the architecture we are leaving.

Bluebut models an entity graph + verbs + one log (the house doctrine: one object, one verb, one log). So the funnel is NOT a pipeline:

• fb_person is the object. Intent, company-ness, reachability are derived properties of it.
• Outreach is a verb (a Job) against that object.
• The DM thread is the log (fb_thread/fb_message).
• The "queue" is a QUERY, never a table. The set of people to contact is a live SELECT over the graph, computed fresh on every tick — exactly like bluebut already computes account-routing and the stalest-target sweep. No status column for a daemon to own, no trigger, no approval-manager, no state-machine drift.

Everything below follows from that.

1. The five layers

HARVEST ──▶ JUDGE ──▶ DECIDE ──▶ ACT ──▶ MEASURE
(wire)      (typed    (a QUERY,  (verb + (eval:
            enrich)   not a tbl) re-read) offline+online+pre-act)

The only genuinely new code is _lib/llm.js, _lib/judge.js, the eval harness, one select helper, one outreach job, and one UI surface. Everything else is machinery bluebut already owns (jobs, scheduler, wire harvest, messenger send, entity graph, job ledger, HyperDX).

2. Layer 1 — Harvest (already wire, already ours)

Two channels feed the graph. Both are WIRE (truth-ladder), both already land in bluebut-db. No change needed except that they stop feeding fb-bot (§9).

• Channel A — admin member-request scraper (scrape-new-members, WIRE): the highest-value channel. The join questionnaire carries explicit intent + email + registration age + city off the wire → fb_person + fb_person_signal. These leads are pre-qualified with intent_confidence='explicit' and need zero LLM.
• Channel B — group feed scraper (get-group-feed, WIRE): every group post → fb_post (real numeric post_id, permalink_url, epoch creation_time, owning User, body) + a lazy fb_person for the poster. Free-text → needs judging.

Both write only bluebut-db. The fbbot-feed.js fan-out to bofrid.dev fb_scraped_posts is deleted (§9).

3. Layer 2 — Judge (TWO LLM passes: classify, then extract)

Same shape as fb-bot's proven flow, on bluebut's rails. Two passes, both to llm.hostbun.cc, because they answer different questions and the second is only worth paying for once the first says "housing":

Pass 1 — CLASSIFY (cheap, every unjudged post). Send the post text to llm.hostbun.cc → typed JSON:

• post_type ∈ rental-listing | listing-seeker | none | unknown
• is_company bool
• confidence 0–100

Gate: confidence ≥ threshold AND post_type ∈ {rental-listing, listing-seeker} → classified (keep). Else → rejected (off-topic / low-confidence / our own post). The threshold lives in a config row, not a const, so it's tunable without a deploy.

Pass 2 — EXTRACT (only for classified posts). A second call with a direction-specific prompt pulls the STRUCTURED object into ONE symmetric table, housing_extract, with a direction:

• rental-listing → direction='offer': the shared dimensions (kommun, lan, rooms, area_m2, price_sek=rent, move_in, property_type) + extras jsonb = the offer-only features (total_rent, floor, furnished, pets, balcony, parking, laundry, contact, is_temporary).
• listing-seeker → direction='demand': the SAME dimensions (price_sek= budget_max) + extras jsonb = the krav (requirements): num_occupants, household_type (single|couple|family|students), has_children, has_pets, employment_status, has_references, can_pay_deposit, duration_months, other[].

One symmetric shape → matching a seeker's krav against listings is a single join on the shared dimensions (§4). scrape-new-members leads carry most of this off the questionnaire wire (explicit) → they enter as direction='demand' extracts without an LLM call.

• _lib/llm.js — the ONE client. POST https://llm.hostbun.cc/v1/chat/completions (OpenAI-compatible, header X-Project: bluebut), model pinned per purpose in an env-overridable const (classify → claude-sonnet-5, extract → claude-haiku-4-5), response_format: json_schema so replies are typed, bounded retry + a backup-model fallback. Every LLM call in the repo goes through here.
• _lib/judge.js — classifyPost(post) (Pass 1) and extractListing(post) / extractSeeker(post) (Pass 2). Build the prompt from the post's wire fields, call llm.js, validate, return the typed struct. Reject phantoms.

3.1 Data model (the robust core — bluebut-db)

The model splits into three layers so it stays sustainable: observation (wire, immutable) never mixes with inference (LLM judgments, append-only + versioned) which never mixes with action (the outreach ledger). Person-level intent / outreach_status are DERIVED caches, never the source of truth. This mirrors the repo's existing fb_person (state) vs fb_person_signal (append-only log) split.

Lead identity is polymorphic. Two channels feed the funnel — a feed post (get-group-feed) and a member request (scrape-new-members, has a person but no post). Everything downstream keys on the agnostic pair (source_kind, source_ref) — the same shape as the job ledger's (target_type, target_id) — so both channels ride one pipeline.

-- ── OBSERVATION — fb_post / fb_person stay PURE WIRE MIRRORS. No judgment columns. ──
-- (unchanged; the entity graph already holds them)

-- ── INFERENCE — append-only, versioned. Re-judge = a new row; nothing is overwritten. ──
CREATE TABLE IF NOT EXISTS llm_prompt (          -- prompts are versioned DATA, not consts
  kind text, version int, text text NOT NULL, model text NOT NULL,
  created_at timestamptz DEFAULT now(), PRIMARY KEY (kind, version));

CREATE TABLE IF NOT EXISTS fb_judgment (         -- one row per (lead, model, prompt) — APPEND ONLY
  id bigserial PRIMARY KEY,
  source_kind text NOT NULL,                       -- 'feed_post' | 'member_request'
  source_ref  text NOT NULL,                        -- post_url | member_request_id (the polymorphic key)
  fb_person_id text,                               -- the poster/requester (denormalized for query)
  judge_kind  text NOT NULL DEFAULT 'classify',
  post_type   text,                                -- rental-listing | listing-seeker | none | unknown
  is_company  boolean,
  confidence  int,                                 -- 0..100
  model       text, prompt_version int,            -- FKs llm_prompt(kind,version) → reproducible + eval-able
  raw         jsonb,                               -- verbatim reply (audit + diff)
  created_at  timestamptz DEFAULT now());
CREATE INDEX IF NOT EXISTS fb_judgment_src ON fb_judgment (source_kind, source_ref, created_at DESC);

CREATE TABLE IF NOT EXISTS housing_extract (     -- ONE symmetric table, both directions
  source_kind text NOT NULL, source_ref text NOT NULL,
  direction   text NOT NULL,                        -- 'offer' (landlord) | 'demand' (seeker)
  fb_person_id text,
  kommun text, lan text, rooms numeric, area_m2 numeric,
  price_sek int,                                    -- rent (offer) | budget_max (demand)
  move_in date, property_type text,                -- apartment | room | house | student
  extras jsonb,                                    -- offer: features · demand: the krav (requirements)
  model text, prompt_version int, confidence int,
  extracted_at timestamptz DEFAULT now(),
  PRIMARY KEY (source_kind, source_ref));

-- ── ACTION — the ledger IS the outreach state (one row per ATTEMPT, multi-touch). ──
CREATE TABLE IF NOT EXISTS fb_outreach (
  id bigserial PRIMARY KEY,
  fb_person_id text NOT NULL,
  source_kind text, source_ref text,               -- the lead this outreach answers
  direction text,                                  -- offer | demand
  profile_id bigint,                               -- which message_profile rendered it
  message_text text,                               -- the exact text sent (audit + eval)
  matched_listings jsonb,                          -- SNAPSHOT of the Bofrid listings shown (for reply-rate eval)
  sender_account text,                             -- the account that sent it (resolved at send)
  job_id text,                                     -- FK to the execution log (job ledger) — like group_post
  status text NOT NULL DEFAULT 'draft',            -- draft | approved | sending | sent | failed | skipped
  skip_reason text,                                -- cooldown | no_runnable_sender | dedup | quiet_hours | blocked
  sent_at timestamptz, reply_at timestamptz,
  created_at timestamptz DEFAULT now(), updated_at timestamptz DEFAULT now());
CREATE INDEX IF NOT EXISTS fb_outreach_person ON fb_outreach (fb_person_id, created_at DESC);

-- fb_person: intent + outreach_status become DERIVED CACHES (refreshed from the logs
-- above), never the source of truth. fb_person_signal stays the intent TIMELINE.
--   intent seeker|landlord|unknown · intent_confidence explicit|inferred · intent_source ...
ALTER TABLE fb_person ADD COLUMN IF NOT EXISTS intent_source text;

-- ── CONFIG — operator-tunable data, no deploy. ──
CREATE TABLE IF NOT EXISTS outreach_config (      -- singleton (id=1)
  id int PRIMARY KEY DEFAULT 1,
  confidence_threshold int DEFAULT 60,
  quiet_start time DEFAULT '23:00', quiet_end time DEFAULT '07:00',
  account_min_interval_s int DEFAULT 600, send_jitter_min_s int DEFAULT 180, send_jitter_max_s int DEFAULT 300);
CREATE TABLE IF NOT EXISTS message_profile (      -- the §4 routing key → templates
  id bigserial PRIMARY KEY,
  direction text, is_company bool, has_match bool, -- routing key: (direction, is_company, has_match)
  version int DEFAULT 1,
  templates text[],                                -- variants; {poster_name}{area}{listings} placeholders
  auto_approve bool DEFAULT false, enabled bool DEFAULT true);
CREATE TABLE IF NOT EXISTS sender_pool (          -- tunable per-account weight; runnable gate is DERIVED at runtime
  account text PRIMARY KEY, weight int DEFAULT 100,-- 0=disabled · <100=probabilistic send-chance (reputation)
  blocked_until timestamptz);

Derived views (the DECIDE layer is a query, not a column):

-- current classification = the latest judgment per lead
CREATE OR REPLACE VIEW v_lead_class AS
SELECT DISTINCT ON (source_kind, source_ref) *
FROM fb_judgment ORDER BY source_kind, source_ref, created_at DESC;

-- seeker leads still needing a first touch = the outreach candidate query
CREATE OR REPLACE VIEW v_seeker_leads AS
SELECT c.*, e.kommun, e.price_sek, e.rooms, e.extras AS krav
FROM v_lead_class c
JOIN housing_extract e USING (source_kind, source_ref)
WHERE c.post_type = 'listing-seeker' AND e.direction = 'demand'
  AND NOT EXISTS (SELECT 1 FROM fb_outreach o          -- dedup from OUR ledger, not bofrid.dev
                  WHERE o.fb_person_id = c.fb_person_id AND o.status IN ('sent','approved','sending'));

classify-posts — a Job that batches leads with no current v_lead_class row: Pass 1 → fb_judgment (append), gate, and on keep Pass 2 → housing_extract + a fb_person_signal intent event. NOTIFY pgrst. Idempotent: same prompt version = no new row; a bumped version appends, so re-running never loses or double-writes. Runs on the recurring scheduler or on demand — every run a row in the job ledger → HyperDX by job_id (fb-bot's realtime trigger was invisible). judge_raw keeps the raw reply so a re-run after a prompt tweak is diffable.

4. Layer 3 — Decide (category → action matrix + krav match)

This is the heart of what you asked for: classification decides the ACTION, the message CONTENT, and whether it auto-sends or waits for review.

4.1 The branch matrix (category → action)

The routing key is (direction, is_company, has_match) (direction derived from post_type: seeker→demand, listing→offer) → look up the enabled message_profile. No matching profile → the lead is skipped (an fb_outreach row with status='skipped', skip_reason), never silently dropped. Same logic fb-bot's pickAutoMessageConfig uses, but as a table you edit, not code.

4.2 The krav → listing MATCH (the value-add)

For a listing-seeker, _lib/match-listings.js takes the extracted krav (kommun, budget_max, rooms_min, household) and finds the top ~3 real live Bofrid listings that fit. This is the ONE legitimate external read — the Bofrid product's own listing inventory (geo-bofrid / the Bofrid platform API), NOT fb-bot. It's read-only and it's the actual product we're recommending. Matches → the personalized seeker-match profile injects them; no match → the generic profile.

4.3 Candidate selection = a query (no daemon-owned queue)

The work-list is the v_seeker_leads view (§3.1): current classification ⋈ housing_extract demand, minus anyone with an open/sent fb_outreach row. Computed fresh, oldest-first (_lib/outreach-select.js) — the stalest-sweep pattern. Dedup reads our own fb_outreach ledger (backstopped by the fb_message log at send), never bofrid.dev.

5. Layer 4 — Act (compose → route sender → wire send → confirm)

A composed DM is a durable row in fb_outreach (the outreach LEDGER — persists the text, chosen sender, status, audit; the honest successor to fb-bot's fb_sent_messages, in bluebut-db). Status machine: draft → approved → sending → sent | failed | skipped. auto_approve profiles land in approved; others in draft for the operator. A send Job drains approved.

5.1 Message CONTENT varies by category

message_profile.templates[] is picked per the §4.1 routing key; one variant chosen at random (spam-hash variation, anti-automation rule #4); placeholders substituted: {poster_name}, {area} (=" i {kommun}" or empty), {listings} (the matched-listing block: • Title – 2 rok, 35 kvm, 8243 kr/mån\n<url>). Seeker-match profiles inject real listings; generic/landlord profiles are plain templates. Content is template-based with DB-driven listing injection — deterministic, auditable, cheap (no LLM at send). (An LLM-authored opener is possible later via llm.js, same gate.)

5.2 SENDER selection (which person sends)

_lib/sender-route.js picks the account per send, improving fb-bot's lane pool:

1. Eligible pool = sender_pool accounts that are cloakbox-runnable (the cdp_catalog run gate — the same one the UI now enforces), have a live session, and are members of the target group where relevant. A no-device account is never chosen (that's the bug you hit today).
2. Weight / reputation — weight 0 = disabled; <100 = probabilistic send-chance (a reputation slider); pick weighted-random among eligible. One account is never driven by two jobs at once (a per-account in-flight lock).
3. Cooldown / quiet hours — per-account floor 1 send / 10 min, global jitter 3–5 min between sends, no sends 23:00–07:00 Europe/Stockholm. A throttled send is released back to approved, never failed.
4. Block auto-disable — an auth/checkpoint/block verdict on the send sets that account's weight=0 + blocked_until (self-heal doctrine), so it drops out of the pool next tick without a human.

5.3 Send + confirm (never trust the send)

Send via messenger/send-message — LightSpeed (plain) or whatsmeow E2EE (type-15), through the anti-detect Chrome + keyvault session, NOT a DOM composer. Then re-read the thread: only mark fb_outreach.status='sent' once fb_message shows our echo (okWith/failWith). Persist sent the instant the browser confirms — before slow screenshot/HAR — so a crash can never double-send (fb-bot's recoverStuckMessages lesson). Write fb_message (the dedup source) + flip fb_person.outreach_status='contacted' + an fb_person_signal. Replies come back via messenger/sync-inbox → outreach_status='replied' (the §6 signal).

6. Layer 5 — Measure

Two cheap signals, no golden-set harness:

• Confidence gate — Pass 1 already drops low-confidence classifications (confidence < threshold → rejected); the threshold is tunable in outreach_config.
• Reply rate by prompt version — this is why judgments are append-only + carry prompt_version: join fb_outreach.reply_at back to the fb_judgment/llm_prompt that produced the lead and GROUP BY prompt_version. A DMed seeker who replies was classified right; a drop after a prompt change = a bad change, caught in the data. The matched_listings snapshot on the ledger also answers "did real listings lift replies vs the generic message". Operator re-tags in the UI store as free corrections.

7. New / changed files

adapters/_lib/llm.js             NEW  one llm.hostbun.cc client (2 pinned models, json_schema, backup fallback)
adapters/_lib/judge.js           NEW  classifyPost (Pass 1 → fb_judgment) + extract (Pass 2 → housing_extract)
adapters/_lib/match-listings.js  NEW  krav → top-N real Bofrid listings (reads the Bofrid product API, read-only)
adapters/_lib/outreach-select.js NEW  candidate query (queue-is-a-query)
adapters/_lib/sender-route.js    NEW  cloakbox-runnable + weighted + cooldown/quiet-hours + block-disable
adapters/facebook/classify-posts.js NEW  batch pending leads → Pass1→fb_judgment, Pass2→housing_extract (a Job)
messenger/jobs/outreach.js       NEW  drain approved fb_outreach → sender-route → send-message → re-read → log
adapters/facebook/SCHEMA.sql     EDIT fb_judgment/housing_extract/fb_outreach/llm_prompt/config tables + views (§3.1)
ui/ (Outreach surface)           NEW  classified leads + matched listings + message draft + approve/send + sender pool
adapters/_lib/fbbot-feed.js      DELETE  (§9)

Every _lib/* module ships a sibling *.test.mjs (house rule). classify-posts and outreach register their ops in _lib/taxonomy.js.

8. UI — the Outreach surface

A batch page / workspace tab (Ui.tsx + MUI, driven by runJob/startJob):

• Leads — classified seekers/landlords, post_type + is_company + confidence, the extracted listing/seeker card, and (for seekers) the matched Bofrid listings.
• Message draft — the composed DM per the profile; editable; approve/decline per lead or bulk. draft → approved here (or auto for auto-approve profiles).
• Sender pool — the sender_pool accounts with weight sliders + live cloakbox status; disabled/blocked ones greyed. This is where the operator tunes reputation.
• Send — one click drains approved via messenger/outreach; every send is a Job, live in the Jobs dock, HyperDX by job_id. Or a recurring job runs it unattended with the same gates. No approval-manager service — the operator/schedule is the trigger, the Job is the unit, the ledger is the trail.

9. Cutover — kill the fb-bot seam

Full cutover, in order, no lead dropped:

1. Ship layers 2–5 (judge, decide, act, measure) writing only bluebut-db.
2. Run in parallel for one week: bluebut classify+outreach AND the legacy fb-bot feed, both live. Compare lead volume + reply rate.
3. Flip the feed off — delete fbbot-feed.js, drop the BLUEBUT_FEED_FBBOT env, remove the feedFbbotScrapedPosts call from get-group-feed. Bluebut no longer writes bofrid.dev fb_scraped_posts.
4. Move dedup home — the two remaining bofrid.dev reads (fb_sent_messages / fb_broadcast_posts in ui/src/lib/util.tsx + ui/src/pages/Messenger.tsx) read fb_message / group_post (bluebut-db) instead. bofrid.dev drops out of the read path entirely.
5. Retire fb-bot — the manager, realtime classifier, and approval UI are decommissioned. bofrid.dev stays read-only ONLY as a historical archive during the transition, then is dropped from the data-planes table in CLAUDE.md.

End state: the funnel is one repo, one DB, one job ledger. No fb-bot, no bofrid.dev, no cross-DB hop.

10. Why this is strictly better than fb-bot

Same proven lifecycle (classify → extract → match → route → send), rebuilt robust: