📄 raw markdown for agents/LLMs: /HOW-E2EE-WORKS.md · all docs

How encrypted Messenger works — CAT, PIN, devices, and how bluebut reads/sends it

⭐ TARGET ARCHITECTURE (2026-07-09): whatsmeow is the ONE path — LightSpeed is being DELETED. bluebut reads + sends Messenger DMs only through whatsmeow (the bluebut-e2ee daemon = a registered encrypted-messaging device per account). The old LightSpeed subsystem (browser sync-inbox, the persistent LS socket, ls-keeper, etc.) is being removed. The daemon decrypts every inbound message and the bridge builds the inbox from that stream (upserts the thread + contact; names come from the fb_person graph). Accepted trade: no thread-list of idle chats, no history, no folders, no plain (non-encrypted) threads. This page explains the mechanics (they don't change) — just note that where it says "LightSpeed does X", X is going away and whatsmeow + the bridge cover the parts we keep.

A from-scratch explainer. If you only remember one thing: to read or send an encrypted Messenger message, you have to BE a device on the account. Facebook's servers physically cannot hand you the plaintext — that's the whole point of end-to-end encryption. So bluebut registers its own device and does the crypto itself, exactly the way the maintained bridge (mautrix-meta) does.

TL;DR

• Messenger has two kinds of thread: plain (thread_type=1, server can read them) and E2EE (thread_type=15, only the participating devices can read them). Most modern 1:1 DMs are E2EE.
• Plain messages ride the LightSpeed wire — a normal socket, no crypto. bluebut reads/sends them directly.
• E2EE messages use the Signal protocol (Meta calls it Labyrinth). Each device holds its own crypto keys. To participate, bluebut runs a whatsmeow client (a Go Signal/Meta client) that registers as a device on the account.
• The CAT (Cryptographic Auth Token) is the connection ticket: it authenticates the E2EE websocket. It lasts ~24h and can only be minted from a real logged-in browser.
• The PIN / secure storage is a separate thing: it protects the encrypted history backup. It is NOT needed to send or receive live messages — only to restore old history.
• So: device (register once) + CAT (refresh ~daily) = we can read and send live E2EE messages, no PIN. Proven live in bluebut.

The picture

Read it as: Facebook relays messages → the daemon's two sockets receive them → decrypted/plain rows land in fb_message. The whatsmeow socket (green) needs a CAT minted by the real browser (amber, once/~24h); the LightSpeed socket is browser-free. The keepers hold both up. The PIN hangs off to the side — history backup only, never on the live path.

1. Two message worlds

Messenger thread
├── thread_type = 1   → PLAIN        server can read it     → LightSpeed wire
└── thread_type = 15  → E2EE         only devices can read  → Signal protocol (whatsmeow)

• Plain (type-1): the classic Messenger. The server sees the plaintext. bluebut reads it off the LightSpeed socket and sends via a SendMessageTask — no crypto involved. (This is the ls-session.go / ls-send path.)
• E2EE (type-15): the message is encrypted on the sender's device and only decryptable on the recipient's devices. The server relays ciphertext it can't read. Facebook has been defaulting new 1:1 chats to E2EE since ~2023–2024, so most DMs you care about are type-15.

Why this split matters: a plain message is just data on a wire — read it. An E2EE message is a sealed envelope addressed to specific devices — you can only open it if you are one of those devices.

A decrypted E2EE message is self-contained. whatsmeow does not hand you "just the body" — the decrypted payload carries its own thread_key, sender_id, timestamp_ms, message_id, and text (see handler.go DecryptedMessage). What whatsmeow does not give you is the inbox around it: the list of threads, folders (inbox/requests/spam), unread counts, contact display names, and participant rosters — plus all your plain (type-1) traffic. That's LightSpeed's job. So the two are complementary: whatsmeow = complete E2EE messages (raw IDs); LightSpeed = the inbox + names + folders + plain messages. Without LightSpeed you'd have decrypted E2EE messages as numeric IDs floating with no threads, no names, and no plain chats.

2. Why E2EE forces us to BE a device

End-to-end encryption means the plaintext exists only on the endpoints (the devices), never on Facebook's servers. There is no API that returns the decrypted body, because Facebook literally does not have it.

So there are only two ways to read an E2EE message:

1. Scrape the rendered browser after it has decrypted the message on-screen (brittle, localized, and needs the PIN for history). ❌ We don't do this.
2. Become a device: register our own E2EE device on the account, receive the ciphertext addressed to it, and decrypt it ourselves with the Signal keys. ✅ This is what bluebut does.

That "own device" is the bluebut-e2ee daemon — a Go service running whatsmeow (the same library that talks to WhatsApp; it also speaks Meta's E2EE). One registered device per account.

Do we need LightSpeed if whatsmeow already gives full messages?

A fair question — a decrypted E2EE message is self-contained (thread_key + sender + ts + text). So what does LightSpeed add? Everything that isn't the message content. Precise split:

The honest answer:

• Just want to receive E2EE message content (a bot that fires DMs and captures replies as raw IDs)? whatsmeow alone is enough.
• Want a usable Messenger — an inbox with threads, names, folders, unread, and your plain (type-1) chats? You need LightSpeed. whatsmeow is realtime-only and delivers new E2EE messages as numeric IDs floating in space — no inbox, no names, no plain traffic, no history.

bluebut is the second case (an operator inbox handling both plain + E2EE), so it runs both: LightSpeed = the inbox + address book + plain messages + history; whatsmeow = the decryptor for the encrypted bodies.

   Other person's phone                         bluebut-e2ee daemon
   ┌───────────────┐   ciphertext (E2EE)   ┌─────────────────────────┐
   │ encrypts for  │ ───────────────────▶  │ whatsmeow device        │
   │ each of the   │      via FB relay     │ = a registered device   │
   │ account's     │                       │ decrypts with its own   │
   │ devices       │                       │ Signal session keys     │
   └───────────────┘                       └────────────┬────────────┘
                                                         │ plaintext
                                                         ▼
                                              bridge → fb_message (is_e2ee)

3. The players

Two things are commonly confused — keep them apart:

• CAT = "is this account allowed to open the E2EE connection right now?" (transport auth, short-lived)
• PIN = "can this device unlock the encrypted history backup?" (history restore, orthogonal)

They have nothing to do with each other. You need the CAT to connect; you never need the PIN to send/receive live.

4. The CAT — the connection ticket (deep dive)

What it is: the encrypted_serialized_cat — a token Facebook mints for a logged-in Messenger web session. When you open facebook.com/messages/, the page embeds it in MessengerWebInitData. whatsmeow sends it as the bearer credential (ClientPayload.FbCat) when it opens the E2EE Noise websocket. It proves "this account is authenticated and this device is authorized to connect."

Why it expires (~24h): it's derived from the web session and is session-scoped. Facebook rotates it. An expired CAT = the websocket won't open (or drops).

Why we can't refresh it headlessly: minting a CAT requires a real logged-in browser page (it's baked into MessengerWebInitData). Facebook's anti-bot rejects a scripted/headless login ("xs cookie was deleted"), so you can't just curl for a fresh CAT. In the daemon, RefreshCAT is a deliberate hard-error stub:

// client.go
e2ee.RefreshCAT = func(ctx context.Context) error {
    // The CAT expires (~24h). We can't refresh it headlessly (that needs the web session).
    return fmt.Errorf("CAT refresh requires a fresh box-side probe (reconnect the account)")
}

How bluebut mints one: cat-connect.js drives the account's real cloakbox browser (cdp_drive → go to /messages/ → read encrypted_serialized_cat + appId off the page), then POSTs the daemon /connect. This is the one and only browser touch in the whole E2EE stack, and it's once per ~24h per account, not per message.

⚠️ The CAT mint MUST drive the public cloakbox gateway (https://cloakbox.hostbun.cc), not a pbox loopback — see cloak-client.js DEFAULT_DRIVE_BASE.

5. The device — register ONCE (ICDC)

Before it can connect, our whatsmeow client must be a registered E2EE device on the account. This is the ICDC flow (Identity Consistency Data Cache):

POST reg-e2ee.facebook.com/v2/fb_icdc_fetch      ← get the account's device list
POST reg-e2ee.facebook.com/v2/fb_register_v2     ← add OUR device, signed
   → returns WADeviceID  (e.g. 1159930757:960@msgr)

• This is authenticated by the CAT (not the web session).
• It happens exactly once per account. It fires a "new device logged in" alert to the account owner — expected, one-time.
• The resulting wa_device_id is stored (WA_DB + keyvault bluebut/e2ee-device/<slug>) and reused on every future connect — so reconnecting never re-registers and never re-alerts ("register-once").

Think of it like linking a WhatsApp companion device: you scan the QR once; after that the device stays linked and just reconnects.

6. Where decryption actually happens — the Signal sessions

Once registered + connected, the daemon holds Signal double-ratchet sessions (in WA_DB), one per conversation partner. This is the actual crypto:

• Sending: our device does an X3DH key agreement with the recipient's device(s) using their published prekeys, derives a shared secret, and encrypts. Facebook relays the ciphertext.
• Receiving: the sender encrypted a copy for our device's key; our whatsmeow decrypts it with the ratchet and advances the chain.
• Multi-device: an account can have several devices (phone, web, our daemon). Each message is encrypted separately for each device. Our device gets its own copy — that's why we can read it without anyone else's keys.

The CAT is transport auth; the Signal session keys are the message crypto. The CAT gets you connected; the session keys (already in WA_DB) do the decryption. That's why a reconnect (fresh CAT) resumes reading immediately — the session keys never left.

7. The PIN / secure storage — history, NOT live

When you enable E2EE, Facebook also creates "secure storage": an encrypted backup of your message history (and some key material), protected by a PIN (6-digit) or device-based key. A new device restores that history by PIN-unlock.

Crucially — and this is the part people trip on:

• The PIN is only for HISTORY. It unlocks the encrypted backup of messages sent before your device existed.
• The PIN is NOT part of live send/receive. New messages flow through the Signal sessions above, which don't touch secure storage.

So the dividing line for "can we read a message?" is not the PIN — it's when our device registered:

message sent AFTER our device registered   → we have a session for it → WE CAN READ IT (no PIN)
message sent BEFORE our device registered   → history → needs secure-storage restore (PIN)

The PIN/restore path in the code is handleMessageWithProtobufsRestoreV2 + the browser-side unlock-e2ee.js — noted as orthogonal and not used by the daemon's live path. bluebut currently does live only, which is all that's needed for send/receive.

8. The full lifecycle

  ┌─ ONCE per account ──────────────────────────────────────────────┐
  │  1. mint CAT (drive the real browser, read encrypted_serialized_cat)
  │  2. ICDC register  → wa_device_id   (fires the one new-device alert)
  │  3. store wa_device_id (WA_DB + keyvault)   ← register-once
  └─────────────────────────────────────────────────────────────────┘
                              │
  ┌─ every connect (after restart / CAT expiry) ─────────────────────┐
  │  4. mint a fresh CAT (browser)                                    │
  │  5. daemon /connect {fbid, app_id, cat, proxy, wa_device_id}      │
  │     → whatsmeow opens the Noise websocket, REUSES the device      │
  │     → resumes the Signal sessions from WA_DB                      │
  └─────────────────────────────────────────────────────────────────┘
                              │
  ┌─ while connected (the standing socket) ──────────────────────────┐
  │  RECEIVE: ciphertext arrives → decrypt with session → fb_message  │
  │           (is_e2ee=true, decrypted_by='daemon')                   │
  │  SEND:    encrypt with session → push over the socket → verified  │
  └─────────────────────────────────────────────────────────────────┘

Everything after step 2 that isn't a fresh CAT is browser-free. The device + Signal keys persist; only the CAT needs the periodic browser mint.

9. What has to be running for realtime send + receive

Per account, two daemon-held sockets (both live in RAM in the bluebut-e2ee daemon, so a daemon restart drops them and they must be re-established):

And the daemon itself must be up. It all runs in PROD (Coolify on hostbun); local dev just talks to the public daemon (E2EE_DAEMON_URL=https://bluebut-e2ee.blpk.cc) — you can't hold these sockets on a laptop.

10. Scenarios (the questions people actually ask)

"Can we read an E2EE message without the PIN?" ✅ Yes, if it was sent after our device registered. Our device has the Signal session, so the daemon decrypts it directly — no PIN. (Proven: philip/pojtie have inbound E2EE messages decrypted_by='daemon', e2ee_no_text=0, zero PIN entered.) The only thing the PIN would add is back-history from before the device existed.

"What happens if we SEND without the PIN?" ✅ Sends fine — our device establishes a fresh Signal session with the recipient and encrypts. The PIN is irrelevant to sending live messages.

"What about on refresh / reconnect?" The device stays registered and its Signal sessions persist in WA_DB. Reconnecting (with a valid CAT) resumes live send/receive with no PIN and no re-registration.

"What about after a daemon restart / deploy?" The in-RAM sockets die → the keepers reconnect. LightSpeed comes back browser-free in ≤5 min; E2EE needs a CAT mint (currently a browser touch — see the optimization below).

"Why did I get a 'new device logged in' alert once?" That's ICDC registration (step 2). It fires once per account, then never again (register-once).

11. How data is saved (the data model)

The flow per received message: whatsmeow decrypts → the daemon emits a DecryptedMessage → the bridge handleE2eeIngest upserts several tables (all in bluebut-db). Every message becomes a row — no type is dropped.

All writes are merge-upserts (Prefer: resolution=merge-duplicates), so re-ingesting the same message is idempotent — safe with the realtime socket's own echo of our sent copies.

⚠️ What is NOT saved yet (honest gaps)

The daemon currently captures the message existence + type + caption + attachment metadata, but not everything whatsmeow could give:

• Media BYTES — we record that an image/video/file exists (type/mime), but do not yet DownloadFB + decrypt + store the actual bytes to MinIO. playable_url is null. (This is the planned next step.)
• Reactions (👍❤️) — not captured. fb_message_reaction stays empty for encrypted threads (the handler doesn't yet process reaction events).
• Full media metadata — dimensions/duration/mime are only partially extracted from the transport ancillary; most media rows carry just attachment_type.
• Reply context, edits, unsends, read receipts — not captured.

These are the backlog. Everything above the line (message + type + caption + basic attachment + thread + contact + name) is live and verified.

12. How bluebut implements it (file map)

13. Gotchas & guardrails

• Plain task into an E2EE thread is dropped. FB accepts a plain SendMessageTask into a type-15 thread and then silently drops it — so send-message must route type-15 to /e2ee-send (whatsmeow), never the plain wire. It auto-detects from fb_thread.
• Register-once is sacred. Re-registering fires another new-device alert and can confuse the account's device list. Always reuse the stored wa_device_id.
• Proxy egress. The E2EE Noise socket must egress the account's own proxy (same IP as the web session) — anti-detect.
• The daemon holds state in RAM. Every daemon deploy drops all sockets. Watch paths already stop unrelated pushes from redeploying it; keep daemon-code changes rare.

14. The open optimization — cache the CAT

Right now the reconnect sweep mints a fresh CAT on every reconnect (a cloakbox browser touch each time). But a CAT is valid ~24h and is reusable across reconnects. So the improvement:

Persist the CAT (with its mint timestamp). On reconnect, try the cached CAT first — if it's still valid (<~20h), connect browser-free; only drive the cloakbox to mint a fresh one when the cached CAT is near expiry.

That would make daemon-restart recovery browser-free and instant, and cut cloakbox CAT mints from "every reconnect" to "once per ~20h per account" — the fewest possible browser touches while staying always-up.