Gmail Sync Architecture

​

Scope

This guide documents the current technical implementation. Product behavior and remaining limitations are canonical in Order Activation.

​

System boundaries

​

Connection verification

The client requests https://www.googleapis.com/auth/gmail.readonly through a Clerk Google external-account connect or reauthorization flow. After OAuth, initGmailSync does not trust the client assertion: on configured remote deployments it queries Clerk’s Backend API, requires a verified Google account, and verifies the exact read-only Gmail scope before writing connected state. Convex increments connectionGeneration on connect and disconnect. Jobs and callbacks must match the active generation, which prevents late work from a revoked connection from writing user data.

​

Historical scan

The historical wizard defaults to the largest configured archive range: up to the past ten years in the current configuration, with preset narrower ranges and a custom range. The lookback can be configured from one through 22 years. Discovery is capped by the server-owned HISTORICAL_IMPORT_MAX_RECEIPTS, defaulting to and never exceeding 2,000 candidates globally for the job. Targeted discovery requires at least one real saved shop with a confirmed receipt domain. The onboarding Find Past Orders action opens this same wizard; inline setup persists the shop/domain before the scan. Broad discovery is separately configurable and always an explicit user choice. startGmailScan validates the date range, normalizes domain inputs, and claims one durable scheduling attempt before contacting Trigger.dev. The claim owns:

• an import job ID;
• a scheduling fingerprint;
• a stable Trigger idempotency key;
• a scheduling generation; and
• pending/scheduled/retryable state.

If no explicit domain list is supplied for a broad scan, Convex derives personal domains from each saved shop’s website, contact email, and emailDomains. Trigger.dev then combines:

1. personal shop domains;
2. community domains supplied by the client-side community-domain loader; and
3. generic cannabis receipt queries unless the scan is explicitly restricted to shop domains.

The scan and classifier post signed, job-bound chunks of at most 100 records. Convex upserts them into gmailImportCandidates, allowing the same job to move from scanning to classifying to ready_for_review without placing a large archive array on gmailImportJobs.

​

Feature flag and purchase boundary

The mobile flag controls presentation only. Convex independently enforces the server flag, visibility, entitlement, and workload limits. The matching EXPO_PUBLIC_HISTORICAL_IMPORT_* values control native labels and presentation limits only. In particular, EXPO_PUBLIC_HISTORICAL_IMPORT_DEFAULT_LOOKBACK_YEARS defaults to ten and is clamped from one through 22. Server values remain authoritative for access, receipt caps, chunking, broad search, and safe confirmation. Users may scan, classify, page through, and select candidates before paying. Immediately before startGmailImport parses selected bodies, production must verify the stored one-time entitlement. Staging/local bypass exercises the same parser, review, commit, research, notebook, and archive path; it skips only the purchase requirement.

​

Incremental Quick Sync

Quick Sync accepts normalized domains from selected saved shops. It rejects an empty domain set. The scheduling claim includes the current Gmail History cursor, selected domain fingerprint, stable idempotency key, generation, and a new or reusable import job. The Trigger.dev task:

• pages Gmail History within explicit page/message/time bounds;
• deduplicates message IDs;
• filters to the selected shop domains;
• uses the prior cursor when valid;
• records whether the attempt used history or bounded fallback; and
• does not advance the cursor when a bounded discovery attempt fails.

Discovered candidates enter the same classification, selection, and batch import model as historical scans. Zero candidates complete the discovery job and may advance the accepted cursor without incrementing totalImported.

​

Candidate review state

gmailImportCandidates is the source-of-truth child ledger for review metadata: message ID, sender, sender domain, subject, date, snippet, match layer, classification fields, selection, and ordinal. Its by_job index backs paged review, while message and selected indexes support idempotent updates and batch selection. gmailImportJobs keeps only bounded preview/progress summaries. Candidate toggles are owner-scoped Convex mutations. Group and select-all operations use owner-scoped actions that apply bounded mutation chunks. Quick Sync results apply the selected ID set before opening import settings. Selection is email-level. Starting import fetches and parses only selected bodies. gmailImportOutcomes stores one durable, idempotent working/terminal outcome per message. Each successful parse becomes a separate gmailOrderReviewDrafts document, so candidate paging, partial retries, and editable review do not inflate one job document and no order exists before confirmation.

​

Import and order commit

startGmailImport claims a durable import scheduling attempt and sends the server-configured chunk size to one parent Trigger task. The parent processes idempotent receipt children in bounded chunks; the child queue concurrency is 10. Its four-hour maxDuration is an active-compute guard; Trigger excludes time spent waiting for triggerAndWait children. Each selected email import:

1. fetches the full body with a fresh verified server-side Google token;
2. calls the Hono receipt parser;
3. posts a signed per-message result;
4. checks the user/message ID compound duplicate key;
5. stages valid parsed lines as ready_for_review; and
6. upserts duplicate, no-items, skipped, failed, or staged progress once in its per-message outcome.

A signed batch.complete means parsing has stopped and moves the job into review. It does not mark a historical import complete while staged drafts remain. Completion is written only after every staged draft is committed or discarded. The user opens each staged receipt in the shared Review & Confirm screen. Name edits re-run matching, unresolved selected names require explicit Yes or No, and a user can save or skip the receipt. Historical drafts that have a valid shop/date and only authoritative, high-confidence strain matches can use confirmSafeHistoricalOrders; one mutation commits at most the next configured chunk, additionally capped at 25. Every other draft remains in individual review. Save calls the same durable commitOrder boundary as text with sourceMessageId idempotency, then finalizeGmailOrderReview marks the draft committed and advances counters. The shared commit preserves all selected lines, never adds historical purchases to active Stash, reserves research only for explicitly confirmed unknown strains, and creates or links the reviewed shop. Each committed historical order requests its own notebook. Finalization attaches the validated candidate sender domain, increments imported totals once, and exposes the order in the searchable Orders archive.

​

Runtime state and idempotency

Each review draft independently transitions pending to committed after a matching order commit, or to discarded after an explicit skip. The separate importRuntimeOutcome records whether the parser batch completed or failed; the parent job stays ready_for_review until its draft ledger is terminal. Signed runtime transitions validate the job, owner, connection generation, idempotency key, scheduling generation, expected current state, and bounded payload. Duplicate or late events become no-ops. resumeHistoricalImport is authenticated and owner-scoped. It returns immediately when the job is already complete or ready for review; otherwise it reclaims a stored scan fingerprint or prepares only resumable import outcomes under the same durable idempotency/generation contract. The native error surface labels this action Resume Import.

​

Counters and cursor rules

• Historical discovery accepts at most the configured global cap, never more than 2,000 candidates for one job.
• totalCandidates is discovery, not import.
• totalImported increases only when a staged receipt is confirmed into a new order ID.
• Duplicates, failures, skipped messages, and no-item results do not increment imported totals.
• The pending Gmail History cursor becomes current only through an accepted discovery/import policy; failure paths preserve the prior cursor.
• A retry of completed partial failure selects only failed message IDs.
• One Gmail message ID can commit at most one order for a user, and each committed historical order requests one notebook through the durable outbox.

​

Privacy lifecycle

Disconnect is a two-phase operation. Convex first marks the connection disconnecting and increments its generation, immediately closing every late write boundary. It then terminalizes active jobs in bounded pages, removes their scheduling keys, and schedules immediate candidate/review-ledger purges. The action tries to cancel each discovered Trigger run and revoke the delegated Google token. If provider revocation cannot be confirmed, the durable state is retryable rather than claiming success. Historical cancellation is a distinct authenticated action. Convex first marks the job cancelled, discards pending drafts, and establishes the authoritative local terminal boundary. It then attempts to cancel the known scan and import Trigger runs. The result includes cancellationIncomplete when provider-side cancellation cannot be fully confirmed; late callbacks still fail job/status or generation validation.

​

Orders archive boundary

Every confirmed receipt creates one ordinary user-owned order. Historical orders are not a parallel archive table: Stash → Orders pages the same order records, provides native shop-name search, and filters by status, source, review state, notebook state, date, and shop. An active-job banner routes historical work back to Historical Import and incremental work back to Quick Sync. Order detail remains the single edit/delete boundary.

​

Staging validation lab

Development and the exact shared staging environment expose an Order Import Lab under Settings → Developer. Production and unknown remote deployments fail closed at both the route and Convex function boundaries. The lab keeps three test concerns separate:

1. approved text fixtures containing real strain names open the production text parser and shared editable order review;
2. deterministic historical fixtures create bounded candidate, outcome, and review-draft ledgers, then enter this same Historical Import completion and commit flow by job ID; and
3. real Gmail buttons use the connected account, verified read-only scope, and actual domain-first or broad discovery tasks.

Mock fixture creation never inserts orders. Known fixture names are resolved through the current strain service, and every order is created only by the ordinary idempotent commit boundary after confirmation. The lab can grant the current staging user temporary Pro access without StoreKit so text parsing and Quick Sync are repeatable, while Historical Import uses the existing exact- staging purchase bypass. Committed fixture orders remain normal user-owned orders and are managed from the Orders archive. ORDER_IMPORT_LAB_ENABLED=false disables the server capability. Temporary access defaults to four hours and mock batches to at most 100 receipts; local or exact-staging operators can lower those bounds with ORDER_IMPORT_LAB_ACCESS_HOURS and ORDER_IMPORT_LAB_MAX_RECEIPTS. An identity-less local harness must also set ORDER_IMPORT_LAB_LOCAL_HARNESS=enabled; exact localhost URLs do not need the extra marker.

​

Implementation map