> ## Documentation Index > Fetch the complete documentation index at: https://docs.highailabs.com/llms.txt > Use this file to discover all available pages before exploring further. # Gmail Sync Architecture > Current architecture for verified Gmail access, historical and incremental discovery, candidate review, idempotent imports, privacy, and disconnect. ## Scope This guide documents the current technical implementation. Product behavior and remaining limitations are canonical in [Order Activation](/help/user-flows/order-upload). ## System boundaries ```mermaid theme={null} flowchart LR A[Expo native wizard and Orders archive] <--> B[Convex sync state and import jobs] A <--> RC[RevenueCat one-time entitlement] B --> CL[Candidate child ledger] B --> OL[Per-message outcome ledger] B --> RD[User-owned staged review drafts] B --> C[Trigger.dev Gmail parent and receipt-child tasks] C --> D[Gmail API] C --> E[Hono receipt parser] C --> F[Supabase community domain intelligence] E --> G[Strain matching services] RD --> H[Convex durable order commit and outbox] H --> OA[Searchable Orders archive] H --> NB[One notebook per historical order] H --> I[Hono strain research batch] I --> J[Trigger.dev strain research pipeline] ``` | System | Responsibility | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Mobile app | Native OAuth launch, onboarding/shop setup, paged candidate selection, staged order editing, safe confirmation, Orders archive, and completion UI | | Clerk | Google external account, approved scope record, server-side OAuth token retrieval | | RevenueCat | One-time `historical_import` entitlement for product `highiq_historical_import`; discovery remains outside the purchase boundary | | Convex | Feature/access enforcement, connection generation, import jobs, child ledgers, scheduling claims, signed-event state transitions, orders, and durable effects | | Trigger.dev | Gmail search/history traversal, candidate classification, chunked email-body parsing children, retries, and best-effort cancellation | | Gmail API | Read-only message discovery and body retrieval | | Hono API | Receipt parsing, strain matching, and later unresolved-strain research admission | | Supabase | Community dispensary domains and the researched strain catalog | ## Connection verification The client requests `https://www.googleapis.com/auth/gmail.readonly` through a Clerk Google external-account connect or reauthorization flow. After OAuth, `initGmailSync` does not trust the client assertion: on configured remote deployments it queries Clerk's Backend API, requires a verified Google account, and verifies the exact read-only Gmail scope before writing connected state. Convex increments `connectionGeneration` on connect and disconnect. Jobs and callbacks must match the active generation, which prevents late work from a revoked connection from writing user data. ## Historical scan The historical wizard defaults to the largest configured archive range: up to the past ten years in the current configuration, with preset narrower ranges and a custom range. The lookback can be configured from one through 22 years. Discovery is capped by the server-owned `HISTORICAL_IMPORT_MAX_RECEIPTS`, defaulting to and never exceeding 2,000 candidates globally for the job. Targeted discovery requires at least one real saved shop with a confirmed receipt domain. The onboarding **Find Past Orders** action opens this same wizard; inline setup persists the shop/domain before the scan. Broad discovery is separately configurable and always an explicit user choice. `startGmailScan` validates the date range, normalizes domain inputs, and claims one durable scheduling attempt before contacting Trigger.dev. The claim owns: * an import job ID; * a scheduling fingerprint; * a stable Trigger idempotency key; * a scheduling generation; and * pending/scheduled/retryable state. If no explicit domain list is supplied for a broad scan, Convex derives personal domains from each saved shop's website, contact email, and `emailDomains`. Trigger.dev then combines: 1. personal shop domains; 2. community domains supplied by the client-side community-domain loader; and 3. generic cannabis receipt queries unless the scan is explicitly restricted to shop domains. The scan and classifier post signed, job-bound chunks of at most 100 records. Convex upserts them into `gmailImportCandidates`, allowing the same job to move from `scanning` to `classifying` to `ready_for_review` without placing a large archive array on `gmailImportJobs`. ## Feature flag and purchase boundary The mobile flag controls presentation only. Convex independently enforces the server flag, visibility, entitlement, and workload limits. | Setting | Default and boundary | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | `EXPO_PUBLIC_FF_HISTORICAL_IMPORT` | Enabled outside production unless explicitly `false`; production requires exactly `true` | | `HISTORICAL_IMPORT_ENABLED` | Enabled in staging/local unless `false`; production requires exactly `true` | | `HISTORICAL_IMPORT_VISIBLE` | Hides the surface when `false`, after server enablement | | `HISTORICAL_IMPORT_STAGING_BYPASS` | Bypasses purchase only when enabled in staging or local and not explicitly `false`; never bypasses production | | `HISTORICAL_IMPORT_MAX_RECEIPTS` | Defaults to 2,000 and is clamped from 1 through 2,000 | | `HISTORICAL_IMPORT_CHUNK_SIZE` | Defaults to 25 and is clamped from 1 through 100 | | `HISTORICAL_IMPORT_ALLOW_BROAD_SEARCH` | Enables the explicit broad-search option unless `false` | | `HISTORICAL_IMPORT_SAFE_CONFIRMATION_CONFIDENCE` | Server-owned safe-confirm threshold, default `0.9`, clamped from `0.5` through `1` | The matching `EXPO_PUBLIC_HISTORICAL_IMPORT_*` values control native labels and presentation limits only. In particular, `EXPO_PUBLIC_HISTORICAL_IMPORT_DEFAULT_LOOKBACK_YEARS` defaults to ten and is clamped from one through 22. Server values remain authoritative for access, receipt caps, chunking, broad search, and safe confirmation. Users may scan, classify, page through, and select candidates before paying. Immediately before `startGmailImport` parses selected bodies, production must verify the stored one-time entitlement. Staging/local bypass exercises the same parser, review, commit, research, notebook, and archive path; it skips only the purchase requirement. ## Incremental Quick Sync Quick Sync accepts normalized domains from selected saved shops. It rejects an empty domain set. The scheduling claim includes the current Gmail History cursor, selected domain fingerprint, stable idempotency key, generation, and a new or reusable import job. The Trigger.dev task: * pages Gmail History within explicit page/message/time bounds; * deduplicates message IDs; * filters to the selected shop domains; * uses the prior cursor when valid; * records whether the attempt used `history` or bounded `fallback`; and * does not advance the cursor when a bounded discovery attempt fails. Discovered candidates enter the same classification, selection, and batch import model as historical scans. Zero candidates complete the discovery job and may advance the accepted cursor without incrementing `totalImported`. ## Candidate review state `gmailImportCandidates` is the source-of-truth child ledger for review metadata: message ID, sender, sender domain, subject, date, snippet, match layer, classification fields, selection, and ordinal. Its `by_job` index backs paged review, while message and selected indexes support idempotent updates and batch selection. `gmailImportJobs` keeps only bounded preview/progress summaries. Candidate toggles are owner-scoped Convex mutations. Group and select-all operations use owner-scoped actions that apply bounded mutation chunks. Quick Sync results apply the selected ID set before opening import settings. Selection is email-level. Starting import fetches and parses only selected bodies. `gmailImportOutcomes` stores one durable, idempotent working/terminal outcome per message. Each successful parse becomes a separate `gmailOrderReviewDrafts` document, so candidate paging, partial retries, and editable review do not inflate one job document and no order exists before confirmation. ## Import and order commit `startGmailImport` claims a durable import scheduling attempt and sends the server-configured chunk size to one parent Trigger task. The parent processes idempotent receipt children in bounded chunks; the child queue concurrency is 10\. Its four-hour `maxDuration` is an active-compute guard; Trigger excludes time spent waiting for `triggerAndWait` children. Each selected email import: 1. fetches the full body with a fresh verified server-side Google token; 2. calls the Hono receipt parser; 3. posts a signed per-message result; 4. checks the user/message ID compound duplicate key; 5. stages valid parsed lines as `ready_for_review`; and 6. upserts duplicate, no-items, skipped, failed, or staged progress once in its per-message outcome. A signed `batch.complete` means parsing has stopped and moves the job into review. It does not mark a historical import complete while staged drafts remain. Completion is written only after every staged draft is committed or discarded. The user opens each staged receipt in the shared Review & Confirm screen. Name edits re-run matching, unresolved selected names require explicit Yes or No, and a user can save or skip the receipt. Historical drafts that have a valid shop/date and only authoritative, high-confidence strain matches can use `confirmSafeHistoricalOrders`; one mutation commits at most the next configured chunk, additionally capped at 25. Every other draft remains in individual review. Save calls the same durable `commitOrder` boundary as text with `sourceMessageId` idempotency, then `finalizeGmailOrderReview` marks the draft committed and advances counters. The shared commit preserves all selected lines, never adds historical purchases to active Stash, reserves research only for explicitly confirmed unknown strains, and creates or links the reviewed shop. Each committed historical order requests its own notebook. Finalization attaches the validated candidate sender domain, increments imported totals once, and exposes the order in the searchable Orders archive. ## Runtime state and idempotency ```mermaid theme={null} stateDiagram-v2 [*] --> SchedulingPending SchedulingPending --> Scheduled: Trigger accepts run SchedulingPending --> Retryable: Trigger call not confirmed Retryable --> SchedulingPending: Same logical retry Scheduled --> Scanning: Runtime observed Scanning --> Classifying: Candidates accepted Classifying --> ReadyForReview: Classifications accepted Scanning --> Failed: Signed failure Classifying --> Failed: Signed failure ReadyForReview --> ImportScheduling ImportScheduling --> Importing: Batch run accepted ImportScheduling --> ReadyForReview: Scheduling rollback Importing --> ReceiptReview: batch.complete and pending drafts Importing --> Completed: batch.complete and no pending drafts Importing --> Failed: Signed terminal failure ReceiptReview --> ReceiptReview: Safe confirm, edit and save, or discard ReceiptReview --> Completed: Every draft committed or discarded Failed --> Scanning: Resume stored scan attempt Failed --> Importing: Resume or retry failed messages only Scanning --> Cancelled: Authenticated cancel Classifying --> Cancelled: Authenticated cancel ReadyForReview --> Cancelled: Authenticated cancel Importing --> Cancelled: Authenticated cancel ReceiptReview --> Cancelled: Cancel remaining drafts ``` Each review draft independently transitions `pending` to `committed` after a matching order commit, or to `discarded` after an explicit skip. The separate `importRuntimeOutcome` records whether the parser batch completed or failed; the parent job stays `ready_for_review` until its draft ledger is terminal. Signed runtime transitions validate the job, owner, connection generation, idempotency key, scheduling generation, expected current state, and bounded payload. Duplicate or late events become no-ops. `resumeHistoricalImport` is authenticated and owner-scoped. It returns immediately when the job is already complete or ready for review; otherwise it reclaims a stored scan fingerprint or prepares only resumable import outcomes under the same durable idempotency/generation contract. The native error surface labels this action **Resume Import**. ## Counters and cursor rules * Historical discovery accepts at most the configured global cap, never more than 2,000 candidates for one job. * `totalCandidates` is discovery, not import. * `totalImported` increases only when a staged receipt is confirmed into a new order ID. * Duplicates, failures, skipped messages, and no-item results do not increment imported totals. * The pending Gmail History cursor becomes current only through an accepted discovery/import policy; failure paths preserve the prior cursor. * A retry of completed partial failure selects only failed message IDs. * One Gmail message ID can commit at most one order for a user, and each committed historical order requests one notebook through the durable outbox. ## Privacy lifecycle | Data | Boundary | | -------------------------- | ---------------------------------------------------------------------------------------------------- | | Gmail OAuth token | Fetched from Clerk inside server-side actions/tasks; never returned to mobile or persisted in Convex | | Raw email body | Trigger.dev task memory only | | Candidate metadata | Temporary, owner-scoped `gmailImportCandidates` children loaded through a paginated query | | Message outcome | Owner-scoped `gmailImportOutcomes` child used for idempotency, partial retry, and recovery | | Parsed order | Owner-scoped staged draft, then Convex order data only after confirmation | | Sender domain intelligence | Supabase shared table, no user ID on the shared row | Disconnect is a two-phase operation. Convex first marks the connection disconnecting and increments its generation, immediately closing every late write boundary. It then terminalizes active jobs in bounded pages, removes their scheduling keys, and schedules immediate candidate/review-ledger purges. The action tries to cancel each discovered Trigger run and revoke the delegated Google token. If provider revocation cannot be confirmed, the durable state is retryable rather than claiming success. Historical cancellation is a distinct authenticated action. Convex first marks the job `cancelled`, discards pending drafts, and establishes the authoritative local terminal boundary. It then attempts to cancel the known scan and import Trigger runs. The result includes `cancellationIncomplete` when provider-side cancellation cannot be fully confirmed; late callbacks still fail job/status or generation validation. ## Orders archive boundary Every confirmed receipt creates one ordinary user-owned order. Historical orders are not a parallel archive table: **Stash → Orders** pages the same order records, provides native shop-name search, and filters by status, source, review state, notebook state, date, and shop. An active-job banner routes historical work back to Historical Import and incremental work back to Quick Sync. Order detail remains the single edit/delete boundary. ## Staging validation lab Development and the exact shared staging environment expose an **Order Import Lab** under Settings → Developer. Production and unknown remote deployments fail closed at both the route and Convex function boundaries. The lab keeps three test concerns separate: 1. approved text fixtures containing real strain names open the production text parser and shared editable order review; 2. deterministic historical fixtures create bounded candidate, outcome, and review-draft ledgers, then enter this same Historical Import completion and commit flow by job ID; and 3. real Gmail buttons use the connected account, verified read-only scope, and actual domain-first or broad discovery tasks. Mock fixture creation never inserts orders. Known fixture names are resolved through the current strain service, and every order is created only by the ordinary idempotent commit boundary after confirmation. The lab can grant the current staging user temporary Pro access without StoreKit so text parsing and Quick Sync are repeatable, while Historical Import uses the existing exact- staging purchase bypass. Committed fixture orders remain normal user-owned orders and are managed from the Orders archive. `ORDER_IMPORT_LAB_ENABLED=false` disables the server capability. Temporary access defaults to four hours and mock batches to at most 100 receipts; local or exact-staging operators can lower those bounds with `ORDER_IMPORT_LAB_ACCESS_HOURS` and `ORDER_IMPORT_LAB_MAX_RECEIPTS`. An identity-less local harness must also set `ORDER_IMPORT_LAB_LOCAL_HARNESS=enabled`; exact localhost URLs do not need the extra marker. ## Implementation map | Area | Files | | -------------------------------------- | --------------------------------------------------------------------------------------- | | Wizard controller and reducer | `apps/mobile/src/_screens/gmail/hooks/useGmailWizardController.ts`, `useGmailWizard.ts` | | Mobile state hooks | `apps/mobile/src/_hooks/convex/useGmailSync.ts`, `useGmailImportJob.ts` | | Convex connection, jobs, and events | `apps/mobile/convex/gmailSync.ts`, `http.ts`, `schema.ts` | | Server rollout and access config | `apps/mobile/convex/utils/historicalImportConfig.ts`, `subscriptions.ts` | | Gmail tasks | `packages/trigger/src/tasks/gmail/` | | Staged review and durable order writer | `apps/mobile/convex/gmailSync.ts`, `apps/mobile/convex/orderPostSave.ts` | | Research admission | `apps/api/src/routes/research/trigger-research.ts` | | Native Orders archive | `apps/mobile/src/_screens/orders-library/`, `apps/mobile/convex/orders.ts` | Canonical implemented product flow. User-facing setup and import guidance.