From Manual OAuth Onboarding to Event-Driven Sync
Evolving a manual OAuth onboarding process into a privacy-safe, serverless, event-driven data sync architecture.
Idempotent upsert pattern
INSERT INTO ext_resource_a (user_email, resource_id, status, updated_at)
VALUES ($1, $2, $3, $4)
ON CONFLICT (user_email, resource_id)
DO UPDATE SET
status = EXCLUDED.status,
updated_at = EXCLUDED.updated_at;
Safe retries for at-least-once event delivery from DynamoDB Streams.
Original publication
This case study was originally published on AWS Builder Center: From Manual OAuth Onboarding to Event-Driven Sync: A Privacy-Safe Serverless Case Study
TL;DR
A manual, support-heavy OAuth onboarding flow was redesigned into a serverless architecture with clear trust boundaries:
- One-time, TTL-bound OAuth state validation.
- Token material isolated in encrypted secret storage.
- Metadata and discovery records managed separately.
- Event-driven projection from DynamoDB into Postgres.
- Nightly reconciliation to ensure long-term correctness.
Problem
The original onboarding path relied on manual hand-offs, account-by-account steps, and brittle mappings. It worked for a few customers but created increasing support load and security concerns at scale.
Constraints
- No raw OAuth tokens in logs or primary metadata tables.
- CSRF-safe OAuth flow with replay protection.
- Low operational overhead and predictable cost.
- Near real-time relational sync for downstream reporting.
- Architecture patterns that can be shared publicly without exposing sensitive implementation details.
Architecture evolution
v1: Serverless OAuth discovery
The first version focused on secure onboarding and automatic resource discovery using API Gateway, Lambda, DynamoDB, and encrypted parameter storage.
Key patterns:
- Single-use OAuth state nonce with explicit consumption and TTL.
- Split data boundary: secrets in secure parameter storage, non-sensitive records in DynamoDB.
- Discovery pipeline that normalizes provider resources after OAuth callback.
v2: Event-driven relational projection
As analytics consumers required SQL workflows, the architecture expanded with DynamoDB Streams and a stream processor Lambda to project updates into Postgres.
Key patterns:
- Idempotent SQL upserts for at-least-once stream delivery.
- Record-level routing for insert, modify, and remove events.
- Soft-delete or inactive-state semantics to preserve historical truth.
- Retry and failure handling (batch tuning, retry limits, failure destination).
Single-use OAuth state validation
record = get_item("OAuthStates", {"state": state})
if not record:
return unauthorized("invalid or reused state")
delete_item("OAuthStates", {"state": state})
if record["ttl"] < now_epoch():
return unauthorized("state expired")The state value is consumed once and checked against TTL to protect against replay.
Security and reliability decisions
- Enforced strict token/metadata separation to reduce blast radius.
- Applied least-privilege IAM by table, path, and function responsibility.
- Implemented logging hygiene to prevent credential leakage.
- Added reconciliation jobs as a correctness safety net for rare drift.
Results
- Onboarding moved from support-driven to self-service.
- Security posture improved through explicit trust boundaries.
- Relational consumers received low-latency synchronized updates.
- Operational model stayed lightweight and cost-aware.
Read more
This page is a concise adaptation for portfolio use. For full diagrams, detailed pseudo-code, and the complete rollout checklist, read the original publication: