Raw Payload Storage

Overview

Raw payloads storage lets you capture the exact JSON payloads received from wearable providers before any processing. This is useful when you need to:

Debug data issues - inspect exactly what a provider sent when something doesn’t look right
Test ingestion changes - replay real payloads against updated parsing logic to make sure nothing breaks
Reproduce bugs - grab the original payload that caused a problem and use it in local development

The feature is disabled by default and adds zero overhead when turned off.

Configuration

Add these environment variables to your .env file:

# Storage backend: "disabled" (default), "log", or "s3"
RAW_PAYLOAD_STORAGE=disabled

# Maximum payload size in bytes (default: 10 MB)
RAW_PAYLOAD_MAX_SIZE_BYTES=10485760

# S3-specific settings (only needed when using "s3" backend)
RAW_PAYLOAD_S3_BUCKET=my-raw-payloads        # Falls back to AWS_BUCKET_NAME if not set
RAW_PAYLOAD_S3_PREFIX=raw-payloads            # S3 key prefix (default: "raw-payloads")
# AWS_ENDPOINT_URL=                         # Shared endpoint for S3-compatible storage
# RAW_PAYLOAD_S3_ENDPOINT_URL=              # Optional raw-payload-only endpoint override

Raw payloads can contain sensitive health data. Make sure your storage destination meets your organization’s data protection requirements.

Storage Backends

Disabled (default)

No payloads are stored. The store_raw_payload() call returns immediately with no overhead.

RAW_PAYLOAD_STORAGE=disabled

Log

Writes each payload as a structured JSON line to stdout. Useful for development, or when your infrastructure already captures application logs (e.g., CloudWatch, Datadog).

RAW_PAYLOAD_STORAGE=log

Example output:

{
  "level": "debug",
  "message": "raw_payload",
  "source": "webhook",
  "provider": "garmin",
  "size_bytes": 4521,
  "trace_id": "a1b2c3d4",
  "payload": "{...}"
}

S3

Uploads each payload as a JSON file to an S3 bucket (or any S3-compatible object store like MinIO or Railway Object Storage).

RAW_PAYLOAD_STORAGE=s3
RAW_PAYLOAD_S3_BUCKET=my-raw-payloads

AWS credentials are reused from the existing application config (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION). See the AWS Setup guide for details.

S3 key format

Files are organized by provider, source, date, and user:

{prefix}/{provider}/{source}/{YYYY-MM-DD}/{user_id}/{file_id}.json

Example:

raw-payloads/garmin/webhook/2025-03-12/_unknown/a1b2c3d4e5f6.json
raw-payloads/apple/sdk/2025-03-12/user-123/f6e5d4c3b2a1.json

_unknown is used when a user ID is not available (e.g., Garmin webhooks)
file_id is a random 12-character hex string

S3 object metadata

Each object includes metadata for filtering and querying:

Key	Description
`source`	Origin type (`sdk`, `webhook`)
`provider`	Provider name (`garmin`, `apple`, `samsung`, etc.)
`size_bytes`	Payload size in bytes
`timestamp`	ISO 8601 UTC timestamp
`user_id`	User identifier (if available)
`trace_id`	Request trace/batch ID (if available)

Supported Ingestion Points

Raw payloads storage is currently enabled at two ingestion points:

Endpoint	Source	Provider	Trace ID
`POST /api/v1/garmin/push`	`webhook`	`garmin`	8-char request trace ID
`POST /api/v1/sdk/users/{user_id}/sync`	`sdk`	`apple`, `samsung`, `google`	UUID4 batch ID

SDK Payload Offload

By default the SDK sync endpoint hands the whole payload to Celery, which keeps it in the Redis broker. A queued backlog then grows with payload size, and a large backfill can push Redis to maxmemory. SDK_PAYLOAD_S3_OFFLOAD uploads the payload to S3 instead and queues only its s3://bucket/key reference. The worker fetches the body when the task runs.

SDK_PAYLOAD_S3_OFFLOAD=false  # default

It needs a bucket (RAW_PAYLOAD_S3_BUCKET, or AWS_BUCKET_NAME) and startup fails without one. It does not need RAW_PAYLOAD_STORAGE=s3 — archival is a debugging aid and must not gate queue reliability, so offload works with archival disabled. For the same reason RAW_PAYLOAD_MAX_SIZE_BYTES does not apply to it. With RAW_PAYLOAD_STORAGE=s3 there is still one object, written to the same key with the same metadata, serving as both the queued payload and the archive. With RAW_PAYLOAD_STORAGE=disabled the object only exists to carry the payload, so the worker deletes it once the batch is committed — a failed batch keeps its payload for diagnosis. This needs s3:DeleteObject on the bucket, otherwise every batch logs a delete failure.

S3 is now on the critical path. Keep any lifecycle expiry above your worst queue lag, and expect 503 from the endpoint when the upload fails.

Error Handling

Raw payloads storage is designed to never break your data ingestion pipeline:

Oversized payloads - Payloads exceeding RAW_PAYLOAD_MAX_SIZE_BYTES are skipped with a warning log. Data processing continues normally.
S3 upload failures - Errors are logged but not propagated. The webhook or sync request continues processing.
Missing S3 bucket - If the S3 backend is configured but no bucket is available, it falls back to disabled with an error log.
S3 client failure - If the S3 client cannot be created (e.g., missing boto3 or invalid credentials), it falls back to disabled.

This covers archival only. Under SDK_PAYLOAD_S3_OFFLOAD the upload is the payload’s only copy, so a failure returns 503 and the SDK retries, rather than a batch being queued without a body.

Example Setup

Development (Log)
Development (MinIO)
Production (S3)

RAW_PAYLOAD_STORAGE=log

Then inspect payloads in Docker logs:

docker compose logs -f backend | grep raw_payload

MinIO is an S3-compatible object store you can run locally. This lets you browse stored payloads via a web UI without needing an AWS account.1. Start MinIO (on the same Docker network as the app):

docker run -d \
  --name minio-open-wearables \
  --hostname minio \
  --network open-wearables_default \
  -p 9002:9000 -p 9003:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address ":9001"

Use --hostname minio to register a clean DNS name in the Docker network. Boto3 rejects hostnames with special characters (e.g. double underscores), so container names like minio__open-wearables won’t work as endpoint URLs.

2. Create the bucket - open the MinIO Console at http://localhost:9003 (login minioadmin / minioadmin) and create a bucket called raw-payloads, or use the CLI:

docker exec minio-open-wearables mc alias set local http://localhost:9000 minioadmin minioadmin
docker exec minio-open-wearables mc mb local/raw-payloads

3. Configure env vars in backend/config/.env:

RAW_PAYLOAD_STORAGE=s3
RAW_PAYLOAD_S3_BUCKET=raw-payloads
RAW_PAYLOAD_S3_ENDPOINT_URL=http://minio:9000
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
AWS_REGION=us-east-1

Payloads will appear in the MinIO Console under the raw-payloads bucket.

RAW_PAYLOAD_STORAGE=s3
RAW_PAYLOAD_S3_BUCKET=mycompany-raw-payloads
RAW_PAYLOAD_S3_PREFIX=open-wearables
RAW_PAYLOAD_MAX_SIZE_BYTES=10485760

# AWS credentials (if not already configured)
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret
AWS_REGION=eu-north-1

Replaying Stored Payloads

Use the replay_raw_payloads.py script to re-send S3-stored payloads through the SDK sync endpoint. This is useful for reproducing issues, re-processing data after a fix, or load testing.

cd backend
set -a && source config/.env && set +a

uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <source-user-uuid> \
    --api-url http://localhost:8000 \
    --api-key sk-...

The script reads AWS/S3 credentials from environment variables automatically.

Filtering

Narrow down which payloads to replay:

uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <uuid> \
    --provider apple \
    --source sdk \
    --date-from 2025-01-01 \
    --date-to 2025-01-31 \
    --limit 10

Replaying to a different user

Use --target-user-id to send one user’s payloads to a different user account:

uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <source-uuid> \
    --target-user-id <destination-uuid> \
    --api-url https://api.openwearables.io \
    --api-key sk-...

Dry run

Preview which payloads would be sent without making any requests:

uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <uuid> \
    --dry-run

Options

Flag	Default	Description
`--user-id`	required	S3 user ID whose payloads to read
`--target-user-id`	same as `--user-id`	User ID to send payloads to
`--api-url`	`http://localhost:8000`	Backend API base URL
`--api-key`	`OPEN_WEARABLES_API_KEY` env var	API key for authentication
`--s3-bucket`	`RAW_PAYLOAD_S3_BUCKET` or `AWS_BUCKET_NAME` env var	S3 bucket name
`--s3-prefix`	`raw-payloads`	S3 key prefix
`--s3-endpoint-url`	`RAW_PAYLOAD_S3_ENDPOINT_URL` or `AWS_ENDPOINT_URL` env var	Custom S3 endpoint for compatible providers
`--aws-region`	`AWS_REGION` env var or `eu-north-1`	AWS region
`--aws-access-key-id`	`AWS_ACCESS_KEY_ID` env var	AWS access key ID
`--aws-secret-access-key`	`AWS_SECRET_ACCESS_KEY` env var	AWS secret access key
`--provider`	all	Filter by provider (`apple`, `samsung`, etc.)
`--source`	all	Filter by source (`sdk`, `webhook`)
`--date-from`	none	Start date filter (YYYY-MM-DD)
`--date-to`	none	End date filter (YYYY-MM-DD)
`--delay`	`1.0`	Seconds between requests
`--limit`	unlimited	Maximum number of payloads to replay
`--dry-run`	off	List payloads without sending

AWS Setup - Configure AWS credentials and S3 buckets
How to Add a Provider - Adding raw payload storage to new providers
Integration Guide - Testing your setup

Getting started

Architecture

Health Scores

Providers

Open Wearables Mobile App

Sync SDK

MCP Server

Developer Guides

Configuration

Deployment

Support

Overview

Configuration

Storage Backends

Disabled (default)

Log

S3

S3 key format

S3 object metadata

Supported Ingestion Points

SDK Payload Offload

Error Handling

Example Setup

Replaying Stored Payloads

Filtering

Replaying to a different user

Dry run

Options

​Overview

​Configuration

​Storage Backends

​Disabled (default)

​Log

​S3

​S3 key format

​S3 object metadata

​Supported Ingestion Points

​SDK Payload Offload

​Error Handling

​Example Setup

​Replaying Stored Payloads

​Filtering

​Replaying to a different user

​Dry run

​Options

​Related Guides

Overview

Configuration

Storage Backends

Disabled (default)

Log

S3

S3 key format

S3 object metadata

Supported Ingestion Points

SDK Payload Offload

Error Handling

Example Setup

Replaying Stored Payloads

Filtering

Replaying to a different user

Dry run

Options

Related Guides