> ## Documentation Index
> Fetch the complete documentation index at: https://openwearables.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Raw Payload Storage

> Capture raw JSON payloads from wearable providers before processing. Useful for debugging issues, testing ingestion changes, and reproducing parsing bugs.

## Overview

Raw payloads storage lets you capture the exact JSON payloads received from wearable providers before any processing. This is useful when you need to:

* **Debug data issues** - inspect exactly what a provider sent when something doesn't look right
* **Test ingestion changes** - replay real payloads against updated parsing logic to make sure nothing breaks
* **Reproduce bugs** - grab the original payload that caused a problem and use it in local development

The feature is **disabled by default** and adds zero overhead when turned off.

## Configuration

Add these environment variables to your `.env` file:

```bash theme={null}
# Storage backend: "disabled" (default), "log", or "s3"
RAW_PAYLOAD_STORAGE=disabled

# Maximum payload size in bytes (default: 10 MB)
RAW_PAYLOAD_MAX_SIZE_BYTES=10485760

# S3-specific settings (only needed when using "s3" backend)
RAW_PAYLOAD_S3_BUCKET=my-raw-payloads        # Falls back to AWS_BUCKET_NAME if not set
RAW_PAYLOAD_S3_PREFIX=raw-payloads            # S3 key prefix (default: "raw-payloads")
# RAW_PAYLOAD_S3_ENDPOINT_URL=              # Optional: custom endpoint for S3-compatible providers
```

<Warning>
  Raw payloads can contain sensitive health data. Make sure your storage destination meets your organization's data protection requirements.
</Warning>

## Storage Backends

### Disabled (default)

No payloads are stored. The `store_raw_payload()` call returns immediately with no overhead.

```bash theme={null}
RAW_PAYLOAD_STORAGE=disabled
```

### Log

Writes each payload as a structured JSON line to stdout. Useful for development, or when your infrastructure already captures application logs (e.g., CloudWatch, Datadog).

```bash theme={null}
RAW_PAYLOAD_STORAGE=log
```

Example output:

```json theme={null}
{
  "level": "debug",
  "message": "raw_payload",
  "source": "webhook",
  "provider": "garmin",
  "size_bytes": 4521,
  "trace_id": "a1b2c3d4",
  "payload": "{...}"
}
```

### S3

Uploads each payload as a JSON file to an S3 bucket (or any S3-compatible object store like MinIO or Railway Object Storage).

```bash theme={null}
RAW_PAYLOAD_STORAGE=s3
RAW_PAYLOAD_S3_BUCKET=my-raw-payloads
```

AWS credentials are reused from the existing application config (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`). See the [AWS Setup guide](/dev-guides/aws-setup) for details.

#### S3 key format

Files are organized by provider, source, date, and user:

```
{prefix}/{provider}/{source}/{YYYY-MM-DD}/{user_id}/{file_id}.json
```

Example:

```
raw-payloads/garmin/webhook/2025-03-12/_unknown/a1b2c3d4e5f6.json
raw-payloads/apple/sdk/2025-03-12/user-123/f6e5d4c3b2a1.json
```

* `_unknown` is used when a user ID is not available (e.g., Garmin webhooks)
* `file_id` is a random 12-character hex string

#### S3 object metadata

Each object includes metadata for filtering and querying:

| Key          | Description                                        |
| ------------ | -------------------------------------------------- |
| `source`     | Origin type (`sdk`, `webhook`)                     |
| `provider`   | Provider name (`garmin`, `apple`, `samsung`, etc.) |
| `size_bytes` | Payload size in bytes                              |
| `timestamp`  | ISO 8601 UTC timestamp                             |
| `user_id`    | User identifier (if available)                     |
| `trace_id`   | Request trace/batch ID (if available)              |

## Supported Ingestion Points

Raw payloads storage is currently enabled at two ingestion points:

| Endpoint                                | Source    | Provider                     | Trace ID                |
| --------------------------------------- | --------- | ---------------------------- | ----------------------- |
| `POST /api/v1/garmin/push`              | `webhook` | `garmin`                     | 8-char request trace ID |
| `POST /api/v1/sdk/users/{user_id}/sync` | `sdk`     | `apple`, `samsung`, `google` | UUID4 batch ID          |

## Error Handling

Raw payloads storage is designed to never break your data ingestion pipeline:

* **Oversized payloads** - Payloads exceeding `RAW_PAYLOAD_MAX_SIZE_BYTES` are skipped with a warning log. Data processing continues normally.
* **S3 upload failures** - Errors are logged but not propagated. The webhook or sync request continues processing.
* **Missing S3 bucket** - If the S3 backend is configured but no bucket is available, it falls back to `disabled` with an error log.
* **S3 client failure** - If the S3 client cannot be created (e.g., missing `boto3` or invalid credentials), it falls back to `disabled`.

## Example Setup

<Tabs>
  <Tab title="Development (Log)">
    ```bash theme={null}
    RAW_PAYLOAD_STORAGE=log
    ```

    Then inspect payloads in Docker logs:

    ```bash theme={null}
    docker compose logs -f backend | grep raw_payload
    ```
  </Tab>

  <Tab title="Development (MinIO)">
    [MinIO](https://min.io/) is an S3-compatible object store you can run locally. This lets you browse stored payloads via a web UI without needing an AWS account.

    **1. Start MinIO** (on the same Docker network as the app):

    ```bash theme={null}
    docker run -d \
      --name minio-open-wearables \
      --hostname minio \
      --network open-wearables_default \
      -p 9002:9000 -p 9003:9001 \
      -e MINIO_ROOT_USER=minioadmin \
      -e MINIO_ROOT_PASSWORD=minioadmin \
      minio/minio server /data --console-address ":9001"
    ```

    <Warning>
      Use `--hostname minio` to register a clean DNS name in the Docker network. Boto3 rejects hostnames with special characters (e.g. double underscores), so container names like `minio__open-wearables` won't work as endpoint URLs.
    </Warning>

    **2. Create the bucket** - open the MinIO Console at [http://localhost:9003](http://localhost:9003) (login `minioadmin` / `minioadmin`) and create a bucket called `raw-payloads`, or use the CLI:

    ```bash theme={null}
    docker exec minio-open-wearables mc alias set local http://localhost:9000 minioadmin minioadmin
    docker exec minio-open-wearables mc mb local/raw-payloads
    ```

    **3. Configure env vars** in `backend/config/.env`:

    ```bash theme={null}
    RAW_PAYLOAD_STORAGE=s3
    RAW_PAYLOAD_S3_BUCKET=raw-payloads
    RAW_PAYLOAD_S3_ENDPOINT_URL=http://minio:9000
    AWS_ACCESS_KEY_ID=minioadmin
    AWS_SECRET_ACCESS_KEY=minioadmin
    AWS_REGION=us-east-1
    ```

    Payloads will appear in the MinIO Console under the `raw-payloads` bucket.
  </Tab>

  <Tab title="Production (S3)">
    ```bash theme={null}
    RAW_PAYLOAD_STORAGE=s3
    RAW_PAYLOAD_S3_BUCKET=mycompany-raw-payloads
    RAW_PAYLOAD_S3_PREFIX=open-wearables
    RAW_PAYLOAD_MAX_SIZE_BYTES=10485760

    # AWS credentials (if not already configured)
    AWS_ACCESS_KEY_ID=your-key
    AWS_SECRET_ACCESS_KEY=your-secret
    AWS_REGION=eu-north-1
    ```
  </Tab>
</Tabs>

## Replaying Stored Payloads

Use the `replay_raw_payloads.py` script to re-send S3-stored payloads through the SDK sync endpoint. This is useful for reproducing issues, re-processing data after a fix, or load testing.

```bash theme={null}
cd backend
set -a && source config/.env && set +a

uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <source-user-uuid> \
    --api-url http://localhost:8000 \
    --api-key sk-...
```

The script reads AWS/S3 credentials from environment variables automatically.

### Filtering

Narrow down which payloads to replay:

```bash theme={null}
uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <uuid> \
    --provider apple \
    --source sdk \
    --date-from 2025-01-01 \
    --date-to 2025-01-31 \
    --limit 10
```

### Replaying to a different user

Use `--target-user-id` to send one user's payloads to a different user account:

```bash theme={null}
uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <source-uuid> \
    --target-user-id <destination-uuid> \
    --api-url https://api.openwearables.io \
    --api-key sk-...
```

### Dry run

Preview which payloads would be sent without making any requests:

```bash theme={null}
uv run --with boto3,httpx python scripts/replay_raw_payloads.py \
    --user-id <uuid> \
    --dry-run
```

### Options

| Flag                      | Default                                              | Description                                   |
| ------------------------- | ---------------------------------------------------- | --------------------------------------------- |
| `--user-id`               | required                                             | S3 user ID whose payloads to read             |
| `--target-user-id`        | same as `--user-id`                                  | User ID to send payloads to                   |
| `--api-url`               | `http://localhost:8000`                              | Backend API base URL                          |
| `--api-key`               | `OPEN_WEARABLES_API_KEY` env var                     | API key for authentication                    |
| `--s3-bucket`             | `RAW_PAYLOAD_S3_BUCKET` or `AWS_BUCKET_NAME` env var | S3 bucket name                                |
| `--s3-prefix`             | `raw-payloads`                                       | S3 key prefix                                 |
| `--s3-endpoint-url`       | `RAW_PAYLOAD_S3_ENDPOINT_URL` env var                | Custom S3 endpoint for compatible providers   |
| `--aws-region`            | `AWS_REGION` env var or `eu-north-1`                 | AWS region                                    |
| `--aws-access-key-id`     | `AWS_ACCESS_KEY_ID` env var                          | AWS access key ID                             |
| `--aws-secret-access-key` | `AWS_SECRET_ACCESS_KEY` env var                      | AWS secret access key                         |
| `--provider`              | all                                                  | Filter by provider (`apple`, `samsung`, etc.) |
| `--source`                | all                                                  | Filter by source (`sdk`, `webhook`)           |
| `--date-from`             | none                                                 | Start date filter (YYYY-MM-DD)                |
| `--date-to`               | none                                                 | End date filter (YYYY-MM-DD)                  |
| `--delay`                 | `1.0`                                                | Seconds between requests                      |
| `--limit`                 | unlimited                                            | Maximum number of payloads to replay          |
| `--dry-run`               | off                                                  | List payloads without sending                 |

## Related Guides

* [AWS Setup](/dev-guides/aws-setup) - Configure AWS credentials and S3 buckets
* [How to Add a Provider](/dev-guides/how-to-add-new-provider) - Adding raw payload storage to new providers
* [Integration Guide](/dev-guides/integration-guide) - Testing your setup