Partner trigger

Overview

As a partner, you tell us when an upload is complete by writing a dotfile named .upload_finished into its <timestamp> directory, after all partitions have been uploaded. This works the same way on Amazon S3, Azure Blob Storage and Google Cloud Storage.

Each night between 01:00 and 04:00 UTC, Greenpixie checks for marked uploads on every platform and starts a run for each one not yet enriched. Each upload is enriched once, and a directory without the marker is never enriched.

On S3 you can also add an event notification on the marker, so an upload starts as soon as it is marked rather than in the nightly check. This is optional, and the nightly check skips any upload the notification has already started.

For the full partner setup, see Partner onboarding.

Triggering an enrichment

1. Upload all partitions into the <timestamp> directory, following the partner directory structure.
2. Once every partition has finished uploading, write an empty file named .upload_finished into the same <timestamp> directory.

input / org_12345 / aws / 1111111111 / 20260722T120000Z / .upload_finished

A marker written before 03:00 UTC is usually picked up that night; one written later may wait until the following night. When many uploads are waiting at once, such as a history backfill, some run on a later night. To enrich an upload again, upload it under a new timestamp.

Optional: on-demand runs on S3

With an event notification on the bucket, each upload starts as soon as its marker is written, at any time of day. Azure Blob Storage and Google Cloud Storage don't support on-demand runs.

1. Send your Greenpixie contact your AWS account ID, your bucket name and the bucket's region. Your contact confirms the region is supported, creates a notification topic that accepts events from that bucket only, and sends you its SNS topic ARN.
2. In the bucket, create an event notification:
   • Event type: s3:ObjectCreated:*
   • Suffix filter: .upload_finished
   • Destination: the SNS topic ARN we provided.