Data format and structure

Overview

This page outlines how our daily enrichment finds and processes your billing data every day.

We run enrichments between 01:00 and 04:00 UTC each day, processing the entire month-to-date, every day.

All dates/times referenced in this document are in UTC timezone.

Storage platforms

Greenpixie can read data from S3, Azure Blob Storage or Google Cloud Storage.

Data from any platform can be placed in any storage provider. You can use the same bucket/container for multiple platforms, or you can provide separate buckets/containers for each platform.

Data format

Billing data can be stored in any of the following formats:

.csv
.csv.gz
.parquet
.snappy.parquet
.zstd.parquet

Data must be split into separate files (partitions) of less than 500MB each. Each partition must have the exact same column names and data types across a single directory. Separate directories can have different schemas, but must still meet the required column names.

Directory structure

Data must be placed in a directory structure that matches the following pattern. Greenpixie data pipelines rely on this structure to infer which platform, account and time period the data is for.

// <bucket_container_name> / <input | output> / <platform> / <account_id> / <year_month> / {partitions here...}

Where:

<bucket_container_name> can be any bucket name you decide
<input | output> can be input or output to indicate whether the data is for input to Greenpixie or output from Greenpixie. Even if you are using separate buckets/containers for input/output, it's essential to still include this base directory.
<platform> is the name of the platform that the billing data relates to from the following list: aws, azure, gcp. Even if you are only sending one platform, it's essential to still include this directory.
<account_id> is the ID of any billing accounts in your environment (e.g. 1234567890). If you do not have separate accounts for a platform, please enter all.
<year_month> is the year and month of the data in the format: YYYY-MM. e.g 2025-01
Partitions can be named anything you wish, but must be less than 500MB each.

If you are providing multiple platforms in separate buckets/containers, you must still include the <platform> directory.

Examples:

ABC Corp, placing AWS billing data in S3 for accounts 1111111111 and 2222222222 in the month of November 2025:

billing-part-0000.parquet

billing-part-0001.parquet

billing-part-0002.parquet

...

billing-part-0000.parquet

billing-part-0001.parquet

billing-part-0002.parquet

...

ABC Corp placing Azure billing data for all accounts (not separated by account) in Azure Blob Storage in the month of November 2025:

billing-part-0000.csv

billing-part-0001.csv

billing-part-0002.csv

...

If you are onboarding as a Greenpixie partner, the directory structure differs from the customer pattern in two ways:

An <org_id> directory sits between <input | output> and <platform>, identifying the organisation the data belongs to. Greenpixie shares these IDs with you during partner onboarding.
The <year_month> directory is replaced by a <timestamp> directory, identifying a single upload. Use ISO 8601 basic format in UTC: YYYYMMDDTHHMMSSZ, e.g. 20260722T120000Z.

// <bucket_container_name> / <input | output> / <org_id> / <platform> / <account_id> / <timestamp> / {partitions here...}

Example: a partner placing AWS billing data for organisation org_12345, account 1111111111, uploaded at 12:00:00 UTC on 22 July 2026:

billing-part-0000.parquet

billing-part-0001.parquet

...

To trigger an enrichment of an uploaded timestamp directory, see Partner trigger.

Required column names

Please include the following column names in your billing data. You can provide more columns if you wish, and they will be returned as provided. Providing extra columns helps to provide context when analysing the results.

AWS

Column names are case sensitive.

line_item_line_item_type
line_item_product_code
line_item_usage_type
line_item_usage_amount
line_item_usage_start_date
line_item_usage_end_date
pricing_unit
product_region
pricing_rate_code
product_sku
line_item_line_item_description

Azure

Column names are not case sensitive.

MeterId
MeterCategory
MeterName
UnitOfMeasure
ChargeType
Date
Quantity
ResourceLocation

GCP

There are two possible column formats for GCP billing data: nested or flat. Column names are case sensitive.

Nested format

sku.id
sku.description
location.location
location.region
location.country
usage.pricing_unit
usage.amount_in_pricing_units
usage_start_time
usage_end_time

Flat format

sku_id
sku_description
region
usage_pricing_unit
usage_amount_in_pricing_units
usage_start_time
usage_end_time

Next steps

At this stage, you should have formatted data in a bucket/container, and assigned permissions to give Greenpixie access.

Next, Greenpixie will check the permissions and minimum columns required for CUR enrichment.

Head back to How it works to continue.

Data format and structure

On this page