Ingest Greenpixie data into Cloudability

This guide takes you from an enriched FOCUS file in your Amazon S3 bucket through to carbon, energy and water reporting inside IBM Cloudability, and on to downstream systems such as Apptio TBM Studio. It is written for a platform, FinOps or data engineer who already has the enriched file. AWS S3 is the worked example; Azure Blob Storage and Google Cloud Storage follow the same shape with provider-specific credential fields.

The integration uses Cloudability's Connect Custom Data (FOCUS Ingress) capability. Greenpixie writes the sustainability metrics into the FOCUS Tags column and delivers the file to S3 with a manifest, with the file's cost and quantity columns zeroed so Cloudability does not double-count against its native data. Cloudability pulls and validates it, and a Business Metric converts the string tag values into numeric metrics that behave like any other metric in the platform.

Before you start

You need:

• An enriched billing file in FOCUS 1.0 or 1.1 format, as Parquet (recommended) or CSV, already written to an S3 bucket you control.
• The sustainability metrics (energy in kWh, carbon in tCO₂e, water in litres) written into the FOCUS Tags column. The formatting of that column is the single most important part of this integration, covered in Format the Tags column.
• Administrator rights in Cloudability to add a vendor credential, create a Business Metric, and edit Tag and Label mappings.
• Admin rights in AWS to create an S3 read policy and an IAM role from a CloudFormation template.

Two constraints from IBM's documentation, worth knowing up front:

• Do not point the FOCUS connector at an AWS, Azure or GCP account you have already credentialed through the native connector. The native connector is more granular and also brings in the utilisation data used for rightsizing and other optimisation features, whereas FOCUS ingress brings cost data only. If the account is already native, use a separate account or bucket for the FOCUS feed.
• Parquet files must be compressed as part of generation, using the framework's default compression (Hadoop or Avro, for example Snappy). Do not compress the Parquet file again afterwards.

Set up the S3 layout and manifest

Cloudability does not scan the bucket freely. It reads a manifest file and follows the paths inside it, so the folder structure and the manifest have to be exact.

Folder structure

The path to the manifest must follow this structure.

<Root Directory>/<Optional_Sub_Directories>/<Vendor>/<Report_Period>/<Epoch Time Folder>/<prefix>-Manifest.json

• Root Directory is the S3 bucket name.
• Optional Sub Directories are any extra prefixes you use. This part is optional.
• Vendor identifies the source, for example aws.
• Report Period is the billing month, formatted YYYYMMDD-YYYYMM(+1)DD. February 2023 is 20230201-20230301.
• Epoch Time Folder identifies the latest drop for the month-to-date data.
• Manifest is the metadata JSON file for that report period. The capital M in Manifest is required.

Every drop must contain the entire month-to-date dataset, not an incremental slice. A worked example path is below.

gpx-cloudability-focus/aws/20260601-20260701/1779440229/gpx-focus-00001-Manifest.json

The manifest file

The manifest tells Cloudability what to read and which FOCUS version it is. A minimal example is below.

{
  "compression": "GZIP",
  "content_type": "Parquet",
  "report_id": "uuid-f431e214843b3c19756368",
  "root_dir": "gpx-cloudability-focus",
  "all_report_keys": [
    "aws/20260601-20260701/1779440229/gpx-focus-00001.snappy.parquet",
    "aws/20260601-20260701/1779440229/gpx-focus-00002.snappy.parquet"
  ],
  "updated_at": "2026-06-30T05:00:22Z",
  "focus_version": "1.0"
}

Field requirements from the IBM docs.

Common mistakes that stop ingestion here:

• all_report_keys pointing at paths that do not exist or have the wrong extension. This surfaces later as an S3 NoSuchKeyException ("The specified key does not exist"): the manifest found the bucket, but the object path inside it was wrong. Recheck the bucket name in root_dir and every entry in all_report_keys.
• A missing epoch or report-period folder. If the folder for the month is not there, the paths Cloudability builds will 404.
• Re-compressing a Parquet file after generation. Keep compression to what the writer produced.

See the FOCUS specification and column definitions.

Format the Tags column

This is the step that most often blocks a first upload, so it is worth being precise.

Cloudability's Tags column accepts strings only. The sustainability metrics have to be written as a flat JSON object of string keys mapped to string values, with numbers stored as their string representation.

{
  "usage_electricity_consumption_kwh": "4.42623188781383e-12",
  "total_tonnes_co2e": "1.9254262434434935e-15",
  "total_water_litres": "6.534493054037568e-12"
}

Two things break the ingest:

• Arrays instead of a flat object. If the tag field is an array such as [{"key": "...", "value": "..."}], Cloudability rejects it with Cannot deserialize value of type java.lang.String from Array value. The value must be a single JSON object of "key": "value" pairs, not a list.
• Null values. Any None or null in the object fails deserialisation, because the parser expects a string. Strip nulls before writing the file, or write an empty string.

If you generate the enriched file from a table where the enriched columns are named per row, the transformation that folds those columns into a single Tags JSON object needs to drop nulls and emit strings.

Zero the cost and quantity columns

The FOCUS feed exists only to carry the sustainability metrics in the Tags column. Cloudability already holds the cost, usage and resource figures from its native data, so the cost and quantity columns in this file must be set to 0, or they will be double-counted. Zero the following:

• BilledCost
• ConsumedQuantity
• ContractedCost
• ContractedUnitPrice
• EffectiveCost
• ListCost
• ListUnitPrice
• PricingQuantity

Set a distinct provider name

Change the FOCUS ProviderName from AWS or Azure to a distinct value such as AWS - Greenpixie or Azure - Greenpixie. Without this, the values conflict with Cloudability's existing provider data in its back end.

You may also need to make the billing account ID and sub-account IDs unique from their native AWS or Azure values. This is not always required, so it is worth testing with and without before settling on an approach.

Credential the bucket in Cloudability

With the file and manifest in place, give Cloudability read access and register the data source.

1. In Cloudability, go to Settings > Vendor Credentials > Add Datasource > Other Data Sources.
2. Select Other Data Sources > Add a Credential. The Add Other Datasources panel opens. Click Next.
3. Choose AWS Simple Storage Service as the storage provider and follow the prompts.
4. Enter the Vendor name for the data being ingested.
5. Enter the AWS Account ID where the files are located.
6. Enter the S3 bucket name under Root Directory.
7. Enter any optional sub prefixes under Sub Directories.
8. Enter the Manifest Prefix.
9. Click Generate template.

The generated template is an AWS CloudFormation template that creates the IAM role and read policy Cloudability uses to reach the bucket. Deploy it in the AWS account that owns the bucket, the same way you would for a standard credential. The mechanics are covered in the AWS customer integration guide.

Azure Blob Storage and Google Cloud Storage use the same flow with provider-specific fields. For Azure these are Tenant ID, Subscription ID, Resource Group, Storage Account and Blob container; for GCP, the billing account and GCS bucket.

A green check mark against the credential means the credential is valid. It does not yet mean the data is in. Cloudability still validates that the billing file and manifest match the required format, and it pulls custom FOCUS data on a daily cycle rather than hourly. If you see no data after 24 hours, raise a case with Apptio Support. If the credential errors with something like "unknown error accessing your bucket ... or its files", the IAM role or bucket name is wrong: redeploy the CloudFormation stack from the current template and confirm the bucket name matches root_dir.

Watch ingestion and diagnose failures

Because the custom FOCUS input reprocesses on a daily cycle, the feedback loop is slow. Use the Data Tracking view in Cloudability to see the state of each pull; a dataset that has landed shows a COMPLETED state. Troubleshooting is a case of correcting the file or manifest and waiting for the next cycle, rather than real-time.

The three failure modes to recognise:

• Data format mismatch: Cannot deserialize value of type java.lang.String from Array value. The Tags object is an array or contains nulls. Fix the Tags formatting.
• Missing data folder: the expected epoch or report-period folder is not present, so the constructed file paths do not resolve. Fix the folder structure.
• File not found: NoSuchKeyException, "The specified key does not exist". The manifest all_report_keys or root_dir points at a path that is not there. Fix the manifest.

A broken custom FOCUS file can also stop the standard cost feed for the same credential if the two are very similar, so treat manifest and format errors as blocking, not cosmetic.

Convert the string tags into numeric metrics

Once the file is ingested, the sustainability values exist as string tags. To use them as numbers in reports, create a Business Metric, which is evaluated at ingestion and behaves like a native metric afterwards.

How Business Metrics work

When Cloudability ingests each cost line item, it evaluates that item against your Business Mapping statements, similar to a case statement. The first statement whose matchExpression matches wins, and its valueExpression is calculated to populate the metric for that line. Each metric has a name, a format of Currency or Number, a default value used when no statement matches, and an ordered list of statements. There is a hard limit of five Business Metrics.

Create one in the UI

1. Go to Business Mapping and select the Metrics tab.
2. Click New Business Metric.
3. Name it so it reads well in reports, for example Carbon (tCO2e).
4. Choose Number as the format for a sustainability quantity. Use Currency only for money.
5. Set a default value. Zero is sensible for an emissions metric, so unmatched lines do not inflate totals.
6. Add statements, each with a match expression and a value expression, ordered from most specific to least.

A statement has two parts:

• Match Expression, the condition that decides when the statement applies. It can reference dimensions, including tags, and supports boolean logic and date ranges. For example, DIMENSION['vendor'] == 'Amazon'.
• Value Expression, the calculation that produces the number. For example, METRIC['unblended_cost'] * 1.15.

Because the enriched value arrives as a string in a tag dimension rather than a native numeric metric, the Business Metric is what surfaces it as a number. The five-metric cap means you choose which measures to promote to first-class numbers, for example total combined emissions, energy and water, rather than every sub-component. A practical approach: at the hourly billing-line grain, take the maximum value each metric reaches for a given resource type, and build the statements to cover that value range.

Create one via the API

The same metric can be created through the v3 Business Mappings endpoint.

curl -X POST 'https://api.cloudability.com/v3/internal/business-mappings/metrics/' \
  -H 'Content-Type: application/json' \
  -u '[auth_token]:' \
  -d @- <<'EOF'
{
  "name": "Carbon (tCO2e)",
  "numberFormat": "number",
  "defaultValueExpression": "0",
  "statements": [
    {
      "matchExpression": "DIMENSION['vendor'] == 'Amazon'",
      "valueExpression": "METRIC['unblended_cost'] * 0"
    }
  ]
}
EOF

Swap the placeholder valueExpression for the mapping that produces your emissions number. To have historic months reflect a new or changed rule, ask Apptio Support or your TAM to reprocess, since by default only new billing data picks up rule changes.

Surface the tags as reporting dimensions

Tag keys become reporting dimensions once mapped. Cloudability allows up to 50 Tag and Label dimensions. Map the sustainability tag keys in Settings under Tag and Label Mapping so they appear as columns and filters in reports.

To inspect the raw string values, use Tag Explorer under Insights, or build a report with the specific Tag (Value) dimension. A (not set) value in a report usually means the resource had no value for that tag, or data was ingested before the tag existed, rather than a bug.

Report and build dashboards

With the numeric Business Metrics created and the tags mapped, carbon, energy and water can be used anywhere a native metric can. Add a Business Metric to a report or dashboard and combine it with dimensions such as service name, account, tag, or ATUM mappings for allocation, showback and chargeback. Trend it month over month, or divide it by a usage or unit metric to produce unit economics such as gCO₂e per dollar of spend.

Push the data downstream

Apptio TBM Studio (ApptioOne)

The Datalink Cloudability connector imports Cloudability cost data into the Costing Standard, Billing Standard and Hybrid Business Management applications, mapped to the Apptio TBM Unified Model (ATUM). The integration is requested through your Apptio support contact, who opens a data channel for your Cloudability environment. Your Cloudability environment and the target Apptio apps must sit in the same Access Administration environment.

The integration request lets you nominate additional fields beyond the defaults, and that list explicitly includes tag and label mappings and Business Metrics. This is the path by which the sustainability Business Metrics travel into TBM Studio and can be allocated against IT services and towers. Billing fields such as resource ID are excluded and cannot be requested. See the Datalink Cloudability Connector Guide.

Cloudability API

Metrics and dimensions, including Business Metrics, are available through the Cloudability v3 API for export to BI tools and other systems. Query the reporting dataset by metrics, dimensions, filters and date ranges, returning JSON or CSV.

Troubleshooting

Next steps