# Load Greenpixie data into Cloudability Source: https://greenpixie.com/docs/guides/cloudability-integration How to integrate your enriched cloud usage report data into Cloudability. This guide takes you from an enriched FOCUS file in your Amazon S3 bucket through to carbon, energy and water reporting inside IBM Cloudability, and on to downstream systems such as Apptio TBM Studio. It is written for a platform, FinOps or data engineer who already has the enriched file. AWS S3 is the worked example; Azure Blob Storage and Google Cloud Storage follow the same shape with provider-specific credential fields. Where the procedure is IBM's, this guide links to IBM's documentation and covers what sustainability data changes. The integration uses Cloudability's [Connect Custom Data, the FOCUS ingress path](https://www.ibm.com/docs/en/cloudability-commercial/cloudability-standard/saas?topic=cloudability-connect-custom-data-focus-ingress). Greenpixie writes the sustainability metrics into the FOCUS `Tags` column and delivers the file to S3 with a manifest, with the file's cost and quantity columns zeroed so Cloudability does not double-count against its native data. Cloudability pulls and validates it, and a Business Metric converts the string tag values into numeric metrics that behave like any other metric in the platform. ## Before you start [#before-you-start] You need: * An enriched billing file in FOCUS 1.0 or 1.1 format, as Parquet, which is preferred, or CSV, already written to an S3 bucket you control. * The sustainability metrics for energy in kWh, carbon in tCO₂e and water in litres, written into the FOCUS `Tags` column. Getting that column's formatting wrong blocks ingestion, so see [Format the Tags column](#format-the-tags-column). * Administrator rights in Cloudability to add a vendor credential, create a Business Metric, and edit Tag and Label mappings. * Admin rights in AWS to create an S3 read policy and an IAM role from a CloudFormation template. Constraints from IBM's documentation: * Do not point the FOCUS connector at an AWS, Azure or GCP account you have already credentialed through the native connector. The native connector is more granular and also brings in the utilisation data used for rightsizing and other optimisation features, whereas FOCUS ingress brings cost data only. If the account is already native, use a separate account or bucket for the FOCUS feed. * Parquet files must be compressed as part of generation, using the framework's default compression, for example Snappy for Hadoop or Avro. Do not compress the Parquet file again afterwards. ## Set up the S3 layout and manifest [#set-up-the-s3-layout-and-manifest] Cloudability reads a manifest file and follows the paths inside it, so the folder structure and the manifest have to be exact. The [Connect Custom Data page](https://www.ibm.com/docs/en/cloudability-commercial/cloudability-standard/saas?topic=cloudability-connect-custom-data-focus-ingress) carries the required path structure and the manifest field definitions, and is the authority on both. A worked Greenpixie path is below. ``` gpx-cloudability-focus/aws/20260601-20260701/1779440229/gpx-focus-00001-Manifest.json ``` Read against IBM's structure, `gpx-cloudability-focus` is the bucket, `aws` the vendor, `20260601-20260701` the billing month, and `1779440229` the epoch folder identifying the latest drop for the month. The capital `M` in `Manifest` is required. Every drop must contain the entire month-to-date dataset, so do not write an incremental slice. A manifest matching that path is below. ```json { "compression": "GZIP", "content_type": "Parquet", "report_id": "uuid-f431e214843b3c19756368", "root_dir": "gpx-cloudability-focus", "all_report_keys": [ "aws/20260601-20260701/1779440229/gpx-focus-00001.snappy.parquet", "aws/20260601-20260701/1779440229/gpx-focus-00002.snappy.parquet" ], "updated_at": "2026-06-30T05:00:22Z", "focus_version": "1.0" } ``` Set `focus_version` to the version Greenpixie produced the file in, `1.0` or `1.1`. Common mistakes that cause an error during ingestion: * `all_report_keys` pointing at a missing path or the wrong extension. The error appears later as an S3 `NoSuchKeyException`, reading "The specified key does not exist", which means the bucket name was right and the object path inside it was wrong. Recheck the bucket name in `root_dir` and every entry in `all_report_keys`. * A missing epoch or report-period folder. If the folder for the month is not there, the paths Cloudability builds will 404. * Re-compressing a Parquet file after generation. Leave the compression as the writer produced it. See the [FOCUS specification](https://focus.finops.org/) and [column definitions](https://focus.finops.org/focus-columns/). ## Format the Tags column [#format-the-tags-column] Cloudability's `Tags` column accepts **strings only**, so the sustainability metrics have to be written as a flat JSON object of string keys mapped to string values, with numbers stored as their string representation. ```json { "usage_electricity_consumption_kwh": "4.42623188781383e-12", "total_tonnes_co2e": "1.9254262434434935e-15", "total_water_litres": "6.534493054037568e-12" } ``` Where ingestion can break: * **Arrays instead of a flat object.** If the tag field is an array such as `[{"key": "...", "value": "..."}]`, Cloudability rejects it with `Cannot deserialize value of type java.lang.String from Array value`. The value must be a single JSON object of `"key": "value"` pairs. * **Null values.** Any `None` or `null` in the object fails deserialisation, because the parser expects a string. Strip nulls before writing the file, or write an empty string. If you generate the enriched file from a table where the enriched columns are named per row, the transformation that folds those columns into a single `Tags` JSON object needs to drop nulls and emit strings. ## Zero the cost and quantity columns [#zero-the-cost-and-quantity-columns] The FOCUS feed exists only to carry the sustainability metrics in the `Tags` column. Cloudability already holds the cost, usage and resource figures from its native data, so the cost and quantity columns in this file must be set to `0`, or they will be double-counted. Zero the following: * `BilledCost` * `ConsumedQuantity` * `ContractedCost` * `ContractedUnitPrice` * `EffectiveCost` * `ListCost` * `ListUnitPrice` * `PricingQuantity` ## Set a distinct provider name [#set-a-distinct-provider-name] Change the FOCUS `ProviderName` from `AWS` or `Azure` to a distinct value such as `AWS - Greenpixie` or `Azure - Greenpixie`. Without a distinct name, the values conflict with Cloudability's existing provider data in its back end. You may also need to make the billing account ID and sub-account IDs unique from their native AWS or Azure values. That is not always required, so test with and without before settling on an approach. ## Credential the bucket in Cloudability [#credential-the-bucket-in-cloudability] With the file and manifest in place, give Cloudability read access and register the data source. The [AWS customer integration guide](https://www.ibm.com/docs/en/cloudability-commercial/cloudability-standard/saas?topic=cloudability-connecting-aws-customer-integration-guide) covers the wizard under **Settings > Vendor Credentials** and the CloudFormation template it generates, which creates the IAM role and read policy Cloudability uses to reach the bucket. Deploy that template in the AWS account that owns the bucket. The wizard asks for the **Root Directory**, being the S3 bucket name, which must match `root_dir` in the manifest, the **Sub Directories**, being any extra prefixes you use, and the **Manifest Prefix**. Its remaining fields come from your existing AWS setup. Azure Blob Storage and Google Cloud Storage use the same flow with provider-specific fields. For Azure these are Tenant ID, Subscription ID, Resource Group, Storage Account and Blob container; for GCP, the billing account and GCS bucket. A green check mark against the credential means the credential is valid. It does not yet mean the data is in. Cloudability still validates that the billing file and manifest match the required format, and it pulls custom FOCUS data once a day. If you see no data after 24 hours, raise a case with Apptio Support. A credential that errors with something like "unknown error accessing your bucket ... or its files" means the IAM role or bucket name is wrong, so redeploy the CloudFormation stack from the current template and confirm the bucket name matches `root_dir`. ## Watch ingestion and diagnose failures [#watch-ingestion-and-diagnose-failures] Because the custom FOCUS input reprocesses on a daily cycle, the feedback loop is slow. Use the **Data Tracking** view in Cloudability to see the state of each pull; a successful pull shows a `COMPLETED` state. Troubleshooting is a case of correcting the file or manifest and waiting for the next cycle, with no real-time feedback. Errors you will see, with the cause and the fix for each: * **Data format mismatch**, `Cannot deserialize value of type java.lang.String from Array value`. The `Tags` object is an array or contains nulls. Fix the Tags formatting. * **Missing data folder**, where the expected epoch or report-period folder is not present, so the constructed file paths do not resolve. Fix the folder structure. * **File not found**, `NoSuchKeyException`, "The specified key does not exist". The manifest `all_report_keys` or `root_dir` points at a missing path. Fix the manifest. A broken custom FOCUS file can also stop the standard cost feed for the same credential if the two are very similar, so treat manifest and format errors as blocking. ## Convert the string tags into numeric metrics [#convert-the-string-tags-into-numeric-metrics] Once Cloudability has pulled the file, the sustainability values are string tags. To use them as numbers in reports, create a [Business Metric](https://www.ibm.com/docs/en/cloudability-commercial/cloudability-premium/saas?topic=mapping-business-metrics-in-cloudability), which is evaluated at ingestion and behaves like a native metric afterwards. That IBM page covers the match and value expression syntax and the UI. To create the same metric through the API, use the [v3 Business Mappings endpoint](https://www.ibm.com/docs/en/cloudability-commercial/api-v3/business_mappings_endpoint.html). What differs when the metric carries a sustainability quantity: * Set the format to **Number**, since Currency is for money. * Set the default value to zero, so lines that match no statement do not inflate totals. * Cloudability enforces a hard limit of five Business Metrics, so choose which measures to promote to first-class numbers, for example total combined emissions, energy and water, and leave the sub-components as string tags. * A practical way to build the statements is to take the maximum value each metric reaches for a given resource type, at the level of a single hourly billing line, then cover that range. By default a rule change applies to new billing data, so historic months keep the values they were loaded with. Ask Apptio Support or your Technical Account Manager to reprocess where history needs to reflect a new or changed rule. ## Surface the tags as reporting dimensions [#surface-the-tags-as-reporting-dimensions] Tag keys become reporting dimensions once mapped. Map the sustainability tag keys in **Settings** under [Tag and Label Mapping](https://www.ibm.com/docs/en/cloudability-commercial/cloudability-standard/saas?topic=spend-tag-label-mapping) so they appear as columns and filters in reports. Cloudability allows up to 50 Tag and Label dimensions, which the sustainability keys count against. To inspect the raw string values, use Tag Explorer under Insights, or build a report with the specific `Tag (Value)` dimension. ## Report and build dashboards [#report-and-build-dashboards] With the numeric Business Metrics created and the tags mapped, carbon, energy and water can be used anywhere a native metric can. Add a Business Metric to a report or dashboard and combine it with dimensions such as service name, account, tag, or Apptio TBM Unified Model (ATUM) mappings for allocation, showback and chargeback. Trend it month over month, or divide it by a usage or unit metric to produce unit economics such as gCO₂e per dollar of spend. ## Push the data downstream [#push-the-data-downstream] Apptio TBM Studio, also known as ApptioOne, imports Cloudability cost data through the Datalink connector, mapped to ATUM. The [Datalink Cloudability Connector Guide](https://www.ibm.com/docs/en/apptio-platform/datalink/saas?topic=connectors-cloudability-connector-guide) covers the request, which is raised through your Apptio support contact. The integration request lets you nominate additional fields beyond the defaults, and that list explicitly includes tag and label mappings and Business Metrics. Nominating them on that list is what makes the sustainability Business Metrics available in TBM Studio, where they can be allocated against IT services and towers. Business Metrics and the mapped tag dimensions are also available through the [Cloudability v3 API](https://www.ibm.com/docs/en/cloudability-commercial/cloudability-premium/saas?topic=api-getting-started-cloudability-v3) for export to BI tools and other systems. ## Troubleshooting [#troubleshooting] | Symptom | Likely cause | Fix | | -------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | Credential errors on save, cannot access bucket | IAM role or bucket name wrong | Redeploy the CloudFormation stack, confirm bucket matches `root_dir` | | `Cannot deserialize value of type java.lang.String from Array value` | `Tags` is an array of objects, or contains nulls | Write `Tags` as a flat JSON object of string key-value pairs, strip nulls | | `NoSuchKeyException`, specified key does not exist | `all_report_keys` or `root_dir` points at a missing path | Correct the manifest paths and file extensions | | Green check but no data after 24 hours | Format or manifest validation still failing, or waiting on the daily cycle | Check Data Tracking, then raise a case with Apptio Support | | Standard cost feed also stopped | A malformed FOCUS file on the same credential blocked the shared pull | Fix the FOCUS file and manifest first | | Historic months do not reflect a new Business Metric rule | Rules apply to new billing data by default | Ask Support or your Technical Account Manager to reprocess history | ## Next steps [#next-steps] - [Enrich an AWS cloud usage report using the API](https://greenpixie.com/docs/guides/enrich-cur-api): Produce the enriched FOCUS file this guide starts from. - [Read the API documentation](https://api.greenpixie.com/v1/docs): Full endpoint reference for the Greenpixie rate card.