Connect Databricks

Connect your Databricks account with one OAuth service principal so leancosts can show what your DBU actually cost, and which clusters, SQL warehouses and instance pools burned it. leancosts is read-only here as everywhere: the only statements it sends to Databricks are SELECT, over the one SQL warehouse you nominate. It never starts, resizes, stops or edits a cluster, a job or a warehouse. For the full posture see Security and data handling.

One service principal covers every workspace in the account. The system tables are metastore-wide, so a single SQL warehouse answers for all of them.

Before you start

• You need to be an account admin in Databricks to create a service principal and grant it access to the system schemas, and the connections-manage permission in leancosts to add a connection.
• Unity Catalog with the system schemas enabled. The connector reads system.billing, system.compute and system.lakeflow. If those schemas do not exist in the account, enable them first.
• One SQL warehouse the principal may use. A serverless 2X-Small is enough: it wakes in seconds when a sync starts and stops again under its own auto-stop.
• If your Azure bill already charges for Azure Databricks, Admin → Connections says so at the top of the page: “Databricks: … on the Azure bill in ”, with a Connect Databricks button. Dismiss for everyone hides it for the whole workspace, and Undo brings it back.
• Databricks support is rolling out. If the Databricks card in the Add-connection picker is disabled and reads “Not enabled in this deployment”, there is nothing to set up yet; it opens when it is ready.

What leancosts reads, and what the principal can do

Read this before you create the secret. The connect form repeats it.

• What we read. system.billing.usage and system.billing.list_prices for DBU and their published rates, and system.compute plus system.lakeflow for the cluster, warehouse and pool inventory and how much of it actually ran. If you grant it, system.query.history for what each SQL warehouse’s statements did per day: how many ran, how many did useful work, how many failed, and the error class of the failures (for example INSUFFICIENT_PERMISSIONS). Never the statement text, never the error message, and nothing about a query’s results.
• What the principal can do. leancosts issues SELECT statements only, over the one SQL warehouse you name, and its HTTP client is held to that by a test that fails the build if it ever gains another verb.
• How the secret is stored. AES-256-GCM encrypted, the same primitive Azure service-principal secrets use, and never logged or echoed back.

1. Create the service principal

In the Databricks account console, open User management → Service principals → Add service principal, give it a name, and note its Application ID. That id is what you paste into leancosts as the client id.

Under the principal’s Entitlements, grant exactly two:

• workspace-access
• databricks-sql-access

No cluster, job or pool permission is needed, and none should be granted.

2. Create the OAuth secret

On the same service principal, open Secrets → Generate secret. Copy the secret once; Databricks will not show it again. It starts with dose.

Caution

Note the expiry you chose. The OAuth API does not report a machine-to-machine secret’s expiry, so leancosts cannot read it back the way it does for Azure. Type it on the connect form and leancosts counts it down for you, always labelled as entered. Rotate before the day, or the sync stops with an authentication error.

3. Grant the system schemas

Run this in a SQL editor in the workspace you will nominate as the query workspace, replacing the id with your principal’s application id:

GRANT USE SCHEMA, SELECT ON SCHEMA system.billing TO `<application-id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.compute TO `<application-id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.lakeflow TO `<application-id>`;
-- Optional: lets leancosts flag a SQL warehouse that bills while none of its
-- statements do useful work. Without it everything else still works.
GRANT USE SCHEMA, SELECT ON SCHEMA system.query TO `<application-id>`;

If system.query does not exist in your account, a metastore admin enables it like the other system schemas, or you skip that line.

4. Grant the SQL warehouse

Open the warehouse you want the statements to run on, then Permissions, and give the service principal CAN USE. Copy the warehouse id from the URL or from the warehouse’s connection details.

That is the whole grant list: the two entitlements, SELECT on three system schemas (four with the optional system.query), and CAN USE on one warehouse.

5. Add the connection

1. In leancosts go to Admin → Connections and click Add connection.
2. Pick Databricks.
3. Fill the nickname, the client id (the application id) and the client secret.
4. Add one row per workspace: its host and the label you want to see on the Databricks page. An Azure host looks like adb-1234567890.7.azuredatabricks.net; Add workspace appends another row.
5. Choose the query workspace from the list. This is the one that runs the statements, so it is the workspace where you ran the grants above.
6. Paste the SQL warehouse id, pick a sync schedule, optionally type the secret’s expiry date, and click Create connection.
7. The row validates in the background: leancosts authenticates the principal on every host you listed, checks the three system schemas are readable, and checks the warehouse exists. A stopped warehouse is fine; the message says it will start on the first sync.

Verify

1. Click Test on the row. Success names every host it reached, confirms system.billing, system.compute and system.lakeflow are readable, says whether the optional system.query is readable, and reports the warehouse’s state.
2. Click Sync on the row, then Open and the Console tab. You should see six phases: the warehouse and list prices, the usage window, the daily usage in chunks, the compute inventory, the ledger bridge, and the summary with the row counts.
3. Open Databricks in the sidebar. The Total band should show DBU and list cost for the last 30 days, and the Breakdowns band should name your workspaces, products and compute objects.

Reading the money

Every figure on the Databricks page is list price: DBU times the published rate from system.billing.list_prices. It is not an invoice, and it does not include any discount you have negotiated.

Note

For an Azure workspace your invoice is on the Azure bill, which leancosts already ingests through the Azure connector. The page shows the two side by side as a reconciliation, with the ratio between them. Never add them - they are two measurements of the same spend. Billed running above list is normal: a serverless SKU’s price includes the virtual machine underneath it.

For an AWS or GCP workspace that Databricks invoices you directly, those rows become cost lines in leancosts under the Databricks provider and appear in your normal cost views.

Note

If you bought Databricks through your cloud’s Marketplace (the AWS Marketplace, or the Google Cloud Marketplace, which is the normal way to buy Databricks on Google Cloud), that cloud invoices it. Once your AWS or Google Cloud connection brings that charge into leancosts, no Databricks cost line is written for the workspaces on that cloud from the charge’s first day on - counting both would double the money - and the page adds an AWS Marketplace reconciliation or Google Cloud Marketplace reconciliation band showing list beside what that cloud charged, for every workspace on it at once. The same rule applies: compare them, never add them. If the bill is in a currency other than USD the band names the currencies and shows no ratio, because list price is in USD.

Until that cloud is connected, those workspaces’ list cost is written as Databricks cost lines, because nothing else in leancosts carries the money. Once the charge arrives, the next Databricks sync removes the lines it had written for those days. If you later remove that connection, the next Databricks sync writes those days as Databricks cost lines again, so the money stays in your cost pages.

The cut-over is the day the Marketplace charge first appears on your cloud bill, not its month. If that is the 18th, the 1st to the 17th stay in your cost pages under Databricks, and the band tells you it covers the 18th onwards only. The band also stops at the last day both the bill and the usage data are complete, because the latest day of each is still filling in, and it names that day.

With two Databricks connections that both have usage on the same cloud, nothing can say which one the Marketplace charge belongs to, so nothing is held back and the band warns you that the total may be counting the same spend twice.

A SKU with no published list price contributes no cost rather than zero, and the page counts how many were unpriced. Money we do not know is never shown as free.

If something is missing

• A host fails validation. The message lists every host with OK or FAIL and the first error. Check the principal exists in that workspace and that the host is spelled as Databricks shows it.
• A system schema is unreadable. The message names the missing schemas. Run the grants from step 3 in the query workspace.
• No usage appears. The first sync goes back 90 days by default; check the console for the window it read, and remember that system.billing.usage is appended late for serverless SKUs, which is why the last few days are always re-read.
• The same principal is already connected. leancosts refuses a second connection on it: one principal reaches the whole account, so two connections would count every workspace twice.