Working in customer data platforms: Snowflake and Databricks

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Snowflake · also covers Databricks, Data infrastructure / data platform, Analytics · Step 23 of 34 in AI forward deployed engineer · version 1

Practice this

On this page

Most enterprise customers already keep their analytical data in Snowflake or Databricks, so AI work starts inside an account someone else owns and governs. Both platforms separate storage from compute, but they differ in how compute is provisioned, how data is stored, and how access is granted. A forward deployed engineer needs to read the architecture quickly, work within the customer's roles and budgets, and find the right tables in a warehouse nobody documented.

Read more

How it is done in practice

On arrival the team requests a service identity, a scoped role, a sandbox schema and a small dedicated warehouse or job cluster, then spends the first days exploring metadata and query history with a customer analyst on call. Findings go into a short data map listing trusted tables, their grain, owners and refresh times. Development happens in the sandbox schema against production data read-only, with cost monitored per warehouse or cluster tag. Before go-live the customer's platform team reviews grants, network configuration and expected monthly spend, and production jobs run under the service identity, not under an engineer's personal login.

Technologies

SnowflakeDatabricksDelta LakeUnity CatalogApache SparkBigQueryLooker

Diagram

flowchart TB
  subgraph Snowflake
    S1[Cloud services: auth, metadata, optimizer]
    S2[Virtual warehouse A]
    S3[Virtual warehouse B]
    S4[Managed columnar storage]
    S1 --> S2
    S1 --> S3
    S2 --> S4
    S3 --> S4
  end
  subgraph Databricks
    D1[Workspace: notebooks, jobs, SQL]
    D2[Unity Catalog governance]
    D3[Clusters and SQL warehouses]
    D4[Delta tables in customer object storage]
    D1 --> D3
    D2 --> D3
    D3 --> D4
  end
  APP[FDE application with scoped role] --> S1
  APP --> D1
flowchart TB
  subgraph Snowflake
    S1[Cloud services: auth, metadata, optimizer]
    S2[Virtual warehouse A]
    S3[Virtual warehouse B]
    S4[Managed columnar storage]
    S1 --> S2
    S1 --> S3
    S2 --> S4
    S3 --> S4
  end
  subgraph Databricks
    D1[Workspace: notebooks, jobs, SQL]
    D2[Unity Catalog governance]
    D3[Clusters and SQL warehouses]
    D4[Delta tables in customer object storage]
    D1 --> D3
    D2 --> D3
    D3 --> D4
  end
  APP[FDE application with scoped role] --> S1
  APP --> D1

Code

sql
-- which tables are alive, and how big are they?
select table_schema, table_name, row_count, last_altered
from information_schema.tables
where table_type = 'BASE TABLE'
order by last_altered desc
limit 50;

-- check the grain of a candidate table
select count(*) as row_count,
       count(distinct order_id) as distinct_orders,
       max(loaded_at) as freshest
from analytics.orders;

Subtopics

Snowflake architecture: storage, virtual warehouses, cloud services

Three layers

Snowflake is a managed service with three layers. Storage holds data in a compressed, columnar internal format that Snowflake reorganizes and manages; you reach it only through SQL. Compute is provided by virtual warehouses, independent clusters that each run queries without sharing compute with other warehouses, so a heavy load job does not slow a dashboard. The cloud services layer coordinates everything: authentication, access control, metadata, query parsing and optimization. The design is a hybrid of shared-disk, with one central data repository, and shared-nothing, with massively parallel compute clusters.

What it means on an engagement

Cost follows warehouse size and running time, so ask which warehouse you may use, set auto-suspend, and avoid leaving it running for an idle service. Scale up for a complex query, scale out for concurrency. Since each warehouse is isolated, request a dedicated one for your workload so its spend and performance are attributable.

Databricks: the lakehouse, workspaces and compute

Open storage with a unified platform

Databricks is a unified platform for data engineering, analytics and AI, built around the lakehouse idea: data sits in the customer's own cloud object storage in open table formats such as Delta Lake, and managed compute reads it in place. A workspace is the working environment holding notebooks, jobs, SQL editors and dashboards. Compute comes in several forms: all-purpose clusters for interactive notebooks, job compute that starts for a scheduled run and stops, SQL warehouses for BI-style queries, and serverless options. Unity Catalog provides governance with a three-level namespace of catalog, schema and table.

Practical differences from a warehouse

Expect to work in Python and SQL side by side, with Spark DataFrames for large data. Since files live in the customer's bucket, network paths, storage credentials and table maintenance matter more than in a fully managed warehouse. Choose job compute for production runs; interactive clusters are for development.

Access, roles and governance in an account you do not own

Ask for the narrowest working grant

Both platforms use role-based access control, with privileges granted on securable objects to roles or groups, never to individuals where it can be avoided. Request a dedicated service principal or service user for your application, a role with read access to named schemas only, a separate schema where you may create objects, and a dedicated compute resource. Prefer key-pair or OAuth authentication to passwords. Sensitive columns may be protected by masking policies and row-level filters, which means the same query returns different results for different roles; confirm which role production will run as before validating outputs.

Governance questions to settle early

Establish whether data may leave the platform, whether model calls to an external provider are permitted for these columns, what region the data must stay in, and who approves new grants. Access requests often take longer than the build, so file them on day one.

Use the platform's own metadata

Documentation is usually stale, but metadata is not. Query the information schema for tables, columns, types and row counts, and sort by last altered time to separate live tables from abandoned ones. Query history shows which tables people and tools really read, how they join them, and which filters they apply; the most frequent queries from the BI service account are the best guide to trusted tables. Lineage views, where enabled, show which jobs write each table. Look at view definitions and existing dbt or notebook code before writing your own logic.

Verify before you trust

For a candidate table, check grain by counting rows against distinct keys, check freshness by the maximum load timestamp, and reconcile one known total, such as last month's revenue, against a number the business already reports. Write down what you learn; it often becomes the customer's first real data dictionary.

Running AI workloads next to the data

Bring the model to the data when you can

Customers often prefer that data never leaves their governed platform. Both vendors offer ways to do that: Databricks positions itself as a platform for building and deploying AI alongside data, with model serving, vector search and experiment tracking in the workspace, and Snowflake offers in-account functions and container-based compute for similar purposes. The alternative is an external application that queries the platform through a driver or SQL API and calls models elsewhere. The first keeps governance simple and reuses existing permissions; the second gives more control over models, latency and tooling. Decide with the customer's security team, and whichever path is chosen, read from curated tables with a dedicated role, never from raw landing zones.

Sources

Related lessons

Practice this
Cleaning messy customer data: data mapping and entity resolutionELT transformations with dbt