Last week Databricks announced the beta of FILE, a native column type that brings documents, images, audio, and video into your lakehouse tables with the same governance you already apply to structured data. It is a small syntax change and a large architectural shift: unstructured data stops living in a separate object store bolted on the side, and becomes something you can SELECT, secure, delete, and feed to AI functions like any other column.
If you have spent the last few years stitching PDFs and screenshots into ad-hoc pipelines, this is the post for you. We will look at what FILE actually stores, the two flavors you can declare, what it unlocks for AI workloads, and where the ecosystem (BigQuery, Lance, Snowflake) is heading in parallel.
The problem: unstructured data has always been a second-class citizen
Most data estates are mostly unstructured. Contracts, product images, call recordings, and video outweigh metrics and transaction logs by a wide margin — but they never get the same treatment. The old playbook was to keep files in cloud storage and store a URL (or worse, the raw bytes) in a table:
A STRING column holding an object key gives you zero governance. Permissions on the table and permissions on the bucket are two different models that drift apart, and deleting a row rarely deletes the file — leaving orphaned data behind.
A BINARY column inlines the bytes, which works for thumbnails up to ~64 KB but collapses under gigabyte-scale video or full contract scans.
So teams defaulted to the real old playbook: store files in one system, build a separate pipeline to parse them into another system, and manage a third index. The Databricks announcement is a bet that this fragmentation is the bottleneck for AI: AI turns unstructured data into something queryable, but only if it is governed and managed alongside everything else.
What FILE actually stores
FILE is not a blob type. It stores a governed reference to a file plus metadata — the file bytes stay in object storage and are read lazily, only when an AI function or UDF actually processes them. A FILE value has five fields:
| Field |
Type |
Meaning |
uri |
STRING |
Location of the file (cannot be null) |
offset |
BIGINT |
Byte offset into the file |
size |
BIGINT |
Size in bytes |
content_type |
STRING |
MIME type |
checksum |
STRING |
Integrity token, e.g. MD5:<digest> or ETAG:"..." |
Because the value is a reference, metadata queries (SELECT file.uri, file.size FROM ...) never touch the actual bytes. That is what keeps query performance intact even when your table references gigabytes of media.
Databricks positions this deliberately against BINARY:
Use FILE when you need to manage and process large unstructured files alongside structured data and pass them to AI functions.
Use BINARY for small objects (up to 64 KB by default) where you want the bytes inline — a thumbnail stored with its row, for example.
FILE EXTERNAL vs FILE MANAGED
You cannot declare a plain FILE column — you must choose one of two lifecycle flavors:
FILE EXTERNAL references files that already exist in a Unity Catalog volume. Nothing is copied, other tools reading the same paths keep working, and permissions are governed by the volume (READ VOLUME). You manage the files' lifecycle yourself — deleting a row does not touch the underlying file. This is the choice when you must not move data or disrupt existing readers.
FILE MANAGED copies files into managed storage (a "FileSpace" volume you declare on the table). Lifecycle is tied to the table: deleting rows makes the referenced files eligible for garbage collection, which is exactly the behavior you want for compliance flows like GDPR right-to-be-forgotten. It costs you a copy, and it is the recommended flavor for ML training and retrieval-augmented generation, where workloads reach the files through the table.
-- Reference files already in a volume, without moving them
CREATE TABLE attachments (
id BIGINT,
document FILE EXTERNAL
);
-- Populate from files that already exist in a volume
INSERT INTO attachments
SELECT row_number() OVER (ORDER BY file.uri), file
FROM read_files('/Volumes/my_catalog/my_schema/my_volume/', format => 'file');
For rows that need the bytes managed by the table:
ALTER TABLE reports ADD COLUMN attachment FILE MANAGED;
One practical note from the docs: automatic garbage collection is not supported in beta — you need a cleanup job for unreferenced managed files. Read the garbage-collection guidance before you assume deletes tidy up after themselves.
Constructing and consuming FILE values
A handful of functions create FILE references: to_file(path) and try_to_file(path) build a reference from a path (returning NULL if the file is missing), create_file() uploads content, and copy_file() copies to a target. You can also cast from VARIANT or a STRUCT with exactly the fields struct<uri:string, offset:bigint, size:bigint, content_type:string, checksum:string>:
SELECT named_struct(
'uri', '/Volumes/my_catalog/my_schema/my_volume/report.pdf',
'size', CAST(19494 AS BIGINT),
'content_type', 'application/pdf',
'checksum', 'ETAG:v1'
)::FILE;
On the consumption side, the interesting part is ai_parse_document(file), which extracts structured content from a file, and the fact that FILE values flow through SQL, Python, and Scala UDFs. Databricks' driving-clips example shows the shape of the future: each row pairs a video FILE column with structured columns (route, scene description, hazard label, embedding) — a single row holding both the media and its derived meaning.
The roadmap doubles down on that: versioning and cloning without copying binaries, streaming datasets directly into PyTorch for GPU-ready tensors, feature engineering (embeddings, classifications) without rewriting tables, and direct search over the table using vector and full-text indexes.
Governance is the real headline
The most important line in the announcement is not the column type — it is that Unity Catalog row- and column-level access control now extends to raw files, and that lifecycle is unified. Delete a row that references a FILE MANAGED value and the file becomes eligible for cleanup, which is the compliance behavior (right-to-be-forgotten) that previously required custom reconciliation jobs between your table and your bucket. FILE type is being developed as an open standard with work underway to integrate it into the Parquet and Delta Lake formats — the explicit goal being portability across the ecosystem rather than lock-in.
The wider race: the multimodal lakehouse
Databricks is far from alone in diagnosing this problem. The July 2026 landscape piece "The Multimodal Lakehouse" maps the field:
Google Cloud announced (April 2026) a cross-cloud lakehouse built on managed Apache Iceberg plus BigQuery ObjectRefs, merging structured Iceberg tables with unstructured Cloud Storage data for unified multimodal analysis — the same gesture as FILE type, from the warehouse side.
LanceDB / Lance argues vector-plus-metadata deserves its own AI-native file format, positioning Lance as a lakehouse for multimodal data rather than a siloed vector database.
Snowflake is extending its warehouse dominance into lakehouse and multimodal territory; Microsoft Fabric is unifying lakehouse, warehouse, BI, and governance under OneLake; Dremio is pitching an "agentic lakehouse" with a semantic layer so AI agents query governed data directly.
The convergence is telling: everyone has concluded that unstructured data must be managed as data, with the same access control, lifecycle, and governance as tables — not parked in an adjacent service.
What it means for data teams
If you are on Databricks with Delta Lake on Runtime 18 LTS+, the practical takeaways:
Start with FILE EXTERNAL. It does not move data or disrupt existing readers, so it is the safe first step for pointing governed references at files that already exist in volumes.
Use FILE MANAGED when the table owns the data. Compliance workflows and RAG/ML workloads that reach files through the table are where the lifecycle guarantees pay off — but plan a cleanup job, because automatic GC is not here yet.
Don't bother parsing everything up front anymore. Lazy loading plus ai_parse_document means you extract meaning when a query demands it, and Declarative Pipelines can do incremental processing so only new or modified documents hit expensive model API calls.
Watch the open-standard angle. If FILE lands natively in Parquet and Delta Lake, the "reference + metadata" pattern becomes the shared substrate for unstructured data across engines — the same way Delta/Iceberg became the substrate for tables.
Structured and unstructured data sharing one governed system was the hardest part of making AI production-ready. With FILE type, Databricks has made that a column declaration. Expect the other lakehouses to answer in kind — the multimodal lakehouse is coming, and it is being built out of columns, not bolt-on indexes.
Sources
