← Blogs

What Data Engineering Is, How We Got Here, and Why AI Depends on It

How data engineering works, how the field moved from one database server to shared data lakes, and why AI systems depend on the data path underneath them.

Ask a chatbot about a company’s revenue growth and you’ll get a confident answer. Whether that answer is right depends on work most people never see: which data sources were collected, how each record was matched to the right company, when the data last changed, and whether the system could tell a current figure from a stale one. That work is data engineering, and in the AI era it matters more than it ever has. A model can only reason over the data it’s given.

I’ve worked with data for years as a data engineer, and more recently on using AI to clean and enrich messy data. The lesson I keep relearning is simple: the model is only as good as the data you feed in. At the 2026 Databricks Data + AI Summit, I presented how my team at YipitData moved entity extraction from a growing set of regex rules to AI agents. The customer story reports that company coverage grew about 20x, from roughly 3,000 to 60,000 in one quarter. Most of the effort went into the data around the model, not the model itself. I’ll come back to that example below.

What data engineering is

Data engineering is the work of moving data from where it is created to where people and systems use it, and keeping it accurate, timely, and understandable along the way. Analysts and data scientists ask questions of data. Data engineers make sure the data exists, has the right shape, arrives on time, and means what people think it means.

The work comes down to five jobs:

  • Ingest: get data out of applications, databases, files, and event streams.
  • Store: keep it in systems built for the reads and writes people actually do.
  • Transform: clean, join, and reshape it into forms others can use.
  • Serve: make it available to dashboards, applications, and models.
  • Govern: track where data came from, who can access it, and whether it can be trusted.

Before the big data era: one server, one bill

For a long time, a company’s data lived in one relational database, and that database did nearly everything. It took in transactions, stored the records, and ran the reports. For reporting, many teams copied data into a separate warehouse on a nightly schedule, and extract, transform, load (ETL) became a discipline in its own right.

The limit was hardware. Storage and compute lived in the same machine, so when reports got slow as data grew, the fix was a bigger server. You paid for storage and computation together, and you paid for peak load all the time. Analysts waited on overnight jobs, and a question that needed a new column could mean a long wait.

Slow answers were the everyday problem. The answers came too slowly to be useful, and that’s what pushed the industry to change.

From MySQL to PostgreSQL to Parquet

The first analytics stack ran on MySQL. MySQL stores rows together, which suits transactions - fetch one customer’s record, update one order. Analytics asks a different question. It usually reads a few columns across millions of rows, such as average order value by region. A row store has to read whole rows to answer that, and most of what it reads goes unused.

We moved to PostgreSQL for scale and better support for analytics. It’s worth being precise about this step. PostgreSQL also stores tables row by row by default, so the move alone didn’t change how the data was laid out on disk. It gave us a more capable engine for the analytical work we were doing, and it was a step toward something better.

The bigger change came with Parquet, a columnar file format. Parquet stores each column’s values together. An aggregation reads only the columns it needs, and values of the same type sit next to each other, which compresses well. Parquet is a file format, not a database. Engines read the files, and the columnar layout lives inside them.

When we moved to Parquet, the aggregations we ran most often got faster. Columnar layout isn’t a win in every case. Workloads that mostly fetch whole records can get little benefit from it. A search for one row might also be slower than using a row base database. Choose the layout by the queries you run.

Parquet arrived as we moved onto object storage, which is the next step.

Why move to the cloud data lake

Three reasons drove the move off on-premises warehouses.

  • Elasticity. Heavy jobs need a lot of compute for a few hours. In the cloud, you can scale up for that job and scale back down afterward. On-premises, you buy for peak load, and the hardware sits idle the rest of the time.
  • Cost of idle hardware. Capacity bought for peak costs money whether it’s used or not. Paying for what you run is easier to justify and easier to change.
  • A shared data lake. One copy of the data can serve many applications and teams. Instead of each team extracting its own copy, they read from the same files.

This pattern shows up across public migrations. GoDaddy moved about 500 TB from an aging Hadoop cluster to Amazon S3 so production jobs could run on EMR. Netflix engineers have described, in conference talks, using S3 as the storage layer of their data warehouse with Parquet files, so that compute and storage could scale independently. By 2018 they reported that the warehouse had grown past 100 PB. The GoDaddy case study is vendor-published and the Netflix accounts come from the company’s own engineers, so read both as the companies’ own telling. The motivations match what we saw.

Object storage made the lake practical: cheap, durable, and independent of any one compute engine. That separation is the key idea behind everything that follows.

Data lakes and lakehouses

A data lake stores raw data in open files, usually on object storage, without deciding everything about it up front. That flexibility is the appeal. The risk is the data swamp. Files pile up with no schema enforcement, no transactions, and no clear record of what changed. Deleting one customer’s records means rewriting files and hoping nothing reads them mid-change.

A lakehouse adds a table layer on top of those files. Table formats such as Delta Lake, Apache Iceberg, and Apache Hudi track which files make up a table, record each change as a transaction, and handle schema changes and time travel. The result is database-like behavior on files you own. The Databricks team made the case for this design in a 2021 CIDR paper, “Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics.”

Parquet is the file format. Delta, Iceberg, and Hudi are table formats that sit on top of Parquet files. Since we are a big databricks customer, we use Delta and in the past, I’ve dug deeply into the implementation of the delta metadata and operations under the hood. The way that delta stores, compact, and optimize the data is critical especially when processing large data.

The payoff is separation in practice. The same Parquet or Delta files can be read by Databricks running Spark and by Amazon Athena, a serverless SQL service that queries files in S3. Two engines, one copy of the data. The data outlives any one tool.

Databricks or Snowflake: open versus locked in

When we chose a platform, the comparison was Databricks against Snowflake. Snowflake deserves a fair description. It separated compute from storage early, and its 2016 paper describes an elastic warehouse that keeps data in cloud object storage while compute scales on its own. So the real difference was never coupled versus separated.

The difference was control over the format and the engine. Snowflake stored data in its own managed format, and you queried it through Snowflake’s engine. Your data was accessible, but not directly as open files that other engines could read. Moving out meant unloading and reloading. Snowflake later added support for open Apache Iceberg tables, which narrows that gap. When we were deciding, the storage format was proprietary.

Databricks was built around Apache Spark, an open-source engine, and Delta Lake, an open table format. The data sits in our own cloud storage in open files. Other engines, including Athena, can read it, and compute can change from one workload to the next. Today both Databricks and Snowflake have worked their way to providing features that each other have and competing in platforms for data and AI.

Open isn’t free. Open formats don’t remove lock-in from catalogs, managed features, or the platform’s own tools. Running a lakehouse also takes more operational work than a fully managed warehouse. So the open choice was a trade: more responsibility for us in exchange for keeping the data portable and not tied to one vendor’s engine.

For a shared data lake serving many applications, that trade was the right one for us. The data had to outlast any single tool. This is our reasoning at the time, not a verdict on either product today.

ETL and ELT

ETL transforms data before it reaches the warehouse. It made sense when warehouse storage and compute were expensive and limited. Transformations ran on a separate server, and only cleaned data was loaded.

ELT reverses the order. You load the raw data first, then transform it inside the lake or warehouse with its compute. It works because storage is cheap and compute is elastic. It has another advantage that matters more than people expect: you keep the raw data. When the business logic changes, you can rebuild the curated tables from the source instead of re-extracting from systems that may no longer have the history.

ELT has costs too. Raw data often contains personal information, so governance has to start at ingestion. Transformation logic becomes a codebase that needs tests and review, which is where SQL transformation tools such as dbt come in.

Most teams use both. The question is where the transformation is cheapest, safest, and easiest to replay.

Why Spark matters

Apache Spark came out of UC Berkeley’s AMPLab. Its 2012 paper introduced resilient distributed datasets, a way to keep intermediate results in memory across a cluster and recompute lost pieces after a failure. Spark was faster than the disk-heavy MapReduce model for many workloads, especially iterative ones. It also gave data engineers one engine for batch SQL, DataFrames, streaming, and machine learning.

For big data, the value is simple. You write a transformation once, and the engine spreads the work across many machines. You don’t manage the distribution yourself. The same code can run on a sample on a laptop or on a cluster over the full dataset.

Spark is central to the work I described above. The talk used Spark user-defined table functions to apply AI models across large batches of records. Distributing model calls across a cluster is a pattern worth knowing for AI pipelines.

Spark isn’t always the answer. For data that fits on one machine, a single-node engine can be simpler and faster to run. Picking the right tool means matching the engine to the data size, not defaulting to the biggest one.

Batch versus streaming

Batch processing collects data over a period and processes it on a schedule, such as hourly or nightly. It’s simpler to build, easier to reason about, and easier to replay when something goes wrong. The cost is latency: an answer can be hours old.

Streaming processes events as they arrive. It’s the right tool when a decision depends on what just happened, such as fraud checks or live operational monitoring. It’s harder. You have to handle events that arrive out of order, late corrections, duplicates, and state that must survive restarts. Kafka popularized the durable event log that many streaming systems read from. At Yipitdata, we don’t really have a use case for live streaming but we do process using structured streaming with spark since we receive data daily. The only streaming infrasture that we have is built on our webscraping application where we are streaming each scrape of our scrapers using Kinesis firehose.

You will probably always start with batch and move to streaming when the business needs lower latency. Many requirements described as “real-time” actually need an answer within minutes, and a well-run batch job can also meet that.

The hard problems in streaming are the same questions you’d ask of any dataset - what does a row represent, how do you recognize a repeat, and what happens when a late correction arrives? Those questions come before the tooling.

Why AI depends on the data path

Models learn from data, and AI systems answer from data. Both depend on the pipeline in between.

Training. A model’s behavior reflects its training data: what was included, what was filtered, how labels were produced, and which duplicates were removed. Those decisions are made in data engineering before the model sees anything.

Retrieval. Most business AI systems that answer questions over company documents use retrieval, which is a pipeline. Documents are collected, parsed (PDFs and tables are hard), split into chunks, embedded, stored, and refreshed. A stale chunk can keep an outdated answer alive. A deleted or restricted document has to stop surfacing. Those failures happen in the data path, not in the model.

Enrichment. The regex-to-agent work in the talk is one example. Entity extraction often needs context to choose between candidates, and that context has to be in the data: reference records, consistent identifiers, and a way to tell when the evidence is missing.

Evaluation and feedback. To know whether an AI system is improving, you need test sets that resemble real inputs, logs of what the system did, and a way to join those logs to outcomes.

Agents. An agent that takes actions needs a reliable record of what has already happened. Without one, a retry can repeat a payment or send the same email twice.

In 2015, a group of Google researchers described hidden technical debt in machine learning systems. In their diagram, the model code is a small box inside a much larger system of data collection, feature extraction, configuration, monitoring, and serving. In most real systems, the model is not the hardest part to build or keep running.

What comes next

Tools change every few years. The questions underneath them mostly don’t. In the world of AI, data becomes even more important and one may argue that proprietary data will be come the biggest moat for companies.