Is it Just Markdown? What AI Agents Actually Do for Data Engineers

Every week brings another story about someone running a thousand agents, followed by a course promising to make engineers 10x more productive. Meanwhile, most data teams are still maintaining old pipelines, investigating broken jobs, and trying to understand undocumented systems.

This talk examines which parts of data engineering agents can handle reliably today and looks at what an AI agent actually consists of: a model, context, tools, an execution loop, and often a surprising number of Markdown files.

I’ll demonstrate real workflows, including using agents with Estuary to create, monitor, and debug data pipelines, as well as agents working across upstream and downstream systems. I’ll also show where these setups failed, required human intervention, or created more work than they actually saved.

The productivity gains are definitely real, but uneven as agents are useful when the task is bounded, the tools are well defined, and the result can be verified but they struggle when context is missing, ownership is unclear, or correctness depends on knowledge that exists only inside someone’s head. For many teams, the highest-value application may not be generating another pipeline, rather, it may be understanding, documenting, and gradually repairing the pipelines they already have.

Sponsored by: Estuary https://estuary.dev/