IN-PERSON: Introduction to the data build tool (dbt)

If you have not used the data build tool yet, you should. It allows companies to transform data via the medallion architecture. Does dbt extract and load data, the answer to this question is not usually.

Most people use a tool like Fivetran or meta data driven pipelines to load the data into the raw quality zone. The seed command can be used to load small static dimensional data that does not change.

In this presentation you will learn about the project, profile, schema, and properties YAML files. Each one is designed for a specific reason. The key to modeling in dbt is the different ways to materialize data. We will be covering seeding, snapshots, tables, views, and materialized views.

The real power of dbt comes from its many adapters. I can run the same model with a target of Databricks SQL Warehouse, Fabric SQL database or Azure PostgreSQL database. Of course, things like dates and currency might different between vendors. We will learn how to use macros to overcome these differences.

Finally, documentation is always lacking in these systems. We will learn how to use Claude Opus to take those database schemas and guess at descriptions. In the end, the tool allows the developer to create documentation on all the models.

Does dbt scale for big data. It depends on the target system. A friend of mine is working with 150 TB of data , 8000 models, and complete documentation for a Snowflake Warehouse that is full of oil and gas information.