I've also been thinking about orchestrators but I've come to a different conclusion. In my mind the current metadata is so underutilized that once it starts being utilized the orchestration layer becomes much more automated and maybe at some point entirely "disappear" and either get absorbed into the data warehouse layer or the BI layer. You don't need a lot of people writing dbt or creating various metric definitions when those can be derived by looking at the queries that are already running on the data warehouse.
That is an interesting thought. And you might be right, the better we get in handling metadata maybe the orchestration part is changing so significantly that I becomes more irrelevant.
Great post Timo. I have been learning about Mage + dbt, and it seems like a great combination for ingestion, orchestration, and transformation. My main focus in selecting this integration was learning a data stack that can be used with several warehouse options and environments.
This is a substack that talks about this interesting symbiosis:
I've also been thinking about orchestrators but I've come to a different conclusion. In my mind the current metadata is so underutilized that once it starts being utilized the orchestration layer becomes much more automated and maybe at some point entirely "disappear" and either get absorbed into the data warehouse layer or the BI layer. You don't need a lot of people writing dbt or creating various metric definitions when those can be derived by looking at the queries that are already running on the data warehouse.
That is an interesting thought. And you might be right, the better we get in handling metadata maybe the orchestration part is changing so significantly that I becomes more irrelevant.
This is a very interesting point. I see Microsoft Fabric going in that direction (the BI or Data Warehouse layer absorving the orchestration part).
Great post Timo. I have been learning about Mage + dbt, and it seems like a great combination for ingestion, orchestration, and transformation. My main focus in selecting this integration was learning a data stack that can be used with several warehouse options and environments.
This is a substack that talks about this interesting symbiosis:
https://castnetanalytics.substack.com/