There are 2 main philosophies about how to build and deploy an ETL pipeline
One is the “ops driven” approach, Dataops, and the other being “metadata driven”.
With a dataops driven pipeline, every step of the ETL pipeline (extraction, transformation, loading and orchestration step) is written and stored as code.
With a metadata driven pipeline, every step of the ETL pipeline is specified with configuration.
With a dataops based approach, there are no constraints to the process, developers are free to process data in anyway they wish. A metadata driven approach only allows steps that are coded within the “ETL engine”.
The dataops approach is infinitely flexible but is slower to on-board and requires experienced developers to implement.
The metadata approach is more limited to what is coded in the pipeline but is faster to on-board new feeds as there is no code to deploy and doesn’t require skilled engineers. It will also be easier to maintain, as there is only one code base as opposed one code base per pipeline.
If you want consistency, between your pipelines and defined “guardrails” for implementors, the metadata approach is the right one.
When building a Data Platform on a Data Mesh paradigm the metadata approach is the right way to go.