Each component in the Data Platform requires a choice in the technology. Each choice will depend on the use-cases you have to deal with. And each layer could be purchased, free software or developed in-house.
The options here are not exhaustive, merely some popular options
Storage Layer
The Storage Layer choice is crucial for a Data Platform and d Data Platform doesnt have to only have on choice of Storage. There may be reasons for selecting different engines for different purposes.
Choices : Snowflake, Databricks, Iceberg Tables, Delta Tables, Google Big Query, OneLake
Orchestration
Choices : Airflow, Dagster, Prefect, Azure Data Factory
Ingestion
Choices : Azure Data Factory, Fivetran, Matallion, AWS Glue
Transformation
As well as tools that specifically carry out the transformation processing, its worth including the current EDM tools in the choice especially if your organisation already uses it and/or has expertise in it. For an EDM to fit in the Data Platform eco-system, it should be able to run transformations locally or push the transformation to the data.
Choices : DBT, Snowflake Openflow, Fivetran, Talend, Matillion, AWS Glue, Azure Data Factory
EDM : Curium Markit EDM, GoldenSource
Data Quality
Choices : Precisly
Data Governance
Lineage Choices : Operlineage.com, DBT, Snowflake
Data Dictionary Choices : Atlam, Dataedo, DataHub
Data Access
REST API : Fast API
Marketplaces : Snowflake Marketplaces, Databricks Marketplaces
Computation layer
Choices : Snowflake & Snowpark, Databricks
Summary
| Layer | Any Cloud | Cloud Specific | |
| Storage Layer | Snowflake, Databricks | Snowflake, Databricks | |
| Orchestration | Airflow, Dagster, Prefect | Azure Data Factory | |
| Ingestion | Fivetran, Matallion | Azure Data Factory, AWS Glue | |
| Transformation | DBT, Snowflake Openflow, Fivetran, Talend, Matillion, | AWS Glue, Azure Data Factory | |
| Data Quality | Precisly, Curium, Markit EDM | ||
| Data Governance : Lineage | |||
| Data Governance : Catalog | Atlan, Dataedo, DataHub | ||
| Data Access | |||
| Computation layer | Snowflake/Snowpark, Databricks |