Photo by Austin Hervias on Unsplash
Engineering project
Financial data-pipeline architecture
Design the processing workflows and infrastructure that turn financial data into models for analysis.
The project
Data-pipeline architecture work on Google Cloud for an international financial-data and index provider. The scope connects workflow orchestration, data transformations and cloud resource definitions.
Engineering approach
SQL models define the tables and metrics produced from source data. The workflow engine schedules their execution and follows their dependencies. The accompanying Google Cloud architecture shows how model execution, orchestration and storage interact.
Work delivered
A pipeline architecture combining Airflow orchestration, dbt transformations, BigQuery analytical storage and Terraform resource definitions.
My contribution
- Architecture of data-processing chains using Airflow, dbt and BigQuery
- Coordination of workflow orchestration, SQL transformations and analytical models
- Infrastructure definition and organisation with Terraform
Related reference architecture
A Google Cloud platform brings market feeds and information extracted from financial documents into a BigQuery warehouse. One application is building a history of financial instruments and portfolios for analysis in Power BI and use by applications through APIs.
Apache Airflow
BigQuery
Cloud Run
Cloud Storage
dbt
GitHub
Power BI
Terraform
Vertex AI
Components and flows
- Source files retained in Cloud Storage and loaded into raw BigQuery tables
- dbt transformations and data quality checks before publishing fact and dimension tables
- Document extraction through Vertex AI, analysis in Power BI and application access through Cloud Run APIs
Discuss your project
A discussion of your project’s design, development and production deployment.