Lakehouse BRD — Chapter 2: Proposed system architecture

2.1 Logical architecture overview

The Lakehouse system follows a standard 7-layer architecture, supporting the flow from ingest to consumption and AI/BI:

Layer	Main function
1. Ingestion Layer	Connect and collect data from source systems (database, API, file, streaming)
2. Processing Layer	Clean, normalize, enrich, transform (batch and real-time)
3. Storage Format Layer	Columnar storage, versioning and schema evolution (Parquet, Delta, Iceberg)
4. Storage Layer	Physical storage (S3/MinIO), partitioning, query optimization
5. Metadata & Catalog	Schema, lineage, data description, and access control
6. Query Layer	High-performance query for dashboard, AI, API
7. Consumption Layer	BI (Superset, Power BI), partner API integration, ML/AI

2.2 Physical architecture diagram

Loading diagram...

• The system can run on-prem, hybrid, or fully in the cloud.
• Data is split into raw, staging, curated, and analytics.

2.3 Processing model: Batch and streaming

Pipeline type	Characteristics	Suggested stack
Batch	Hourly/daily ETL from core systems	Airbyte + dbt + Spark
Streaming	Real-time data (scoring, transactions)	Kafka + Flink + Iceberg

Technology complexity per layer:

▲ Complexity score (0–100) — Processing & Query layers require the most tuning

2.4 Sample pipeline: CRM → Lakehouse → BI dashboard

1. Airbyte ingests from PostgreSQL CRM (CDC).
2. Spark processes and normalizes customer data.
3. Data is stored in MinIO as Iceberg (curated layer).
4. DataHub records metadata and lineage.
5. Trino queries build materialized views.
6. Superset serves reports.

Loading diagram...

Layer	Example
Raw	crm_raw.customer_profile
Staging	crm_stg.customer_profile_cleaned
Curated	crm_cur.customer_profile_enriched
Analytics	crm_ana.customer_segmentation_dashboard

Data quality progression across layers:

2.5 Environment configuration

Environment	Purpose	Data policy
DEV	Pipeline testing, validation	Anonymized data
STAGING	UAT, close to PROD	Full data, delayed timing
PROD	Production operations	Real data, audit and control

CI/CD: deploy pipelines via CI, rollback on failure, version tags and full change logs.