Day 2 • 11:30 R0
AI Coding 不只是工具升級-區域醫院醫療資訊系統協作開發與資訊團隊轉型實踐
ApplicationChinese
Max Lai
原來蟒蛇不會吃鴨子? 一段關於輕量 ETL 分析的故事
蘇揮原 Mars Su
Day 1 • Sat, Oct 17
13:20 - 13:50
Location
R1
Language
Chinese (English slides)
Category • Level
Application • Medium
Ever spun up a Spark cluster just to process a few GB of parquet — only to realize it was complete overkill?
As data engineers, we've long lived between two extremes: pandas is too slow, Spark is too heavy. But there's a duck quietly waiting in that gap — and in 2026, there's also a fast-moving Rust-based challenger, Polars, sitting right next to it.
This talk is grounded in a real migration I led: an existing pandas-based ETL pipeline processing for anomaly analysis was migrated to DuckDB. You'll see a complete ETL pipeline running on a single machine using Python + DuckDB — reading multi-format data directly from Apache Iceberg on MinIO, transforming it with pure SQL, and loading it downstream without Spark cluster
Demo version：DuckDB v1.5.3 + Iceberg extension 0.11.1
I'll also show where DuckDB's Python API goes beyond "just SQL" — zero-copy interop with pandas/Polars DataFrames, type handling across engines.
Description
Why this Session, Why Me?
I am a staff data engineer (Also Tech Leader) and in charge of design data architecture in network anomaly data team. I have introduced duckdb light weight pipeline mechanism so that data analyst & scientist can easy to process production telemetry data with Apache Iceberg OTF Lakehouse for anomaly detection algorithm. They would not be create on-demand spark cluster to extract data, which achieve cost & process effective (From minute level to seconds level).
What This Session Is About
One pip install duckdb. A complete ETL pipeline on a single machine — reading Apache Iceberg in MinIO, transforming with pure SQL, writing to your downstream target for analysis. No JVM. No cluster. No config files.
What You'll Take Away
- Replace the Extract and Transform steps of an existing pandas pipeline with DuckDB in under 30 minutes
- Query Apache Iceberg OTF (Silver/Gold data) directly from object storage (MinIO/S3) via extensions — skipping the download–process–upload cycle.
- A decision framework covering pandas / Polars / DuckDB / Spark
- Python-specific engineering practices around DuckDB: DataFrame interop, type/schema handling, error handling, and how to write tests for SQL-based transforms.
Reference
A Staff Data Engineer in TrendMicro. With over 8 years of experience in data engineering and machine learning. Mainly includes design & implement various data architecture. I am willing to pursue new knowledge and skill and share it so that I can make greater contributions in Data Engineering and ML field in the future.
