Day 1 • 11:30 R2
The hidden current context: understanding `contextvars` through real-world runtime problems
Best Practices & PatternsEnglish
Yuichiro Tachibana
What I Learned (and Broke) Migrating Large-Scale Production Data From SQL to NoSQL via PySpark
PITI CHAMPEETHONG
Day 1 • Sat, Oct 17
10:50 - 11:20
Location
R2
Language
English
Category • Level
Best Practices & Patterns • Medium
Moving production applications from relational SQL to NoSQL document models demands a fundamental shift in data modelling. At scale, standard row-by-row scripts fail, forcing teams to rely on distributed engines like PySpark. However, without careful tuning, PySpark's parallel efficiency can easily overwhelm target clusters, saturate legacy database connections, or cause severe memory overhead.
In this talk, we present a practical post-mortem of a large-scale data migration from on-premise Microsoft SQL Server to MongoDB Atlas. We deep dive into using PySpark on Google Cloud Platform (GCP) to transform flat relational tables into optimized, nested NoSQL structures. We will demonstrate how to leverage the DataFrame API (struct, array, collect_list) to map relational data natively, optimize partition parallelism to prevent write-throttles on Atlas clusters, and tune JDBC read parameters to safely handle live SQL instances without impacting system uptime.
Description
This session presents a practical post-mortem of a large-scale data migration from on-premise Microsoft SQL Server to a scalable MongoDB Atlas document store. While the migration stack is specific, the engineering patterns we discuss, including PySpark partition tuning, managing JDBC read saturation, and avoiding the missing index trap, are applicable to any distributed database migration.
Rather than presenting an idealized happy path tutorial, this talk provides a transparent look at real-world production challenges. We will analyze how we navigated massive structural data translation and infrastructure scaling bottlenecks without impacting active system uptime.
Third-party Libraries & Core Technologies:
PySpark: Utilized as the high-throughput engine for data extraction and complex schema manipulation. PySpark SQL Functions: Leveraged to transform relational rows into flexible, high-performance nested sub-documents.
I've been working with databases and software development for 20 years. I'm currently a senior MongoDB consulting engineer based in Singapore. I've previously spoken at conferences such as PyCon Lithuania, PyCon APAC/Asia, PyCon SG, PyCon Thailand, and Global Azure Thailand. I’m also part of the community leader team for the MongoDB and PyLanna (the Python) User Group in Thailand, which brings together over 3,000 developers. Piti is also a Microsoft MVP (Developer Technologies), recognised for his expertise and valuable contributions to the developer community.
