What I Learned (and Broke) Migrating Large-Scale Production Data From SQL to NoSQL via PySpark

PITI CHAMPEETHONG

Yuichiro Tachibana
Python programs often rely on invisible "current" state: the current request, user, transaction, directory, runtime, and so on. In synchronous code, this may look simple, but in async applications, web frameworks, and systems running multiple logical executions in one Python environment, it quickly becomes confusing. This talk explains Python's contextvars module practically. We will start with why thread-local state is not enough when multiple asyncio tasks share one thread, then look at logging and tracing as typical use cases. The main case study comes from Stlite, an in-browser Streamlit runtime powered by Pyodide/WebAssembly, which I created and maintain. Based on this experience, I will show how contextvars helps model runtime problems and how to design around global state.
Python has many ways to represent "current" state: globals, threading.local(), request-local values, tracing spans, transactions, the current directory, or the current runtime. These ideas are easy to name, but hard to make correct when multiple async tasks share one Python thread or runtime.
The contextvars module, introduced by PEP 567, provides a mechanism for context-local state. It is often explained through examples such as request IDs in logs, but the underlying idea is more general: it lets code carry invisible execution-local state across async call chains.
In this talk, I will first explain the mental model of ContextVar, Context, and task context propagation. Then I will show common examples such as request-local logging, tracing, and framework-level state.
The case study is Stlite, a Streamlit runtime that runs in the browser on Pyodide/WebAssembly, which I created and maintain.
Although Stlite is the concrete example, the lesson is not Stlite-specific: if you build any runtime, framework, or library where multiple logical executions share one Python process, you need to define what "current" means. In Stlite, multiple logical async web servers may share one Python environment and thread. APIs such as the current working directory and environment variables are still global, so Stlite uses ContextVar to remember task-specific runtime information and a coroutine proxy to apply the right directory state when a coroutine is resumed.
This case study shows both the power and the limitation of contextvars. It is based on practical knowledge I gained while designing and debugging Stlite’s runtime behavior. contextvars is useful for modeling logical execution context, but it does not automatically solve problems caused by global APIs. When the underlying API is global, we still need to wrap, restore, or avoid global side effects explicitly.

Yuichiro is a professional software developer with a deep passion for open source software. After working on software development for several machine learning startups, he joined Hugging Face in 2023. In 2026, he joined the Research Center for Advanced Science and Technology, The University of Tokyo. He develops and maintains several OSS projects, including Streamlit-WebRTC, Stlite, and Awesome Emacs Keymap, and contributes to other OSS projects such as Streamlit and Gradio. He has also been actively involved in the global PyCon community, attending PyCons around the world and sharing his work through talks at several Python conferences.