From CPython to PySpark

For people who have never worked on large open source projects, these projects can feel a little mysterious. We use them every day, but many people do not know how their developers joined the project, what they usually work on, or what someone can do to become part of the community.

I first started to contribute to a large open source project in 2022 and became a Python Core Developer in 2024. In 2026, I became an Apache Spark Committer. Through my own story, I hope to make these large open source projects feel less mysterious. I also hope these experiences can give people who want to join similar projects a practical path to follow.

In this talk, I will first look back at how I went from being just a programmer to a Python Core Developer. I will share how someone with no previous open source experience can start contributing to an open source project.

Then I will talk about the work I have done on PySpark over the past year. Spark started as a Scala project, but its Python part, PySpark, is both important and still rough in many places. As a programmer brought in specifically to improve the Python experience, what kinds of things can I work on?

Many of these tasks are things you may also be able to do, whether in your own project, in Spark, or in another large open source project. Understanding this kind of work can help you find a good place to start contributing. It can also help you improve the products built by you or your company.