Day 1 • 14:00 R1
打造企業級的金融 AI 服務鏈: Python 串起 ML、XAI 與 LLM 的實戰經驗
FinTechChinese
游騰林
Avoiding Zero-Trade Policies in RL with a Decoupled MLOps Architecture
Sho Tanaka
Day 2 • Sun, Oct 18
11:45 - 12:15
Location
R2
Language
English
Category • Level
FinTech • Medium
Reinforcement Learning (RL) looks perfect for algorithmic trading until you add reality. In 75+ Python experiments (mainly DQN and PPO), adding transaction costs and switching rewards to realized PnL repeatedly produced a stable but useless policy: never trade. I call this failure mode "Zero-Trade Collapse". In this talk I'll show why it happens and the pragmatic fix: stop making RL micromanage exits. Split the system into (1) a signal model that outputs a continuous confidence score and (2) a deterministic, unit-testable execution engine. This boundary lets you compare RL vs classic baselines apples-to-apples and spot overfitting early.
What you'll learn
- Early warning metrics for buy-and-hold traps and zero-trade equilibria
- How to iterate safely across dozens of experiments (reproducibility + refactoring; tool-agnostic LLM assistance)
Description
Building on 75+ Python experiments with DQN and PPO, this talk examines Zero-Trade Collapse: a failure mode where RL agents learn never to trade because holding no position minimizes risk under shaped reward functions.
How the collapse happens The collapse is triggered by realistic reward shaping: once transaction costs are added and rewards are switched to realized PnL, every trade tends to produce a negative return. The agent quickly learns that not trading avoids negative PnL — and a "never trade" policy becomes the stable optimum. The failure is visible in the realized PnL curve going negative during early training, followed by the agent converging to zero trades.
The architectural fix — a clean Python module boundary
The fix is not another reward tweak. It is a separation of concerns at the Python module level:
- Signal Generator: RL model (DQN/PPO) outputs a continuous confidence score (0.0–1.0), not a discrete trade action.
- Execution Engine: a deterministic Python module with four classes:
Position,Order,RiskManager,ExitPolicy.
A non-alpha reference implementation of the execution engine will be published on GitHub before the conference.
A Lead Developer Advocate at Snowflake, focused on AI/ML and data engineering. He previously worked at Google (gTech) delivering ML/Data solutions across Japan, APAC and global. Google Developer Expert (AI/ML) and a co-founder of the MLOps community in Japan, where he has organized monthly events for over five years for a growing network of 6,000+ members. He enjoys turning messy real-world ML projects into reproducible, production-minded architectures.