Day 2 • 13:20 R1
The Agent Failure Playbook: Production Gone Wrong
Machine LearningEnglish
Kushal Vijay
Learn Japanese with Manga! Building a Multimodal Search Engine with Gemini Embedding 2
@x0mg
Day 2 • Sun, Oct 18
10:50 - 11:20
Location
R0
Language
Chinese (English slides)
Category • Level
Machine Learning • Medium
Imagine typing "dramatic battle scene" and finding the matching manga panel, uploading a manga-style screenshot and jumping to the corresponding video scene, or searching an anime-style opening with a short audio clip. This talk shows how to build that cross-modal search app in Python with FastAPI, Streamlit, asyncio/aiohttp, psycopg2, pgvector, and Google's Gemini Embedding 2, which maps text, images, audio, and video into one vector space.
We will go beyond "the results look better" by evaluating a small hand-labeled query set with Recall@10/MRR, then comparing 768, 1536, and 3072 dimensions to show the real quality-vs-cost trade-offs of Matryoshka Representation Learning. Along the way, we will cover the production pieces: batch ingestion, schema design, pgvector indexes, async concurrency limits, cached demo fallbacks, and why the same design works on vanilla PostgreSQL + pgvector, with AlloyDB as the production deployment choice.
Description
Background
One of the most effective ways to learn a language is through engaging content — manga and anime are perfect media. But traditional search (filenames, tags, episode numbers) can't capture semantic meaning. Google's Gemini Embedding 2 can embed text, images, audio, and video into a shared space, making cross-modal semantic search practical for language-learning workflows.
Talk Content
This talk is based on the open-source project manga-lingo, a multimodal search app for learning Japanese/Korean through manga.
Part 1 — What are multimodal embeddings? A compressed intuition section covering embeddings, vector spaces, cosine similarity, and why Gemini Embedding 2 makes "search images with text" or "search video with audio" possible.
Part 2 — System architecture & implementation. The full tech stack: Streamlit frontend → FastAPI backend → Gemini Embedding 2 API → PostgreSQL/AlloyDB + pgvector. Focus on schema design (three vector columns for three dimensions), IVFFlat index creation, embedding generation code, and why AlloyDB is optional rather than required.
Part 3 — Matryoshka embeddings in practice. Side-by-side comparison of search results at 768/1536/3072 dimensions for the same query, backed by Recall@10/MRR on 20-30 hand-labeled text/image/audio queries to show the precision vs. performance trade-offs.
Part 4 — Live Demo. Search manga panels with Japanese text queries, upload a manga-style screenshot to find the matching video scene (with timestamps), run a tiny audio-search demo, and compare results across dimensions. All demo paths have pre-cached local results so the talk remains smooth if network APIs are slow.
Data note. The demo uses a small, local, educational sample set. The talk will explicitly call out copyright boundaries: do not redistribute copyrighted media, use owned/licensed/public-domain samples for public demos, and keep generated embeddings/results separate from copyrighted assets when publishing code.
