Tom Ouyang

Tom Ouyang works on speech-to-speech models for Gemini, connecting machine understanding with the practical demands of human interaction. He introduced himself as an engineer working on Gemini speech-to-speech in his joint AI Engineer presentation with Valeria Wu Fon, where both were credited with Google DeepMind. A supplied September 2026 Gemini Audio announcement, co-authored with Malini Jaganathan, identifies him as a principal engineer. His research has spanned hand-drawn diagrams, mobile keyboards, autonomous-driving perception, and spoken language.

Sketch recognition at MIT

Ouyang graduated from Northwestern University in computer science in 2005, working with Kenneth Forbus on analogical reasoning and problem solving. At MIT’s Computer Science and Artificial Intelligence Laboratory, he conducted graduate research with Randall Davis in artificial intelligence and human-computer interaction, earning a computer-science master’s degree in 2007. His sketch-recognition research addressed how computers could interpret drawings without requiring people to abandon the pen-based interaction they already used to think and work.

With Davis, he developed software that recognized hand-drawn chemical structures and electrical circuits. The chemistry effort originated in a collaboration with Pfizer and led to ChemInk’s work on real-time recognition. Interpreting the marks made the drawing useful to other software: a scientist could search for research on a molecule, or an engineer could analyze whether a circuit would function. Their work on chemical and circuit drawings connected a familiar interface to computational search and analysis.

Mobile keyboards and Gboard

At Google, Ouyang helped develop mobile text-entry systems that combined interface research with statistical language modeling. He belonged to the mobile-interaction research group whose work informed Android’s Gesture Typing and co-authored research on two-handed gesture keyboards and keyboards that adapted to different hand postures and users. These systems had to infer intended words from imperfect finger paths and touch locations while responding quickly enough to keep writing fluid. His mobile-interaction work treated recognition accuracy and the physical experience of entering text as connected problems.

• Keyboard decoding: His work on Gboard brought methods from speech recognition into keyboard decoding. With David Rybach, Françoise Beaufays, and Michael Riley, he developed a finite-state transducer framework supporting autocorrection, word completion, next-word prediction, and corrections informed by later context under tight memory and latency constraints. Spatial evidence about where someone touched could be interpreted alongside language probabilities about what they intended to write. He also contributed to Gboard’s neural spatial models and transliteration support for Indian languages, allowing users to write those languages through Latin-letter keyboards.
• Learning new vocabulary: Fixed vocabularies posed another problem: people continually introduce words that a keyboard’s dictionary does not contain. Ouyang co-authored federated learning research on out-of-vocabulary words, using a character-level neural model trained on data held on mobile devices. The approach enabled vocabulary discovery without transferring sensitive user text to servers, addressing changing language while keeping the underlying text on users’ devices.

Combining LiDAR views at Waymo

By 2019, Ouyang was working on autonomous-driving perception at Waymo. He co-authored LiDAR object-detection research that combined complementary views of the same point cloud. A bird’s-eye view preserves useful information about objects’ physical dimensions, while a perspective view can help reveal small or distant objects that are sparsely represented from above. Fusing the representations at individual points improved detection over a comparable single-view baseline.

Gemini speech-to-speech and multimodal interaction

His speech-to-speech work returns to the question of how recognition becomes useful interaction. In their joint presentation on Gemini speech research, Ouyang and Wu Fon trace the shift from specialized speech-recognition pipelines to models trained jointly on audio, video, and text. A transcription-focused model converts speech into words; translation, responses, visual understanding, and information about tone or emotion require further capabilities. Interleaved multimodal training teaches relationships among these forms of input and output within a shared model. A training example might ask for a summary of an audiovisual story with timestamps, connecting what the model hears and sees to a textual answer.

Streaming translation makes those demands concrete. The model must begin translating before an utterance is complete, accommodate changing languages and multiple speakers, preserve the source speaker’s voice, and cope with noise. The presentation argues that unified multimodal training makes these combinations easier to request through prompts than to assemble from separately engineered recognition and synthesis components. It also describes the team’s observation that streaming translation quality approached offline systems that had access to the full utterance; this is a reported result from the presentation, rather than an independently established comparison.

The joint talk frames voice interaction as a balance among conversational latency, task intelligence, and multimodal input and output. More reasoning can improve an answer while delaying the first spoken response. Demonstrations show how the requirements change across applications: visual search uses streamed video and audio to answer questions about a sofa and retrieve search cards, while a roadside-assistance agent must accurately handle alphanumeric details and distinguish conversational input from background noise. Conversational avatars extend the work into visual presence. These examples illustrate the team’s research direction; they do not establish Ouyang’s individual authorship of each application.

Reasoning while speaking in Gemini Live

Ouyang and Jaganathan’s September 2026 introduction of Gemini 3.8 Live and Live Extended Thinking describes background tool execution and a model designed to reason and speak simultaneously, acknowledging requests and narrating progress during longer tasks. This addresses a concrete difficulty in conversational software: useful computation can take time, but the user still needs a responsive interaction while the work proceeds.