AI Engineer World's Fair 2024
The Multimodal Future of Education
Loading slides…
Read the talk
The Multimodal Future of Education
From children training their own classifiers to Gemini asking questions about handwritten equations, Stefania Druga explores AI that helps learners experiment without taking over.
From a talk by Stefania Druga
Children are already using AI. What are they learning?
How many children are already playing with generative AI? Stefania Druga opens by asking parents in the audience that question. The educational challenge begins with adoption that is already happening, rather than a hypothetical future classroom. Her opening slide invokes a Salesforce survey, but reverses its denominator: the September 2023 survey found that 70% of Gen Z respondents used generative AI, not that Gen Z accounted for 70% of users. It surveyed more than 4,000 people across the United States, United Kingdom, Australia and India.
That adoption sits alongside urgent educational needs. Druga points to early literacy, pandemic learning losses and gaps in mathematics and reading. The reading statistic needs a similar correction: the World Bank’s June 2022 estimate was that 70% of ten-year-olds in low- and middle-income countries could not understand a simple written text. Druga also cites roughly 60% of children and teenagers being left behind in reading and mathematics. That figure should not be treated as a measured COVID effect: the related UNESCO finding about widespread failure to reach minimum proficiency was published in 2017.
The need extends beyond K–12 to adults who must reskill. Meanwhile, Druga reports that students already use AI for homework, recommend it to peers and sometimes prefer it to human tutoring. These observations create a practical design question: how can an attractive, readily available tool support learning rather than simply become a source of completed answers?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What children think an AI knows
Generative AI enters a world where children have already encountered voice assistants, chatbots and smart toys. Druga calls these children the first AI generation, situating their experience in the spread of household voice assistants and citing adoption in half of US households. Her research on children’s relationships with these devices began at MIT in 2015. The central observation is that a child’s perception of a device influences how they interact with it and what they learn from it.
Children generally saw the devices as friendly, but friendliness did not imply a uniform judgment of intelligence. Druga describes a striking age difference in her research:
| Children | Reported perception of voice assistants |
|---|---|
| Ages four to six and a half | More skeptical of how smart Google Home or Alexa was |
| Older children entering school | More likely to describe the devices as smarter than themselves |
These observations concern voice assistants, before the talk turns to generative AI. A learner’s mental model shapes both the questions they ask and how much they trust the answers. Teaching AI literacy therefore means helping children investigate how a system works, not merely teaching them to operate its interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the AI something a child can change
To make that investigation possible, Druga built Cognimates, which she dates to 2016: a free, open-source platform extending Scratch’s block-based programming environment. One child’s program turns a robot into a hide-and-seek partner. Add a loop, and the child can run around the room while the robot repeatedly tries to find her. The behavior becomes something the learner constructs and revises.
Cognimates also lets children program smart lights and voice assistants, then move from controlling devices to training custom models. They supply image or text examples and use the resulting models inside their own games and applications. One student trains an image classifier to distinguish unicorns from narwhals. When a drawing is presented, the program exposes both the predicted class and its confidence. Druga points out that confidence on the drawing is low. The child can now inspect a model’s uncertain judgment instead of receiving a label that appears unquestionable.
The same environment supports very different projects: inspecting what is in food, playing rock-paper-scissors and making a robot speak like Shakespeare. Across these examples, the useful common structure is a short path from a child’s idea to behavior they can observe. Programming and model training become materials for play.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tinkering changes the judgment of intelligence
Druga reports that Cognimates was used internationally and translated into more than 30 languages. To study its educational effects, she conducted a longitudinal study in public and private schools. Children answered questions about AI before participating in learning activities, then answered the same questions afterward. The activities included text training, image training and smart-home programming. Afterward, children were less certain that an AI device or model was smarter than they were.
An embedded classroom video makes the activities concrete. A child describes playing rock-paper-scissors through a camera with a robot. Another participant compares learning through repeated attempts with the way people improve at unfamiliar tasks. Parents and teachers describe AI knowledge as a necessary skill, even while acknowledging that the technology can be intimidating. One participant offers a mechanical analogy: a father once took apart a car to understand it; children should similarly be able to investigate technologies that adults usually build for them.
Returning to the study, Druga describes a significant before-and-after change in children’s perceptions, without giving a numerical effect size. Her explanation is the learning process the platform enables: children form hypotheses, test them and iterate quickly. A suitable sandbox makes those actions accessible. Skepticism emerges from encountering the system’s behavior directly, including its mistakes and uncertainty.
Children are not the only learners in that sandbox. Parents and teachers also need to understand AI, and pandemic home learning created opportunities for families to explore it together. An early Cognimates tutorial illustrates another possibility: the programmable system itself helps the learner find the Ask block needed to ask a question. The thing being programmed becomes a collaborator in learning how to program it.
That interaction suggests a richer relationship with a chatbot. When an answer is unsatisfactory or inappropriate for a child’s age, the learner needs ways to influence the system. They may also want to bring their own language, culture or unusual interests into the interaction. The educational opportunity is to make that relationship understandable and adjustable, rather than leaving the learner to accept whatever the model produces.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Help families create without taking over
Before building a programming copilot for families, Druga investigated what children and parents wanted from one. She describes a multiweek study involving families in ten US states. The closely matching AI Friends study used a researcher-operated assistant, so these findings establish design requirements rather than the autonomous capabilities of a model. One of the most valued roles was helping a family begin: an AI friend in Scratch could suggest coding ideas when a project stalled at the blank-page stage.
The next role was translating an intention into code constructs. A child might want a bear to jump over a hedgehog but not know how to express the movement. A small Scratch script for the bear illustrates the kind of construct a helper could introduce:
scratchblocks
when green flag clicked
repeat (10)
change x by (4)
change y by (5)
end
repeat (10)
change x by (4)
change y by (-5)
end
The two loops separate ascent from descent while maintaining forward motion. The learner can change the movement and place the hedgehog along its path. The larger design requirement is creative ownership: children should experience themselves as the people doing the coding, with the assistant helping them through a point of difficulty.
The assistant also has a social role. Druga describes it prompting a child to consider a parent’s opinion, encouraging turn-taking and helping the family work together. That third participant can make collaboration easier, but its presence must remain optional. Families need to be able to switch it off when it becomes distracting or when they want to work independently.
Some limits are outside the assistant’s reach. It cannot solve siblings fighting over a shared laptop. It also cannot always break a complex concept into manageable steps; parents remain important sources of explanation and support. A family copilot therefore needs to fit around human relationships and assistance, rather than assume that generating a response resolves the learning problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate the help, then expand the medium
With those family requirements in view, Druga turns to evaluating generative models. She reports that leading models were useful for explaining Scratch programs, generating ideas and asking questions that helped families explore and test new games. She also describes a published benchmark; the matching Scratch Copilot Evaluation assesses assistance on bounded programming tasks. The relevant distinction is between the quality of that assistance and measured improvements in children’s learning. The talk provides no model-by-model scores.
Multimodality expands the possible working material beyond code. Druga sketches several directions:
- Games and simulations: A creative sidekick could participate in Minecraft, other games, or physics and science simulations.
- Physical making: Photographs of flowers and preferred colors could inspire 3D models that a learner prints and paints.
- Craft projects: A generative model could help someone explore ideas for knitting.
These are proposed extensions of the collaborative pattern: the learner brings an interest or an artifact, receives help developing it, and continues making.
A learning companion can also address a more specific instructional problem: identifying math misconceptions. Druga describes work with Nancy Otero on a benchmark intended to capture common K–12 difficulties and evaluate whether models recognize them in chatbot conversations. She presents it as a first benchmark of its kind and offers a download link. Its purpose is diagnostically different from checking whether a model can produce a correct answer: the system must notice what the learner misunderstands.
The broader invitation runs from Cognimates to tools people explore through Hugging Face: make AI tinkering available at every age. Druga then moves from research to three experiments built the previous night with the then-latest Gemini API. Their premise is to use AI for learning through play and making, with live success still uncertain. The first experiment works with science drawings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A drawing becomes the next science question
The envisioned setup is a webcam over a drawing surface, where a learner can draw and revise in real time. Without a table onstage, Druga uses prepared drawings instead. She presents a balance scale with a weight on each side. The system asks what would happen if another five kilograms were added. She then proposes a revised drawing with ten kilograms on each side. The drawing is both the input to the model and the medium through which the learner can respond.
The interaction has a simple instructional cycle:
- Present a drawing of a situation.
- Receive a question grounded in that drawing.
- Revise the drawing to explore a possible answer.
- Present the changed drawing and continue the exchange.
Druga next tries water and CO₂, asking what happens when they mix. She returns to the idea of drawing beneath a webcam: the value is the possibility of moving fluidly between physical marks and model-generated questions.
The Earth-and-moon example exposes the fragility of visual interpretation. The initial interaction describes Earth as being hit by something. Druga adds another arrow, and the system eventually recognizes the moon. That correction matters to the learning loop: before an assistant can ask a useful science question, it must interpret the learner’s representation well enough to discuss the intended situation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompt the learner’s next action
The math experiment applies the same pattern to successive pieces of written work. Druga presents an expression and receives a prompt to solve what is inside the parentheses. She shows a version with that step completed, then receives guidance about multiplication involving parentheses. With another example, the interaction moves to simplification and the rule that a nonzero number divided by itself equals one. The intended behavior is to offer a question or a next-step hint while leaving the learner to do the work.
The live interaction is uneven. Druga notes that she needs a better background, and a more complicated example produces an unhelpful response about four on an axis. She had hoped for a better question. Recognizing the input and choosing a pedagogically useful response are separate requirements; a visual tutor must do both to sustain the learning exchange.
The third experiment shifts from solving problems to encouraging curiosity. It asks about what a lady is doing, the colors of a flag and the shape of a star; Druga notes that it has asked about Jordan. With an apple, it asks about the kind of apple and then its smell. The smell question extends beyond information the camera can supply: it invites the learner to inspect the physical object and contribute an observation. An intended origami example does not happen because Druga cannot find it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a handwritten equation to an adaptable template
Druga invites the audience to suggest another science, math or object example. Someone requests a system of equations, but the expression supplied for the demonstration is a single equation:
She writes it and presents it to the camera. The tutor prompts her to subtract seven from both sides. She attempts that step, after which the next prompt is to divide both sides by two.
The algebra behind those prompts is straightforward:
At this point, dividing by two is the next proposed learner action. The demonstration’s useful result is the transition from one piece of handwritten work to a specific next-step instruction, rather than immediately displaying the final value of x.
Druga says the demo template is open source and fewer than 100 lines. Her setup instructions are correspondingly small: create an API key and supply your own instructions. She describes the key as free at the time of the demonstration; that historical setup is not a statement about current Gemini access or pricing. Custom instructions are the adaptation point for changing what kind of help the learner receives.
The closing challenge is to think beyond chatbot interfaces and delegating questions. Put tools into young people’s hands so that they can tinker with AI and learn how it works. A child who programs a behavior, changes an example or redraws a problem has a way to investigate the technology while learning with it. Druga closes by pointing to her research website and inviting questions afterward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Scratch-based interface for family activities involving games, robots and training AI models.
Four-country survey reporting generative AI adoption by generation, including usage among 70% of Gen Z respondents.
Further reading
- AI FriendsPaper
Three-week family study of a researcher-operated coding assistant, examining creative agency, parental support and idea generation.
Evaluation of language-model assistance for explaining Scratch code, debugging and generating project ideas.
World Bank explanation of pandemic-era reading difficulties among ten-year-olds in low- and middle-income countries.
Updates since the talk
Later paper documenting 55 middle-school algebra misconceptions and errors, 220 diagnostic examples, and model evaluations informed by educators.
Full transcript
Loading transcript…