← All speakers

Roey Ben Chaim

Conference affiliation: Zenity

Roey Ben Chaim is a staff engineer at Zenity and a co-author of SkillsBench, which measures how well agents apply skills across tasks. A skill can give an agent instructions, resources, and code to use on demand. Roey’s security work examines what that added capability does once the agent uses it, including actions the user may never see in the conversation.

Roey spent a decade building large-scale security systems at Microsoft before joining Zenity, where he designs systems that govern how agents access data and operate within limits. He contributes to Presidio, an open-source privacy toolkit, and co-organizes AI Tinkerers Tel Aviv. Alongside his engineering work, he has pursued AI safety research as a MATS fellow.

What an agent’s activity can tell you

An official Azure diagnostic accessed sensitive cloud configuration and identity information during legitimate troubleshooting. In Roey’s account of Zenity’s agent skill research, that example shows why a record of sensitive access is a starting point for investigation. A reviewer still has to ask whether the access was needed for the task the user authorized.

The Zenity research and engineering team built an isolated environment to observe agents using skills. Early tests exposed two blind spots. A skill could stop before doing meaningful work if the test environment lacked a service or tool it needed. And the agent’s own record could show an ordinary package installation while missing the files or network activity of a separate process started during that installation. The team made the test environments more representative and recorded activity on the machine alongside the agent’s trace. That gave reviewers a better view of what happened in a particular run; a quiet run still could not establish that the skill would behave safely under a different task or in a different environment.

Roey recommends treating third-party skills as software dependencies. Review what they load, restrict what they can access, and observe what they do when used. Reviewing the installed file is only the first step: if a skill loads instructions from elsewhere, the agent can receive different instructions later even when the local file has not changed.

1 conference talk

Loading more talks and highlights…