A software agent can know Python, call an API, and still fail at the part a human learns by watching someone work: the sequence of small decisions that turns a blank canvas into a finished result.

Resource2Skill turns human tutorials into executable agent skills

Microsoft Research's new Resource2Skill paper attacks that gap with a useful idea: stop treating tutorial videos as passive reference material. Convert them into reusable skills. The system ingests tutorial videos, repositories, articles, documentation, and reference artifacts, then builds a hierarchical Skill Wiki. Each entry can contain instructions, code, visual examples, metadata, and provenance. An agent retrieves the relevant entries, composes them, and uses them while operating real software.

That sounds like a better version of retrieval augmented generation, but the distinction matters. RAG usually gives a model passages to quote. Resource2Skill tries to give an agent procedures it can execute. A video can show the order of operations, the visual result after each step, and the little UI choices that a transcript loses. A repository can contribute working code. An article can explain why a particular approach works. The system keeps those signals together instead of flattening everything into text.

The missing layer between instructions and action

The paper frames a skill as procedural knowledge: how to break a task into steps, which tool or API pattern to use, what intermediate state to inspect, and how to recover after a failure. That is exactly where many current agents feel brittle. They can produce a plausible plan, then stumble when the task depends on application state, visual layout, or a tool with awkward conventions.

Resource2Skill builds the library offline, but it also has an online path. When the wiki lacks coverage for a user request, the same resource-to-skill process can search for new material, distill another skill, validate it, and add it to the library. The result is meant to grow instead of remaining a fixed prompt folder.

The hierarchy is doing real work here. A flat pile of skill files creates a retrieval problem: the agent has to search every entry and decide which abstraction level fits. The wiki first narrows the domain and category, then selects candidate skills before the language model reads and composes them. That design resembles how a good engineer navigates documentation. You do not search every page about a design tool when you already know you are editing a spreadsheet chart.

The representation is also deliberately multimodal. Text explains when a skill applies. Code gives the agent something it can adapt or run. Visual examples preserve details such as spacing, style, motion, and the before-and-after state of an edit. Provenance tells the agent where the procedure originated, so a human can inspect it.

The result is strong, with an important asterisk

Across seven authoring areas, the authors report an average overall-score gain of 11.9 percentage points over the same agents without skills. The evaluation covers Web, Excel, PowerPoint, Blender, CAD, UE5, and REAPER-style audio work. Resource2Skill wins 26 of 28 main aggregate model-domain comparisons against strong harness baselines.

The pairwise results are more revealing than the headline average. In the reported user study, the skill-enabled agent won 68.0% of 200 comparisons, tied 20.5%, and lost 11.5%. Excluding ties, its win rate was 85.5%. The gains vary by domain. The system reports a 95.7% win rate excluding ties on UE5 tasks, 88.5% on Excel, 87.0% on Blender, and 77.8% on CAD. CAD is the weakest of those listed results, which makes sense: a procedure that works in a tutorial may still break when geometry, constraints, or hidden application state changes.

The benchmark is not a generic test of autonomous computer use. It focuses on practical authoring tasks where public tutorial material exists and the agent has a programmatic tool interface. That makes the result easier to reproduce, but it also sets a boundary around the claim. Resource2Skill may be excellent at teaching an agent software workflows while offering little help in a domain with scarce tutorials, conflicting documentation, or no stable interface.

There is another caveat. The paper's scores route through automated judges, including vision-capable models for rendered artifacts, with a smaller blinded human study used as a cross-check. That is better than trusting one automatic score, but generated slides, web pages, scenes, and audio projects are still hard to judge. A visually polished artifact can hide a broken workflow. A correct spreadsheet can look boring. The difference between a useful skill and a memorized demo will show up in long-running use, not just in a curated task set.

Why this could change agent engineering

The practical shift is from writing skills by hand to mining them from the internet's existing instructional layer. Every mature tool has a pile of tutorials, walkthroughs, sample projects, and reference files. Most agents can retrieve those sources, but they still leave the human to translate a tutorial into a reliable procedure. Resource2Skill makes that translation the product.

The open release is concrete enough to try. Microsoft's repository provides the runtime and domain tooling, while the Hugging Face dataset contains structured skill entries, executable assets, a lock file, and provenance metadata. The released domains include Web, PowerPoint, Excel, Blender, and REAPER-style workflows. A fresh Python 3.11 environment is the sensible starting point, and the project exposes commands for listing domains and validating one before running a case.

I like this direction because it attacks a less glamorous failure mode than model quality. Agents often do not need another general capability. They need a reliable procedure for the exact software in front of them. The catch is that a skill wiki becomes a new attack surface and maintenance burden. A stale tutorial can encode a bad habit. A copied repository can carry unsafe code. A visual example can teach the wrong style while still scoring well. Provenance and validation are not decorative fields; they are the part that decides whether this becomes an engineering asset or a well-organized pile of hidden prompt injection.

Resource2Skill also raises a question the paper does not settle: who gets to update the wiki? If the system learns online from arbitrary resources, skill acquisition needs permissions, review, versioning, and rollback. Otherwise an agent can quietly turn one bad tutorial into a reusable behavior that spreads across every future task. The most important component may end up being the change log, not the multimodal encoder.

The broader lesson is simple. General models provide a lot of raw ability, but software work depends on local knowledge: where a setting lives, what order the tools expect, which intermediate state matters, and what a good result looks like. Tutorials already contain that knowledge. The interesting move is finally treating them as executable material.

Sources