Microsoft Research has published details of a research prototype that provides a different type of interaction between humans and AI tools. Magentic-UI keeps the human in the loop and more in control.  

Magentic-UI is powered by a multi-agent system that can browse and perform actions on the web, generate and execute code, and generate and analyze files. What makes it different is the interaction between the tool and the user. 

Microsoft says the tool is designed to be especially useful for web tasks that require actions on the web such as filling in a form or customizing a food order. It is also aimed at tasks such as deep navigation through websites not indexed by search engines, perhaps narrowing down which flights would work when planning a trip, or tasks that need web navigation and code execution such as generating a chart from online data.

Where Magentic-UI takes a different approach is that it acts more like a human assistant would for a boss who likes to keep close control. So, for instance, the researchers show an example where the tool is asked to find a birthday present for a five year old nephew living in Seattle who loves bikes.

The tool comes up with a multi-stage plan: first search for popular bike-related gifts for the right age such as bike helmets with fun designs, or accessories such as bells or baskets. 

The second step in the plan is to search for retailers either in the Seattle area or national chains that can deliver the chosen items in time for the birthday. And finally, the tool will provide a summary of the top three choices, their prices and delivery options, and why they're a good choice.

The key elements start with the collaborative planning with users. You can directly modify the plan either through a plan editor or by providing textual feedback before Magentic-UI executes any actions. Once the tool is undertaking the task, you can pause it and provide feedback in natural language or demonstrate it by directly taking control of the browser.

Action guards are also included; Magentic-UI seeks user approval before executing potentially irreversible actions, and the user can specify how often Magentic-UI needs approvals.

Microsoft says Magentic-UI can also learn and save plans from previous interactions to improve task completion for future tasks.

Magentic-UI is built using AutoGen, Microsoft's framework for creating multi-agent AI applications that can act autonomously or work alongside humans. Magentic-UI is based on Magentic-One, a tool launched by Microsoft last year that is designed to carry out intricate web and file-based tasks. It is based on a collection of agents that work together to create a modular system, starting with the lead agent, Orchestrator. This is powered by a large language model (LLM) that performs co-planning with the user, decides when to ask the user for feedback, and delegates sub-tasks to the remaining agents to complete.

The second agent, WebSurfer, is an LLM agent equipped with a web browser that it can control. Given a request by the Orchestrator, it can click, type, scroll, and visit pages in multiple rounds to complete the request from the Orchestrator. The other agents include Coder, an LLM agent equipped with a Docker code-execution container; and FileSurfer, an LLM agent equipped with a Docker code-execution container and file-conversion tools. 

Magentic-UI is available on GitHub now.  

More Information

Magentic-AI On GitHub

Magentic-AI Research Paper

Related Articles

Azure RAGChat Deep Dive

The Microsoft AI Agents for Beginners Course

OpenAI Adds New Tools For Building Agents

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info