Add three-step tool discovery mode - #3289
Conversation
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
github-actions
Bot
added
server-Azure.Mcp
b49e444 to
950eaeb
Compare
Adds an opt-in three-step tool discovery mode to the Azure MCP Server, alongside Vally-based end-to-end comparison tooling between two-step and three-step discovery. - Three-step discovery: a top-level learn=true query now returns only tool description/metadata (no inputSchema or inner-command list) unless three-step mode is enabled via the new CLI/server option; a learn=true query with a specific command still returns full command details including inputSchema. Default two-step behavior is unchanged unless the option is set. - Added focused unit tests covering default two-step and three-step-enabled behavior, including top-level learn and individual-command learn schemas. - Added Vally evaluation tooling (Invoke-VallyEvalTests.ps1, Compare-VallyToolDiscoveryMetrics.ps1) to run and compare two-step vs. three-step discovery costs end-to-end, including per-turn context/token/ AI-credit/MCP-byte metrics. - Updated relevant documentation (docs/testing-with-vally.md, e2eTestPrompts.md, azmcp-commands.md) where directly appropriate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
950eaeb to
4380512
Compare
…vements - Implement three-step tool discovery mode (ThreeStepToolDiscovery option) with size-based automatic fallback (ThreeStepToolDiscoveryThresholdBytes) and an opt-out (DisableAutomaticThreeStepToolDiscovery) - Add Default/TwoStep/ThreeStep environments to .vally.yaml and CI workflow - Add 3-mode support, build-once optimization, aggregate/average summary tables, and -SkipEval/-Warmup switches to Compare-VallyToolDiscoveryMetrics.ps1 - Add -Warmup switch to Invoke-VallyEvalTests.ps1 to prime model provider prompt cache before measured runs, reducing cross-mode AI-credit noise - Add -Architectures parameter to Build-Local.ps1 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| ? """ | ||
| This tool is a hierarchical MCP command router. | ||
| Sub commands are routed to MCP servers that require specific fields inside the "parameters" object. | ||
| To invoke a command, set "command" to the exact command name (e.g. "appconfig_account_list") and wrap its args in "parameters". |
There was a problem hiding this comment.
Something we need to stop doing is having explicit command names in descriptions. Often the MCP client modifies the name we provide to prevent collisions across MCP servers, so we don't get a nice appconfig_account_list but more a <guid-like-string>_appconfig_account_list. Aside from that, it also just adds another maintenance spot we're likely to miss in a refactoring.
There was a problem hiding this comment.
Interesting. The LLM suggested adding this text to prevent the host LLM from guessing the tool names, it proposed adding text to the instructions to explain how to construct the command name.
I'm surprised that the MCP host would change the tool name, but it does make sense in the face of tool ambiguity. So would a better pattern be something like <tool_name>_add_kv?
| Do not guess a "command" name from natural language. First call this tool with "learn=true" and no "command" to get the exact list of command names and descriptions. | ||
| Then call again with "learn=true" and the matching "command" from that list to get its full input schema before invoking it. |
There was a problem hiding this comment.
This is really good guidance but also may be overly prescriptive and forcing the agent to learn again when it's possible it can just skip a step as it had enough context
| new TextContentBlock | ||
| { | ||
| Text = $""" | ||
| Here are the available commands for '{namespaceName}' tool. | ||
| The "tool" field of each entry below is the exact "command" value to use -- do not guess or invent a | ||
| different name. Pick the entry matching your task, then run again with "learn=true" and "command" set | ||
| to that exact "tool" value to get its full input schema. | ||
|
|
||
| {learnToolsJson} | ||
| """ | ||
| } |
There was a problem hiding this comment.
Hmm, something we could change here, but would explicitly require three turns is to simplify learnToolsJson returned here to just the tool name and description. Right now, we're returning the input schema as well. So, this current implementation is more a 2.5 turn type implementation.
2 participants
Summary
Related