feat: adds docs skill#55
Conversation
WilliamBergamin
left a comment
WilliamBergamin
left a comment
There was a problem hiding this comment.
Awesome work 💯 this project is starting to take shape 🏗️
It keeps surprising me how much overlap the skills have! The eval test seems to be catching a lot of those potential overlap issues. Figuring out the scope of each skill is starting to become the trickiest part of this project
Left a few comments, I think we are going in the right direction, the main blocker is around the description, when running the evals locally I found this description was conflicting with other skills 😅
| --- | ||
|
|
||
| ## Step 2: Read the Page (fetch markdown) | ||
|
|
There was a problem hiding this comment.
🤔 slack-cli vs slack-docs overlap
Needs a boundary decision (the skill-slack-cli-socket-mode failure). We previously expected slack-cli for "Search the Slack developer documentation...", but slack-docs is arguably the better pick
slack-cli's own description still claims "searching the Slack developer documentation for any topic (socket mode, the Events API, OAuth, …)", which is now redundant with this skill.
Maybe we should move general doc-search to slack-docs and narrow the slack-cli's description to "search docs from the terminal via slack docs search". Then update the skill-slack-cli-socket-mode scenario's expected_tool to slack-docs.
| The response is JSON: | ||
|
|
||
| ```json | ||
| { |
There was a problem hiding this comment.
💡 (Lower priority)
Verified live: a socket mode search actually returns SDK pages (/tools/java-slack-sdk/…, /tools/node-slack-sdk/…) above the events-api guide shown here. 🤔
Claude claims "The example oversells the ranking.", we might need to improve the search endpoint 😅
There was a problem hiding this comment.
maybe quick fix is to have it filter to guides - there is a filter param in endpoint
There was a problem hiding this comment.
jk i made a filter on site but not on endpoint hmmmm
There was a problem hiding this comment.
no longer jk, aded filter to endpoint and then told it to filter to guides
|
|
||
| --- | ||
|
|
||
| ## Fast Path (the developer already has a URL) |
There was a problem hiding this comment.
💡 (Lower priority) The "drop #anchor, append .md, WebFetch" logic is duplicated here and in Step 2. Consider making Step 2 the single home for path→.md→fetch and having the Fast Path just hand off to it
Goal is to have less to maintain
There was a problem hiding this comment.
brilliant. made it so!
| PROMPT = "How does Socket Mode work in Slack? Point me to the docs." | ||
|
|
||
|
|
||
| class TestSlackDocs: |
|
|
||
| # Skill inventory (single source of truth) | ||
| EXPECTED_SKILLS = ("create-slack-app", "block-kit", "slack-api", "slack-cli") | ||
| EXPECTED_SKILLS = ("create-slack-app", "block-kit", "slack-api", "slack-cli", "slack-docs") |
Co-authored-by: William Bergamin <william.bergamin.coen@gmail.com>
The docs search endpoint now supports a `category` param. Lead with it in Step 1 rather than treating it as optional: an uncategorized search skews heavily toward SDK reference pages (queries like "socket mode" or "oauth" can return a top 10 that is entirely Bolt/Node pages), burying the conceptual and reference content most questions are about. - Step 1 (discovery): make `category` the default path, with guidance on picking the starting category (guides for conceptual, reference for a named method/event/scope, a tool category once the SDK is known) and how to widen the search when a categorized query misses. - Step 3 (tool/SDK scoping): map each tool to its matching `category` so the agent can scope discovery to a language family. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Make Step 2 the single home for the page-reference → .md → fetch → cite routine, accepting both a site-relative path from search and a full URL the developer pasted. The Fast Path now hands off to Step 2 instead of duplicating the append-.md/WebFetch logic. Less to maintain, and Step 2 now covers the anchor-stripping case that only the Fast Path mentioned before. Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
3 participants
This PR adds a docs skill.
It starts by using the search endpoint and then falling back to going through sitemap-lls.txt. Discussions were hard about swapping that but testing so far seems really snappy. Can reassess though ofc ofc
Ran some questions by it about methods and claude defaulted to the api skill!