feat(api): add /v1/detokenize endpoint

[Dennisadira]

Summary

Closes #1649. Mirror of /v1/tokenize for the inverse direction: take a list of token IDs and return the detokenized text, requested by @benniekiss in the issue thread for "complete API workflow" use cases that need to turn token IDs back into text without local processing.

The proto/handler shape was discussed in #1649 (comment). @benniekiss reacted positively; landing this with the strict-mirror-of-tokenize precedent in mind. Happy to adjust if the proto naming or response shape should differ.

What's added

• Proto (backend/backend.proto): new Detokenize(DetokenizeRequest) returns (DetokenizeResponse) RPC, with DetokenizeRequest{repeated int32 tokens} and DetokenizeResponse{string content}. The Go bindings are regenerated by make protogen-go (gitignored as usual).
• llama.cpp backend (backend/cpp/llama-cpp/grpc-server.cpp): handler that calls common_token_to_piece per token and concatenates — the same primitive TokenizeString already uses internally at the same file.
• Other backends: inherit the default Unimplemented from pkg/grpc/base.Base — same pattern as Detect, Rerank, etc. Backends can opt in later.
• Go plumbing: pkg/grpc/{interface,server,backend,client,embed}.go + pkg/grpc/base/base.go updated alongside their TokenizeString counterparts.
• HTTP: POST /v1/detokenize in core/http/endpoints/localai/detokenize.go and core/http/routes/localai.go. Request {"model": "...", "tokens": [...]}, response {"content": "..."}.
• Auth: entry in RouteFeatureRegistry gated by the existing FeatureTokenize — no new feature flag.
• Discovery: added under ai_functions in the routes index.
• Swagger regenerated; authentication.md updated to list the new endpoint.

Test plan

• make protogen-go regenerates clean
• go build ./core/... ./pkg/grpc/... clean
• go vet ./core/... ./pkg/grpc/... clean
• go test -c -o /dev/null ./core/services/nodes/... clean (the existing testcontainers-based suite needs Docker; only updated the two interface mocks so the test package still compiles)
• make swagger regenerates with the new endpoint visible
• Manual round-trip: POST /v1/tokenize → POST /v1/detokenize returns the original text on a llama.cpp model

Assisted-by: Claude:claude-opus-4-7

[mudler]

this requires a test addition to our e2e-backend test suite where we exercise a mocked backend via api

[Dennisadira]

Done — added a Detokenize method to the mock gRPC backend and two e2e tests in the MockBackend suite (0024a9c): one that POSTs known token IDs and asserts a non-empty content response, and a round-trip that tokenizes first then detokenizes the returned IDs.

[Dennisadira]

Rebased onto current master (05e8e1e). The only conflicts were in the generated swagger files — upstream had added Diarization types; I merged both sets in alphabetical order. All other files applied cleanly. Ready for another look when you have a moment.

[Dennisadira]

Hi @mudler — just checking in on this one. The e2e mock backend tests you requested are in (commit 349c9d2), and the branch was rebased onto current master on May 21. Happy to address any further feedback whenever you get a chance to take another look.

[Dennisadira]

Rebased onto current master — branch now has 2 commits (the endpoint + e2e tests) cleanly on top of master with no unrelated changes.

Also checked: no other PR has landed a /v1/detokenize implementation in master. The feature is still unaddressed upstream (related open issue: #1649), so this PR is still relevant.

[Dennisadira]

Hi @mudler — just flagging that the e2e tests requested in the review have been added in the second commit (c168bcba). The test suite adds a Detokenize method to the mock gRPC backend and covers both a basic POST with known token IDs and a tokenize→detokenize round-trip.

Happy to make any further changes if needed.