fix: always send USERAUTH_FAILURE on reject

[ejohnstown]

RFC 4252 section 5.1 has the server answer a request it does not accept with USERAUTH_FAILURE. A callback returning WOLFSSH_USERAUTH_REJECTED now always gets that reply.

NO_FAILURE_ON_REJECTED suppressed it. The macro was never set by configure, named in a header, or documented, and it guarded all four method handlers alike.

• drop the macro and its four guards
• add test_UserAuthRejectedSendsFailure(), asserting on the wire, since the handler returns WS_USER_AUTH_E either way

Issue: F-11672

[Copilot]

Pull request overview

This PR updates wolfSSH’s server-side user authentication handling to comply with RFC 4252 §5.1 by ensuring a rejected authentication attempt still results in a USERAUTH_FAILURE reply on the wire, and adds a unit test that asserts this behavior.

Changes:

• Removed NO_FAILURE_ON_REJECTED guards so WOLFSSH_USERAUTH_REJECTED consistently triggers USERAUTH_FAILURE sending across auth methods.
• Added test_UserAuthRejectedSendsFailure() to validate the on-the-wire response when the userauth callback rejects.
• Wired the new test into the unit test runner.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/internal.c	Always sends USERAUTH_FAILURE when userauth callback returns WOLFSSH_USERAUTH_REJECTED (removes macro guards).
tests/unit.c	Adds a unit test that captures outgoing packets and asserts MSGID_USERAUTH_FAILURE is sent on callback rejection.

Suppressed comments (1)

src/internal.c:8756

• Setting authFailure=1 on WOLFSSH_USERAUTH_REJECTED will cause SendUserAuthFailureCount() to run later, but the function currently forces ret back to WS_USER_AUTH_E when authRejected is set. That can mask a WS_WANT_WRITE from SendUserAuthFailureCount(), despite nearby comments stating a blocked send is non-fatal.

                else if (ret == WOLFSSH_USERAUTH_REJECTED) {
                    WLOG(WS_LOG_DEBUG, "DUARPW: password rejected");
                    authFailure = 1;
                    authRejected = 1;
                    ret = WS_USER_AUTH_E;
                }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1202

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1202

Scan targets checked: wolfssh-bugs, wolfssh-src

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Fenrir's latest completed scan found no issues; clearing the prior automated change request.

[philljj]

merge conflict in unit.c, and skoll High.

[ejohnstown]

Rebased onto master, so the unit.c conflict is gone. The two versions of
the request builder are merged: it takes the service name, and keeps the
guard the publickey case needs.

The skoll High is answered on its thread -- real, but not new to this PR,
and the fix belongs in the teardown flush, which has to follow the channel
EOF/close PR in flight.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1202

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1202

Scan targets checked: wolfssh-bugs, wolfssh-src

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Fenrir's latest completed scan found no issues; clearing the prior automated change request.