Skip to content

Call reaches Establishing then immediately Terminated with 500/1203002 "Server Internal Error" — app-hosted media join, no other diagnostic detail (relates to #772, #839) #870

Open

Description

@ftacorn

Title

Call reaches Establishing then immediately Terminated with 500/1203002 "Server Internal Error" — app-hosted media join, no other diagnostic detail (relates to #772, #839)

Body

Describe the issue

Building a minimal app-hosted-media calling bot (throwaway feasibility spike, not derived from
any sample repo directly, but following the architecture of RecordingBot/EchoBot) to prove
whether app-only Teams meeting join + media works on our tenant. Auth, app registration, and the
join request itself all work correctly — the call is accepted and transitions to Establishing
— but it then fails to Terminated within a couple of seconds every single time, before ever
reaching Established, with no other error surfaced anywhere (no exception thrown locally, no
webhook/notification content beyond the state change itself).

This looks identical to #772 and #839 — same code (500), same subcode (1203002), same
"immediately after Establishing" timing. Neither of those threads got a maintainer response, so
opening a fresh one with a from-scratch tenant/environment to add a third data point, since #839
already ruled out ngrok/tunneling as the cause (real Azure VM + public IP + real cert, same
result) and I can independently confirm the same on a different tenant.

Setup

  • Local dev machine (Windows 11), signaling via an ngrok HTTP tunnel (static/reserved hostname,
    paid tier), media via a separate ngrok TCP tunnel.
  • Per the official local-dev guide
    (learn.microsoft.com/en-us/microsoftteams/platform/bots/calls-and-meetings/debugging-local-testing-calling-meeting-bots),
    MediaPlatformInstanceSettings.ServiceFqdn is set to a dedicated hostname we control
    (CNAME'd to the ngrok TCP tunnel's hostname), separate from the HTTPS signaling hostname.
    InstancePublicIPAddress is IPAddress.Any (0.0.0.0), matching the SDK sample pattern, not
    a resolved literal IP.
  • Certificate is a real CA-signed cert (win-acme / Let's Encrypt via DNS-01), subject/SAN
    matching ServiceFqdn exactly, installed in LocalMachine\My, thumbprint passed as
    CertificateThumbprint.
  • Azure AD app registration has Calls.AccessMedia.All, Calls.InitiateGroupCall.All,
    Calls.JoinGroupCall.All (all application permissions, admin-consented), plus a companion
    Azure Bot Service resource with the Teams channel added and Calling explicitly enabled
    (required separately from the Graph AD app registration — error 7503 without it).
  • Joining a real scheduled Teams meeting via Calls().AddAsync() with AudioSocketSettings
    (StreamDirection.Recvonly, AudioFormat.Pcm16K), no video/VBSS sockets.

Code snippet (relevant excerpt)

clientBuilder.SetMediaPlatformSettings(new MediaPlatformSettings
{
    ApplicationId = clientId,
    MediaPlatformInstanceSettings = new MediaPlatformInstanceSettings
    {
        CertificateThumbprint = certificateThumbprint,
        InstancePublicIPAddress = IPAddress.Any,
        InstancePublicPort = mediaPublicPort,       // ngrok TCP tunnel's public port
        InstanceInternalPort = mediaInternalPort,    // local port media platform binds
        ServiceFqdn = mediaServiceFqdn,               // dedicated hostname, CNAME'd to the ngrok TCP host
    },
});

Call-state handler logs ResultInfo on every transition (added specifically to chase this
issue down, since the bare state field alone gives no reason):

call.OnUpdated += async (sender, args) =>
{
    var state = args.NewResource.State;
    Console.WriteLine($"[spike] call state -> {state}");

    var resultInfo = args.NewResource.ResultInfo;
    if (resultInfo is not null)
    {
        Console.WriteLine($"[spike] result info -- code: {resultInfo.Code}, subcode: {resultInfo.Subcode}, message: {resultInfo.Message}");
    }
    ...
};

Expected behavior

Call transitions Establishing → Established, media session starts receiving audio frames.

Actual behavior

[spike] call state -> Establishing
[spike] call state -> Terminated
[spike] result info -- code: 500, subcode: 1203002, message: Server Internal Error
[spike] call terminated. Total audio frames received: 0

ResultInfo is the only diagnostic surfaced — no exception on our side, nothing else in the
/api/calling notification payload beyond the state change and this result info.

Package versions

<PackageReference Include="Microsoft.Graph.Communications.Calls.Media" Version="1.2.0.17950" />
<PackageReference Include="Microsoft.Identity.Client" Version="4.83.1" />
<PackageReference Include="Microsoft.IdentityModel.Protocols.OpenIdConnect" Version="8.16.0" />
<PackageReference Include="Microsoft.Skype.Bots.Media" Version="1.31.0.225-preview" />
  • Target framework: net8.0
  • OS: Windows 11 (dev machine), ngrok for tunneling

What we've already ruled out

  • Not a networking/tunneling issue — TLS handshake to the media endpoint completes fine, and
    Another 500#1203002 error when teams connects to unmixed meeting (8445/MediaProcessor/v1) #839 reproduces the identical error on a real Azure VM with a public IP and zero tunneling
    involved.
  • Not the ServiceFqdn/InstancePublicIPAddress config — corrected both to match the SDK's own
    documented sample (dedicated CNAME'd hostname distinct from signaling, 0.0.0.0 for the
    public IP field) per the local-dev guide linked above; identical failure before and after.
  • Not a missing native-DLL/build issue — confirmed the media platform initializes (TLS
    handshake succeeds, call is accepted into Establishing) before failing.
  • Not a permissions/licensing issue — no 401/403, no 7503/7504, the call is genuinely accepted
    and only fails once media negotiation starts.

Additional context

Two existing issues on this exact code/subcode (#772, #1203002-tagged #839) are open/closed with
no maintainer resolution. Happy to provide a fresh Fiddler/Wireshark capture, full server log, or
try any diagnostic the team wants — this is currently a hard blocker on evaluating app-hosted
media bots for our use case.

Activity

  1. ftacorn commented on Aug 23, 2026

    @ftacorn
    Author

    Follow-up after digging further — wanted to rule out the mundane causes before assuming this is upstream, since I found two other issues on this exact DiagCode (#764, #775) that turned out to be config problems, not SDK bugs. Ran all three checks this exact setup could produce:

    1. Full SDK diagnostic logging — enabled GraphLogger(name, redirectToTrace: true) with a console trace listener (matching EchoBot's own BotHost.cs setup). Result: no additional detail beyond what ResultInfo already reports. Worth noting for anyone else chasing this: redirectToTrace bridges managed SDK-side events (auth, Graph HTTP calls, call-state transitions) — it does not appear to surface whatever happens server-side when a call goes Establishing → Terminated this fast. If your case is like ours, don't expect this flag alone to reveal more.

    2. Certificate chain validation — not just confirming the thumbprint loads, but building and validating the full chain:

    $chain = New-Object System.Security.Cryptography.X509Certificates.X509Chain
    $chain.Build($cert)
    $chain.ChainStatus        # empty array
    $chain.ChainElements | ForEach-Object { $_.Certificate.Subject }
    # leaf -> Let's Encrypt intermediate -> ISRG root, fully trusted

    ChainStatus came back completely empty — no untrusted root, no partial chain, nothing. This rules out #764's root cause for us.

    3. Port configuration — double-checked InstancePublicPort/InstanceInternalPort (our public/internal port env vars) weren't swapped, which was #775's actual root cause. Confirmed correct, matching what was intended.

    All three come back clean, and the failure is still identical: Establishing → Terminated, 500/1203002, zero audio frames, no further detail available anywhere in the notification payload. Sharing in case it saves someone else the same three checks, and in case it helps narrow this down from the maintainer side — happy to provide anything else that would help (fresh repro, packet capture, etc.) if there's a next step you'd want to see.

  2. ftacorn commented on Aug 23, 2026

    @ftacorn
    Author

    Follow-up to my earlier comment (local-dev diagnostics) — went further and deployed to a real Azure VM with a genuine public IP, removing ngrok/tunneling entirely, plus tried every media-config variable I could think of. Sharing the full result since it rules out a lot of ground for anyone else chasing this.

    Setup: real Windows Server VM, static public IP, two separate hostnames (signaling/media) each with their own DNS A record pointing straight at the VM, a CA-signed multi-SAN cert covering both, NSG rules opening the needed ports, and — this tripped me up, worth flagging for anyone else on Azure — Windows Firewall on the VM itself silently blocked inbound traffic separately from the NSG rule. The NSG allowing a port through is not sufficient; you need a matching New-NetFirewallRule on the guest OS too, or the app just hangs with zero signal (no exception, no incoming request logged — looks exactly like a normal join for anyone not watching closely). Cost some time to catch since it fails silently rather than erroring.

    With everything confirmed reachable end to end (inbound Graph notification arrived in ~100-200ms consistently, TLS handshake fine, cert fully trusted chain), I tried every combination of the two settings most likely to matter:

    1. InstancePublicIPAddress: tried both the VM's actual real static IP, and IPAddress.Any (the value used in Microsoft's own ngrok/local-dev sample). Identical failure both ways.
    2. Single port vs. MediaPortRange: tried the single InstancePublicPort/InstanceInternalPort model first (matches what got furthest via ngrok), then switched to MediaPlatformInstanceSettings.MediaPortRange with a UDP range opened on both NSG and Windows Firewall. One useful, undocumented-as-far-as-I-found constraint: the SDK itself validates and rejects a range under 1000 ports (ArgumentException: Port range must be at least 1000) — my first attempt at a 20-port range failed on this before ever reaching a join attempt. Widened to 1001 ports, passed validation, ran the join — identical failure again.

    Every single combination produces exactly the same result: call state -> Establishing then immediately Terminated, 500/1203002, zero audio frames, no further detail in ResultInfo or anywhere else in the notification payload.

    At this point I've eliminated: tunneling/NAT, DNS/CNAME indirection, certificate trust and chain validity, port configuration (single port and multiple valid ranges), and both documented values for the public IP setting. Genuinely out of client-side variables to try. If anyone from the team can shed light on what else might cause this DiagCode specifically, or confirm whether this needs a different kind of tenant-side provisioning, that would be hugely appreciated — happy to provide anything else useful (fresh repro, full logs, packet capture) on request.

  3. InDieTasten commented on Aug 24, 2026

    @InDieTasten

    Ft Acorn (@ftacorn) Since you have tried a lot of things already and also eliminated tunneling from the equation (most likely problem for anyone new to Teams bot development), could you try one of the samples with exposed port (no tunneling via ngrok) to rule out implementation problems? You have already gone so far, and without looking into it from the Platform side of things, that's the best angle to investigate further.

    I could also look into the code base, if you care to share, though I wouldn't do that for free. If there is any interest in that, my profile contains contact details.

  4. ftacorn commented on Aug 24, 2026

    @ftacorn
    Author

    InDieTasten Appreciate you looking at this — to close the loop, I already ran that exact scenario. This comment above (2026-08-23) covers a full deployment to a real Azure VM with a static public IP, zero ngrok/tunneling anywhere in the path, verified reachability at both the NSG and OS firewall layers, a fully-chained trusted cert, and three separate media configurations (single port, IPAddress.Any, and a valid MediaPortRange). All of them hit the identical 500/1203002 that the ngrok setup did. So tunneling's been ruled out pretty thoroughly at this point — I think this is genuinely server-side, but happy to be wrong if there's something else worth trying. I'll pass on the paid review for now, but thanks for the offer.

  5. ssulzer commented on Aug 25, 2026

    @ssulzer

    Ft Acorn (@ftacorn)
    Can you share a Scenario-Id or Graph Call-id please?

    How is the application configuring its media session? In particular, the AudioSocketSettings.StreamDirection?

    A bot which creates just an AudioSocket configured with StreamDirection.Inactive will fail the error you are seeing. A bot must have at least a RecvOnly AudioSocket.

    Thanks.

  6. ftacorn commented on Aug 25, 2026

    @ftacorn
    Author

    Thanks for taking a look, Stephen Sulzer [MSFT] (@ssulzer).

    AudioSocketSettings: confirmed we're not hitting that specific case -- our media session is created with a single AudioSocketSettings block set to StreamDirection.Recvonly (not Inactive):

    var mediaSession = client.CreateMediaSession(
        new AudioSocketSettings
        {
            StreamDirections = StreamDirection.Recvonly,
            SupportedAudioFormat = AudioFormat.Pcm16K,
        },
        videoSocketSettings: new List<VideoSocketSettings>(),
        vbssSocketSettings: null,
        mediaSessionId: Guid.NewGuid());

    No video/VBSS sockets configured at all, just the one RecvOnly audio socket.

    Scenario-Id / Call-id: from a real, confirmed failure on our end (real Azure VM, public static IP, no tunnel):

    • Call ID: 3e005780-e4ed-40fd-9072-883e97baddbf
    • Scenario ID: d14ca12e-c1e3-48aa-a71a-a20825da0410

    That run had MediaPlatformInstanceSettings.MediaPortRange set to a UDP range (50000-51000) rather than a single fixed port, in case that mattered -- result was identical to every other configuration we've tried (single TCP port, IPAddress.Any vs. a real static IP, with/without a tunnel): Establishing -> Terminated, 500/1203002, 0 audio frames. Happy to pull IDs from additional runs (single-port config, different InstancePublicIPAddress settings, etc.) if any of those would be more useful to compare against.

  7. ssulzer commented on Aug 25, 2026

    @ssulzer

    FYI - I see there was also an earlier failed call attempt, that looks a bit different.

    Scenario-Id: d5986f3f-a366-4e58-8cec-11d874447e73
    Call-id: 1b005c80-7631-4234-80c9-a8a214993adf
    around 02:13 UTC on 2026-08-23

    This call had an InstancePublicPort value that was not 8445. In that call attempt, I could see that our platform service made a successful connection to the bot app, whereas in subsequent attempts (against port 8445), I don't think we even got that far.

  8. ftacorn commented on Aug 25, 2026

    @ftacorn
    Author

    Thanks for digging into that earlier call -- that's a really useful data point.

    You're right that the successful connection was on the ngrok-tunnel run, where the public media port was whatever ngrok's TCP tunnel assigned dynamically (not 8445) -- every VM attempt since has used a fixed port (8445, or a UDP range), which was the one variable none of our prior fallback experiments actually isolated. We're going to test a couple of different fixed single ports on the VM (leaving everything else -- signaling, cert, DNS, static public IP -- unchanged) and see if that alone changes the outcome.

    One caveat we're keeping in mind: the ngrok run differed in more than just the port number (tunnel + CNAME vs. a direct static IP), so this isn't a perfectly isolated variable -- but it's the most concrete lead we've had, so worth ruling in or out directly. Will report back with results either way, including Scenario-Id/Call-id for whichever ports we try.

    Out of curiosity -- is there anything about port 8445 specifically, or fixed high ports in general, that would make the platform behave differently than a dynamically-assigned one? Any port range you'd recommend trying (or avoiding)?

  9. ssulzer commented on Aug 25, 2026

    @ssulzer

    There's nothing special about TCP port 8445: it is just a historical convention this port has typically been specified for the bot media SDK's InstanceInternalPort. Which is the port the bot media SDK uses to run a local HTTPS/TCP listening endpoint as part of the platform's control plane.

    The InstancePublicPort specifies the TCP port on the bot service's load balancer or external edge which maps (via, e.g., inbound NAT port forwarding) to a specific VM instance. So, if you deploy a cluster of VMs for the bot service, each bot media SDK listens locally on port 8445 and each VM instance is assigned a port on the load balancer which maps to it. If your bot deployment consists of just a single VM, omitting a load balancer, then often InstancePublicPort == InstanceInternalPort == 8445.

    After deploying the bot, a simple connection test is to run the following Powershell command to verify you can connect to the bot VM's media SDK listening endpoint:
    Test-NetConnection -ComputerName <mediaServiceFqdn> -Port <mediaPublicPort>

    (However, this test won't work if your bot deployment has firewalled the mediaPublicPort to allow only inbound connectivity from the Teams/Office365 business IP address space.)

    What you choose for the UDP media port range is less of a concern, as long the length of the range is at least 1000 ports. 50000-51000 is fine.

    I suspect - once we have ruled out the inbound TCP connectivity to the bot via the InstancePublicPort - there is another issue causing the call failure. It might be a dependency conflict. Since it looks like you are using the latest version of the Microsoft.Graph.Communications.Calls SDK, please try upgrading to the latest version of the media SDK - 1.33.0.0258-preview.

  10. ftacorn commented on Aug 26, 2026

    @ftacorn
    Author

    Update, and thank you -- this is resolved.

    Bumping Microsoft.Skype.Bots.Media from 1.31.0.225-preview to 1.33.0.258-preview, per your suggestion, was the actual fix. First run afterward reached Established for the first time in this entire investigation -- auth, join, and media negotiation all working, no more 500/1203002.

    For anyone else who finds this thread: reaching Established surfaced a second, unrelated issue -- updateRecordingStatus returning 403/8506 ("API call not allowed for non-Compliance Recording call"). That turned out to be expected behavior, not a bug: that API only works on calls established through Teams' policy-based compliance recording, which requires the bot to be invited in via Calls().OnIncoming/AnswerAsync() (matching the PolicyRecordingBot sample) rather than proactively joining via Calls().AddAsync(). Once that was set up correctly, real audio came through end to end.

    Appreciate you sticking with this one -- the version bump was something we'd genuinely have had no way to find on our own. Closing this out on our end.

  11. ssulzer commented on Aug 26, 2026

    @ssulzer

    Same issue reported here: #861 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions