[nexus] tune libpq keepalive to catch dead CockroachDB connections fa…

[jam-mad]

Tune libpq keepalive so dead CockroachDB connections are detected in minutes, not hours

Fixes #10668.
Closes #10668

Summary

Nexus's connections to CockroachDB use libpq's default TCP keepalive
settings, which on illumos means a dead connection can go undetected for
about 2 hours before the OS even sends a probe, then another 8 minutes
before it gives up. This adds four keepalive parameters to the existing
connection args in make_postgres_connector(), cutting that detection
window to roughly 2 minutes.

All of the diagnostic work here is @davepacheco's, from a real incident
where this hung a rack update for 45+ minutes. This PR implements the
fix he identified and adds test coverage for it, including an automated
test that opens a real connection and reads back the actual kernel TCP
keepalive settings, the same check davepacheco did by hand with mdb -k
during the original investigation. It works on Linux, macOS, and
illumos.

The problem

During a self-service rack update, one Nexus instance hung during
handoff, holding two CockroachDB connections for 47+ minutes with zero
sagas running. omdb nexus quiesce show --stacks showed both threads
blocked inside libpq's PQgetResult, waiting on a socket read that was
never going to complete.

davepacheco tracked this down (full details in #10668).

The change

make_postgres_connector() in nexus/db-queries/src/db/pool.rs already
builds a list of extra libpq connection parameters (previously just
sslmode=disable). That list gets appended as query parameters onto the
Postgres connection URL in DieselPgConnector::to_url()
(nexus/db-queries/src/db/pool_connection.rs), which Diesel hands
straight to libpq. Nothing about that plumbing needed to change; this
just adds four more entries to the same list:

let args = vec![
    ("sslmode", "disable"),
    ("keepalives", "1"),
    ("keepalives_idle", "10"),
    ("keepalives_interval", "10"),
    ("keepalives_count", "12"),
];

keepalives_idle=10 sends the first probe 10 seconds after the
connection goes quiet. keepalives_interval=10 and keepalives_count=12
give up after 120 more seconds of unanswered probes. Total worst-case
detection time: about 2 minutes, instead of roughly 2 hours.

These numbers are a starting point in the same ballpark davepacheco
suggested (10 seconds to first probe, 120 seconds to abort), not a final
answer. Worth a second opinion before merging, since this is a production
reliability configuration. If it's too aggressive it risks tearing down connections
during a legitimate CockroachDB pause, too lax doesn't fix the problem.

Testing

nexus/db-queries/src/db/pool_connection.rs had no test coverage at all
on DieselPgConnector's URL-building logic before this. Added two
tests, in increasing order of how much they actually prove:

• to_url_includes_keepalive_args: builds a connector with the new args
  and asserts the resulting connection string contains all four
  keepalive parameters. Fast, offline, no database needed. This only
  proves we build the right string.

• connection_actually_gets_tcp_keepalive_settings: opens a real
  connection to a real (test) CockroachDB with that exact connection
  string, then reads back the TCP socket's actual kernel keepalive
  settings and asserts they match what we configured. This is the
  automated version of the mdb -k check davepacheco did on a
  live connection while diagnosing the original incident.

  Diesel's PgConnection hides the underlying socket, so the test can't
  inspect the connector's connection directly. It opens a second,
  plain libpq connection with the same connection string instead. libpq
  is what actually applies these settings via setsockopt, so this
  checks real behavior. It gets the raw socket via libpq's own PQsocket()
  (a supported, public part of libpq's C API), dup()s the fd so nothing
  double-closes it, and reads it back with socket2.

  This works on Linux, macOS, and illumos (production?) with no
  platform-specific code. socket2's keepalive_time /
  keepalive_interval / keepalive_retries already handle each OS's
  different option names. For example, macOS reports TCP_KEEPIDLE as
  TCP_KEEPALIVE. illumos has its own older TCP_KEEPALIVE_THRESHOLD,
  but also supports the same TCP_KEEPIDLE/TCP_KEEPINTVL/
  TCP_KEEPCNT names Linux uses (confirmed by checking illumos's actual
  constants in the libc crate). I've only run this on Linux myself;
  illumos should work the same way, gated by socket2's own
  cfg(target_os = "illumos"), but I haven't confirmed it there
  directly.

Also ran the pre-existing db::pool::test tests
(test_pool_can_be_terminated, test_pool_drop_does_not_panic), which
connect to a real ephemeral CockroachDB instance. Both still pass with
the new connection args in place.

New dev-dependencies: socket2 (already a workspace dependency with the
all feature enabled, just not previously used by this crate) and
libc (also already workspace-managed). pq-sys was already a direct
dependency of this crate. Nothing new added to the crates.io dependency
graph; this just wires up existing workspace dependencies for the first
time here, and only in test code.

How to re-run

# String-only test, no live DB needed:
cargo test -p nexus-db-queries --lib pool_connection

# Both pool_connection tests, including the real-kernel-state one
# (needs cargo-nextest, spins up a real ephemeral CockroachDB):
cargo nextest run -p nexus-db-queries -E 'test(pool_connection)'

# Existing pool lifecycle tests against a real CockroachDB:
cargo nextest run -p nexus-db-queries -E 'test(db::pool::test)'

Risk / non-goals

• Small, contained change: 2 files touched for the fix itself
  (pool.rs, pool_connection.rs), plus a Cargo.toml update to wire
  up two already-workspace-managed dependencies for the first time in
  this crate. The connection-string code already existed. This only adds
  parameters to an existing list.
• The kernel-state test only ran on Linux during development. It should
  work unmodified on illumos given socket2's own platform gating
  includes it, but that hasn't been confirmed by actually running it
  there. Worth a CI run on the real target before merging.
• Doesn't address the separate, possibly-related duplicate-connection
  detail sunshowers raised in CockroachDB lost quorum on critical ranges during racklette update #10658. That looked like a different issue.
• The exact timing values are open to adjustment; see the note above.