feat(pubsub): add publish telemetry headers for publish attempt observability

[tonyyyycui]

Adds client-side publish telemetry headers to provide observability into publish latency and hedging attempts.

NOTE: Stacked on #13735. Review commit 885792e for the isolated diff.

[gemini-code-assist]

Code Review

This pull request introduces publish hedging for the Google Cloud Pub/Sub Java client, adding configuration settings (HedgingSettings), a coordinator (CancellationSharer), and queue management (HedgedRequest) to manage and rate-limit hedged attempts. Feedback on these changes highlights several critical issues, including a race condition in scheduleQueueProcessing that could leave tasks unscheduled, a logic error in CancellationSharer where a failure of the original attempt prematurely cancels active hedged attempts, a thread-safety issue with queueProcessingFuture lacking volatile visibility, and a hardcoded 10-second timeout cap that ignores the user's configured maximum RPC timeout.

[gemini-code-assist]

There is a race condition in scheduleQueueProcessing when scheduling a new hedged request. If hedgingQueue.peek() returns null (because the queue was empty at that moment), but another thread adds an item to the queue and calls scheduleQueueProcessing right before isQueueProcessingScheduled.set(false) is executed, the compareAndSet in the second thread will fail (since it is still true). The first thread will then set the flag to false and return. As a result, the newly added item will remain in the queue without any scheduled task to process it. Using a loop to re-check the queue after setting the flag to false resolves this issue.

  private void scheduleQueueProcessing() {
    while (true) {
      if (!isQueueProcessingScheduled.compareAndSet(false, true)) {
        return;
      }
      HedgedRequest nextItem = hedgingQueue.peek();
      if (nextItem == null) {
        isQueueProcessingScheduled.set(false);
        if (!hedgingQueue.isEmpty()) {
          continue;
        }
        return;
      }

      long delay = Math.max(0, nextItem.getSendAfterMs() - clock.millisTime());

      queueProcessingFuture =
          executor.schedule(
              new Runnable() {
                @Override
                public void run() {
                  processQueue();
                }
              },
              delay,
              TimeUnit.MILLISECONDS);
      return;
    }
  }

[gemini-code-assist]

The queueProcessingFuture field is written to by both the publisher thread (via startHedgedCall) and the executor thread (via processQueue) without any synchronization or volatile visibility. This can lead to data races and visibility issues, particularly during shutdown where queueProcessingFuture is read and cancelled. Marking this field as volatile ensures proper thread visibility.

Suggested change

	private ScheduledFuture<?> queueProcessingFuture;
	private volatile ScheduledFuture<?> queueProcessingFuture;

[gemini-code-assist]

Capping the attempt timeout at a hardcoded 10000 (10 seconds) ignores the user's configured maxRpcTimeoutDuration in retrySettings. It should instead be capped at retrySettings.getMaxRpcTimeoutDuration().toMillis() to respect the publisher's configuration.

Suggested change

	long attemptTimeoutMs = Math.min(10000, remainingTimeoutMs);
	long attemptTimeoutMs = Math.min(retrySettings.getMaxRpcTimeoutDuration().toMillis(), remainingTimeoutMs);