Skip to content

test(bigtable): speed up the integration test suite - #14344

Open
mutianf wants to merge 2 commits into
googleapis:mainfrom
mutianf:speedup-integration-tests
Open

test(bigtable): speed up the integration test suite#14344
mutianf wants to merge 2 commits into
googleapis:mainfrom
mutianf:speedup-integration-tests

Conversation

@mutianf

@mutianf mutianf commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

The bigtable integration suite was taking > 1 hour to run with the presubmit's
profile list (.kokoro/presubmit/bigtable-integration.cfg). Measured from a full run:
emulator-it 66s + prod-it 2414s + prod-batch-it 905s ≈ 56 min wall clock.

Three separate problems, all fixed here.

1. The suite was running single threaded

google-cloud-bigtable/pom.xml sets forkCount=4 on failsafe, but the
enable-verbose-grpc-logs profile overrode it back to forkCount=1. Profile
plugin config wins, so every run with that profile serialized the entire IT
suite into one JVM. The forked-JVM banner in the logs only ever mentions
"forked JVM 1", confirming it.

The override dates back to #1004 / #2295, when TestEnvRule installed a JUL
appender that wrote grpc logs to the console and interleaved output across
forks. That appender is gone — TestEnvRule.grpcLogHandler is declared and
read in teardownLogging() but never assigned — and
redirectTestOutputToFile=true now gives each fork its own -output.txt.
So cross-process forking is safe; only in-JVM parallel needs to stay off
for log attributability.

The profile now keeps <parallel>none</parallel> and
<redirectTestOutputToFile>true</redirectTestOutputToFile> but no longer
pins forkCount. Fork count is also now a property
(-Dbigtable.it.fork-count=8) since the ITs are RPC bound and can go well
above the core count.

2. The prod-batch pass was redundant

bigtable-prod-batch-it re-ran the entire data.v2.it package against
batch-bigtable.googleapis.com — 905s, ~27% of the presubmit's wall clock.

It was added in 484b62a (#892, "fix: jwt authentication on
batch-bigtable.googleapis.com") to prove end to end that self-signed JWT auth
worked against an endpoint whose service host and JWT audience diverge. That
rationale no longer holds:

  • EnhancedBigtableStubSettings.Builder.setJwtAudienceMapping — the
    endpoint-to-audience table the profile guarded — is now @Deprecated and a
    no-op that just returns this.
  • batch-bigtable.googleapis.com appears nowhere in main source.
  • The profile passed bigtable.data-jwt-audience=${bigtable.cfe-jwt-audience},
    the same value as prod-it, and that property defaults to empty. So the only
    difference between the two passes was the data endpoint hostname.

The auth behaviour for that endpoint is still covered hermetically by
EnhancedBigtableStubTest, which points the stub at
batch-bigtable.googleapis.com:443 with JWT credentials against an in-process
server and asserts the emitted authorization header.

Removes the bigtable-prod-batch-it profile, the
internal-bigtable-prod-batch-it-prop-helper helper profile and the
bigtable.cfe-data-batch-endpoint property.

Note: .kokoro/presubmit/bigtable-integration.cfg still lists
bigtable-prod-batch-it in INTEGRATION_TEST_ARGS and needs the profile
dropped from that list. It is the only CI config referencing it
(bigtable-integration-dp.cfg uses bigtable-directpath-it and is
unaffected).

3. Slow tests

  • LargeRowIT.testSkipLargeRow (242s prod / 79s batch) sent 200
    sequential 3 MiB mutateRow calls — 600 MiB of serial round trips. The
    writes target distinct qualifiers and are independent, so they now go out
    with a bounded 16-deep in-flight window.
  • BigtableBackupIT (485s) had two flat 2-minute sleeps. RestoreTable
    only kicks off the optimize-restored-table operation once the backup is a
    couple of minutes old, and each restore test slept through that window
    itself. The backups are now created in setUpClass() and the tests only
    wait for whatever is left of the window after the rest of the class has run.
  • BigtableCmekIT (422s) polled key status on a {5, 10, 50, 100, 150, 200, 250, 300} second backoff, so it could oversleep 95s+ past the point
    where the status was already OK. Now a 10s fixed interval against a 360s
    deadline.
  • BigtableMaterializedViewIT, BigtableLogicalViewIT,
    BigtableSchemaBundleIT, BigtableAuthorizedViewIT
    each waited for a
    delete to propagate on a doubling {2, 4, ... 1024} backoff (2046s worst
    case). The resource normally disappears within seconds, so these are now a
    2s interval with a 60 attempt cap.

Follow-ups not done here

  • reuseForks=false costs ~20s of JVM boot per test class (~763s across 36
    cloud forks). The pom comment says BuiltinMetricsIT mutates process-global
    metrics state, so enabling reuse means first splitting that test into its
    own isolated execution.
  • TestEnvRule.cleanUpStale() runs after every test class (36+ times per run)
    and does a project-wide listInstances + listClusters + listTables +
    listAppProfiles scan, but only deletes resources older than a day. It is
    logically a once-per-run job.
  • The admin view ITs (BigtableMaterializedViewIT, BigtableLogicalViewIT,
    BigtableSchemaBundleIT) create and delete a fresh table in @Before/
    @After for every test method. No test mutates that table — it is only
    referenced by name in the view query — so one table per class would do.

@mutianf
mutianf requested review from a team as code owners September 10, 2026 02:43

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a MillisTimestampInterceptor to truncate client-generated timestamps to millisecond granularity, enabling several integration tests to run successfully against the Bigtable emulator. It also optimizes test execution times by parallelizing writes in LargeRowIT, pre-creating backups in BigtableBackupIT, and replacing slow exponential backoffs with faster polling intervals across multiple IT classes. The review feedback correctly identifies a missing import for java.util.concurrent.TimeUnit in LargeRowIT.java that would cause a compilation failure, and suggests a performance optimization in MillisTimestampInterceptor to avoid rebuilding request objects when no timestamp truncation is required.

Comment on lines +44 to 45
import java.util.ArrayList;
import java.util.List;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The class java.util.concurrent.TimeUnit is used in the newly added asynchronous write loop (e.g., TimeUnit.MINUTES), but it is not imported in this file. To prevent compilation failures, we should add the import.

Suggested change
import java.util.ArrayList;
import java.util.List;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;

The presubmit IT run was taking ~56 minutes for ~40 minutes of actual test
time, almost all of it serialized.

The dominant cause was the enable-verbose-grpc-logs profile pinning failsafe
to forkCount=1. That override dates back to when TestEnvRule installed a JUL
appender itself and in-JVM parallelism would interleave the log files; that
code is gone, and forking across *processes* was never the problem (each fork
gets its own JUL config and its own -output.txt via
redirectTestOutputToFile). Dropping it restores the module's intended
forkCount=4, which is now also overridable via -Dbigtable.it.fork-count.

On top of that, the slowest individual tests:

- LargeRowIT#testSkipLargeRow sent 200 sequential 3 MiB mutateRow calls
  (600 MiB of round trips) to build two oversized rows. The cells use
  distinct qualifiers, so they now go out asynchronously with a bounded
  in-flight window.
- BigtableBackupIT slept 2 minutes twice, waiting for a freshly created
  backup to become old enough for RestoreTable to trigger an optimize
  operation. Both backups are now created in setUpClass so the wait overlaps
  the rest of the class.
- BigtableCmekIT polled the CMEK key status on a {5,10,50,100,150,200,250,300}
  backoff, so it could sleep for minutes after the status was already OK.
  Replaced with a 10s poll under a 6 minute deadline.
- The four delete-propagation loops in the view/schema-bundle admin ITs used
  a {2,4,...,1024} doubling backoff for a resource that disappears within
  seconds. Replaced with a 2s poll under a 2 minute deadline.
@mutianf
mutianf force-pushed the speedup-integration-tests branch from 334bb75 to 62ccebb Compare September 10, 2026 02:50
`bigtable-prod-batch-it` re-ran the entire `data.v2.it` package against
`batch-bigtable.googleapis.com`, costing ~905s (~27% of the presubmit's
wall clock) on every run.

It was added in 484b62a (googleapis#892, "fix: jwt authentication on
batch-bigtable.googleapis.com") to prove end-to-end that self-signed JWT
auth worked against an endpoint whose service host and JWT audience
diverge. That rationale no longer holds:

- `EnhancedBigtableStubSettings.Builder.setJwtAudienceMapping` — the
  endpoint-to-audience table the profile guarded — is now `@Deprecated`
  and a no-op that just returns `this`.
- `batch-bigtable.googleapis.com` appears nowhere in main source.
- The profile passed `bigtable.data-jwt-audience=${bigtable.cfe-jwt-audience}`,
  the same value as `prod-it`, and that property defaults to empty. So the
  only difference between the two passes was the data endpoint hostname.

The auth behaviour for that endpoint is still covered hermetically by
`EnhancedBigtableStubTest`, which points the stub at
`batch-bigtable.googleapis.com:443` with JWT credentials against an
in-process server and asserts the emitted authorization header.

Removes the `bigtable-prod-batch-it` profile, the
`internal-bigtable-prod-batch-it-prop-helper` helper profile and the
`bigtable.cfe-data-batch-endpoint` property.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant