Skip to content

Commit 2c6876c

Browse files
TEST-162 Update Smarter Testing onboarding docs (#10164)
This change addresses feedback from onboarding sessions. - Add verification step after pushing to CI in getting started. - Clarify that `discover` and `run` are still required for TIA. - Fix the locally generated impact data file name. - Clarify how to validate TIA locally. - Add verification step after pushing to CI in TIA. - Move "Next Steps" above setup examples to better show those examples are optional and TIA is now setup.
1 parent 41069af commit 2c6876c

2 files changed

Lines changed: 51 additions & 8 deletions

File tree

docs/guides/modules/test/pages/getting-started-with-smarter-testing.adoc

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -422,6 +422,14 @@ jobs:
422422

423423
Commit both `.circleci/test-suites.yml` and `.circleci/config.yml` to your feature branch and push to your VCS.
424424

425+
=== Verify in CI
426+
427+
After your pipeline runs, verify the test job is behaving as expected before merging.
428+
429+
. In the CircleCI web app, navigate to your pipeline and open the test job.
430+
. Check the *Tests* tab and confirm the number of test results matches what you see when running tests without the `testsuite` command. A difference in test results could be a misconfiguration in your `discover` command.
431+
. Compare the job duration to a recent run without the `testsuite` command. The runtime should be similar - a significant difference may indicate a misconfiguration in your `test-suites.yml`.
432+
425433
== Next steps
426434

427435
At this stage, all of your tests should successfully run locally and in CI using the `testsuite` command.

docs/guides/modules/test/pages/set-up-test-impact-analysis.adoc

Lines changed: 43 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ Before enabling test impact analysis, ensure you have completed the xref:getting
6464

6565
== 1. Enable test impact analysis option in your `.circleci/test-suites.yml` file
6666

67-
Update the testsuite from the getting started to include an `analysis` command and `test-impact-analysis` option.
67+
Update the testsuite from the getting started to include an `analysis` command and `test-impact-analysis` option, keep the `discover` and `run` commands in the config.
6868

6969
Some starter configs are shown below. Choose the one for your test runner and copy/paste the `analysis: ...` line into `.circleci/test-suites.yml`.
7070

@@ -220,7 +220,7 @@ Remove the `--doctor` flag and add the following flags: `--local --select-tests=
220220
$ circleci run testsuite "ci tests" --verbose --local --select-tests=none --analyze-tests=impacted
221221
----
222222

223-
Then inspect the locally stored impact data in `.circleci/impact-ci\ tests.json` to see the mapping between tests and files. This is a JSON file with a form like:
223+
Then inspect the locally stored impact data in `.circleci/ci tests-impact.json` to see the mapping between tests and files. This is a JSON file with a form like:
224224
[source,json]
225225
----
226226
{
@@ -230,6 +230,10 @@ Then inspect the locally stored impact data in `.circleci/impact-ci\ tests.json`
230230
"path": ".circleci/test-suites.yml",
231231
"hash": "c9684be83632a628"
232232
},
233+
"2": {
234+
"path": "src/foo.ts",
235+
"hash": "c9684be83632a628"
236+
},
233237
...
234238
},
235239
"edges": {
@@ -238,13 +242,33 @@ Then inspect the locally stored impact data in `.circleci/impact-ci\ tests.json`
238242
}
239243
----
240244

241-
Try modifying a source file whose ID appears in the list for a test atom and then run test selection to verify only impacted test atoms are selected.
245+
* `files` are a map of IDs to files with the `path` relative to the working directory and a `hash` of the contents.
246+
* `edges` are a map of test atoms with an array of file IDs that impact this test atom.
247+
248+
Try modifying a source file whose ID appears in the `edges`, then run test selection to verify only impacted test atoms are selected.
242249

243250
[source,console]
244251
----
245252
$ circleci run testsuite "ci tests" --verbose --local --select-tests=impacted
246253
----
247254

255+
Look for the "Selecting tests..." section in output to see why a test was selected.
256+
257+
[source,console]
258+
----
259+
==> Selecting tests...
260+
--> Selecting 'test-atom-one' due to modified file: 'src/foo.ts'
261+
==> - 0 new test atoms
262+
==> - 0 test atoms impacted by new files
263+
==> - 1 test atoms impacted by modified files
264+
==> - 0 test atoms impacted by removed files
265+
==> - 0 test atoms failed previously
266+
==> - 0 test atoms with no source file mappings in impact data
267+
==> - 0 test atoms impacted by include rule
268+
==> - 0 test atoms impacted by full test run paths
269+
==> Selected 1 test atoms, Skipped 123 test atoms in 168ms
270+
----
271+
248272
=== Local flags
249273

250274
[#local-cli-flags]
@@ -352,12 +376,27 @@ jobs:
352376
path: test-reports
353377
----
354378

355-
Commit both `.circleci/test-suites.yml` and `.circleci/config.yml` to your feature branch and push to your VCS. Follow your usual process to merge to your default branch.
379+
Commit the restored `.circleci/config.yml` to your feature branch and push to your VCS. Follow your usual process to merge to your default branch.
380+
381+
NOTE: The local impact JSON files generated from running the local CLI should not be checked in to VCS.
382+
383+
=== Verify in CI
384+
385+
After analysis completes on your feature branch, verify test impact analysis is working correctly.
386+
387+
. In the CircleCI web app, navigate to your pipeline and open the test job.
388+
. Check the job was successful and analyzed all tests. A successful analysis run will output "Found n files impacting tests" for each test atom.
389+
. Modify a source file that appears in the impact data, then push. Only the tests that cover the modified file should be selected to run. Look for the "Selecting tests..." output in the job to confirm the correct tests were selected and the reason for selection.
356390

357391
*Test impact analysis is now set up for your test suite*. Feature branches will run the tests impacted by code changes, and your default branch will first run all tests and then analyze the tests impacted by code changes.
358392

359393
There are xref:#set-up-examples[other options] available if these defaults don't suit your project.
360394

395+
== Next steps
396+
397+
* xref:use-dynamic-test-splitting.adoc[Use Dynamic Test Splitting] to evenly split tests across parallel nodes.
398+
* xref:auto-rerun-failed-tests.adoc[Auto Rerun Failed Tests] to automatically retry flaky tests.
399+
361400
[#set-up-examples]
362401
== Common setup examples
363402

@@ -487,7 +526,3 @@ jobs:
487526
path: test-reports
488527
----
489528

490-
== Next steps
491-
492-
* xref:use-dynamic-test-splitting.adoc[Use Dynamic Test Splitting] to evenly split tests across parallel nodes.
493-
* xref:auto-rerun-failed-tests.adoc[Auto Rerun Failed Tests] to automatically retry flaky tests.

0 commit comments

Comments
 (0)