[janedbal]

Follow-up to #6963. That PR streams the cached collected data to the rules on CollectedDataNode. The fresh data from the workers still stayed in the main process as arrays. On a full analysis, that is all collected data of the project.

Main gains

• Less memory when files are analysed. The main process writes the collected data from each worker batch to a spool file in the tmpDir. It keeps only where each value list is, with its hash.
• The most memory goes away where the most data is fresh: a full analysis, and a run with many changed files.
• Dead Code Detector does not need offloadCollectorData any more. With this PR, true and false use the same memory in every scenario.

Measurements

A large private codebase (~30k analysed files) with Dead Code Detector, whose rule reads its data with iterate() (shipmonk-rnd/dead-code-detector#489). Peak memory of the main process:

	2.3.x	this PR
full analysis, DCD default (offloadCollectorData: true)	3.99 GB	3.65 GB
full analysis, offloadCollectorData: false	4.74 GB	3.64 GB
2,472 files changed, true	3.37 GB	3.28 GB
2,472 files changed, false	3.57 GB	3.27 GB
no file changed, both	3.18–3.19 GB	3.18–3.19 GB

The time does not change.

All scenarios, times and the setup

Linux CI with 14 CPUs, Dead Code Detector 1.4.2 with its get() calls replaced by iterate(). Both PHARs are built in the same job from 2.3.x at bcca2b0, without and with this PR. All runs report the same errors.

A PHAR built this way does not match the version of the phpstan_turbo extension, so turbo is off in both. That is why a full analysis takes ~170 s here, not ~105 s as with the release PHAR.

Peak memory of the main process and wall time:

	2.3.x, true	this PR, true	2.3.x, false	this PR, false
full analysis	3.99 GB, 171–181 s	3.65 GB, 172–178 s	4.74 GB, 168–171 s	3.64 GB, 169–182 s
no file changed	3.19 GB, 28–29 s	3.19 GB, 28–29 s	3.18 GB, 26–27 s	3.18 GB, 26–28 s
1 file changed	3.19 GB, 53 s	3.19 GB, 53 s	3.18 GB, 52 s	3.18 GB, 52 s
2,472 files changed	3.37 GB, 85 s	3.28 GB, 86 s	3.57 GB, 84 s	3.27 GB, 84 s

The full analysis ran 3–4 times in each mode, alternating the modes. The first full run of each job took ~181 s, in one job with 2.3.x, in the other with this PR. The step after the analysis (the rules on CollectedDataNode and the save) took 22.3–23.1 s with this PR and 23.1–23.2 s without it with false. With true it varied between 24 and 32 s in both.

What changes

• CollectedDataSpool (new, internal) writes each value list to the spool file when a worker sends it, and keeps its offset, length and hash. The file is deleted when nothing refers to it.
• SpooledCollectedData (new, internal) is the index of the spool: collector type → file → offset, length, hash.
• LazyCollectedData gets the spooled part next to the cached part and the fresh arrays. iterate(), hashCollector() and toArray() read it like the cached part.
• ParallelAnalyser writes to the spool instead of the arrays.
• ResultCacheManager::save() copies the spooled value lists into the result cache as they are, one file at a time. Only a collector with paths in its values (CollectorWithPaths) is serialized again, with relative paths. The saved file is the same, byte for byte, as with the arrays.

Details:

• The data of a trait is stored under the trait's file, and each file that uses the trait can be in a different job. The spool merges those value lists at the end of the analysis.
• The order of the files and the hashes are the same as with the arrays.
• Editor mode and the single-process analysis keep the arrays: editor mode renames the temporary file later and analyses few files.
• When the spool cannot be written (a full disk), the data goes back to memory, as without the spool.
• A forked worker holds a copy of the spool object. It does not delete the file of the main process.

Known trade-offs

• The spool takes disk space in the tmpDir for the duration of the run: about the size of the collected data section of the result cache.
• A run that is killed leaves the spool file behind. It is named after the process, as the temporary file of the result cache is, so a later run with the same process ID overwrites it.

Co-Authored-By: Claude Code