Repository navigation
Reduce the memory every process keeps for directory scans and normalized paths - #6983
Open
ondrejmirtes wants to merge 3 commits into
Open
ondrejmirtes wants to merge 3 commits into
ondrejmirtes wants to merge 3 commits into
Conversation
FileHelper's cache of normalized paths kept an array per original path, holding a second copy of the path - about 500 bytes per entry. Every file FileFinder finds goes through it, 6,653 paths for Sylius before analysis starts, which made it 3.2 MB. Keyed by the directory separator first, an entry is one hash bucket, and a path that normalizes to itself - nearly all of them - is the same string as its key: 0.3 MB for those 6,653 paths, and lookups are slightly faster. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b250-2318-7374-8dd7-2834f59aa720
OptimizedDirectorySourceLocatorFactory kept every file it checked, with its hash, stat signature and symbols, for the rest of the process, so that a file reachable from two locators would be looked at once. A directory is scanned once per process (OptimizedDirectorySourceLocatorRepository), so that kept 5.4 MB for 3,810 files in every Sylius worker with little to reuse. The files that were hashed or scanned are still shared by the locators of one scan, the ones vouched for by their stat signature are checked against each locator's cache - another stat for a file two locators reach. A locator none of whose files waits for the scan is filled in right away, so that on a warm run only one directory's cache entry is in memory at a time. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b250-2318-7374-8dd7-2834f59aa720
DirectoryWalker loaded the listings of every directory it ever walked into memory on the first walk and kept them for the rest of the process - 2.3 MB for Sylius's 3,067 directories, in the main process and every worker. Each walked directory now has its own file with the listings under it, loaded when it is walked and released after. A walk reads and writes only the listings it needs, and the directories it no longer reaches are left out of its file. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b250-2318-7374-8dd7-2834f59aa720
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every PHPStan process, including each parallel worker, kept about 10 MB more than it needs from the first symbol lookup on. Bisecting pointed to three caches, and this PR changes how each one is stored. Each speedup stays:
FileHelper's normalized path cache (from NodeDependencies: cache normalized filenames #4815). It kept an array per original path plus a second copy of the path, about 500 bytes per entry.FileFindersends every found file through it, so on Sylius it held 6,653 paths (3.2 MB) before analysis started. It is now keyed by the directory separator first, and a path that normalizes to itself, which is nearly every path, is stored once as both key and value. Those 6,653 paths now take 0.3 MB, and cached lookups are slightly faster (0.46 ms vs 0.58 ms per pass over 6,653 paths).OptimizedDirectorySourceLocatorFactory's$checkedFiles. It kept every checked file's hash, stat signature and symbols for the life of the process: 5.4 MB for 3,810 files in every Sylius worker. A directory is scanned only once per process anyway (OptimizedDirectorySourceLocatorRepository), so it is now local to one scan. A locator whose files are all unchanged gets filled as soon as its cache entry is checked. On a warm run, that means only one directory's cache entry is in memory at a time instead of all of them. The files that were hashed or scanned are still shared by the locators of one scan. A file that two locators reach and that its stat signature vouches for costs one extrastat().DirectoryWalker's listings. Every listing ever walked was loaded on the first walk and kept: 2.3 MB for Sylius's 3,067 directories, in the main process and in every worker. Each walked directory now has its own listings file, loaded for that walk and released after it.No caps or limits were added. None of the changed classes is turbo-shadowed. The C++
NodeScopeResolveronly calls back into the PHPFileHelper::normalizePath(), soturbo-extneeds no change.Measurements
Sylius (level 6, its own
phpstan.neon.distwith phpstan-doctrine, phpstan-symfony and phpstan-webmozart-assert, extension versions that every PHAR below accepts), PHP 8.4, 16 cores, no turbo extension, warm caches, result cache cleared before each run. Best of 5. Peak memory ismemory_get_peak_usage()recorded at shutdown of each process; max RSS and wall time come from/usr/bin/time. The 2.3.x-dev and this-PR PHARs were built withcompiler/bin/prepare+ box, as inphar.yml.One-line file (
function foo(mixed $c): mixed { strlen("a"); return $c; }, analysed with one spawned worker)--debug(one process)Full
src/analysis (16 workers)In PHAR mode, 2.1.34 uses more heap than 2.3 because its PHAR members have mtime 0. OPcache refuses to cache such members, so all the compiled code stays in the heap; since then
resign.phpstamps the members, and 2.3 enables OPcache in its workers. The source-mode row (withopcache.enable_cli=1) compares the analyser itself: this PR brings it back to within about 2 MB of 2.1.34.Breakdown of what the one-line source-mode run keeps after the scan, on 2.3.x-dev:
$checkedFiles5.4 MB,FileHelpercache 3.2 MB (now 0.3 MB), directory listings 2.3 MB.