Skip to content

Reduce the memory every process keeps for directory scans and normalized paths - #6983

Open
ondrejmirtes wants to merge 3 commits into
2.3.xfrom
reduce-per-process-memory
Open

ondrejmirtes wants to merge 3 commits into
2.3.xfrom
reduce-per-process-memory

Conversation

@ondrejmirtes

Copy link
Copy Markdown
Member

Every PHPStan process, including each parallel worker, kept about 10 MB more than it needs from the first symbol lookup on. Bisecting pointed to three caches, and this PR changes how each one is stored. Each speedup stays:

  • FileHelper's normalized path cache (from NodeDependencies: cache normalized filenames #4815). It kept an array per original path plus a second copy of the path, about 500 bytes per entry. FileFinder sends every found file through it, so on Sylius it held 6,653 paths (3.2 MB) before analysis started. It is now keyed by the directory separator first, and a path that normalizes to itself, which is nearly every path, is stored once as both key and value. Those 6,653 paths now take 0.3 MB, and cached lookups are slightly faster (0.46 ms vs 0.58 ms per pass over 6,653 paths).
  • OptimizedDirectorySourceLocatorFactory's $checkedFiles. It kept every checked file's hash, stat signature and symbols for the life of the process: 5.4 MB for 3,810 files in every Sylius worker. A directory is scanned only once per process anyway (OptimizedDirectorySourceLocatorRepository), so it is now local to one scan. A locator whose files are all unchanged gets filled as soon as its cache entry is checked. On a warm run, that means only one directory's cache entry is in memory at a time instead of all of them. The files that were hashed or scanned are still shared by the locators of one scan. A file that two locators reach and that its stat signature vouches for costs one extra stat().
  • DirectoryWalker's listings. Every listing ever walked was loaded on the first walk and kept: 2.3 MB for Sylius's 3,067 directories, in the main process and in every worker. Each walked directory now has its own listings file, loaded for that walk and released after it.

No caps or limits were added. None of the changed classes is turbo-shadowed. The C++ NodeScopeResolver only calls back into the PHP FileHelper::normalizePath(), so turbo-ext needs no change.

Measurements

Sylius (level 6, its own phpstan.neon.dist with phpstan-doctrine, phpstan-symfony and phpstan-webmozart-assert, extension versions that every PHAR below accepts), PHP 8.4, 16 cores, no turbo extension, warm caches, result cache cleared before each run. Best of 5. Peak memory is memory_get_peak_usage() recorded at shutdown of each process; max RSS and wall time come from /usr/bin/time. The 2.3.x-dev and this-PR PHARs were built with compiler/bin/prepare + box, as in phar.yml.

One-line file (function foo(mixed $c): mixed { strlen("a"); return $c; }, analysed with one spawned worker)

2.1.34 2.3.0 2.3.x-dev this PR
peak memory, worker 101.42 MB 74.09 MB 73.97 MB 63.42 MB
peak memory, main process 82.96 MB 55.97 MB 55.97 MB 51.19 MB
max RSS 134.6 MB 167.6 MB 168.4 MB 158.5 MB
wall time 1.96 s 2.14 s 2.15 s 2.14 s
peak memory, from source, --debug (one process) 24.87 MB – 37.63 MB 27.01 MB

Full src/ analysis (16 workers)

2.1.34 2.3.0 2.3.x-dev this PR
wall time 8.59 s 6.28 s 6.95 s 6.18 s
peak memory, largest worker 213.40 MB 158.47 MB 162.79 MB 146.94 MB
peak memory, main process 131.56 MB 95.91 MB 95.91 MB 91.11 MB
max RSS 240.7 MB 265.1 MB 269.2 MB 252.6 MB

In PHAR mode, 2.1.34 uses more heap than 2.3 because its PHAR members have mtime 0. OPcache refuses to cache such members, so all the compiled code stays in the heap; since then resign.php stamps the members, and 2.3 enables OPcache in its workers. The source-mode row (with opcache.enable_cli=1) compares the analyser itself: this PR brings it back to within about 2 MB of 2.1.34.

Breakdown of what the one-line source-mode run keeps after the scan, on 2.3.x-dev: $checkedFiles 5.4 MB, FileHelper cache 3.2 MB (now 0.3 MB), directory listings 2.3 MB.

FileHelper's cache of normalized paths kept an array per original path,
holding a second copy of the path - about 500 bytes per entry. Every file
FileFinder finds goes through it, 6,653 paths for Sylius before analysis
starts, which made it 3.2 MB.

Keyed by the directory separator first, an entry is one hash bucket, and a
path that normalizes to itself - nearly all of them - is the same string as
its key: 0.3 MB for those 6,653 paths, and lookups are slightly faster.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b250-2318-7374-8dd7-2834f59aa720
OptimizedDirectorySourceLocatorFactory kept every file it checked, with its
hash, stat signature and symbols, for the rest of the process, so that a
file reachable from two locators would be looked at once. A directory is
scanned once per process (OptimizedDirectorySourceLocatorRepository), so
that kept 5.4 MB for 3,810 files in every Sylius worker with little to reuse.

The files that were hashed or scanned are still shared by the locators of
one scan, the ones vouched for by their stat signature are checked against
each locator's cache - another stat for a file two locators reach. A
locator none of whose files waits for the scan is filled in right away, so
that on a warm run only one directory's cache entry is in memory at a time.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b250-2318-7374-8dd7-2834f59aa720
DirectoryWalker loaded the listings of every directory it ever walked into
memory on the first walk and kept them for the rest of the process - 2.3 MB
for Sylius's 3,067 directories, in the main process and every worker.

Each walked directory now has its own file with the listings under it,
loaded when it is walked and released after. A walk reads and writes only
the listings it needs, and the directories it no longer reaches are left
out of its file.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b250-2318-7374-8dd7-2834f59aa720
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant