Memoize file contents by path in CachedParser to skip redundant reads

[SanderMuller]

CachedParser::parseFile() reads the whole file via FileReader::read() on every call, because the contents are the key of the AST cache. The same file is parsed many times (a trait file once per class that uses it), so the read repeats even when nothing changed. Measured on current 2.2.x with an instrumented build, cold: a large Laravel project does 143,613 parseFile() reads for 6,624 distinct files, with one hot trait file read 94,007 times; a large doctrine/symfony project does 24,155 reads for 6,251 distinct files.

This memoizes the contents by path, keyed by mtime and size, and skips the re-read when the file is unchanged. Following the direction of the recent cache work (LRU eviction, source-byte caps), the memo is bounded: total memoized source is capped at 512 KB (MEMOIZED_SOURCE_BYTES_LIMIT) with least-recently-used eviction, and files larger than the cap are never memoized. Hot trait files stay resident by definition, so the bound costs little: the read counts drop to 8,444 on the Laravel project (94% fewer) and 8,266 on the doctrine/symfony one (66% fewer).

Keying by size as well as mtime catches same-second edits that change the length in long-running processes (PHPStan Pro, fixer worker); filesize() is served from the stat cache populated by filemtime(), so it costs no extra syscall. A same-second, same-length edit is the remaining undetectable case, pinned in a test.

Effect, measured on Shopware v6.7.6.2 (9205 files, 9 workers, cold result cache), phars built from 2.2.x and from this branch:

	2.2.x	this PR
real	155.6 s	146.3 s
user	666.5 s	613.3 s
sys	103.9 s	54.4 s
total CPU	770.4 s	667.7 s (-13.3%)
FileReader::read() calls	1,970,713	54,149
bytes read	15,269 MB	217 MB
max RSS	851 MB	782 MB

Output is byte-identical. block input operations is 0 in both runs, so none of that 15 GB reaches the disk - the page cache serves it, and what the memo removes is the syscall round trips, the copy into a fresh userspace buffer and PHP's stream/string overhead. Re-reading a warm 929-byte file costs ~13.5 us per call against ~0.8 us for the stat pair the memo replaces it with, so the cost is per call rather than per byte.

A trait-light project sees much less: the same PR is flat on a Doctrine/Symfony application, and was within noise on the Laravel project this PR was originally measured against (sys 7.3 s -> 5.4 s, total CPU unchanged).

Memory: at most 512 KB per process (the earlier revision of this PR held contents unbounded, about +4 MB; that is gone). Sweeping the cap on Shopware shows the knee is at or below 64 KB - 64 KB already captures 98.0% of the achievable read reduction and every larger cap up to unbounded lands on the same 98.3% plateau - so 512 KB is on the plateau rather than at a cliff, and is 0.06% of the run's footprint.

Tests cover: unchanged file not re-read, size change detected with unchanged mtime, newer mtime re-read, oversized files never memoized, and LRU eviction at the byte cap.

[staabm]

How to reproduce/measure the performance improvement?

[SanderMuller]

Here's the A/B I used to measure it. Swap only CachedParser.php between this branch and its parent (bad7874ec), keep the same vendor, run cold (delete the tmp/cache dir before each run) and single-process, and compare sys and total CPU (user+sys) rather than wall. Wall is overlapped by the parallel workers and is too noisy on a shared machine to read a few-percent change from.

# in the phpstan checkout, swap just the one file between the two versions:
git show <ver>:src/Parser/CachedParser.php > src/Parser/CachedParser.php   # <ver> = this branch, then bad7874ec
# from the target project, cold + single-process, 3 reps:
rm -rf <tmpDir> && /usr/bin/time php <phpstan>/bin/phpstan analyse -l 8 --no-progress <paths>

The mechanism: parseFile() is called once per class that uses a given trait (and for other shared includes), so the same file's contents get read from disk many times in a single run. The memo keys the contents by path + mtime and reads each file once.

So the win tracks how much cross-file re-read redundancy a codebase has, which makes it corpus-dependent. Single-process, cold, 3 reps, median:

• Tempest (multi-package framework, many files pulling the same shared base classes/traits): sys 2.62s → 1.79s (-32%), total CPU 34.9s → 34.0s (-2.6%), user flat. The saving is in read syscalls, as expected; user time doesn't move because parsing itself is unchanged.
• rector-src (~1150 files, parses each roughly once): no measurable change (sys ~1.60s either way).

So it's a real win on high-fan-in codebases and neutral on low-redundancy ones. The content memo costs about 4MB regardless of whether the project benefits. If that always-paid cost is the concern, a bounded variant (memoize only files that get read more than once) would make the memory track the benefit; happy to do that if you'd prefer it.

The red CI check is the flaky RegressionBench wall-time assertion (it fails the same way on unrelated PRs that can't affect analysis time), not a real regression.

[SanderMuller]

Reworked and re-measured on top of the current 2.2.x (the LRU and source-byte-cap changes rewrote this file under the PR, so the numbers below are fresh, from instrumented builds, cold runs).

The call volume that motivates it: a large Laravel project does 143,613 parseFile() disk reads for 6,624 distinct files, with one hot trait file read 94,007 times; a large doctrine/symfony project does 24,155 reads for 6,251 distinct files. The contents are the AST-cache key, so the read happens on every call regardless of the AST cache policy.

Changes since your question:

• the memo is now bounded like the AST cache: 512 KB total with LRU eviction, files above the cap never memoized. The +4 MB unbounded retention from the first revision is gone. Reads still drop 94% (143,613 to 8,444) on the Laravel project and 66% on the doctrine/symfony one, since the hot trait files stay resident.
• keyed by mtime and size instead of mtime alone, so same-second edits that change the length are caught in long-running processes; filesize() comes from the stat cache, no extra syscall.

Measured effect on the current base (cold, single process, interleaved): sys 7.3 s to 5.4 s (-26%) on the Laravel project, every run with the change below every run without; total CPU and wall within noise. So this is a bounded syscall/IO reduction in the spirit of the recent duplicate-work PRs, not a latency win. Output stays byte-identical; the full test suite passes (17,565 tests).

[staabm]

I don't get what these 3 lines achieve here

[SanderMuller]

These three lines are the LRU touch on a cache hit: the unset plus re-assign moves the entry to the end of the array so it becomes most-recently-used. The eviction loop lower down drops array_key_first() (the oldest-touched entry), so touching on hit keeps a hot file resident while cold one-off reads get evicted first.

It's the same bookkeeping markRecentlyUsed() already does for the AST cache (the unset + re-append at lines 183-185). It's load-bearing for exactly the case this PR targets: a trait file read once per using class is interleaved with a stream of other files, so under the 512 KB bound plain FIFO would evict it between uses while LRU keeps it resident.

I inlined it rather than sharing a helper because the two caches key differently (source string vs file path). I can add a short comment here mirroring the one on markRecentlyUsed() so it's self-explanatory, or fall back to FIFO eviction if you'd rather keep this minimal.

[staabm]

this logic duplicates the markRecentlyUsed method.
lets extract a simple class which implements a LRU cache, backed by a array.

[SanderMuller]

Done in 622413e: PHPStan\Internal\LruCache, array-backed, and both caches use it.

get(string $key): mixed          // touches the entry, null when absent
set(string $key, mixed $value, int $weight): list<string>   // evicts until it fits, returns what it evicted
replace(string $key, mixed $value): void                    // same weight, refreshed position, evicts nothing
count(): int
all(): array<string, TValue>

Insertion order is the LRU order, so get() re-inserts and array_key_first() is the victim - the same trick both copies were doing by hand. set() returning the evicted keys is what lets CachedParser keep parsedByString in step without the cache knowing about it, and replace() exists so the parseFile-upgrades-a-parseString-entry path stays eviction-free the way it was.

Both bounds moved in unchanged: the AST cache keeps entry count + total source bytes + the floor that stops one oversized source from flushing everything, the file memo keeps its byte bound. That is why markRecentlyUsed() and evictLeastRecentlyUsed() are gone from CachedParser rather than just deduplicated.

Checks, since this touches a hot path:

• the 11 existing CachedParserTest tests pass untouched, including the exact post-eviction counts (50/33/20/21)
• new LruCacheTest covers touch-on-get, count eviction, weight eviction, the floor, replace, and the evicted-key list
• same number of file reads before and after: 2342 on the same analysis, counted with a probe in FileReader::read, so the refactor is inert with respect to what the PR optimizes
• full suite 21334, self-analysis clean, phpcs clean

I put it in src/Internal/ next to the other generic helpers; happy to move it under PHPStan\Parser if you would rather keep it local to the only caller.

[SanderMuller]

Rebased onto current 2.2.x — it was 314 commits behind, so its CI status was stale. Applied cleanly this time (the two-hunk conflict from the byte-cap refinements is gone), diff unchanged at +161/-1.

Gates: full suite green (21308), CachedParserTest 11 tests / 52 assertions, self-analysis clean, phpcs clean.

The measurement in the description is from early July. This one is a cold-run IO win, so it is worth re-measuring on current 2.2.x before you spend time on it — tell me and I will post a fresh interleaved set.

[staabm]

Please check whether/how much impact has this PR on analyzing the shopware codebase (which also uses traits heavily)

[SanderMuller]

Measured it on Shopware. Short answer: -12.7% CPU, and the output is byte-identical. It is the biggest win I have measured for this PR, and your hunch about traits is exactly why.

Setup

shopware/shopware at v6.7.6.2 — the same ref the integration test uses — installed with composer install, composer run framework:schema:dump and php src/Core/DevOps/StaticAnalyze/phpstan-bootstrap.php, then analysed with its own phpstan.neon.dist (9205 files under src + tests, 839 reported errors). Two phars built from upstream/2.2.x and from this branch, alternated, cold result cache every run, CPU as user+sys.

run	2.2.x	this PR
round 1	650.7s	568.7s
round 2	665.1s	579.6s
extra base run	674.4s
median	657.9s	574.1s

-12.7% (1.15x), and every PR run beat every base run. JSON output is identical between the two (same sha256, 423,440 bytes, 839 file errors + 2 errors), so nothing is being skipped.

Why it is so large here — the numbers behind it

Shopware is 107 trait definitions against 2322 use SomeTrait; statements, and parseFile() runs once per class using a trait. Instrumenting FileReader::read() in both phars, summed over the 9 worker processes of one full analysis:

	2.2.x	this PR
FileReader::read() calls	1,970,792	54,877
bytes read from disk	15,269 MB	217 MB

So analysing a 9205-file project currently reads 15 GB off disk; with the memoization it reads 217 MB. That is 36x fewer read calls and 70x fewer bytes, which is where the 12.7% comes from.

The PR still does 54,877 reads against roughly 20,800 distinct paths, because the memo is capped at MEMOIZED_SOURCE_BYTES_LIMIT (512 KB) with LRU eviction, so hot files get re-read after eviction. Raising that cap would shrink the remainder further — I have not tried it, and the current cap is deliberately conservative about memory.

Caveats

• PHP 8.5 locally, whereas the integration test pins 8.4 for Shopware.
• I used Shopware's own phpstan.neon.dist rather than the e2e/integration/shopware.neon wrapper (baseline + editor links), so absolute times will not line up with CI. Both sides used the identical config, so the delta is unaffected.
• composer install needed --no-security-blocking locally: with no composer.lock committed, resolution picks versions that now carry advisories my Composer blocks and CI's did not at its last green run. Worth knowing independently of this PR — it means the Shopware integration job is one advisory away from failing to install.

For contrast, the same PR is roughly flat on a Doctrine/Symfony application I benchmark with, so trait-heavy is indeed the distinguishing factor. If you want, I can rerun with the CI shopware.neon wrapper or on 8.4 before you decide.

[staabm]

why does this PR improve performance?
why is the OS kernel not able to properly cache the file-reads?
why is MEMOIZED_SOURCE_BYTES_LIMIT 512 KB? how did you come up with this concrete value?

[staabm]

Total CPU and wall are within noise, so this is a syscall/IO reduction, not a latency win.[10:43 Uhr]

how can I measure the number of syscalls a PHPStan run needs on macos to reproduce the before/after PR syscall counts?

[SanderMuller]

Good questions — the third one especially, because the honest answer was "convention, not measurement". I re-measured everything on Shopware (v6.7.6.2, 9205 files, 9 workers, cold result cache) with phars built from 2.2.x and from this branch.

First, a correction to my own description: "total CPU and wall are within noise" is out of date. That was measured on a Laravel project in July. On Shopware it is not noise:

	2.2.x	this PR
real	155.6 s	146.3 s
user	666.5 s	613.3 s
sys	103.9 s	54.4 s
total CPU	770.4 s	667.7 s (-13.3%)
max RSS	851 MB	782 MB

I will update the description.

1. Why does it improve performance?

parseFile() is called once per class using a trait, and it reads the whole file every time because the contents are the AST cache key. On Shopware that is 1,970,713 FileReader::read() calls for ~20,374 distinct paths, or 15.3 GB read; with the memo it is 54,149 calls / 217 MB.

Per avoided read (1,916,564 of them): 25.9 µs of sys time and 27.7 µs of user time. Sys time nearly halves, which is the syscall side; the user-time half is PHP's stream layer, the zend_string allocation for each result, and copying the bytes out of the page cache.

2. Why doesn't the OS page cache make this free?

It already does the part it can — and the measurement says so directly: block input operations is 0 in both runs. Nothing touches the disk in either case; all 15.3 GB comes from the page cache.

What the page cache cannot remove is the cost of asking for it: the open/fstat/read/close round trips, the copy from kernel pages into a fresh userspace buffer, and PHP's own stream-wrapper and string allocation on top.

That cost is dominated by the per-call overhead rather than the file size. Re-reading a 929-byte file 100k times, warm:

file_get_contents()      13450 ns/op
array lookup                10 ns/op
clearstatcache + 2 stats    841 ns/op   <- what the memo pays instead

An 11.8 KB file costs 15363 ns/op — 13x the bytes for 14% more time. So it is ~13 µs of fixed cost per call whatever the file, against ~0.8 µs for the stat pair the memo replaces it with. (That 13 µs is a floor measured in a tight loop where every cache is hot; in a real run, interleaved across 20k files, the observed cost is the ~54 µs above.)

3. Why 512 KB?

Honestly: because the surrounding cache work had just introduced byte caps with LRU eviction and I matched it. So I swept it. Read counts are deterministic, so they are the clean signal; CPU across a single run at each cap is noisy and I would not read anything into differences of a few seconds.

cap	FileReader::read() calls	share of the achievable reduction	max RSS
off	1,970,713	—	803 MB
64 KB	58,828	98.0%	828 MB
256 KB	54,986	98.3%	816 MB
512 KB	54,149	98.3%	828 MB
1 MB	53,881	98.3%	788 MB
4 MB	52,842	98.3%	865 MB
unbounded	52,601	98.3%	864 MB

So the knee is at or below 64 KB, and everything from there up is the same plateau — 512 KB is comfortably on it but there is nothing special about that number. It is 0.06% of the ~800 MB the run uses anyway, so the cap is not what bounds memory here. I am happy to drop it to 64 KB (same result, smaller promise) or raise it; tell me which you prefer and I will change it and re-measure.

One thing the sweep exposed that I did not expect: even unbounded, reads stay at ~2.6x the distinct-path count (52,601 vs 20,374 summed over the workers). So something re-reads files beyond the memo's reach — most likely more than one CachedParser instance per process, each with its own memo. That is a separate potential win and not something this PR addresses.

4. Measuring syscalls on macOS

The two I actually used here, neither needing root:

• /usr/bin/time -l — sys time is the aggregate syscall cost, and block input operations tells you whether anything reached the disk (0 = all page cache). This is where the 103.9 s -> 54.4 s number comes from.
• Counting at the PHP level — a static counter in FileReader::read() plus register_shutdown_function appending to a file, so each of the 9 workers reports its own count. Exact, portable, and it attributes the reads to the call site rather than to the process.

For real syscall counts there is sudo dtruss -c -f -- php ... (aggregate counts per syscall) or sudo fs_usage -w -f filesys -p <pid>. I could not verify either on this machine — SIP is enabled and I do not have passwordless sudo here — so I would rather not hand you a command I have not run. If you want those numbers I can get them and post the before/after.

[staabm]

I did also some measures on my macbook m4 pro on shopware/shopware

2.2.x at 9e4cf9d needs ~1min 5s
2.2.8 needs ~1min 4s
this PR: ~56s

PHP 8.5.9 (cli) (built: Jul 28 2026 13:06:52) (NTS)

so this proves the promise

[staabm]

thank you

[staabm]

@mitelg @keulinho @shyim this change should speedup shopware analysis by 5-10% with the next release

see #5928 (comment) and #5928 (comment)

[staabm]

@SanderMuller maybe there are more places in the phpstan-src codebase which can make use of this class

[SanderMuller]

Went looking for them - the tell is array_key_first() plus an unset/re-insert on a cache hit. Four sites, three of which are a real fit:

site	shape today	fit
PhpClassReflectionExtension::touchMemberCacheKey()	order map over four member caches, count-bounded	yes - this is the one that needs set()'s evicted-key list, since the four value maps have to drop the same key
FileTypeMapper::$memoryCache	touches on hit, count-bounded, manual $memoryCacheCount	yes - count() replaces the counter too
UsefulTypeAliasResolver::$resolvedLocalTypeAliases	touches on hit, count-bounded	yes
FileTypeMapper::$resolvedPhpDocBlockCache	no touch on hit, so it evicts in insertion order	no - it is FIFO, not LRU, despite sitting right next to one. Moving it to LruCache would change its eviction policy, which wants measuring rather than refactoring

I prototyped the first one because it is the odd shape (one order map governing four value maps) and I did not want to claim the API fits without checking:

private function touchMemberCacheKey(string $cacheKey): void
{
	if ($this->memberCacheOrder->get($cacheKey) !== null) {
		return;
	}

	foreach ($this->memberCacheOrder->set($cacheKey, true, 0) as $evictKey) {
		unset(
			$this->methodsIncludingAnnotations[$evictKey],
			$this->nativeMethods[$evictKey],
			$this->propertiesIncludingAnnotations[$evictKey],
			$this->nativeProperties[$evictKey],
		);
	}
}

LruCache<true> as a pure order keeper, 3158 reflection tests green and self-analysis clean. The other two are plain LruCache<TValue> with a count bound.

I would rather not fold these into this PR - all three are hot caches and this one is about the parser - so shall I open a follow-up with the three conversions once this lands? The FIFO one I would leave alone until someone measures whether LRU is actually better there.

[staabm]

please send a new PR

[SanderMuller]

Opened as #6240 — I had missed that this was already merged when I asked, so I just did it: the three true-LRU sites converted, resolvedPhpDocBlockCache left alone because it is FIFO.

One thing that came out of doing it rather than eyeballing it: nameScopeMapMemoryCacheCountMax: 0 did not mean "no limit" the way it does in the other three caches. Its eviction loop ran before the insertion, so 0 emptied the cache and put a single entry back — a one-entry cache. #6240 preserves that (new LruCache(1)) rather than silently giving anyone who set it to 0 an unbounded cache; normalising the four to agree is a behaviour change and would want its own PR.

Footprint is identical, which is the part equal output cannot show: 1168 name-scope map misses, 2 local type alias misses, 910 member-cache evictions on both sides.

[staabm]

@calebdw @canvural this PR might also positively affect analyze-performance of laravel projects which utilize a lot of traits