Repository navigation
Memory issue on full cache flush #45
Description
We've noticed that running bin/typo3 cache:flush sometimes runs into memory exhaustion issues.
I'm not sure if the problem is with our TYPO3 instance or if it's an issue with this extension.
In https://github.com/b13/proxycachemanager/blob/master/Classes/Cache/Backend/ReverseProxyCacheBackend.php#L128-L136 all cached URLs are being fetched from table cache_tx_proxy:
$conn = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable($this->cacheTable);
$stmt = $conn->select(['content'], $this->cacheTable);
while ($url = $stmt->fetchOne()) {
$urls[] = $url;
}Question 1: does it really make sense to fetch them one by one? That's one hella DB queries, isn't it?
This big fat array of URLs is then being handed over to $this->reverseProxyProvider->flushAllUrls($urls);.
Question 2: would it be better to implement chunking on $urls and iterating over those chunks on your extension already or rather on our custom CurlHttpProxyProvider?
Looking forward to hearing back from you guys!
Activity
ischmittis commentedon Aug 25, 2025
+1 for Chunking especially for big installations when Num of URLs exceeds 10 k(Num of URLs = Number of pages * Number of Languages)
bmack commentedon Aug 28, 2025
Hey, fetching 10K DB records should be a no-brainer in 2025 - both in terms of performance and memory. Although I agree this is suboptimal in terms of coding, I haven't found any perf issues here
The main problem we have / had a lot is the amount of time it takes to do the Invalidation on the Edge Servers. A few numbers from the top of my head from my client installations:
For CloudFront, we've implemented grouping and chunking on the client. Example: We have 100 URLs that all start with www.example.com/en/products/dish-washers/ (and then /dish-washer-1/ etc) - then it's actually important we can do this grouping on the PHP side with all "to be flushed" URLs. We then send one API Request to invalidate
www.example.com/en/products/dish-washers/*(also to save money, as they charge per invalidation).For some of our custom providers, we've implemented a queue worker now to flush the caches in the background. this might be a solution for the future in general: "deferred cache flush" which does the cache flush in a separate process in general. Especially when editors publish something within a workspace, the UI is frozen (= due to the CDN API HTTP requests).