Repository navigation
Hung task timeout reported by CI #3387
Description
.===========================>>
[ 3318.829119] kernel: INFO: task kworker/u16:0:17963 blocked for more than 122 seconds.
[ 3318.829227] kernel: Not tainted 5.16.0-rc1-daily-default-20220111-0 #55744127
[ 3318.829239] kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<<===========================
Activity
marc-hb commentedon Jan 21, 2022
A few weeks ago I spent some time investigating a similar report for a volume-basic-test. My preliminary conclusion was that the storage was overwhelmed by the volume of kernel logs.
kv2019i commentedon Jan 28, 2022
marc-hb commentedon Feb 3, 2022
Just found by chance that journald.conf has some rate limit options, worth a try.
marc-hb commentedon Feb 11, 2022
Another side effect: on some devices we have less than 1 week of logs:
Would some log level tuning make sense?
cc: @fredoh9 , @keqiaozhang , @aiChaoSONG , @kv2019i
plbossart commentedon Feb 12, 2022
new one: https://sof-ci.01.org/linuxpr/PR3338/build7152/devicetest/?model=CML_SKU0955_HDA&testcase=check-playback-3times
looks like file system issues again
keqiaozhang commentedon Feb 17, 2022
Another one:
http://sof-ci.sh.intel.com/#/result/planresultdetail/10270?model=CML_SKU0955_HDA&testcase=volume-basic-test-50
plbossart commentedon Feb 17, 2022
yes, but look further and it's again ext4 locking issues. Nothing apparently to do with SOF.
marc-hb commentedon Apr 8, 2022
Here's another interesting issue that happened in daily 11687?model=TGLH_SKU0A70_HDA&testcase=volume-basic-test-50
Note the 35 minutes gap!
@plbossart is there an easy way to reduce the log volume of this test?
sof-logger has the ability to fine-tune per component log levels at run timesilly me these are kernel logs.Tracepoints maybe? #3482
plbossart commentedon Apr 8, 2022
Indeed this GLB_TRACE_MSG: DMA_POSITION is not very useful, we should check if we can remove it.
marc-hb commentedon May 10, 2022
This recent one also has a ton of
WARN dai_copy(): Copy_bytes 0 + avail bytes 0 < period bytes 384, possible glitch(logs also going to the disk)https://sof-ci.01.org/softestpr/PR903/build21/devicetest/?model=CML_SKU0955_HDA&testcase=check-pause-resume-playback-10
marc-hb commentedon May 10, 2022
Other interesting logs just found by chance.
Also linking to
which could help mitigate a bit?
Daily 12389?model=CML_SKU0955_HDA&testcase=check-ipc-flood
19 remaining items
keqiaozhang commentedon May 25, 2022
Thanks for @ujfalusi suggestion.
@marc-hb and @plbossart
I verified this issue with latest
linux-nextkernel https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/ and this issue cannot be reproduced anymore.Here is the first round of test result for 2 Dell platforms which are easy to hit this issue: https://sof-ci.sh.intel.com/#/result/planresultdetail/12875
There's only one failure and caused by the new debugging code for checking the write speed of storage.
So I triggered another round of test on all 4 Dell platforms and also remove the workaround(pagecache setting), the results are really good and consistent.
Please refer to: https://sof-ci.sh.intel.com/#/result/planresultdetail/12877
This is really a good news, it seems that this issue has been fixed in latest upstream kernel.
plbossart commentedon May 25, 2022
@keqiaozhang can you try with 5.18 to see if that already fixes the problem? linux-next includes stuff that will be provided later, or in 5.19-rc1 and that in itself will add other issues.
marc-hb commentedon May 25, 2022
Maybe a newer kernel has marginally better performance but that's IMHO only hiding the real, device-specific issue here.
As noticed by @plbossart , we've observed these hangs only on certain TGL devices with (normally!) very high performance NVMe storage. I just discovered that the storage performance on some of our old BYT devices is ONE HUNDRED times lower than on those recent TGL devices:
Yet we never ever noticed anything like this on BYT!? Something does not add up.
Anyway we're not in the storage business so whatever hides this issue is good enough for me.
EDIT: CML TIMEOUT in
syncin https://sof-ci.01.org/linuxpr/PR3645/build166/devicetestbase-config: add FRAMEBUFFER_CONSOLE
keqiaozhang commentedon May 26, 2022
Be noticed that some BYT devices use the SD card for storage, not SSD or NVME. please check the storage size for the low performance BYT devices. the storage size should be 16G or 32G if it uses SD card.
marc-hb commentedon May 26, 2022
Very interesting but that's not my point. My point is: incredibly slow BYT storage never triggered any "hung task" or TIMEOUT. So this is not just a "slow storage" issue, something else is really wrong with these devices and we still have no idea what it is. Maybe storage hangs sometimes, or maybe it's not even storage. So neither the newer kernel versions or
vm_dirtychanges can really fix these issues, they only make it less likely. So these failures will be back.keqiaozhang commentedon May 26, 2022
I tested with https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git/log/?h=for-5.18 kernel before, the hung task timeout issue is still reproducible.
The last update of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git/log/?h=for-5.18 is 14 days ago. Same commit ID as I tested before.
Then I tested with today's
for-next, the hung task timeout issue remains.I also checked with
linux-nexthttps://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tag/?h=v5.18, the hung task timeout is still exists.So it seems that the fix only landed on
linux-nextmaster branch.marc-hb commentedon May 27, 2022
Reproduced again in daily 12927?model=TGLU_SKU0A32_SDCA&testcase=multiple-pause-resume-50
I had a look at the results of the quick storage test I just added in thesofproject/sof-test@34ca191b92a32 and write speed was blazingly fast as usual before and after this failure, zero issue found with actual storage.
Start Time: 2022-05-26 22:28:02 UTC
End Time: 2022-05-27 04:34:14 UTC
Kernel Branch: topic/sof-dev
Kernel Commit: 46f61a08b40c
SOF Branch: main
SOF Commit: 13b659259de2
marc-hb commentedon May 27, 2022
This issue is intermittent, so the lack of reproduction can be just luck.
marc-hb commentedon Jun 2, 2022
Whatever the problem is, it is definitely triggered by more intense logging. The
sudo syncI recently added makes that obvious.In daily test run 13008?model=TGLH_SKU0A70_HDA&testcase=check-ipc-flood
after 10,000 lines like this:
Same on the day before; 10,000 lines like these in only 2 seconds and then hang after
sync.plbossart commentedon Jun 29, 2022
@marc-hb I don't recall having seen this in recent PR or daily tests, should we close or is this still current?
marc-hb commentedon Jun 29, 2022
It's been root-caused to failing hard drives, probably because of SSD firmware bugs, like for instance this one:
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=754y5
(This is just random example; I'm not saying we use this particular model)
See internal issue 233 for more details.
Flash storage is complex, so it is buggy:
https://en.wikipedia.org/wiki/Write_amplification
https://www.bunniestudios.com/blog/?p=3554
It is still happening on some devices that have not been "serviced" yet, however the failure signature is usually very different since I added this quick storage test: thesofproject/sof-test#910
marc-hb commentedon Jun 30, 2022
And now for something different:
https://sof-ci.01.org/sofpr/PR5959/build785/devicetest/?model=TGLU_UP_HDA_ZEPHYR&testcase=check-suspend-resume-with-playback-5
Never a dull moment...