Time limit sensitivity should go both ways #414

Activity

1. gkreitz commented on Apr 27, 2026

   

   gkreitz

   on Apr 27, 2026

   Contributor

   More actions

   A TLE case below the time limit is definitely an error in my mind (and how I read the spec, both legacy and 2025-09), so I definitely do not want to turn that into a warning rather than an error.

   In fact, the current warning is likely too soft compared to the spec language. The intent, as I understand it, has always been that TLE submissions should run in > TL * time_safety_margin (or time_limit_to_tle` in the new version). I guess (without digging too much into history) the error was made a warning to make things close to TL * time_safety_margin not flip-flop between error and OK. I don't think the intent was ever to treat that warning as being OK to regularly hit.

   👍React with 👍2Reacted by Fredrik Niemelä and mzuenni
2. gkreitz commented on Apr 28, 2026

   

   gkreitz

   on Apr 28, 2026

   Contributor

   More actions

   Thinking a bit more about this - I wonder if we should improve the current check by making it harsher.

   The current code just has a generic "sensitive to time limit" warning, which tells you if you get different results between TL and TL * safety_margin. I think it may be valuable to have a special case for TLE in that check, to make the messaging more clear.

   Two ideas (I think both are good, but happy to take opinions):

   • Change the "sensitive to time limit" message for submissions which should be TLE. Let the message be something to the effect that TLE submissions should have a runtime > time limit * safety margin. Word smithing suggestions very welcome, it's hard to phrase stuff like this in a one-liner which is comprehensible.
   • Move the spot where we make this warning into an error, perhaps 50% into the safety margin. Regardless of where we put the breakpoint, if people are near it, they'll flip-flop. I think the current behavior of not flagging an error directly at time_limit * saftey margin is good (even if strict standard adherence probably would make this an error) to reduce flip-flopping.
3. gkreitz

   linked a pull request that will close this issueImprove messages when submissions change status around the time limit #500on Oct 9, 2026
4. pehrsoderman

   closed this as completedin #500on Oct 10, 2026