Fix extension() and localize() giving wrong results with fermitools 2.4-2.5.2

[ndilalla]

Summary

gta.extension() and gta.localize() silently returned incorrect results with Fermi ScienceTools 2.4.0 through 2.5.2. Extension fits reported nonsensical best-fit values with nan errors and ~zero TS_ext; localization reported positional uncertainties several times too large. Both worked correctly with fermitools 2.2.0.

This was caused by a regression in fermitools itself, now confirmed fixed in fermitools 2.5.3 (currently available as a fermi/label/dev conda build). This PR:

1. Works around the bug on affected fermitools versions (< 2.5.3) by using a slower but correct code path.
2. Automatically detects the installed fermitools version at runtime and uses the original, fast code path on 2.5.3+, where the bug is fixed — no configuration needed, and no permanent performance cost once users upgrade.

Symptom

Running the IC 443 extension tutorial:

Best-fit extension: 2.3829 +    nan -    nan
TS_ext:        0.000

Expected (matches Abdo et al. 2010, and what fermipy 1.4.1 / fermitools 2.2.0 produces):

Best-fit extension: 0.2734 + 0.0048 - 0.0036

Same pattern in localize(): for a TS~42 point source, reported pos_err/r68 were ~5-10x too large (0.38°/0.58° vs. the expected ~0.04-0.1°), with the position refinement scan's DeltaLogLike pinned near 0.

Root cause (upstream, in fermitools)

Both extension() and localize() implement fast width/position scans by calling BinnedLikelihood.setSourceMapImage() to overwrite a source's per-pixel counts map in place at each trial value, then re-fitting the normalization — this avoids the overhead of deleting and re-adding the source at every step.

A fermitools regression, introduced in fermi-lat/Likelihood@9f574d768 ("Fix slowdown by reverting reloadIfCleared()", 2024-08-22, first shipped in fermitools 2.4.0), caused SourceMap::reloadIfCleared() to no longer reset its internal m_dataCleared flag after regenerating a source's map. Since BinnedLikelihood::addSourceCounts() calls reloadIfCleared() unconditionally on every source on every likelihood evaluation, and m_dataCleared gets set the first time any source transitions out of the fixed-source list (a routine event — e.g. every source GTAnalysis.optimize() touches), the flag effectively became stuck true forever afterward. From that point on, the affected source's map was silently regenerated from its original spatial definition on every fit, discarding anything written via setSourceMapImage().

We verified this directly: overwriting a free source's map and reading it back before vs. after a fit() call showed the map correctly reflecting the injected value pre-fit, then reverting to a fixed value post-fit regardless of what was injected — with SourceMap.model_is_local flipping True -> False across the fit.

Filed upstream: <link to fermitools issue, if filed>. Confirmed fixed in fermitools 2.5.3.

What changed

Since the fast (use_pylike=False) path is unsafe on fermitools < 2.5.3, but correct (and considerably faster) on 2.5.3+, this PR adds a runtime version check and only forces the slow (use_pylike=True, recreate the source) path when needed.

• fermipy/gtutils.py (new)

  • FERMITOOLS_SRCMAP_FIX_VERSION = (2, 5, 3).
  • fermitools_srcmap_update_is_broken(): parses fermipy.get_st_version() and returns whether the installed ScienceTools predates the fix. Cached (lru_cache) since the installed version doesn't change at runtime. Fails safe (assumes the bug is present) if the version string can't be parsed.
  • use_pylike_srcmap_workaround(logger=None): the function callers actually use. Returns the same boolean, and — the first time (and only the first time) it returns True in a process — logs a logger.warning(...) explaining that the slower method is being used and that upgrading to fermitools >= 2.5.3 restores the fast path.

• fermipy/extension.py

  • _scan_extension_fast / _scan_extension_fast_ebin: use_pylike = reoptimize and use_pylike_srcmap_workaround(self.logger). The fast in-place path is used whenever just evaluating the likelihood without a re-fit (reoptimize=False, always safe), or whenever a re-fit follows but the installed fermitools has the fix; forced to the recreate-source path only when a re-fit follows on a broken fermitools version.
  • Post-scan state restoration mirrors the same condition: if the recreate-source path was used during the scan, restore through the same add/delete-source path (a captured raw pixel array may no longer match in size, which would raise RuntimeError: Wrong size for input model map on the old fast-path restore); otherwise restore the raw array directly as before.
  • The set_source_morphology(...) calls in _extension() / _fit_extension_full() that directly precede a _fit() call now use use_pylike_srcmap_workaround(self.logger) instead of a hardcoded value.
  • Removed a stray use_pylike=False override on the _fit_position(...) call in _fit_extension_full (the fit_position=True combined mode), which was also unnecessarily forcing the TS-map coarse-localization step into the unsafe path.

• fermipy/sourcefind.py

  • _scan_position (used by localize()'s refinement scan and by _fit_position()): use_pylike = use_pylike_srcmap_workaround(self.logger) (this loop always re-fits, so no reoptimize gate is needed here). Removed the now-dead _create_srcmap_cache call that only served the fast path when it was unconditionally disabled.

Validation

IC 443 extension fit (3FGL J0617.2+2234e), width scan [0.25, ..., 0.30]:

	ext	ext_err	TS_ext	time
Published (fermipy 1.4.1 / fermitools 2.2.0)	0.2734	+0.0048/-0.0036	large	--
Before this PR (fermitools 2.5.1, bug present)	2.3829	nan	0.0002	~20s
With workaround forced (fermitools 2.5.1)	0.2810	±0.0039	4286	~120s
With workaround forced (fermitools 2.5.2)	0.2745	±0.0037	4094	~110s
This PR, auto-detected fast path (fermitools 2.5.3, fixed)	0.2733	+0.0049/-0.0036	4098	~10s

On fermitools 2.5.3 the result is effectively an exact match to the published value, and runtime is back to the original ballpark (~10s vs. ~120s under the forced workaround) — confirming both that the upstream fix works and that this PR correctly re-enables the fast path automatically.

localize() on 3FGL J0619.4+2242 (TS~42), fermitools 2.5.1 (workaround forced):

	pos_err	r68	r95	r99	ΔLogLike (refinement)
Before this PR	0.380°	0.577°	0.930°	1.154°	0.004
With workaround forced	0.039°	0.060°	0.096°	0.119°	1.716

No warning is logged on fermitools 2.5.3; the one-time warning was confirmed to fire correctly on 2.5.1/2.5.2.

Tested in three environments: python 3.11 / fermitools 2.5.1 (bug), python 3.12 / fermitools 2.5.2 (bug), python 3.12 / fermitools 2.5.3-dev (fixed).

[codecov]

Codecov Report

❌ Patch coverage is 80.00000% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.89%. Comparing base (191a6f2) to head (5f58fd0).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
fermipy/extension.py	55.55%	4 Missing ⚠️
fermipy/gtutils.py	87.50%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #680      +/-   ##
==========================================
- Coverage   51.21%   50.89%   -0.32%     
==========================================
  Files         145      145              
  Lines       27443    27469      +26     
==========================================
- Hits        14054    13980      -74     
- Misses      13389    13489     +100     

Flag	Coverage Δ
unittests	50.89% <80.00%> (-0.32%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.