Repository navigation
2026-10-08 Engineering Team #147
Description
Agenda
ActivitySim
-
PR follow-ups:
- #1083 Shadow-pricing segmentation fix: Sijia’s updated SANDAG example model, need to confirm updated targets are correct then merge.
- #1103 RNG: Jeff needs to respond to remaining review comments; then abench testing is required.
- #1110 Peak-memory reduction: Jan reviewed and approved, pending some small concerns.
abenchresults from MTC model are provided, results from SANDAG model or sufficient proxy should also be reviewed. - #1119 Extension loading: Reviewed and merged.
- Run manifest #1120: Jeff’s proposed scope and next steps; carried over from last week.
- Python support and installation: Jan’s expanded CI matrix, Windows/Python 3.13 failure(?), Sharrow’s Python 3.14 blocker, and Windows installer removal/feedback.
-
Numerical consistency: follow up on the PSRC Sharrow/non-Sharrow discrepancy and park-and-ride availability behavior discussed September 24, does #1123 resolve this.
-
New PRs need review:
PopulationSim
- Stabilization #219: final review/merge status, cross-platform Python testing, and unblocking intermediate geographies #217.
Release planning and Phase 12
- Phase 11 closeout: what's left? what schedule?
- Release policy
- Engineering input to Phase 12 planning: discuss now or defer to a new bench?
Activity
jpn-- commentedon Oct 12, 2026
Meeting Overview
Agenda
Participants
Duration
Approximately 61 minutes.
Executive Summary
The team reviewed remaining Phase 11 work and preparations for Phase 12, including a retrospective, more consistent issue documentation, and engagement with the incoming consultant bench. The shadow pricing fix is nearing completion, while the faster RNG pull request remains contingent on performance testing. Jeff demonstrated ABench, prompting discussion of release review, shared benchmarking infrastructure, and memory optimization; no cloud budget was approved. The team also discussed a versioned run manifest, Python compatibility, successful PSRC testing of PR #1129, and unresolved school shadow pricing questions.
Meeting Notes
Phase 11 Retrospective and Phase 12 Planning
Joe reported that the evaluation committee had submitted consultant bench recommendations to the PMC. Voting was scheduled to close the day after the meeting, with notifications to proposers planned afterward. Joe intends to circulate draft contracts shortly after the announcement.
Jeff proposed bringing the incoming consultants and consortium members together to discuss Phase 12 priorities and engineering practices once the bench is known.
Joe outlined two preparations discussed by the Product and Community team:
Amir Samimi’s scenario testing has identified additional requests for Phase 12. Joe encouraged documenting those as issues with the desired level of detail. The intended sequence is to improve the issue inventory and review Phase 11 while preparing to engage the new bench over the next few weeks.
Shadow Pricing Fix and Example Model Tests
Jeff relayed Sijia Wang’s update that the SANDAG example model now has targets consistent with the corrected shadow pricing implementation. He considers the substantive work largely complete, with verification and branch coordination remaining before merge.
The remaining checks include:
maininto WSP’s branch to pick up fixes for unrelated MTC test failures.Jan pointed out that SANDAG’s testing discussion is in its own repository. Jeff distinguished agency validation of model behavior from the consortium’s CI checks: both matter, but the consortium must verify the exact code and example-model inputs it distributes.
Joe noted that some charts and subsequent SANDAG cleanup discussed alongside the fix concern separate issues. He recommended keeping Bhargava Sana involved so that the agency and consortium remain coordinated. Jeff indicated that he would add Bhargava as a reviewer.
Faster RNG and ABench Performance Testing
Jan has approved the faster RNG pull request, with a few minor comments. Jeff will hold the merge until benchmarking demonstrates that it is at least as fast as the existing implementation.
Jeff demonstrated ABench’s automated comparison of a pull request against
main. The tool creates separate clean ActivitySim installations, runs the model, and reports runtime and memory use directly on the pull request.The example shown tested the explicit chunking changes using the MTC model with 500,000 households, all zones, and all model components. It showed substantial memory savings and some runtime improvement. The final summarize step remains a memory problem requiring a separate solution.
The SANDAG model exceeded the memory available on Jeff’s laptop. Jeff is integrating the smaller benchmark model prepared by Joe Flood, which retains more scale than the usual CI example and was intended to run in roughly half an hour on a laptop. This would support convenient benchmarking of both one-zone and two-zone models.
ABench lives in a separate repository so that changes to the benchmarking tool remain independent of the ActivitySim versions being compared. Jeff noted that its AI-generated implementation has not yet undergone thorough code review, so the benchmark results and methodology still need validation. Jan has also used the tool for the recent release and run benchmarks on AWS.
Release Review and Benchmark Records
The group discussed making performance assessment a more systematic part of pull request review and release planning.
Proposals included:
Jan suggested that compact submodel summaries could support checking output consistency as well as performance. Joe asked for an issue or feature request describing the proposed workflow so it can be considered during Phase 12 prioritization. No task owner or final scope was established.
Benchmarking Infrastructure, Costs, and Memory Optimization
Jeff clarified that running A-Bench requires compute resources, not AI tokens. AI assistance was used to develop the tooling, but benchmark execution itself is a separate expense.
The group discussed rough cloud cost estimates:
Jeff favored selective benchmarking of relevant pull requests before merge rather than running expensive tests on every push.
Alex questioned whether consortium-funded cloud infrastructure offered enough benefit to justify its risks and costs, including future price increases and uncertainty about comparability across years. Joe emphasized the value of a consistent hardware platform. Stefan noted that cloud instance specifications can be recorded and that consortium resources could be limited to explicitly authorized benchmarking uses.
Stefan also reported that PSRC primarily runs its models on owned servers because routine cloud model runs are too expensive, while retaining cloud access for selected uses.
No cloud funding decision was made. Joe proposed developing the idea for Phase 12 consideration.
For AI assistance costs, Jeff recommended treating consultant subscriptions or token usage as consultant overhead. Alex agreed with that approach; no formal funding policy was adopted.
Jeff also argued that reducing ActivitySim’s memory requirements may now offer greater value than relying on larger machines. Joe noted that those improvements would benefit local users as well as cloud benchmarking.
Extension Loading and Run Manifest Proposal
Jeff reported that the extension-loading pull request has been merged following Sijia’s approval and an AI review.
Jeff also described his issue proposing a versioned JSON manifest of the resolved run plan. External monitoring tools can currently record memory use over time, but interpreting those measurements is difficult when multiple processes are running different components and writing separate logs.
The proposed manifest would describe, at the beginning of a run:
This would let monitoring tools understand the run without independently reconstructing ActivitySim’s internal logic from settings files. A stable, versioned schema would also help tools work across ActivitySim versions.
Jeff requested review from multiple developers before implementation, particularly around what information external tools need. Potential uses extend beyond monitoring to settings checks and validation. No final schema was agreed upon.
Python Compatibility
Jan reported that the Windows Python 3.13 CI failure was resolved through a dependency update.
Python 3.14 support remains blocked by Sharrow. Jan expects the fix may be straightforward and agreed to investigate. Jeff asked him to report back if it proves more involved, noting that the compatibility issue will need to be addressed either way.
Zero Probabilities and School Shadow Pricing
Stefan reported that PSRC tested PR #1129, which allows zero probabilities, and that it appears to resolve the park-and-ride problem they had encountered. The affected lots are recoded to
-1, and the previously problematic values no longer cause the crash.Stefan also described failures in school location choice using simulation-based shadow pricing. He attributed these to very unfavorable utilities interacting with Sharrow’s 32-bit floating-point calculations. The resulting failed households remained below the default failure threshold.
Stefan asked whether shadow pricing can be disabled specifically for schools. Joe thought it might already be configurable, but this was not confirmed. Jeff offered to investigate and said that, if unsupported, the option should be feasible to add.
Joe also flagged Amir’s recent email about zero probabilities as potentially relevant, though the connection was not established. He proposed a broader diagnosis of reported shadow pricing problems to distinguish implementation defects, configuration issues, and behavior inherent to the simulation-based method.
For schools, Stefan emphasized that limited alternatives and capacity constraints can push students to much more distant locations. He would prioritize realistic distance distributions over assigning every student to the exact school, particularly given uncertainty in enrollment and capacity data. Jeff agreed that practical configuration options and smoother model behavior deserve attention. No broader change to the school modeling approach was decided.
Action Items
main.