Skip to content

Optimize _get_faces_constLat_intersection_info - #1799

Open
cmdupuis3 wants to merge 11 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/optimize_integrate
Open

cmdupuis3 wants to merge 11 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/optimize_integrate

Conversation

@cmdupuis3

@cmdupuis3 cmdupuis3 commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1570

Finishes off #790

Overview

This PR moves the per-face work in uxarray/grid/integrate.py into Numba, mainly for the robust zonal-weights path (zonal_mean(..., use_robust_weights=True)). That path used to build several small polars DataFrames for every candidate face, and that fixed per-call cost was nearly all of its runtime. Each latitude is now two Numba calls:

  • One finds the longitude intervals of every candidate face, reusing one set of scratch buffers for the whole latitude.
  • One runs the sweep line that resolves overlapping intervals.

zonal_mean output is bit-identical to main for both the robust and default paths on oQU480 and oQU120. In ASV, ZonalAverage.time_zonal_average_robust drops from 2.10 s to 13.2 ms at 120 km (about 160×) and from 530 ms to 5.46 ms at 480 km (about 97×). Locally the robust path is now as fast as the default one or faster. Both per-latitude kernels release the GIL (nogil=True): four concurrent calls from four threads take 1.14× the wall time of one.

The default path gets a smaller cleanup. _compute_face_arc_length builds its points and segments in fixed-size arrays instead of numba typed Lists, which makes the default weights about 1.6× faster on a single thread. Cold JIT compilation of the first robust zonal_mean drops from 2.5 s to 1.7 s. The private helpers now take and return arrays instead of DataFrames:

  • _zonal_face_weights_robust returns the weight array directly.
  • _get_zonal_face_interval returns (start, end) rows.
  • _process_overlapped_intervals takes starts, ends and face indices, and returns per-face contributions.

The only change outside integrate.py is the matching one-line update in core/zonal.py.

Behaviour changes, all in private helpers:

  • A face whose crossing longitudes can't be paired raises ValueError instead of polars' ShapeError.
  • When every candidate face only touches the latitude, robust weights are NaN instead of raising ZeroDivisionError. That matches the default path, and zonal.py already skips those latitudes.
  • The concave-face check is removed: by construction it could never fire.
  • The error path no longer prints diagnostics to stdout.

Two changes on main needed adapting. After #1727, gca_const_lat_intersection returns a tuple of tuples. After #1705, the error messages format the face array, which Numba can't do, so the raises moved into the Python wrappers. Two pole tests in test_zonal_weights.py were each defined three times with identical bodies, so only the last copy ever ran; the extra copies are gone. polars stays a dependency because io/_healpix.py, io/_scrip.py and grid/validation.py still use it.

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • There is adequate test coverage of changes from this PR (add new tests if needed)
  • If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation and Examples

  • Docstrings updated with any function changes, and included in all new functions
  • User (public) functions added to docs/api.rst; internal (private) function names start with an underscore (_)

AI Disclosure

AI Usage: Claude Opus 5

  • I have tested and take responsibility for all AI-generated content in my PR.

@cmdupuis3 cmdupuis3 self-assigned this Sep 30, 2026
@cmdupuis3 cmdupuis3 added scalability Related to scalability & performance efforts run-benchmark Run ASV benchmark workflow labels Sep 30, 2026
cmdupuis3 and others added 2 commits September 30, 2026 13:30
Adapt the Numba intersection kernel to two changes on main:

- UXARRAY#1727 changed gca_const_lat_intersection to return a tuple of tuples
  instead of a (2, 3) array, so the kernel now reads points as
  intersections[r][i] rather than intersections[r, i].
- UXARRAY#1705 made the no-intersection and concave-face errors format the face
  array into their messages, which Numba cannot do, so both raises move
  from the kernel into the Python wrapper. The kernel now also returns the
  number of non-dummy edges that the concave check needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [709c98d] After [275800c] Ratio Benchmark (Parameter)
- 613±10μs 529±7μs 0.86 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
- 636±30μs 574±5μs 0.9 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
- 2.10±0.01s 13.2±0.09ms 0.01 mpas_ocean.ZonalAverage.time_zonal_average_robust('120km')
- 530±4ms 5.46±0.3ms 0.01 mpas_ocean.ZonalAverage.time_zonal_average_robust('480km')

Benchmarks that have stayed the same:

Change Before [709c98d] After [275800c] Ratio Benchmark (Parameter)
1.26±0.02ms 1.28±0.02ms 1.02 connectivity.Connectivity.time_edge_face('120km')
461±10μs 472±6μs 1.02 connectivity.Connectivity.time_edge_face('480km')
5.06±0.1ms 4.96±0.2ms 0.98 connectivity.Connectivity.time_edge_node('120km')
1.13±0.02ms 1.12±0.07ms 0.99 connectivity.Connectivity.time_edge_node('480km')
63.1±9μs 64.3±8μs 1.02 connectivity.Connectivity.time_face_edge('120km')
58.3±3μs 61.7±7μs 1.06 connectivity.Connectivity.time_face_edge('480km')
818±20μs 825±30μs 1.01 connectivity.Connectivity.time_face_face('120km')
337±30μs 320±10μs 0.95 connectivity.Connectivity.time_face_face('480km')
68.4±3μs 70.5±7μs 1.03 connectivity.Connectivity.time_face_node('120km')
58.3±6μs 57.0±7μs 0.98 connectivity.Connectivity.time_face_node('480km')
675±50μs 591±50μs ~0.88 connectivity.Connectivity.time_n_nodes_per_face('120km')
385±20μs 390±10μs 1.01 connectivity.Connectivity.time_n_nodes_per_face('480km')
1.06±0.04ms 1.05±0.06ms 0.99 connectivity.Connectivity.time_node_edge('120km')
416±6μs 432±10μs 1.04 connectivity.Connectivity.time_node_edge('480km')
73.9±2ms 73.5±0.8ms 0.99 connectivity.Connectivity.time_node_face('120km')
4.45±0.05ms 4.38±0.04ms 0.99 connectivity.Connectivity.time_node_face('480km')
1.42M 1.42M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('120km')
105k 105k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('480km')
6.48M 6.48M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('120km')
419k 419k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('480km')
2.05k 2.05k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('120km')
2.05k 2.05k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('480km')
1.6M 1.6M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('120km')
101k 101k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('480km')
2.18k 2.18k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('120km')
2.18k 2.18k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('480km')
240k 240k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('120km')
26k 26k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('480km')
1.9M 1.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('120km')
127k 127k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('480km')
11.9M 11.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('120km')
747k 747k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('480km')
7.31±0.2ms 7.10±0.05ms 0.97 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
2.39±0.05ms 2.40±0.02ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
8.11±1ms 7.99±9ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.48±0.04ms 1.46±0.02ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
57.3k 57.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
12.3k 12.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
123k 123k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
128 128 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.27M 1.27M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
50.1k 50.1k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.48M 1.48M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
712 712 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.77M 1.77M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
1.75M 1.75M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.91M 1.91M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
35.3k 35.3k 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.14±0.02μs 1.14±0.02μs 1.00 geometry_kernels.AccucrossKernels.time_accucross
2.62±0.04μs 2.62±0.03μs 1.00 geometry_kernels.AccucrossKernels.time_accucross_pair
396±30ns 397±60ns 1.00 geometry_kernels.EFTPrimitives.time_acc_sqrt_re
415±9ns 416±60ns 1.00 geometry_kernels.EFTPrimitives.time_diff_of_products
382±90ns 346±40ns ~0.91 geometry_kernels.EFTPrimitives.time_two_prod
354±70ns 375±50ns 1.06 geometry_kernels.EFTPrimitives.time_two_sum
645±30ns 574±70ns ~0.89 geometry_kernels.GCAConstLatIntersection.time_accux_constlat_kernel
758±50ns 754±30ns 0.99 geometry_kernels.GCAConstLatIntersection.time_gca_const_lat_intersection
870±30ns 814±30ns 0.94 geometry_kernels.GCAConstLatIntersection.time_try_gca_const_lat_intersection
827±40ns 826±30ns 1.00 geometry_kernels.GCAGCAIntersection.time_accux_gca_kernel
942±20ns 937±30ns 1.00 geometry_kernels.GCAGCAIntersection.time_gca_gca_intersection
1.09±0.02μs 1.13±0.04μs 1.04 geometry_kernels.GCAGCAIntersection.time_try_gca_gca_intersection
31.5±0.7μs 32.3±0.5μs 1.03 geometry_kernels.OrientPredicates.time_on_minor_arc
32.0±0.7μs 33.0±0.6μs 1.03 geometry_kernels.OrientPredicates.time_orient3d_on_sphere
3.07±0ms 3.05±0ms 1.00 geometry_samebody.SameBodyConstLat.time_accux_dispatch
1.16±0ms 1.16±0ms 1.00 geometry_samebody.SameBodyConstLat.time_accux_kernel
2.31±0.02ms 2.29±0ms 0.99 geometry_samebody.SameBodyConstLat.time_fp64_dispatch
148±0.3μs 168±20μs ~1.14 geometry_samebody.SameBodyConstLat.time_fp64_kernel
44.5±0.1ms 44.6±0.2ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_dispatch
4.16±0.03ms 4.17±0.04ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_kernel
41.0±0.07ms 41.1±0.09ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_dispatch
546±6μs 524±10μs 0.96 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_kernel
1.02±0.03s 1.01±0.02s 0.99 import.Imports.timeraw_import_uxarray
1 1 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_isel
2 2 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_open_grid_chunked
96.8±2ms 93.0±0.8ms 0.96 lazy_grid_construction.OpenGridChunked.time_open_grid('120km')
129±3ms 129±2ms 1.00 lazy_grid_construction.OpenGridChunked.time_open_grid('480km')
1.99M 2.01M 1.01 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('120km')
2.3M 2.32M 1.01 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('480km')
2.45±0.06ms 2.52±0.08ms 1.03 mpas_ocean.AzimuthalAverage.time_azimuthal_average('120km')
1.00±0.02ms 1.01±0.01ms 1.01 mpas_ocean.AzimuthalAverage.time_azimuthal_average('480km')
2.90±0.07ms 2.66±0.04ms 0.92 mpas_ocean.CheckNorm.time_check_norm('120km')
2.04±0.02ms 2.04±0.06ms 1.00 mpas_ocean.CheckNorm.time_check_norm('480km')
1.33±0.03ms 1.21±0.02ms ~0.91 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
755±10μs 725±20μs 0.96 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
3.30±0.01ms 3.27±0.01ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
2.38±0.01ms 2.36±0.01ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
82.8±0.07ms 83.4±0.5ms 1.01 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
7.32±0.03ms 7.46±0.1ms 1.02 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
18.1±0.2ms 17.8±0.02ms 0.98 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
898±10μs 892±40μs 0.99 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
9.88±0.08ms 10.0±0.06ms 1.01 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
555±7μs 560±20μs 1.01 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
451±2ms 453±3ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 1)
231±3ms 229±3ms 0.99 mpas_ocean.CrossSections.time_const_lat('120km', 2)
117±0.8ms 117±0.4ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 4)
387±3ms 390±3ms 1.01 mpas_ocean.CrossSections.time_const_lat('480km', 1)
192±0.2ms 194±1ms 1.01 mpas_ocean.CrossSections.time_const_lat('480km', 2)
98.4±0.2ms 98.9±0.3ms 1.00 mpas_ocean.CrossSections.time_const_lat('480km', 4)
32.4±0.2ms 32.3±0.3ms 1.00 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.52±0.01ms 3.58±0.03ms 1.02 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
11.3±0.5ms 11.1±0.2ms 0.98 mpas_ocean.FaceAreas.time_face_areas('120km')
3.86±0.08ms 3.87±0.1ms 1.00 mpas_ocean.FaceAreas.time_face_areas('480km')
229k 229k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('120km')
14.3k 14.3k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('480km')
2.12M 2.12M 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('120km')
674k 674k 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('480km')
33.0±0.3ms 34.3±0.4ms 1.04 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'exclude')
93.9±1ms 94.1±1ms 1.00 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'split')
3.37±0.1ms 3.32±0.1ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'exclude')
17.8±0.2ms 17.6±0.2ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'split')
13.7±0.2ms 13.4±0.08ms 0.98 mpas_ocean.Gradient.time_gradient('120km')
2.33±0.02ms 2.31±0.02ms 0.99 mpas_ocean.Gradient.time_gradient('480km')
457k 457k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('120km')
28.7k 28.7k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('480km')
3.44M 3.44M 1.00 mpas_ocean.Gradient.track_peakmem_gradient('120km')
226k 226k 1.00 mpas_ocean.Gradient.track_peakmem_gradient('480km')
281±10μs 266±6μs 0.95 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
114±3μs 117±1μs 1.02 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
138±1μs 139±2μs 1.01 mpas_ocean.Integrate.time_integrate('120km')
117±2μs 117±1μs 1.00 mpas_ocean.Integrate.time_integrate('480km')
18.4M 18.4M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('120km')
1.2M 1.2M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('480km')
111±0.9ms 110±1ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
112±0.9ms 109±0.9ms 0.98 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
111±2ms 109±1ms 0.98 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
8.01±0.04ms 7.80±0.03ms 0.97 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
8.04±0.03ms 8.59±0.4ms 1.07 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
7.98±0.05ms 7.86±0.08ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
234±0.4ms 237±1ms 1.02 mpas_ocean.NeighborhoodBuild.time_build('120km', 1.0)
486±2ms 483±2ms 0.99 mpas_ocean.NeighborhoodBuild.time_build('120km', 5.0)
12.6±0.02ms 12.7±0.1ms 1.01 mpas_ocean.NeighborhoodBuild.time_build('480km', 1.0)
16.1±0.3ms 15.6±0.01ms 0.97 mpas_ocean.NeighborhoodBuild.time_build('480km', 5.0)
232±2ms 230±0.5ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 1.0)
486±2ms 479±3ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 5.0)
12.5±0.06ms 12.4±0.08ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 1.0)
15.6±0.2ms 15.4±0.1ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 5.0)
612.76 612.76 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 15.0)
74.17 74.17 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 5.0)
37.29 37.29 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 15.0)
6.57 6.57 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 5.0)
141M 141M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 15.0)
17.4M 17.4M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 5.0)
563k 563k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 15.0)
123k 123k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 5.0)
145M 145M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 15.0)
21.5M 21.5M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 5.0)
824k 824k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 15.0)
384k 384k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 5.0)
39.0±0.5ms 39.0±0.7ms 1.00 mpas_ocean.NeighborhoodDask.time_mean('120km', 'grid_chunks')
18.1±0.5ms 21.8±5ms ~1.20 mpas_ocean.NeighborhoodDask.time_mean('120km', 'numpy')
35.5±0.6ms 35.3±0.2ms 0.99 mpas_ocean.NeighborhoodDask.time_mean('120km', 'time_chunks')
10.2±0.2ms 10.3±0.1ms 1.01 mpas_ocean.NeighborhoodDask.time_mean('480km', 'grid_chunks')
647±10μs 602±10μs 0.93 mpas_ocean.NeighborhoodDask.time_mean('480km', 'numpy')
7.04±0.09ms 7.04±0.06ms 1.00 mpas_ocean.NeighborhoodDask.time_mean('480km', 'time_chunks')
5.82M 5.75M 0.99 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'grid_chunks')
2.75M 2.75M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'numpy')
5.67M 5.68M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'time_chunks')
744k 687k 0.92 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'grid_chunks')
177k 177k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'numpy')
520k 528k 1.01 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'time_chunks')
4.77±0.06s 4.39±0.02s 0.92 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'mean')
4.89±0.02s 4.50±0s 0.92 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'median')
117±0.5ms 117±0.9ms 1.00 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'mean')
120±0.7ms 121±0.9ms 1.01 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'median')
509±0.9ms 491±4ms 0.96 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'mean')
535±3ms 526±2ms 0.98 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'median')
16.6±0.2ms 15.9±0.03ms 0.96 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'mean')
16.7±0.2ms 16.5±0.1ms 0.99 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'median')
4.70±0.05ms 4.61±0.03ms 0.98 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'mean')
37.7±0.07ms 37.3±0.1ms 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'median')
239±50μs 196±1μs ~0.82 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'mean')
484±20μs 475±8μs 0.98 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'median')
234k 234k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'mean')
235k 235k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'median')
19.5k 19.5k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'mean')
19.6k 19.6k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'median')
391±10μs 368±10μs 0.94 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
357±30μs 353±30μs 0.99 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
299±20μs 352±5μs ~1.18 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
326±30μs 328±7μs 1.01 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
104±0.6ms 102±0.7ms 0.98 mpas_ocean.RemapDownsample.time_bilinear_remapping
16.7±0.2ms 16.4±0.2ms 0.98 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
15.1±0.2ms 15.3±0.2ms 1.01 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
1.11±0.01s 1.11±0s 1.00 mpas_ocean.RemapUpsample.time_bilinear_remapping
27.8±3ms 28.2±2ms 1.01 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
12.0±0.3ms 13.0±1ms 1.08 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
8.07±0.3ms 7.58±0.08ms 0.94 mpas_ocean.ZonalAverage.time_zonal_average('120km')
4.24±0.07ms 4.13±0.08ms 0.97 mpas_ocean.ZonalAverage.time_zonal_average('480km')
1.04763630688818 1.0288776568041553 0.98 nogil_scaling.GILScaling.track_gil_scaling
929±9μs 959±9μs 1.03 open_dataset.OpenDataset.time_open_dataset('dataset', 'dataset')
1.78±0.01ms 1.83±0.01ms 1.03 open_dataset.OpenDataset.time_open_dataset('dataset', 'file')
3.67±0.02ms 3.73±0.01ms 1.02 open_dataset.OpenDataset.time_open_dataset('file', 'dataset')
4.43±0.04ms 4.50±0.02ms 1.02 open_dataset.OpenDataset.time_open_dataset('file', 'file')
151±3μs 152±3μs 1.01 open_dataset.OpenDataset.time_open_dataset('grid', 'dataset')
925±20μs 938±10μs 1.01 open_dataset.OpenDataset.time_open_dataset('grid', 'file')
1.29±0.01s 1.29±0.01s 1.00 plots.PlotContour.time_contour_projected
228±1ms 226±2ms 0.99 plots.PlotHealpix.time_default_plot(6)
2.17±0.01s 2.16±0.01s 0.99 plots.PlotHealpix.time_default_plot(8)
28M 28M 1.00 plots.PlotHealpix.track_peakmem_default_plot(6)
443M 443M 1.00 plots.PlotHealpix.track_peakmem_default_plot(8)
10.2±0.2ms 9.41±0.3ms 0.93 quad_hexagon.QuadHexagon.time_open_dataset
8.60±0.4ms 8.16±0.2ms 0.95 quad_hexagon.QuadHexagon.time_open_grid
408 408 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_dataset
392 392 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_grid
71.3k 71.3k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_dataset
71k 71.1k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_grid
10.5±0.3ms 10.5±0.09ms 1.00 subset.BoundingBox.time_bounding_box('120km')
5.93±0.2ms 6.04±0.2ms 1.02 subset.BoundingBox.time_bounding_box('480km')
2.38M 2.38M 1.00 subset.BoundingBox.track_peakmem_bounding_box('120km')
203k 203k 1.00 subset.BoundingBox.track_peakmem_bounding_box('480km')
4.01±0.1ms 3.99±0.1ms 0.99 subset.BoundingBoxHealpix.time_bounding_box(6)
14.8±2ms 14.9±0.3ms 1.01 subset.BoundingBoxHealpix.time_bounding_box(8)
2.18M 2.18M 1.00 subset.BoundingBoxHealpix.track_peakmem_bounding_box(6)
13.2M 13.2M 1.00 subset.BoundingBoxHealpix.track_peakmem_bounding_box(8)
658±3ms 651±4ms 0.99 to_raster.ToRaster.time_to_raster((10000.0, 10.0))
1.39±0.01s 1.38±0.01s 0.99 to_raster.ToRaster.time_to_raster((10000.0, 100.0))
1.62±0.01s 1.62±0.01s 1.00 to_raster.ToRaster.time_to_raster((100000.0, 100.0))
270M 270M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 10.0))
680M 680M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 100.0))
765M 765M 1.00 to_raster.ToRaster.track_peakmem((100000.0, 100.0))
122±4μs 123±3μs 1.00 to_xarray.UxDataArrayToXarray.time_to_xarray(10)
328±10μs 320±10μs 0.98 to_xarray.UxDatasetToXarray.time_to_xarray((100, 10))
3.12±0.1ms 2.85±0.07ms 0.91 to_xarray.UxDatasetToXarray.time_to_xarray((1000, 100))
682±7μs 674±9μs 0.99 to_xarray.UxDatasetToXarray.time_to_xarray_healpix((100, 10))
6.71±0.2ms 6.38±0.2ms 0.95 to_xarray.UxDatasetToXarray.time_to_xarray_healpix((1000, 100))

Benchmarks that have got worse:

Change Before [709c98d] After [275800c] Ratio Benchmark (Parameter)
+ 304M 388M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
+ 305M 388M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
+ 307M 388M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
+ 305M 388M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
+ 278M 389M 1.4 import.Imports.track_peakmem_import_uxarray
+ 327M 388M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 1)
+ 328M 388M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 2)
+ 328M 388M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 4)
+ 307M 388M 1.26 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 1)
+ 308M 388M 1.26 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 2)
+ 307M 388M 1.26 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 4)
+ 327M 390M 1.19 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('120km')
+ 301M 390M 1.29 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('480km')
+ 328M 390M 1.19 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('120km')
+ 308M 390M 1.26 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('480km')

cmdupuis3 and others added 8 commits September 30, 2026 14:08
… Numba

The robust path built several tiny polars DataFrames per candidate face,
and that fixed per-call overhead was nearly its whole runtime. Each
latitude is now two Numba calls: one computing every candidate face's
longitude intervals, and one running the overlap sweep line. Results are
bit-identical, and robust zonal means are 52-86x faster, now within
1.3-2.7x of the default path instead of 100-200x slower.

Both new per-latitude kernels release the GIL. Each call does milliseconds
of work, so from 4 threads the robust weights take 1.09x the single-thread
wall time instead of 4.03x.

The private helpers now use arrays instead of DataFrames:
_zonal_face_weights_robust returns the weights, _get_zonal_face_interval
returns (start, end) rows, and _process_overlapped_intervals takes
starts/ends/face indices and returns per-face contributions. A face whose
crossing longitudes cannot be paired into intervals now raises ValueError
rather than polars' ShapeError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Robust zonal weights allocate their scratch space once per latitude
instead of three to four small arrays per face, compute each
intersection's longitude once, replace np.unique with an in-place sort,
and sort the sweep-line events once instead of twice. The default
path's per-face arc length builds its points and segments in fixed-size
arrays instead of numba typed Lists. Results are bit-identical on oQU480
and oQU120 across 168 latitudes. Robust weights are 2.2x faster, default
weights 1.6x faster single-threaded, and the first robust zonal_mean
compiles in 1.7 s instead of 2.5 s.

Also:
- Robust weights return NaN when every candidate face is only touched
  by the latitude, matching the default path, instead of raising
  ZeroDivisionError. zonal.py already skips non-finite weights.
- Drop the concave-face check, which could not fire, the error branch
  matching a message nothing raises, and the error helper's prints.
- Combine nested ifs flagged by ruff SIM102.
- Remove two pole weight tests that were each defined three times with
  identical bodies, so only the last copy ever ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3
cmdupuis3 marked this pull request as ready for review November 20, 2026 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-benchmark Run ASV benchmark workflow scalability Related to scalability & performance efforts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unoptimized routines in integrate.py

1 participant