GH-50779: [C++][Parquet] Replace remaining RapidJSON usage with simdjson by Reranko05 · Pull Request #50781 · apache/arrow

Reranko05 · 2026-08-03T10:41:09Z

Rationale for this change

This PR continues the simdjson migration by replacing the remaining RapidJSON usages under cpp/src/parquet with simdjson and JsonWriter. It also removes the remaining unnecessary RapidJSON dependencies from the Parquet test target and Meson build configuration.

What changes are included in this PR?

Replace the remaining RapidJSON parsing logic in reader_test.cc with simdjson.
Replace RapidJSON parsing and serialization in geospatial/util_json_internal.cc with simdjson and JsonWriter.
Replace RapidJSON string escaping in types.cc with JsonWriter.
Remove the unused rapidjson_dep dependency from cpp/src/parquet/meson.build.
Remove the remaining RapidJSON dependency from the Parquet test target in CMakeLists.txt.
GitHub Issue: [C++][Parquet] Replace remaining RapidJSON usage with simdjson #50779

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-08-03T10:44:42Z

⚠️ GitHub issue #50779 has been automatically assigned in GitHub to PR creator.

Reranko05 · 2026-08-03T14:42:40Z

@rok @taepper

I'm migrating parquet/geospatial/util_json_internal.cc and ran into an ondemand question.

GeospatialGeoArrowCrsToParquetCrs() needs to:

inspect the crs object (e.g. check the id.authority/code fields to detect EPSG:4326/CRS84), and
if it isn't a recognized lon/lat CRS, serialize the same object back to JSON.

With RapidJSON this was straightforward because the DOM is reusable, but with simdjson::ondemand the object has already been consumed by the time I want to serialize it, leading to OUT_OF_ORDER_ITERATION.

Is the expected approach here to restructure the code into a single pass, or is there another way to serialize an already-inspected ondemand::object?

Thanks!

rok · 2026-08-03T15:08:43Z

As per docs here you could probably use the .reset() in case you don't get desired CRS fields.

  // Inspect crs_object...

  // If it was not a recognized lon/lat CRS:
  ARROW_ASSIGN_OR_RAISE(
      auto ignored,
      ::arrow::internal::GetSimdjsonResult(
          crs_object.reset(), "Failed to reset 'crs' object: "));

  ARROW_ASSIGN_OR_RAISE(
      auto raw_crs,
      ::arrow::internal::GetSimdjsonResult(
          crs_object.raw_json(), "Failed to get raw 'crs' JSON: "));

  return std::string(raw_crs);

Check also rewind and consider what's best for your usecase.

Reranko05 · 2026-08-03T16:38:13Z

Thanks @rok, reset() + raw_json() fixed the iteration issue.

However, one test still fails because raw_json() preserves the original formatting (e.g. {"key0": "value0"}), whereas the previous RapidJSON Writer minified it to {"key0":"value0"}. LogicalType::Equals() compares the CRS string literally.

Would you prefer keeping the previous compact formatting or preserving the original JSON formatting?

rok · 2026-08-03T16:55:05Z

@Reranko05 let's avoid behavior changes. Perhaps we can use simdjson::minify?

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-08-04T06:26:04Z

@kou @rok @pitrou Could you review this when you have time?

rok

Perhaps we can remove rapidjson from here also?

rok · 2026-08-04T14:49:39Z

I also see many RapidJSON references in cpp/cmake_modules/ThirdpartyToolchain.cmake. Are we ready to remove those yet? (I don't know, hence the question)

kou · 2026-08-04T21:33:04Z

-  json << R"(, "crs": )" << buffer.GetString();
+  ::arrow::json::JsonWriter writer;
+  writer.String(crs);
+  json << R"(, "crs": )" << writer.GetString().ValueUnsafe();


Can we use JsonWriter for all JSON build instead of mixing manual JSON build and JsonWriter build?

Good point. That would require refactoring the entire ToJSON() implementation instead of only the crs field. I wasn't sure if that was in scope for this PR. Should I include that refactoring here?

Reranko05 · 2026-08-05T03:21:00Z

I also see many RapidJSON references in cpp/cmake_modules/ThirdpartyToolchain.cmake. Are we ready to remove those yet? (I don't know, hence the question)

@rok RapidJSON is no longer used by Parquet after this change, so I've removed the Parquet-specific dependency. However, there are still many RapidJSON users elsewhere in Arrow (e.g. arrow/json, arrow/integration, arrow/extension, arrow/flight/sql), so I don't think we're ready to remove the ThirdpartyToolchain references yet.

wgtmac · 2026-08-05T06:23:45Z

+
+  auto padded_json = simdjson::padded_string(json_string);
+
+  if (auto error = parser.iterate(padded_json).get(document)) {


CheckJsonValid() no longer validates the full JSON output: it creates an On-Demand document but never consumes it, so only stage-1 errors are detected. This can let malformed printer output pass the test. The previous parser also intentionally enabled full precision and NaN/Inf, so the replacement should preserve that contract while actually traversing and validating the document.

The NaN/Inf case is a very unfortunate one. While not being part of the official JSON RFC, it will be added to the next simdjson release: simdjson/simdjson#2696

I added behavior preserving json parsing to our testing helpers in from_string.cc:

#50653

Maybe that work-around is also acceptable here?

(in the case we really want to keep supporting NaN/Inf here, the tests were using json to generate test data for our Float computations so it was required. Again, it is not part of the JSON RFC so dropping support for it here is defendable, although many JSON libraries such as RapidJSON do support it)

I did not preserve the previous NaN/Inf behavior, since simdjson doesn't currently support it. Should we preserve the previous NaN/Inf behavior here?

wgtmac · 2026-08-05T06:25:27Z

I've posted several review comments (some are posted by Codex as I'm not that familiar with simdjson). Let me know what you think.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI lite review requested due to automatic review settings August 3, 2026 10:41

Reranko05 requested review from pitrou and wgtmac as code owners August 3, 2026 10:41

Copilot AI reviewed Aug 3, 2026

github-actions Bot added the awaiting review Awaiting review label Aug 3, 2026

github-actions Bot added Component: Parquet Component: C++ labels Aug 3, 2026

Reranko05 marked this pull request as draft August 3, 2026 10:55

Reranko05 added the CI: Extra: C++ Run extra C++ CI label Aug 3, 2026

Replace remaining RapidJSON usage with simdjson

0230946

Reranko05 force-pushed the gh-35460-pr5-updated branch from d026d1f to 0230946 Compare August 3, 2026 12:54

Preserve GeoArrow CRS JSON formatting

777999e

Reranko05 force-pushed the gh-35460-pr5-updated branch from 270af67 to 777999e Compare August 4, 2026 05:40

Reranko05 marked this pull request as ready for review August 4, 2026 06:23

Copilot AI review requested due to automatic review settings August 4, 2026 06:23

Copilot AI reviewed Aug 4, 2026

rok reviewed Aug 4, 2026

View reviewed changes

Comment thread cpp/src/parquet/CMakeLists.txt Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 4, 2026

kou reviewed Aug 4, 2026

View reviewed changes

Address feedback

3ccd746

Copilot AI review requested due to automatic review settings August 5, 2026 04:17

wgtmac reviewed Aug 5, 2026

View reviewed changes

Comment thread cpp/src/arrow/util/simdjson_internal.h

Copilot AI review requested due to automatic review settings August 5, 2026 11:17

Copilot AI reviewed Aug 5, 2026

Copilot AI review requested due to automatic review settings August 5, 2026 12:08

Reranko05 force-pushed the gh-35460-pr5-updated branch from 487d90f to 742921a Compare August 5, 2026 12:08

Copilot AI reviewed Aug 5, 2026

Reranko05 force-pushed the gh-35460-pr5-updated branch from 742921a to d57370a Compare August 5, 2026 12:19

Copilot AI review requested due to automatic review settings August 5, 2026 12:19

Copilot AI reviewed Aug 5, 2026

Reranko05 force-pushed the gh-35460-pr5-updated branch from d57370a to 38042c2 Compare August 5, 2026 13:10

Copilot AI review requested due to automatic review settings August 5, 2026 13:10

Copilot AI reviewed Aug 5, 2026

Reranko05 force-pushed the gh-35460-pr5-updated branch from 38042c2 to f0953e2 Compare August 5, 2026 13:29

Copilot AI review requested due to automatic review settings August 5, 2026 13:29

Copilot AI reviewed Aug 5, 2026

Copilot AI review requested due to automatic review settings August 5, 2026 13:46

Reranko05 force-pushed the gh-35460-pr5-updated branch from f0953e2 to 57bf182 Compare August 5, 2026 13:46

Copilot AI reviewed Aug 5, 2026

Copilot AI review requested due to automatic review settings August 5, 2026 14:15

Reranko05 force-pushed the gh-35460-pr5-updated branch from 57bf182 to 32d9dac Compare August 5, 2026 14:15

Copilot AI reviewed Aug 5, 2026

Copilot AI review requested due to automatic review settings August 5, 2026 15:27

Reranko05 force-pushed the gh-35460-pr5-updated branch from 32d9dac to 4121d9a Compare August 5, 2026 15:27

Copilot AI reviewed Aug 5, 2026

Address feedback 2

0fcef4d

Reranko05 force-pushed the gh-35460-pr5-updated branch from 4121d9a to 0fcef4d Compare August 5, 2026 16:13

Reranko05 requested review from taepper and wgtmac August 5, 2026 16:38


		auto padded_json = simdjson::padded_string(json_string);

		if (auto error = parser.iterate(padded_json).get(document)) {

Uh oh!

Conversation

Reranko05 commented Aug 3, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

Reranko05 commented Aug 3, 2026

Uh oh!

rok commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reranko05 commented Aug 3, 2026

Uh oh!

rok commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Aug 5, 2026

Uh oh!

wgtmac Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

taepper Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

taepper Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wgtmac commented Aug 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Reranko05 commented Aug 3, 2026 •

edited by github-actions Bot

Loading

rok commented Aug 3, 2026 •

edited

Loading

Reranko05 commented Aug 4, 2026 •

edited

Loading

rok commented Aug 4, 2026 •

edited

Loading