Group JSON ingestion schema state#1281
Closed
jwils wants to merge 1 commit into
Closed
Conversation
jwils
force-pushed
the
joshuaw/json-ingestion-state-container
branch
2 times, most recently
from
c443df8 to
d660189
Compare
jwils
force-pushed
the
joshuaw/json-ingestion-state-container
branch
from
d660189 to
ee70fc0
Compare
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
Fills in the `elasticgraph-proto_ingestion` gem with the Protocol Buffers schema artifact generation logic. Running `schema_artifacts:dump` emits: - `schema.proto` — the generated Protobuf schema for the indexed types - `proto_field_numbers.yaml` — a sidecar that reserves field numbers and enum value numbers so they stay wire-stable as the schema evolves (including across field renames, and reserving removed values so numbers are not reused) Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `protobuf` to configure custom scalars. - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid. The extension's schema definition state lives on a ProtoIngestionState container exposed via a single `proto_ingestion_state` reader, mirroring the JSONIngestionState approach from #1281. The generator emits `proto3` by default and can emit `proto2` via `syntax: :proto2` (which labels every field `optional`/`repeated`). Arbitrary file-level headers (e.g. `option` declarations) can be injected verbatim via `headers:`, so language-specific options can be set without baking any particular convention into the gem.
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
Fills in the `elasticgraph-proto_ingestion` gem with the core Protocol Buffers schema generation logic. Running `schema_artifacts:dump` emits a `proto3` `schema.proto` artifact covering the schema's indexed types. Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `protobuf` to configure custom scalars (resolved via `type_ref.with_reverted_override` so built-in scalars renamed with `type_name_overrides` keep working). - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid. - `schema.proto` uses the public GraphQL field names, and validates proto package names (each dot-separated segment must be a valid identifier). Field and enum value numbers are assigned sequentially in definition order; keeping them wire-stable across schema evolution comes in a stacked follow-up that adds a `proto_field_numbers.yaml` sidecar artifact. The extension's schema definition state lives on a ProtoIngestionState container exposed via a single `proto_ingestion_state` reader, mirroring the JSONIngestionState approach from #1281.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Keep JSON ingestion-specific schema definition state off the shared
SchemaDefinition::Statesurface. This follows up on #1259 by grouping the extension state behind one accessor.What
JSONIngestionStateand attach it viaState#json_ingestion_stateRisk Assessment
Low — internal schema-definition state refactor; public DSL behavior remains unchanged.
References
Validation:
script/lint --fixscript/spellcheck -wscript/type_checkRUBYOPT=--disable-frozen-string-literal bundle exec rake site:doctestbundle exec rspec elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/deprecated_element_spec.rb elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/json_schema_spec.rb elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/indexing/json_schema_with_metadata_spec.rbscript/run_gem_specs elasticgraph-json_ingestionscript/run_gem_specs elasticgraph-schema_definitionandscript/quick_build; local datastore boot is blocked because Docker is not running (~/.docker/run/docker.sockis unavailable).RUBYOPT=--disable-frozen-string-literal bundle exec rake site:validate; the CI-relevant doctest failure is fixed, but local execution later stops insite:npm_installwith an npm dependency resolution issue.