Generate a protobuf schema artifact from ElasticGraph schemas

[jwils]

Note

First of a four-PR stack. Supporting changes merged earlier (#1296 artifact comment prefixes, #1297 gem scaffold), so this diff is contained entirely to the elasticgraph-proto_ingestion directory. Per review feedback the original PR has been split: wire-stable field/enum-value numbering, proto2 + custom headers, and enum sourcing/external references are now stacked follow-ups.

Why

• continue the pluggable ingestion serializer proposal after pulling JSON Schema into its own gem
• revive the earlier protobuf prototype from Add elasticgraph-proto gem with stable proto field-number mappings #1056 on top of the new serializer extension points

What

• fill in the elasticgraph-proto_ingestion extension gem with core generation: schema_artifacts:dump emits a proto3 schema.proto covering the schema's indexed types
• map built-in ElasticGraph scalars to proto types, with t.protobuf type: for custom scalars (resolved via type_ref.with_reverted_override so built-ins renamed with type_name_overrides keep working)
• generate messages for object/interface/union types and enums (with a zero-valued *_UNSPECIFIED entry), escaping proto reserved words and wrapping lists of lists so the output stays valid
• keep schema.proto on public GraphQL field names; validate proto package names
• hold extension state on a ProtoIngestionState container behind a single proto_ingestion_state reader (matching Group JSON ingestion schema state #1281)

Field and enum value numbers are assigned sequentially in definition order in this PR; the stacked follow-up adds the proto_field_numbers.yaml sidecar that keeps them wire-stable across schema evolution.

Stacked follow-ups

1. this PR — core schema.proto generation
2. wire-stable field/enum value numbers via a proto_field_numbers.yaml sidecar
3. syntax: :proto2 support and custom file-level headers:
4. Source enum values from external proto enums and reference external enum types #1286 — enum value sourcing from existing proto enums + external proto type references

Verification

• script/run_gem_specs elasticgraph-proto_ingestion (100% line + branch coverage at this commit)
• script/type_check, script/lint, script/spellcheck
• script/quick_build green at the stack head (whose tree is identical to the previously reviewed single-PR revision)

References

• Feature Proposal: Support for Protobuf ingestion #1059
• Add elasticgraph-proto gem with stable proto field-number mappings #1056
• Pluggable ingestion serializers with json_ingestion #1079

[myronmarston]

Not done reviewing but wanted to submit what I have so far.

[myronmarston]

EG today doesn't have any concept of proto enum mappings, and doesn't have any logic to support exclusions, expected_extras or name_transform. Instead of importing an existing data structure designed for tests within Block internal code, can we invert this?

• Design and implement whatever features make sense for the elasticgraph-proto_ingestion gem as a stand-alone, open source library.
• Update the internal testing support inside Block to leverage what elasticgraph-proto_ingestion provides.

For example, instead of it being provided as a global data structure, maybe we can do something like:

schema.enum_type "Currency" do |t|
  t.external_proto_enum "v1.money.Currency", # ... plus other options as needed like `exclusions`, etc.
end

With that said, I don't yet understand how this gem uses this mapping...I haven't gotten that far.

[jwils]

Makes sense. Going to move enums out of this pr so we can review this independently.

[myronmarston]

You can also think about applying this tag automatically to specs under elasticgraph-proto_ingestion/spec, using this technique:

elasticgraph/elasticgraph-json_ingestion/spec/spec_helper.rb

Lines 16 to 18 in 4cdc065

	config.define_derived_metadata(absolute_file_path: %r{/elasticgraph-json_ingestion/}) do |meta|
	meta[:json_ingestion_schema_definition] = true
	end

[myronmarston]

Should we use https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp for this?

[myronmarston]

One potential concern is that string is a much wider type than Date. Something like 0.0001% of valid strings are valid ISO8601 Date strings.

Have you given thought to how invalid values will get handled? With JSON schema we could specify validations for this kind of thing.

[myronmarston]

I'm not following this.

schema.object_type "Team" do |t|
  t.field "players", "[Player!]!" do |f|
    f.mapping type: "nested"
  end

  t.field "seasons", "[Season!]!" do |f|
    f.mapping type: "object"
  end
end

Is this saying that a wrapper message type will be generated for the players field but not the seasons field? Why would that be? And what would the proto look like?

[jwils]

nested as in the list of lists not the mapping type, so the issue exists for both player and seasons. you can't do repeated repeated for a field. Updating test.