When adapting expressions to an older physical schema, a missing nullable column is expected and is replaced with a null literal. The current adapter calls Schema::index_of to check for that column. On a miss, Arrow formats an error containing all field names, which the adapter immediately discards. This makes schema adaptation increasingly expensive as physical schemas grow.
Use the non-error-producing field lookup for this expected miss and retain the existing behavior for present columns and non-nullable missing columns. A small benchmark covering present and missing columns in 16- and 128-field schemas would make the cost visible.
When adapting expressions to an older physical schema, a missing nullable column is expected and is replaced with a null literal. The current adapter calls
Schema::index_ofto check for that column. On a miss, Arrow formats an error containing all field names, which the adapter immediately discards. This makes schema adaptation increasingly expensive as physical schemas grow.Use the non-error-producing field lookup for this expected miss and retain the existing behavior for present columns and non-nullable missing columns. A small benchmark covering present and missing columns in 16- and 128-field schemas would make the cost visible.