Stop using the OpenAPI schema to drive the UI (?)

[dantownsend]

Related to piccolo-orm/piccolo#1086

The UI of Piccolo Admin is largely driven by OpenAPI schemas (generated via create_pydantic_model in Piccolo).

We augment the OpenAPI schemas with Piccolo specific metadata (under the extra attribute).

Pydantic allows us to do this, and it has worked pretty well up until now. But other libraries (Litestar) complain if an OpenAPI schema has any custom attributes.

It should be possible to add custom attributes, as long as they start with x-, but we can't work out how to do it and still keep Litestar happy.

Which just leaves us with these options:

1. Completely replace the current schema endpoint with our own custom one.
2. Keep the current /schema endpoint in Piccolo Admin (minus all of the custom attributes), and have a second endpoint (something like /meta) which contains all of the custom attributes. The downside is it means 2 API calls instead of 1.
3. Have our own custom endpoint, but embed the existing schema JSON within it. Something like {"schema": ..., "custom": ...}.

[Compro-Prasad]

Ref: litestar-org/litestar#3762

[Compro-Prasad]

1. Completely replace the current schema endpoint with our own custom one.
2. Keep the current /schema endpoint in Piccolo Admin (minus all of the custom attributes), and have a second endpoint (something like /meta) which contains all of the custom attributes. The downside is it means 2 API calls instead of 1.
3. Have our own custom endpoint, but embed the existing schema JSON within it. Something like {"schema": ..., "custom": ...}.

I think the first one is the way to go. But why is the OpenAPI schema important to drive the UI? What was your plan when you implemented it using OpenAPI schema?

If you think the OpenAPI schema is still useful then having it as a separate endpoint makes sense as you mentioned in the third point.

The second point isn't too bad in today's day and age as most websites already are too bloated. But still, I would like to know why having 2 API calls is important if we could get it done using single endpoint.

[diklios5768]

What about add a custom param and default value to create_pydantic_model? For example: add field_json_schema_extra_name:str='extra'

then change Field json_schema_extra content like this:

field = pydantic.Field(
    json_schema_extra={field_json_schema_extra_name: extra},
    **params,
)

Litestar use additional_properties:create_pydantic_model(field_json_schema_extra_name='additional_properties')
Other libraries start with x-:create_pydantic_model(field_json_schema_extra_name='x-extra')
Piccolo change nothing:create_pydantic_model()

[dantownsend]

I tried the x- prefix but couldn't get it to work with Litestar. I might have been doing something wrong though.

So if we use additional_properties instead of extra it will work with Litestar?

[diklios5768]

I tried the x- prefix but couldn't get it to work with Litestar. I might have been doing something wrong though.

So if we use additional_properties instead of extra it will work with Litestar?

I tried the additional_properties instead of extra and work with Litestar like this:

field = pydantic.Field(
    json_schema_extra={"additional_properties": extra},
    **params,
)

But if we add a custom param, then it can be applied to other frameworks or modules as well.

Just like the code I provided above.

field = pydantic.Field(
    json_schema_extra={field_json_schema_extra_name: extra},
    **params,
)

[diklios5768]

Indeed, if you use x-, it does not work in litestar, it seems that it does not fully follow the extensions rules of OpenAPI.

Litestar only support those Schema fiedls: https://github.com/litestar-org/litestar/blob/main/litestar/openapi/spec/schema.py

[dantownsend]

@diklios5768 That's good to know - thanks!

[diklios5768]

Litestar use litestar.plugins.pydantic.plugins.schema.PydanticSchemaPlugin by default when not set in plugins

OpenAPI schema find plugin and use it by this way: 1 -> 2 -> 3 -> 4

Solved by extend PydanticSchemaPlugin and no need to modify piccolo.utils.pydantic.create_pydantic_model:

from dataclasses import replace

from litestar.openapi.spec import Schema
from litestar.params import KwargDefinition
from litestar.plugins.pydantic.plugins.schema import PydanticSchemaPlugin
from litestar.plugins.pydantic.utils import get_model_info
from litestar.typing import FieldDefinition


class PiccoloCompatiblePydanticSchemaPlugin(PydanticSchemaPlugin):
    """Ignore Piccolo's field-only admin metadata during OpenAPI generation."""

    @staticmethod
    def _strip_piccolo_field_extra(field_definition: FieldDefinition) -> FieldDefinition:
        kwarg_definition = field_definition.kwarg_definition
        if not isinstance(kwarg_definition, KwargDefinition):
            return field_definition

        schema_extra = kwarg_definition.schema_extra
        if not schema_extra or 'extra' not in schema_extra:
            return field_definition

        cleaned_schema_extra = dict(schema_extra)
        cleaned_schema_extra.pop('extra', None)

        return replace(
            field_definition,
            kwarg_definition=replace(
                kwarg_definition,
                schema_extra=cleaned_schema_extra or None,
            )
        )

    @classmethod
    def for_pydantic_model(cls, field_definition: FieldDefinition, schema_creator) -> Schema:
        model_info = get_model_info(field_definition.annotation, prefer_alias=schema_creator.prefer_alias)
        property_fields = {
            name: cls._strip_piccolo_field_extra(property_field)
            for name, property_field in model_info.field_definitions.items()
        }

        return schema_creator.create_component_schema(
            field_definition,
            required=sorted(field.name for field in property_fields.values() if field.is_required),
            property_fields=property_fields,
            title=model_info.title,
            examples=None if model_info.example is None else [model_info.example],
        )


piccolo_pydantic_schema_plugin = PiccoloCompatiblePydanticSchemaPlugin()

__all__ = ['PiccoloCompatiblePydanticSchemaPlugin', 'piccolo_pydantic_schema_plugin']

And then use this Plugin in Litestar APP:

from litestar import Litestar

app=Litestar(plugins=[piccolo_pydantic_schema_plugin])