Model serialization drops keys

[lucas-bremond]

There may be an issue affecting the serialize_model methods of the Pydantic models in this library.

Taking the DocumentContent model as an example, we see:

class DocumentContent(BaseModel):
    full_text_list: Annotated[
        Optional[List[str]], pydantic.Field(alias="fullTextList")
    ] = None
    r"""The plaintext content of the document."""

    @model_serializer(mode="wrap")
    def serialize_model(self, handler):
        optional_fields = set(["fullTextList"])
        serialized = handler(self)
        m = {}

        for n, f in type(self).model_fields.items():
            k = f.alias or n
            val = serialized.get(k)

            if val != UNSET_SENTINEL:
                if val is not None or k not in optional_fields:
                    m[k] = val

        return m

This model uses a field alias that, when constructing the Pydantic object from an API response, will map the fullTextList field of the JSON object to the full_text_list field of the Pydantic object.

However, the model serializer uses:

...
k = f.alias or n
val = serialized.get(k)
...

which means that the field alias (fullTextList) will be used to extract the value rather than the Pydantic field name. This results in value being None and in missing keys in the returned dictionary m when the field name and its alias are different.

To support this claim, please find attached a documents.json file that contains an anonymized response collected from the Glean API (/rest/api/v1/getdocuments endpoint).

And below is a simple debug.py script to run alongside it:

import pathlib

from glean.api_client import models
from glean.api_client.utils.unmarshal_json_response import unmarshal_json_response


class DummyHttpResponse:
    def __init__(self, text):
        self.status_code = 200
        self.text = text


with pathlib.Path("documents.json").open("r") as f:
    http_res = DummyHttpResponse(
        text=f.read(),
    )


documents_response = unmarshal_json_response(models.GetDocumentsResponse, http_res)


assert isinstance(documents_response, models.GetDocumentsResponse)
assert documents_response.documents is not None
assert isinstance(
    documents_response.documents["https://company.com/Test"].content,
    models.DocumentContent,
)
assert (
    documents_response.documents["https://company.com/Test"].content.full_text_list[0]
    == "This is a test document."
)

serialized_document_response = documents_response.model_dump()

assert isinstance(serialized_document_response, dict)
assert serialized_document_response["documents"] is not None

# Here's the problem: no `full_text_list` or `fullTextList` in the serialized response!
assert (
    len(serialized_document_response["documents"]["https://company.com/Test"]["content"])
    > 0
)

Running it yields:

$ ls    
debug.py  documents.json

$ python debug.py
Traceback (most recent call last):
  File "/workspace/app/debug/debug.py", line 40, in <module>
    len(serialized_document_response["documents"]["https://company.com/Test"]["content"])
    > 0
AssertionError

Note that I'm using:

pydantic_core==2.41.5
pydantic==2.12.5
glean-api-client==0.11.27

[lucas-bremond]

Digging a little further...

The models of this library are auto-generated using Speakeasy.

And as hinted here, the reverse operation (transforming Pydantic back to dictionaries) should use model_dump(by_alias=True) instead.

And indeed, changing the above debug.py script with:

serialized_document_response = documents_response.model_dump(by_alias=True)

...works!

However...!

What led me to this library is the glean-agent-toolkit.

When using it with Google ADK, as documented, the Glean tool ends up providing missing responses to the LLM. And this is because of the issue above:

1. The Glean API Client returns a Pydantic object back to Google ADK
2. Google ADK serializes the Pydantic object using convert_to_dict which uses model_dump WITHOUT by_alias=True
3. And therefore, any field differing between snake_case and camelCase are stripped from the Glean tool output and never reach the LLM.

[dhruv7539]

Opened a PR with a fix for the reported DocumentContent.model_dump() key loss and a regression test:

• PR: Fix null likedBy parsing and DocumentContent model_dump key loss #121
• Change: in DocumentContent.serialize_model, fall back to field-name lookup when alias key is not present
• Test: test_document_content_model_dump_keeps_alias_field_value