Skip to content

mask is not applied to create_dataset_item or create_score(comment=) #1896

Description

@hchittanuru3

Describe the bug

mask is documented as the client-level redaction hook (MaskFunction: "A function that masks data"), but it is only applied to span-backed payloads. Two client methods send user-supplied data to the API without ever calling it:

method unmasked fields
Langfuse.create_dataset_item() input, expected_output, metadata
Langfuse.create_score() comment

For anyone using mask as a privacy/compliance control, this is a silent gap — the hook is configured once on the client, so the natural reading is that it covers what the client sends.

Evidence (v4.14.1)

_mask is referenced only in _client/span.py and _client/span_exporter.py:

$ grep -rn "_mask" langfuse/ --include='*.py' | grep -v "/span.py" | grep -v _process_media
langfuse/_client/client.py:310:    _mask: Optional[MaskFunction] = None
langfuse/_client/client.py:438:        self._mask = self._resources.mask
langfuse/_client/span_exporter.py:106:        self._mask_otel_spans = mask_otel_spans   # the separate OTel hook

create_dataset_item passes its arguments straight through (client.py:3627):

result = self.api.dataset_items.create(
    dataset_name=dataset_name,
    input=input,                 # not masked
    expected_output=expected_output,
    metadata=metadata,
    ...
)

create_score builds the score body and hands it to add_score_task with comment untouched.

For contrast, create_event is not affected — it constructs a LangfuseEvent (span-backed), so it does go through _process_media_and_apply_mask. The boundary is "span-backed vs direct API call", which is an implementation detail rather than something a caller can infer.

Reproduction

from langfuse import Langfuse

def mask(*, data, **kwargs):
    return "[MASKED]"

lf = Langfuse(public_key=..., secret_key=..., mask=mask)

# masked, as expected
with lf.start_as_current_observation(name="s", as_type="span", input={"x": "secret"}):
    pass

# NOT masked — "secret" is sent verbatim
lf.create_dataset_item(dataset_name="d", input={"x": "secret"})
lf.create_score(name="quality", value=1, comment="secret", trace_id=lf.create_trace_id())

Expected behaviour

Either of these resolves it; I have no strong preference:

  1. Apply mask to these payloads too, so the hook means what its name implies. The change in create_dataset_item is a couple of lines against the existing _mask attribute.
  2. Document the boundary explicitly — on MaskFunction, on the mask constructor arg, and in the masking docs — stating that mask covers span/generation/event payloads only, and that dataset items and score comments are the caller's responsibility.

Happy to open a PR for (1) if you'd take it — say which of the two you'd prefer and I'll follow the repo's conventions.

Why this came up

We register a masking function so nothing unredacted leaves the process, then discovered our own codebase had grown a comment warning callers that "dataset items go straight to the API and are never masked" — a documented workaround for an undocumented boundary. That seemed worth reporting rather than keeping local.

Environment

  • langfuse 4.14.1, Python 3.14, Langfuse Cloud (US)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions