Skip to content

Add _source filtering for LIST_CHART to prevent oversized OpenSearch responses #1796

@mjabascal10

Description

@mjabascal10

Acknowledgements

Describe the bug

LIST_CHART visualizations currently fetch full documents from OpenSearch, including extremely large fields such as event.original, winlog.event_data.*, and message.
This results in response payloads exceeding 100MB, causing the OpenSearch Java client (HttpClient5) to fail.

LIST_CHART is a hybrid visualization:

  • It uses aggregations to build the hierarchical structure.
  • It uses documents to populate the table.

Because documents are required, size > 0 must remain.
However, the _source must be restricted to avoid returning massive payloads.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

LIST_CHART visualizations should return lightweight documents containing only the fields required by the table, while still using aggregations to build the hierarchical structure.
The OpenSearch response should remain well below 100MB, allowing the Java client (HttpClient5) to process it without errors.

Current Behavior

LIST_CHART currently fetches full documents, including extremely large fields such as:

  • event.original
  • winlog.event_data.*
  • message
  • stacktrace

This results in OpenSearch responses exceeding 100–150MB, causing the client to fail with:

UtmVisualizationResource.run: ElasticsearchService.search: OpenSearch.search: entity content is too long [143527767] for the configured buffer limit [104857600]

Reproduction Steps

  1. Create or open a LIST_CHART visualization with deep aggregations
    (e.g., name → severity → status → category → dataSource → timestamp).
  2. Apply filters that match Windows Event Log data.
  3. Ensure the visualization requests documents (size > 0).
  4. Execute the request through the backend (not directly in OpenSearch Dashboards).
  5. Observe the backend error due to oversized _source payload.

Possible Solution

Implement _source filtering only for LIST_CHART, excluding all fields and re‑including only the fields required by the visualization.

Additional Information/Context

No response

UTMStack Version

11.2.3

Operating System and version

Ubuntu

Hypervisor and Version | Server Vendor and Model

Browser and version

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

Status

👀 In review

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions