Dataflow Engine Pipeline configuration stabilization #2031

lquerel · 2026-02-12T18:53:39Z

Change Summary

Sorry in advance, this is a fairly large PR, but it's for a good reason as it aims to stabilize our configuration model, which we discussed during our SIG meetings.

Reworked node identity to use type: NodeUrn and removed the old kind/plugin_urn split.
Evolved NodeUrn from a type alias to a concrete parsed type (namespace, id, kind) with zero-cost part access and canonical URN reconstruction.
Moved URN normalization/parsing logic into the node_urn module and cleaned up obsolete URN plumbing.
Fully removed node-level out_ports wiring from NodeUserConfig.
Externalized graph wiring into top-level connections in PipelineConfig.
Simplified connection syntax:
- removed out_port field from connections
- default source output is implicit (default)
- multi-output selection stays explicit via from: node["output"]
Standardized naming around output ports:
- config fields use outputs and default_output
- default output name is default
- outputs/default_output are optional for single-output nodes
Replaced connection fanout schema with policy-oriented schema:
- policies.dispatch with one_of (default) and broadcast. I believe one_of better reflect the underlying implementation (was never really a round robin strategy as the channel receivers were competing together).
- broadcast is currently parsed but rejected for multi-destination edges (reserved for future support)
- single-destination edges treat dispatch as no-op
Refactored PipelineConfigBuilder API for readability in tests:
- one_of(src, targets) and broadcast(src, targets) for default output
- one_of_output(src, output, targets) and broadcast_output(...) for explicit output
- added to(src, dst) and to_output(src, output, dst) aliases
Updated engine wiring internals and channel identity labeling to use dispatch policy terminology (one_of/broadcast) consistently.
Updated docs and examples to the new model:

To do: update the configuration of our continuous benchmarks.

What issue does this PR close?

How are these changes tested?

All unit tests passed

Are there any user-facing changes?

The structure of the configuration files have changed.

…ut syntax

lquerel · 2026-02-12T22:20:32Z

rust/otap-dataflow/crates/config/src/lib.rs


 /// The URN of a node type.
-pub type NodeUrn = Cow<'static, str>;
+pub use node_urn::NodeUrn;


NodeUrn is now a concrete type

lquerel · 2026-02-12T22:23:30Z

rust/otap-dataflow/crates/config/src/node_urn.rs

+    kind: NodeKind,
+}
+
+impl NodeUrn {


Most of the methods here were present in other files before. I just group them under this new NodeUrn type.

# Conflicts: # rust/otap-dataflow/crates/config/src/pipeline.rs

jmacd · 2026-02-13T01:17:19Z

rust/otap-dataflow/crates/config/src/pipeline.rs

+    OneOf,
+    /// Send each message to every destination.
+    Broadcast,


Looks good.

jmacd · 2026-02-13T01:23:24Z

rust/otap-dataflow/crates/config/src/pipeline.rs

+        // The fanout processor expects each destination port to map to a single downstream node.
+        // Multiple targets on the same source port would introduce load-balancing semantics.


It looks like the fanout processor is a special case --

Not great. Need to find a better approach.

Introducing a WiringContract directly in the node factory. By default, there would be no additional wiring restrictions beyond those already in place for orphan nodes, cycle detection, and similar safeguards. The fanout processor would be able to declare a constraint such as: "pipeline builder, do not accept more than one connection per output".

jmacd · 2026-02-13T01:25:01Z

rust/otap-dataflow/crates/otap/src/batch_processor.rs

+        nuc.add_output("main_output");
+        nuc.set_default_output("main_output");


This is great, see!

codecov · 2026-02-13T01:28:40Z

Codecov Report

❌ Patch coverage is 88.80937% with 172 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.63%. Comparing base (75a2f71) to head (ecaa3ff).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2031      +/-   ##
==========================================
+ Coverage   86.62%   86.63%   +0.01%     
==========================================
  Files         527      528       +1     
  Lines      168879   169481     +602     
==========================================
+ Hits       146283   146823     +540     
- Misses      22062    22124      +62     
  Partials      534      534

Components	Coverage Δ
otap-dataflow	`88.69% <88.80%> (+<0.01%)`	⬆️
query_abstraction	`80.61% <ø> (ø)`
query_engine	`90.29% <ø> (ø)`
syslog_cef_receivers	`∅ <ø> (∅)`
otel-arrow-go	`53.50% <ø> (ø)`
quiver	`91.84% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lquerel added 8 commits February 10, 2026 18:18

Replace kind and plugin_url fields with a single type field

df3fa3c

Replace NodeUrn type alias with a concrete type

3f6d9bd

Remove out_ports and replace it with output and connections with outp…

a4e27c4

…ut syntax

Update documentations

ad3014b

More documentation updates

933c707

Configuration cleanups

2d5ff54

Rename out port to output port

cb47ea2

Refactor connection policies

a0c825e

github-project-automation bot added this to OTel-Arrow Feb 12, 2026

github-actions bot added the rust Pull requests that update Rust code label Feb 12, 2026

lquerel self-assigned this Feb 12, 2026

lquerel removed this from OTel-Arrow Feb 12, 2026

lquerel added this to OTel-Arrow Feb 12, 2026

lquerel added the engine-capability Internal engine features label Feb 12, 2026

lquerel commented Feb 12, 2026

View reviewed changes

lquerel added 4 commits February 12, 2026 14:32

Update continuous benchmark config files

2de6dbd

Update continuous benchmark config files

9c8f576

Make errors more structured

7428562

Merge remote-tracking branch 'upstream/main' into cfg-stabilization

fa624c2

# Conflicts: # rust/otap-dataflow/crates/config/src/pipeline.rs

jmacd approved these changes Feb 13, 2026

View reviewed changes

Fix a merge issue

4f6a89f

lquerel added 3 commits February 12, 2026 17:32

Fix markdown issues

db2805d

Fix markdown issues

db7d3f6

Intrroduce WiringContract to make fanout constraints explicit

995dfd4

lquerel marked this pull request as ready for review February 13, 2026 02:18

lquerel requested a review from a team as a code owner February 13, 2026 02:18

lquerel changed the title ~~[WIP] Dataflow Engine Pipeline configuration stabilization~~ Dataflow Engine Pipeline configuration stabilization Feb 13, 2026

Intrroduce WiringContract to make fanout constraints explicit

d1a25d3

lquerel and others added 2 commits February 12, 2026 19:05

Improve the topology validation process

91229a1

Merge branch 'main' into cfg-stabilization

ecaa3ff

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataflow Engine Pipeline configuration stabilization #2031

Dataflow Engine Pipeline configuration stabilization #2031

lquerel commented Feb 12, 2026 •

edited

Loading

Uh oh!

lquerel Feb 12, 2026

Uh oh!

lquerel Feb 12, 2026

Uh oh!

jmacd Feb 13, 2026

Uh oh!

jmacd Feb 13, 2026

Uh oh!

lquerel Feb 13, 2026

Uh oh!

lquerel Feb 13, 2026

Uh oh!

jmacd Feb 13, 2026

Uh oh!

codecov bot commented Feb 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// The fanout processor expects each destination port to map to a single downstream node.
		// Multiple targets on the same source port would introduce load-balancing semantics.

		nuc.add_output("main_output");
		nuc.set_default_output("main_output");

Dataflow Engine Pipeline configuration stabilization #2031

Are you sure you want to change the base?

Dataflow Engine Pipeline configuration stabilization #2031

Conversation

lquerel commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

What issue does this PR close?

How are these changes tested?

Are there any user-facing changes?

Uh oh!

lquerel Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

lquerel Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

jmacd Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

jmacd Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

lquerel Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

lquerel Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

jmacd Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lquerel commented Feb 12, 2026 •

edited

Loading

codecov bot commented Feb 13, 2026 •

edited

Loading