Feature: Per-service/namespace control for Java and Node.js agent injection

[gk-clode]

Problem Statement

Currently, the Java agent (javaagent) and Node.js agent (nodejs) injectors can only be enabled/disabled globally via:

• OTEL_EBPF_JAVAAGENT_ENABLED=false
• OTEL_EBPF_NODEJS_ENABLED=false

This is problematic when:

1. Some applications have library conflicts - e.g., Neo4j bundles caffeine cache which conflicts with OBI's shaded io.opentelemetry.obi.com.github.benmanes.caffeine.* classes, causing NoClassDefFoundError
2. Users want eBPF traces but not agent injection for specific services
3. Different namespaces have different requirements - production apps may need full instrumentation while third-party databases should be excluded

Current Workaround

Users must either:

1. Disable agents globally (losing functionality for ALL Java/Node.js apps)
2. Use exclude_instrument to exclude the namespace entirely (losing ALL eBPF traces for that service)

Neither option is ideal.

Proposed Solution

Add per-service agent injection control to the instrument discovery configuration, similar to existing per-service options like exports, sampler, and routes.

Option 1: Unified control

discovery:
  instrument:
    - k8s_namespace: juno
      k8s_statefulset_name: neo4j
      agent_injection: false  # Disable both Java + Node.js agents
      
    - k8s_namespace: production
      agent_injection: true   # Enable (default)

Option 2: Granular control

discovery:
  instrument:
    - k8s_namespace: juno
      javaagent_enabled: false
      nodejs_enabled: true
      
    - k8s_pod_labels:
        app: legacy-app
      javaagent_enabled: false

Option 3: Exclusion list (similar to exclude_instrument)

javaagent:
  enabled: true
  exclude:
    - k8s_namespace: juno
    - k8s_statefulset_name: neo4j

nodejs:
  enabled: true
  exclude:
    - k8s_namespace: kube-system

Implementation Notes

Based on code analysis:

1. GlobAttributes struct (pkg/appolly/services/attr_glob.go) already supports per-service config for exports, sampler, routes, metrics. Adding javaagent_enabled and nodejs_enabled fields would follow the same pattern.

2. attacher.go (pkg/appolly/discover/attacher.go:110-115) currently calls injectors unconditionally:

   ta.nodeInjector.NewExecutable(&instr.Obj)
   if ta.javaInjector != nil {
       if err := ta.javaInjector.NewExecutable(&instr.Obj); err != nil {
           // ...
       }
   }

   This would need to check the per-service config before calling the injectors.

3. Namespace is already available in instr.Obj.FileInfo.Service.UID.Namespace and Metadata[attr.K8sNamespaceName] at injection time.

Use Case

We run Neo4j (a Java application) in Kubernetes. OBI's Java agent injection causes:

java.lang.NoClassDefFoundError: io/opentelemetry/obi/com/github/benmanes/caffeine/cache/BoundedBuffer$RingBuffer

This happens because Neo4j bundles caffeine cache, and OBI's shaded caffeine classes conflict with it.

We want to:

• ✅ Keep eBPF network-level traces for Neo4j (HTTP, Bolt protocol)
• ✅ Keep Java agent injection for our own Java microservices
• ❌ Disable Java agent injection ONLY for Neo4j

Currently impossible without disabling Java agent globally.

Additional Context

• OBI version: v0.4.1
• Deployment: Kubernetes DaemonSet
• Affected application: Neo4j 5.x (uses caffeine cache internally)

[grcevski]

This is a great idea @gk-clode! I think we should add this.

On the error you are seeing with neo4j, can you please tell me how to reproduce this? Does it happen with just running neo4j? Does it have to be configured with TLS or doesn't matter.

The reason I am asking is that out caffeine is shaded as you can see from the class name and it won't conflict. It must be another issue

[gk-clode]

Hi @grcevski, thanks for looking into this!

Reproduction Steps

1. Deploy Neo4j 5.26-enterprise on Kubernetes
2. Enable OBI v0.4.1 with OTEL_EBPF_JAVAAGENT_ENABLED=true
3. Wait ~1 minute for Neo4j's internal scheduler to run tasks
4. Errors appear repeatedly in Neo4j pod logs

No special TLS configuration needed - the issue occurs with default Neo4j setup. Neo4j uses SSL internally for various operations.

Full Stack Trace

java.lang.NoClassDefFoundError: io/opentelemetry/obi/com/github/benmanes/caffeine/cache/BoundedBuffer$RingBuffer
    at io.opentelemetry.obi.com.github.benmanes.caffeine.cache.BoundedBuffer.create(BoundedBuffer.java:53)
    at io.opentelemetry.obi.com.github.benmanes.caffeine.cache.StripedBuffer.expandOrRetry(StripedBuffer.java:265)
    at io.opentelemetry.obi.com.github.benmanes.caffeine.cache.StripedBuffer.offer(StripedBuffer.java:149)
    at io.opentelemetry.obi.com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:1166)
    at io.opentelemetry.obi.com.github.benmanes.caffeine.cache.BoundedLocalCache.put(BoundedLocalCache.java:2147)
    at io.opentelemetry.obi.com.github.benmanes.caffeine.cache.BoundedLocalCache.put(BoundedLocalCache.java:1993)
    at io.opentelemetry.obi.com.github.benmanes.caffeine.cache.LocalManualCache.put(LocalManualCache.java:130)
    at io.opentelemetry.obi.java.instrumentations.data.SSLStorage.trackTask(SSLStorage.java:154)
    at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:146)
    at org.neo4j.kernel.impl.scheduler.ThreadPool.submit(ThreadPool.java:121)
    at org.neo4j.kernel.impl.scheduler.ScheduledJobHandle.submitIfRunnable(ScheduledJobHandle.java:148)
    at org.neo4j.kernel.impl.scheduler.TimeBasedTaskScheduler.scheduleDueTasks(TimeBasedTaskScheduler.java:146)
    at org.neo4j.kernel.impl.scheduler.TimeBasedTaskScheduler.tick(TimeBasedTaskScheduler.java:129)

Observations

You're right that caffeine is shaded - the package io.opentelemetry.obi.com.github.benmanes.caffeine appears in the stack trace. However:

• Parent class loads OK: BoundedBuffer (shaded) is found
• Inner class fails: BoundedBuffer$RingBuffer throws NoClassDefFoundError

This happens when OBI's SSLStorage.trackTask() intercepts Neo4j's scheduler submitting tasks.

Possible Causes

1. Inner class shading issue - BoundedBuffer$RingBuffer may not be properly relocated in the shade plugin config
2. Neo4j's caffeine interfering - Neo4j bundles its own unshaded caffeine (com.github.benmanes.caffeine). When OBI's agent loads the shaded parent class but tries to instantiate the inner class, the classloader might resolve incorrectly.

Environment

• Kubernetes: EKS 1.31
• OBI: v0.4.1 (DaemonSet)
• Neo4j: 5.26-enterprise
• Java: OpenJDK (bundled with Neo4j image)

Happy to provide more details or test a patched version!