README

                   -----------------------------------------
                      The HAProxy OpenTelemetry filter (OTel)
                                 Version 2.3.0
                          ( Last update: 2026-09-16 )
                   -----------------------------------------
                           Author : Miroslav Zagorac
                     Contact : mzagorac at haproxy dot com


SUMMARY
--------

  0.    Terms
  1.    Introduction
  2.    Build instructions
  3.    Basic concepts in OpenTelemetry
  4.    OTel configuration
  4.1.    OTel scope
  4.2.    "otel-instrumentation" section
  4.3.    "otel-scope" section
  4.4.    "otel-group" section
  5.    Examples
  5.1.    Benchmarking results
  6.    OTel CLI
  7.    Known bugs and limitations


0. Terms
---------

* OTel: The HAProxy OpenTelemetry filter

OTel is the HAProxy filter that allows you to send telemetry data (traces,
metrics and logs) to observability backends via the OpenTelemetry protocol.


1. Introduction
----------------

Nowadays there is a growing need to divide a process into microservices and
there is a problem of monitoring the work of the same process.  One way to solve
this problem is to use a distributed tracing service in a central location.

The OTel filter is the successor to the OpenTracing (OT) filter and is built on
the OpenTelemetry standard, which unifies distributed tracing, metrics and
logging into a single observability framework.  Unlike the older OpenTracing
filter which relied on vendor-specific tracer plugins, the OTel filter uses the
OpenTelemetry protocol (OTLP) to export data directly to any compatible backend.

The OTel filter is a standard HAProxy filter, so what applies to others also
applies to this one (of course, by that I mean what is described in the
documentation, more precisely in the doc/internals/filters.txt file).

The OTel filter activation is done explicitly by specifying it in the HAProxy
configuration.  If this is not done, the OTel filter in no way participates in
the work of HAProxy.

As for the impact on HAProxy speed, this is documented with test results located
in the test directory (see section 5.1).  The speed of operation depends on the
way the filter is used and the complexity of the configuration.  In typical
production use with a rate limit of 10% or less, the performance impact should
be negligible (see the 'rate-limit' keyword).

The OTel filter allows intensive use of ACLs.  An ACL may be declared in the
HAProxy configuration, in the otel-instrumentation section or in one otel-scope,
and the place of the declaration decides where the name reaches: everywhere, in
every otel-scope, or in that one scope only.  The ACL has to be defined by the
time the line that uses it is read.  Each name of a condition is looked up on
its own, in the scope first, then in the otel-instrumentation and then in the
HAProxy configuration, so one condition may mix them.  Thus, it is possible to
use the filter only for those connections that are of interest to us.

The OTel filter is distributed under the GNU General Public License, version 2
(reproduced in the LICENSE file) or any later version; the copyright is held by
HAProxy Technologies.


2. Build instructions
----------------------

OTel is the HAProxy filter and as such is compiled together with HAProxy.  It
supports all HAProxy versions from 3.2 onward; the sources are available at
https://github.com/haproxy/haproxy/ .

To communicate with an OpenTelemetry compatible backend, the OTel filter uses
version 3.4.0 of the OpenTelemetry C Wrapper library (which in turn uses the
OpenTelemetry C++ SDK version 1.26.0 or newer).  This means that we must have
the library installed on the system on which we want to compile or use HAProxy.

Note that the build does not enforce this version: pkg-config is asked only
whether the library exists.  At startup the filter verifies instead that the
linked library and the header files that it was compiled against carry the same
version string, and refuses to initialize on a mismatch.

Instructions for compiling and installing the required library can be found at
https://github.com/haproxytech/opentelemetry-c-wrapper .

The OTel filter is built as a standalone addon outside the HAProxy source tree
and is plugged into the HAProxy build via the EXTRA_MAKE variable, which lists
directories whose Makefile.mk fragments are included by the HAProxy top-level
Makefile.  The OTel addon's Makefile.mk sits at the top of this source tree, so
EXTRA_MAKE must point to that directory.

The examples below assume that the OTel addon checkout sits next to the HAProxy
source tree and that make is run from the HAProxy directory, referencing the
addon as "../haproxy-opentelemetry".  Adjust the path if your layout differs.

The OTel filter can be more easily compiled using the pkg-config tool, if we
have the OpenTelemetry C Wrapper library installed so that it contains
pkg-config files (which have the .pc extension).  If the pkg-config tool cannot
be used, then the path to the directory where the include files and libraries
are located can be explicitly specified.

Below are examples of the two ways to compile HAProxy with the OTel filter, the
first using the pkg-config tool and the second explicitly specifying the path to
the OpenTelemetry C Wrapper include and library.

Note: prompt '%' indicates that the command is executed under an unprivileged
      user, while prompt '#' indicates that the command is executed under the
      root user.

Example of compiling HAProxy using the pkg-config tool (assuming the
OpenTelemetry C Wrapper library is installed in the /opt directory):

  % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry"

The OTel filter can also be compiled in debug mode as follows:

  % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_DEBUG=1

HAProxy compilation example explicitly specifying path to the OpenTelemetry C
Wrapper include and library:

  % make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_INC=/opt/include OTEL_LIB=/opt/lib

In case we want to use debug mode, then it looks like this:

  % make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_INC=/opt/include OTEL_LIB=/opt/lib OTEL_DEBUG=1

To enable OpenTelemetry context propagation via HAProxy variables (in addition
to HTTP headers), add the OTEL_USE_VARS=1 option:

  % PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_USE_VARS=1

If the library we want to use is not installed on a unix system, then a locally
installed library can be used (say, which is compiled and installed in the user
home directory).  In this case instead of /opt/include and /opt/lib the
equivalent paths to the local installation should be specified.  Of course, in
that case the pkg-config tool can also be used if we have a complete
installation (with .pc files).

Last but not least, if the pkg-config tool is not used when compiling, then the
HAProxy executable may not be able to find the OpenTelemetry C Wrapper library
at startup.  This can be solved in several ways, for example using the
LD_LIBRARY_PATH environment variable which should be set to the path where the
library is located before starting the HAProxy.

  % LD_LIBRARY_PATH=/opt/lib /path-to/haproxy ...

Another way is to add RUNPATH to HAProxy executable that contains the path to
the library in question.

  % make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_INC=/opt/include OTEL_LIB=/opt/lib OTEL_RUNPATH=1

After HAProxy is compiled, we can check if the OTel filter is enabled:

  % ./haproxy -vv | grep -i opentelemetry
  --- command output ----------
  Built with OpenTelemetry support (filter version 2.3.0, C++ version 1.26.0, C Wrapper version 3.4.0-1028).
          [OTEL] opentelemetry
  --- command output ----------

A summary of all OTel build options:

  EXTRA_MAKE    - path to the OTel addon directory (enables the filter)
  OTEL_DEBUG    - compile the filter in debug mode
  OTEL_INC      - force path to opentelemetry-c-wrapper include files
  OTEL_LIB      - force path to opentelemetry-c-wrapper library
  OTEL_RUNPATH  - add opentelemetry-c-wrapper RUNPATH (needs OTEL_LIB)
  OTEL_STATIC   - pass --static to pkg-config for static linking
  OTEL_USE_VARS - enable context propagation via HAProxy variables

Finally, the filter can also be compiled without the OpenTelemetry C Wrapper
library being available at all.  The 'dummy' subdirectory of this source tree
holds a stand-in implementation of the wrapper API that is built as a static
archive in the same make pass once OTEL_INC and OTEL_LIB are pointed at it:

  % make -j8 TARGET=linux-glibc EXTRA_MAKE="../haproxy-opentelemetry" OTEL_INC=../haproxy-opentelemetry/dummy/include OTEL_LIB=../haproxy-opentelemetry/dummy

This is meant for compile testing only, for example in continuous integration
or when working on the filter where the OpenTelemetry C++ SDK is not installed.
The resulting executable parses its configuration and runs, and the filter's own
parsing, scope and event handling is exercised, but no telemetry is produced and
the OTel configuration file is not even read.  Such an executable reports the
C++ version as "none", which tells it apart from a real build:

  % ./haproxy -vv | grep -i opentelemetry
  --- command output ----------
  Built with OpenTelemetry support (filter version 2.3.0, C++ version none, C Wrapper version 3.4.0-0).
          [OTEL] opentelemetry
  --- command output ----------

OTEL_DEBUG is handed on to the stand-in build automatically, so the filter and
the archive cannot disagree on the debug macros.  See dummy/README for the whole
story, including what the stand-in implements and what it merely accepts.


3. Basic concepts in OpenTelemetry
-----------------------------------

Basic concepts of OpenTelemetry can be read on the OpenTelemetry documentation
website https://opentelemetry.io/docs/concepts/ .

Here we will list only the most important elements of distributed tracing.

A 'trace' is a description of the complete transaction we want to record in the
tracing system.  A 'span' is an operation that represents a unit of work that is
recorded in a tracing system.  A 'span context' is a group of information
related to a particular span that is passed on to the system (from service to
service).  Using this context, we can add new spans to already open trace (or
supplement data in already open spans).

An individual span may contain one or more attributes, events, links and baggage
items.

An 'attribute' is a key-value element that is valid for the entire span.
Attributes describe properties of the span such as HTTP method, URL, status
code, and so on.

A span 'event' is a named key-value element that allows you to write some data
at a certain time within the span's lifetime.  It can be used for debugging or
recording notable occurrences.

A 'link' is a reference to another span (possibly in a different trace) that is
causally related to the current span.  Unlike the parent-child relationship,
links represent non-hierarchical associations between spans.

A 'baggage' item is a key-value data pair that can be used for the duration of
an entire trace, from the moment it is added to the span.

A span 'status' indicates the outcome of the operation: unset (default), ok
(successful) or error (failed).  An optional description string can accompany
the error status.

Note: telemetry delivery is best-effort.  The OpenTelemetry specification does
not mandate reliable delivery, so there is no guarantee that all telemetry
reaches its destination.  Data may be dropped at any stage -- collection,
processing or export -- through sampling, buffer or queue limits, network
failures or backend unavailability.  Treat occasional loss as a normal
operational condition rather than a fault, and do not rely on OTel data where
completeness is critical (for example billing or auditing).


4. OTel configuration
----------------------

The OTel filter must also be included in the HAProxy configuration, in the
proxy section (frontend / listen / backend):

   frontend otel-test
     ...
     filter opentelemetry [id <id>] config <file> [<name>]
     ...

If no filter id is specified, 'otel-filter' is used as default.  The 'config'
parameter must be specified and it contains the path of the OTel filter
configuration file.  This file defines the OTel scopes, groups and
instrumentation sections (see section 4.1).  The YAML configuration for the
OpenTelemetry SDK is a separate file, referenced by the 'config' keyword inside
the "otel-instrumentation" section (see section 4.2).  A filter line with no
id is accepted with a warning.

An optional section name after the file path selects the top-level OTel scope
to use; it defaults to the filter id, so several filters may share one scope.
Each of 'id' and 'config' stands once on the line, and at most one name follows
the file; a repeated keyword or a further word there is a configuration error.
Neither the filter id nor the section name may be the word 'if' or 'unless':
the two words introduce conditions and are reserved, so no name in the OTel
configuration, of whatever kind, may be written as either word.  An ACL may
not be called 'or' either, whatever its case, that word joining the terms of a
condition.  A name is limited in length too, to 127 characters, every key and
variable name included, a variable counted with its scope prefix and the key
inside a 'set-var-ctx' field left aside, which follows the W3C limits given with
that keyword; an ACL name, an event name, an exception type, the description
of an 'instrument' and the text of a sample expression carry no such limit, and
neither the filter id, the context name on the 'config' line nor the name of
the top-level scope is measured.  Two filters of one proxy cannot share a scope
that injects a context: the resolved inject names must be unique across the OTel
filters of a proxy.  Filters that never run on one stream may reuse the names
freely, the injected variables included: the variables the filter generates
carry the transaction scope, so they end with the stream.  A 'set-var' name
carries the scope the writer chose, and one in the 'proc' scope outlives the
stream.

Whatever a keyword allows across its lines, a clause of it stands once on a
line: a second 'time' on a 'log-record', a second 'unit' on an 'instrument' or
a second 'root' on a 'span' is a configuration error.  So is a name or a key
written twice in one list, an 'attr' key, a 'link' target or an 'unset-var'
variable among them; the same name may still stand in two lists of one line, as
a span's 'parent' and its inline 'link'.

Note: the parser does not limit the number of OTel filters; it only requires
each filter id to be unique.  Since the wrapper version 3.0.0, every filter
instance holds its own library context, so several filters, each with its own
configuration, may operate simultaneously in one HAProxy process.


4.1. OTel scope
----------------

If the filter id is defined for the OTel filter, then the OTel scope with the
same name should be defined in the configuration file.  In the same
configuration file we can have several defined OTel scopes.

Each OTel scope must have a defined (only one) "otel-instrumentation" section
that is used to configure the operation of the OTel filter and define the used
groups and scopes.

OTel scope starts with the id of the filter specified in square brackets and
ends with the end of the file or when a new OTel scope is defined.  An OTel
scope name is unique: opening a second scope with an already used name is a
configuration error.  The whole configuration lives inside OTel scopes, so the
file begins with a scope line; a section or keyword outside of any OTel scope
is a configuration error.  The filter reads only the scope its line selects and
checks the rules below in that one; of the others it reads the name alone, so an
unread scope may hold anything from its first section header on until a filter
line selects it; a line above every header is an error HAProxy itself reports.

For example, this defines two OTel scopes in the same configuration file:
  [my-first-otel-filter]
    otel-instrumentation instrumentation1
    ...
    otel-group group1
    ...
    otel-scope scope1
    ...

  [my-second-otel-filter]
    ...

Several OTel scopes may coexist in a single configuration file, each declared
filter selecting and running one of them.  Since the wrapper version 3.0.0,
every filter instance holds its own library context, so several filters may
operate simultaneously in one HAProxy process, each with its own scope.


4.2. "otel-instrumentation" section
-------------------------------------

Only one "otel-instrumentation" section must be defined for each OTel scope; a
second one is a configuration error whatever its name.

Unless stated otherwise, each keyword of this section may be given only once
and a repetition is a configuration error.  The 'groups', 'scopes' and 'acl'
keywords may be repeated, every name that they declare being unique, and each
'option' may likewise be set only once.  An 'option' line also takes the HAProxy
keyword prefixes 'no' and 'default', which keep the meaning section 4.1 of the
HAProxy Configuration Manual gives them; a prefixed line counts as the one use
the option is allowed.

The mandatory 'config' keyword defines the YAML configuration file for the
OpenTelemetry SDK.  This file specifies the telemetry pipeline: exporters,
processors, samplers, readers, providers and signals.

Through optional keywords can be defined ACLs, logging, rate limit, and groups
and scopes that define the tracing model.


otel-instrumentation <name>
  A new OTel instrumentation with the name <name> is created.

  Arguments :
    name - the name of the OpenTelemetry instrumentation section


  The following keywords are supported in this section:
    - mandatory keywords:
      - config

    - optional keywords:
      - acl
      - debug-level
      - groups
      - [no] log
      - [no|default] option disabled
      - [no|default] option dontlog-normal
      - [no|default] option hard-errors
      - [no|default] option noflush
      - [no|default] option require-context
      - rate-limit
      - scopes


acl <aclname> <criterion> [flags] [operator] <value> ...
  Declare an access list.  Each <aclname> may be declared only once within the
  section; a duplicate name is a configuration error.  Everything after the
  criterion is pattern text, so an 'if'/'unless' there is taken as a pattern
  value and not as a condition.

  The name may be used in the conditions of every otel-scope, once the section
  has defined it; an otel-instrumentation placed below an otel-scope comes too
  late for that scope.

  To configure and use the ACL, see section 7 of the HAProxy Configuration
  Manual.


config <file> [context]
  The mandatory keyword associated with the OTel instrumentation configuration.
  This keyword sets the path of the YAML configuration file for the
  OpenTelemetry SDK and, optionally, the name of the signals context to select
  within that file.  Without the argument, the wrapper serves the 'default'
  context.  The YAML file defines the complete telemetry pipeline including the
  exporters, samplers, processors, readers, providers and signal routing.  The
  file is parsed already when the configuration is loaded: 'haproxy -c' rejects
  a broken file.  A context name that matches no signal entry falls back to the
  'default' entry of each signal, with a warning at startup; a signal with no
  'default' entry has nothing to fall back to, so 'haproxy -c' rejects the name.
  A trailing 'if'/'unless' condition is not accepted.

  The YAML configuration file supports the following top-level sections:

  'exporters' - defines telemetry data destinations.  Supported exporter types
  are:
    - otlp_grpc     : export via OTLP over gRPC
    - otlp_http     : export via OTLP over HTTP (JSON or Protobuf)
    - otlp_file     : export to local files in OTLP format
    - zipkin        : export to Zipkin-compatible backends
    - elasticsearch : export to Elasticsearch
    - ostream       : write to a file (text output, useful for debugging)
    - memory        : in-memory buffer (useful for testing)

  'samplers' - defines trace sampling strategies.  Supported types:
    - always_on            : sample every trace
    - always_off           : sample no traces
    - trace_id_ratio_based : sample a fraction of traces (set by ratio)
    - parent_based         : sampling decision based on parent span

  'processors' - defines how telemetry data is processed before export:
    - batch  : batch spans before exporting (configurable queue size, export
               interval and batch size)
    - single : export each span individually

  'readers' - defines metric readers with configurable export interval and
  timeout.

  'providers' - defines resource attributes (service name, version, instance ID,
  namespace, etc.) that are attached to all telemetry data.

  'signals' - binds the above components together for each signal type (traces,
  metrics, logs), specifying which exporter, sampler, processor, reader and
  provider to use.  Under each signal type, the bindings are grouped into named
  signals contexts, and the optional [context] argument selects the one that
  serves the filter ('default' when absent).

  Arguments :
    file - the path of the YAML configuration file
    context - the name of the signals context to select ('default' if absent)


debug-level <value>
  This keyword sets the value of the debug level related to the display of debug
  messages in the OTel filter.  The 'debug-level' value is a bitmask, ie a
  single value bit enables or disables the display of the corresponding debug
  message that uses that bit.  The default value is set via the
  FLT_OTEL_DEBUG_LEVEL macro in the include/config.h file.  Debug level value is
  used only if the OTel filter is compiled with the debug mode enabled,
  otherwise it is ignored and a warning at load says so.  The value itself is
  checked in every build, so a configuration parses the same with and without
  the debug build.

  Arguments :
    value - bitmask value (hexadecimal notation, e.g. 0x77f)


groups <name> ...
  A list of "otel-group" groups used for the currently defined instrumentation
  is declared.  Several groups can be specified in one line, and the keyword
  may be repeated on multiple lines.  Every name must be a defined "otel-group"
  section and is unique in the instrumentation; a repeat is a configuration
  error.  Every defined section must stand here as well: one that no 'groups'
  line names is a configuration error too.  A trailing 'if'/'unless' condition
  is not accepted.

  Arguments :
    name - the name of the OTel group


log global
log <addr> [len <len>] [format <fmt>] [sample <ranges>:<size>]
    [profile <prof>] <facility> [<level> [<minlevel>]]
no log
  Enable per-instance logging of events and traffic.

  HAProxy allows the 'no' and the 'default' prefix on this keyword; 'no log'
  leaves the filter's own logging off, 'default log' works like a plain line,
  and either one uses up the single line the section allows.

  To configure and use the logging system, see section 4.2 of the HAProxy
  Configuration Manual.


option disabled
no option disabled
default option disabled
  Keyword which turns the operation of the OTel filter on or off.  By default
  the filter is on.


option dontlog-normal
no option dontlog-normal
default option dontlog-normal
  Drop the below-error log lines (warnings and informational) while keeping the
  error-level messages.  By default, this option is disabled and all levels are
  logged.  For this option to be considered, logging must be turned on.

  See also: 'log' keyword description.


option hard-errors
no option hard-errors
default option hard-errors
  During the operation of the filter, some errors may occur, caused by incorrect
  configuration of the instrumentation or some error related to the operation of
  HAProxy.  By default, such an error will not interrupt the filter operation
  for the stream in which the error occurred.  If the 'hard-errors' option is
  enabled, the operation error prohibits all further processing of events and
  groups in the stream in which the error occurred.


option noflush
no option noflush
default option noflush
  When the filter is deinitialized, the telemetry still buffered in the OTel
  library is flushed to the exporters, and that final flush waits for them,
  so an unreachable endpoint at that moment delays the shutdown or the reload
  by the export timeouts.  With this option enabled the teardown sets a zero
  flush budget on the OTel handles instead: the exporters are shut down at once
  and the buffered telemetry is dropped.  By default the option is disabled
  and the buffered telemetry is flushed on deinitialization.


option require-context
no option require-context
default option require-context
  By default, a stream that carries no incoming context starts a new trace,
  with its first span becoming the trace root.  With this option enabled, a
  stream produces no telemetry at all -- no trace, no metric and no log record
  -- until a configured 'extract' finds a valid upstream context: a scope that
  runs before the extract emits nothing, and a scope carrying extracts none of
  which yields a valid context disables the filter for the whole stream.  At
  least one used scope must therefore hold an 'extract', and an 'otel-scope'
  bound to an event that runs before the request headers become readable is
  rejected at configuration time; group-driven scopes fire at their action's
  rule location instead and are simply checked at run time.  Enable this option
  on a downstream HAProxy that should only continue a trace propagated from
  upstream, never originate one of its own.


rate-limit <value>
  This option allows limiting the use of the OTel filter, ie it can be
  influenced whether the OTel filter is activated for a stream or not.
  Determining whether or not a filter is activated depends on the value of this
  option that is compared to a randomly selected value when attaching the filter
  to the stream.  By default, the value of this option is set to 100.0, ie the
  OTel filter is activated for each stream.  A value below 100.0 is accepted
  with a warning: not every stream then starts a trace, so a span context does
  not always reach a downstream HAProxy.

  Arguments :
    value - floating point value ranging from 0.0 to 100.0


scopes <name> ...
  This keyword lists the "otel-scope" definitions used by the currently defined
  instrumentation.  Multiple scopes may be given on one line, and the keyword
  may be repeated on multiple lines.  Every name must be a defined "otel-scope"
  section and is unique in the instrumentation; a repeat is a configuration
  error.  A trailing 'if'/'unless' condition is not accepted.  A defined section
  that neither a 'scopes' line nor a declared otel-group names is reported as
  unused, with a warning at load.

  Arguments :
    name - the name of the OTel scope


4.3. "otel-scope" section
--------------------------

Stream processing begins with filter attachment, then continues with the
processing of a number of defined events and groups, and ends with filter
detachment.  The "otel-scope" section is used to define actions related to
individual events.  However, this section may be part of a group, so the event
does not have to be part of the definition.

A keyword that names a span or an extracted context -- the 'parent' and the
inline 'link' of a span, the standalone 'link', 'finish', 'set-var-ctx' and
the 'span' of a 'log-record', which takes a span alone -- refers to a name that
some otel-scope defines, this one or another.  The name is resolved at run time
against the spans and the contexts the stream has opened by then, so a name
that no otel-scope defines anywhere can never resolve and is refused when the
configuration is checked.  The 'finish' wildcards are the exception, standing
for whatever the stream holds at that moment.


otel-scope <name>
  Creates a new OTel scope definition named <name>.  The name is unique within
  its top-level scope; another top-level scope may reuse it.

  Arguments :
    name - the name of the OTel scope


  The following keywords are supported in this section:
    - acl
    - attribute
    - baggage
    - event
    - exception
    - extract
    - finish
    - idle-timeout
    - inject
    - instrument
    - link
    - log-record
    - otel-event
    - otel-stop
    - set-var
    - set-var-ctx
    - span
    - status
    - unset-var


acl <aclname> <criterion> [flags] [operator] <value> ...
  Declare an access list.  Each <aclname> may be declared only once within the
  section; a duplicate name is a configuration error.  Everything after the
  criterion is pattern text, so an 'if'/'unless' there is taken as a pattern
  value and not as a condition.

  The name may be used only in the conditions of this otel-scope, where it hides
  an otel-instrumentation ACL of the same name.  Each name of a condition is
  looked up on its own, in the scope first, then in the otel-instrumentation
  and then in the HAProxy configuration, so one condition may mix them.  The ACL
  has to exist when the line is read: an otel-instrumentation section below this
  scope, or a HAProxy 'acl' line below the 'filter' line, comes too late and the
  name is refused.

  To configure and use the ACL, see section 7 of the HAProxy Configuration
  Manual.


attribute <key> <sample> ... [{ if | unless } <condition>]
  This keyword allows setting an attribute for the currently active span.  The
  first argument is the name of the attribute (key) and the rest are its value.
  A value can consist of one or more sample expressions.  If the value is only
  one sample, then the type of that data depends on the type of the HAProxy
  sample.  If the value contains more samples, then the data type is string.
  The data conversion table is below:

   HAProxy sample data type | the OpenTelemetry data type
  --------------------------+----------------------------
            BOOL            |        BOOL
            SINT            |        INT64
            IPV4            |        STRING
            IPV6            |        STRING
            STRING          |        STRING
            METH            |        STRING
            BINARY          |        UNSUPPORTED
  --------------------------+----------------------------

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  covers the attribute; it is set only when the condition holds at request time.
  The keyword may be repeated, one line per condition: the lines of one key are
  tried in configuration order, the first whose condition holds sets that key,
  and a line without a condition, placed last, acts as the key's default.

  Arguments :
    key       - key part of a data pair (attribute name)
    sample    - sample expression (value part of a data pair), at least
                one sample must be present
    condition - an optional ACL-based 'if' or 'unless' condition


baggage <key> <sample> ... [{ if | unless } <condition>]
  Baggage items allow the propagation of data between spans, ie allow the
  assignment of metadata that is propagated to future children spans.  This data
  is formatted in the style of key-value pairs and is part of the context that
  can be transferred between processes that are part of a server architecture.

  This keyword allows setting the baggage for the currently active span.  The
  data type is always a string, ie any sample type is converted to a string.
  The exception is a binary value that is not supported by the OTel filter.  A
  sample that yields nothing leaves its pair out, since the carrier refuses an
  empty value and would drop the other pairs of the span with it.

  See the 'attribute' keyword description for the data type conversion table.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  decides whether the baggage is set.  The keyword may be repeated, one line
  per condition: the lines of one key are tried in configuration order, the
  first whose condition holds sets that key, and a line without a condition,
  placed last, acts as the key's default.

  Arguments :
    key       - key part of a data pair; letters, digits, '_', '.' and '-'
    sample    - sample expression (value part of a data pair), at least one
                sample must be present
    condition - an optional ACL-based 'if' or 'unless' condition


event <name> [time [<unit>] <sample>] <key> <sample> ... [{ if | unless } <condition>]
  This keyword allows adding a span event to the currently active span.  A span
  event is a named, timestamped annotation with optional attributes.  The data
  type is always a string, ie any sample type is converted to a string.

  See the 'attribute' keyword description for the data type conversion table.

  An optional 'time [<unit>] <sample>' clause, placed between the event name
  and the first key, overrides the event timestamp.  The unit is 's' (default),
  'ms', 'us' or 'ns', and the sample must evaluate to a number; without it the
  event takes the time of the scope run that adds it, the span's own start only
  in the run that creates the span.  A 'time' token directly after the name is
  read as this clause, so the key cannot be called 'time'.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  covers the event.  The keyword may be repeated, and every line whose condition
  holds is applied.  The lines of one span definition are grouped by <name>:
  each distinct name produces a single span event, and the <key>/<sample> pairs
  of all lines sharing that name become the attributes of that one event.  The
  timestamp is set by the first such line whose 'time' clause evaluates, and
  the later lines keep it.  Within one event a key may be named across several
  lines, one per condition: the first line whose condition holds supplies the
  key, and a line without a condition, placed last for its key, acts as the
  default.  The same key may appear in different events independently.

  Arguments :
    name      - name of the span event
    time      - optional explicit timestamp (with optional unit selector)
    key       - key part of a data pair (attribute name within the event)
    sample    - sample expression (value part of a data pair), at least one
                sample must be present
    condition - an optional ACL-based 'if' or 'unless' condition


extract <name-prefix> [use-vars | use-headers]
  For a more detailed description of the propagation process of the span
  context, see the description of the keyword 'inject'.  Only the process of
  extracting data from the carrier is described here.

  The default carrier is HTTP headers.  If OTEL_USE_VARS is enabled at compile
  time, the 'use-vars' option can be used instead to extract context from
  HAProxy variables.  A header whose name is only the prefix itself is ignored,
  as stripping the prefix would leave an empty data name.  No span may carry the
  context name: a reference resolves to the spans first, so the context could
  never be reached.  A prefix is unique within its scope and a repetition is a
  configuration error, while another otel-scope may reuse the name.

  Header extraction (use-headers) needs a channel to read from and is rejected
  on the two events that fire without one, on-stream-start and on-stream-stop.
  A scope that carries no 'otel-event' is not affected, since the 'otel-group'
  action that runs it always brings a channel.  Variable extraction (use-vars)
  carries no such restriction, and a prefix with an upper-case letter or a '-'
  draws the same variable-name warning as under 'inject'.

  When the OpenTelemetry context is transferred via the HTTP header and the
  first character of the context name is '-', the context name is excluded
  from the header.  This behavior enables the use of OpenTelemetry contexts
  with external services that do not expect a HAProxy-specific name prefix on
  the propagation headers.

  Arguments :
    name-prefix - data name prefix (ie key element prefix)
    use-vars    - data is extracted from HAProxy variables
    use-headers - data is extracted from the HTTP header


  Below is an example of using HAProxy variables to transfer span context data:

  --- test/ctx/otel.cfg -----------------------------------------------
      ...
      otel-scope client_session_start_2
          extract "otel_ctx_1" use-vars
          span "Client session" parent "otel_ctx_1"
      ...
  ---------------------------------------------------------------------


finish <name> ...
  Closing a particular span or span context.  Instead of the name of the span,
  there are several specially predefined names with which we can finish certain
  groups of spans and span contexts.  So it can be used as the name '*req*' for
  all open spans and contexts related to the request channel, '*res*' for all
  of them related to the response channel and '*' for all of them regardless of
  which channel they are related to.  Several spans and/or span contexts can be
  specified in one line.  A name may be given only once in the scope, whether
  on one line or on several, and a repetition is a configuration error; another
  otel-scope may finish the same span.  The keyword does not accept a trailing
  'if'/'unless' condition; one given after the names is rejected.

  Arguments :
    name - the name of the span or span context


inject <name-prefix> [use-vars] [use-headers]
  In OpenTelemetry, the transfer of data related to the tracing process between
  microservices that are part of a larger service is done through the
  propagation of the span context.  The basic operations that allow us to access
  and transfer this data are 'inject' and 'extract'.

  'inject' allows us to extract span context so that the obtained data can be
  forwarded to another process (microservice) via the selected carrier. 'inject'
  in the name actually means inject data into carrier.  Carrier is an interface
  here (ie a data structure) that allows us to transfer tracing state from one
  process to another.

  Data transfer can take place via one of two selected storage methods, the
  first is by adding data to the HTTP header and the second is by using HAProxy
  variables (the latter requires OTEL_USE_VARS=1 at compile time).  Only data
  transfer via HTTP header can be used to transfer data to another process (ie
  microservice).  All data is organized in the form of key-value data pairs.

  When neither 'use-vars' nor 'use-headers' is given, injection defaults to
  'use-headers'.  A span carries a single context, so 'inject' may appear at
  most once per span of one otel-scope; a second 'inject' on the same span in
  the same scope is a configuration error, while a span re-activated in another
  otel-scope may carry a second context there.  The resolved context name is, in
  addition, one that no other 'inject' of the configuration or of the other OTel
  filters of the proxy resolves to, two names differing only in case counting as
  one because both carriers keep the name in lower case, and at most one context
  on the proxy that injects headers may use a name with the leading '-', whose
  headers carry no name at all; a second one of either kind is a configuration
  error.

  Header injection (use-headers) is only available on events that provide a
  writable HTTP header carrier when they fire.  Using it on any other event is
  a configuration error, reported at startup.  It is rejected for these events:

    on-stream-start, on-stream-stop, on-idle-timeout, on-backend-set,
    on-client-session-end, on-server-unavailable, on-server-session-start,
    on-tcp-response, on-http-end-request, on-http-end-response, on-http-reply,
    on-server-session-end

  A scope that carries no 'otel-event' at all is rejected the same way: it runs
  wherever the 'otel-group' action that reaches it stands, which the OTel
  configuration file cannot know, so such a scope can inject the context into
  variables only.

  Variable injection (use-vars) carries no such restriction and may be used
  with any event.

  No matter which data transfer method you use, we need to specify a prefix for
  the key element.  Letters, digits and the characters '_', '.' and '-' can be
  used to construct the data name prefix, with uppercase letters converted to
  lowercase when the prefix is created.  The special prefix '-' can be used to
  generate the name automatically from the scope's event name or the span name.
  The span name is checked like a written prefix, so it has to hold only those
  characters and fit the length limit.  The generated name keeps a leading '-',
  so by the rule below the injected headers carry only the bare propagation
  headers, with no HAProxy-specific name prefix.

  When the OpenTelemetry context is transferred via the HTTP header and the
  first character of the context name is '-', the context name is excluded
  from the header.  This behavior enables the use of OpenTelemetry contexts
  with external services that do not expect a HAProxy-specific name prefix on
  the propagation headers.

  Arguments :
    name-prefix - data name prefix (ie key element prefix), or '-' for automatic
                  naming
    use-vars    - HAProxy variables are used to store and transfer data
                  (requires OTEL_USE_VARS=1)
    use-headers - HTTP headers are used to store and transfer data


  Below is an example of using HTTP headers and variables to propagate the span
  context.

  --- test/ctx/otel.cfg -----------------------------------------------
      ...
      otel-scope client_session_start_1
          span "HAProxy session" root
              inject "otel_ctx_1" use-headers use-vars
      ...
  ---------------------------------------------------------------------

  Because HAProxy does not allow the '-' character in the variable name (which
  is automatically generated by the OpenTelemetry API and on which we have no
  influence), it is converted to the letter 'D'.  We can see that there is no
  such conversion in the name of the HTTP header because the '-' sign is allowed
  there.  Due to this conversion, initially all uppercase letters are converted
  to lowercase because otherwise we would not be able to distinguish whether the
  disputed sign '-' is used or not.  A name holding an upper-case letter or a
  '-', which every autogenerated name does, is accepted with a warning about
  this mapping.

  Thus created HTTP headers and variables are deleted when executing the
  'finish' keyword or when detaching the stream from the filter.

  Independently of 'inject', a build with OTEL_USE_VARS=1 registers the HAProxy
  variable 'sess.otel.uuid' on every stream the filter attaches to and stores
  the stream's runtime-context UUID in it, so a configuration can read a stable
  per-stream identifier through 'var(sess.otel.uuid)' (several test examples
  use it as a baggage value).  Without this build option the variable does not
  exist and the fetch yields an empty value.


instrument <type> <name> [aggr <aggregation>] [desc <description>] [unit <unit>] value <sample> [bounds <bounds>] [{ if | unless } <condition>]
instrument update <name> [attr <key> <sample> ...] [{ if | unless } <condition>]
  This keyword allows creating or updating metric instruments within the scope.
  Metric instruments record numerical measurements that are exported alongside
  traces.

  To create a new instrument, give the instrument type, a name, and a sample
  expression for the measurement value, preceded by the 'value' keyword.  The
  'value' takes one expression and refuses a second one, since the measurement
  it records is a number.  An optional description ('desc') and unit string
  ('unit') may also be given.  The 'aggr', 'desc', 'unit', 'value' and 'bounds'
  keywords may appear in any order.  A name begins with a letter and carries
  letters, digits and the punctuation '_', '.', '-' and '/' alone, and a unit
  is at most 63 ASCII characters: the metric SDK refuses anything else, and an
  instrument it refuses records nothing at all.

  An aggregation type can be specified using the 'aggr' keyword followed by one
  of the supported aggregation types listed below.  When specified, a metrics
  view is registered with the given aggregation strategy.  If no aggregation
  type is specified, the SDK default is used.

  For histogram instruments (hist_int), optional bucket boundaries follow the
  'bounds' keyword as a double-quoted string of space-separated numbers, which
  are sorted internally; any duplicate boundaries are then rejected.  With no
  explicit aggregation type set, histogram aggregation is applied automatically.
  On any other instrument type the 'bounds' clause is a configuration error.

  To update an existing instrument (a create form defined in any scope, the
  same one included), use 'update' followed by the instrument name.  Optional
  attributes can be added using the 'attr' keyword, followed by a key and a
  single sample expression evaluated at runtime; the keyword may stand once
  before the first pair or again before each of them, so no key may be called
  'attr'.

  Supported instrument types.  The recorded <value> is applied differently by
  each one, so choose the type that matches how the measurement behaves:

    cnt_int (counter, uint64) -- adds <value> to a monotonically rising total.
    Use it for cumulative counts such as requests served or bytes forwarded;
    <value> is the non-negative amount to add, not the running total.  The
    default aggregation is a sum.

    udcnt_int (up-down counter, int64) -- adds a signed <value> to a total that
    may rise and fall.  Use it for levels tracked as deltas, such as in-flight
    requests or queue depth.  The default aggregation is a non-monotonic sum.

    hist_int (histogram, uint64) -- files each <value> as one observation in a
    bucketed distribution.  Use it for value spreads such as request latencies
    or payload sizes, when percentiles matter.  Bucket boundaries follow the SDK
    default unless set with 'bounds', or switched to a base-2 exponential layout
    with 'aggr exp_histogram'.

    gauge_int (gauge, int64) -- replaces the reported value with the latest
    <value>, a snapshot of a level that rises and falls where only the most
    recent reading matters, such as a current queue length.  The default
    aggregation is the last value.

  Supported aggregation types:
    - drop          : measurements are discarded
    - histogram     : explicit bucket histogram
    - last_value    : last recorded value
    - sum           : sum of recorded values
    - default       : SDK default for the instrument type
    - exp_histogram : base-2 exponential histogram

  Observable (asynchronous) instruments are not supported.  The OpenTelemetry
  SDK invokes their callbacks from an external background thread that is not
  a HAProxy thread.  HAProxy sample fetches rely on internal per-thread-group
  state and return incorrect results when called from a non-HAProxy thread.

  Double-precision types are not supported because HAProxy sample fetches do
  not return double values.

  A create form registers the instrument and supplies the value that is to be
  recorded, but emits no measurement on its own; a data point is produced only
  when an 'update' form written for that instrument runs.  A create-form name
  may repeat, in the same or another scope, and compares case-insensitively: the
  lines of one name share one instrument, so they must agree on the type, the
  aggregation, the description, the unit and the bounds, and a disagreement is a
  configuration error.  At run time exactly one otel-scope ever creates the
  instrument, and a create line of another otel-scope that finds it already
  created is a runtime error, not a line without effect.  Within one scope the
  create lines of a name follow the first-match rule, one line per condition
  with the bare line last as the default: the first line whose condition holds
  registers the instrument, and an update form records its value, taken from the
  chain of its own scope; without one there it follows the scope that created
  the instrument, and until it exists, the first scope of the file that runs and
  defines the name.  An update that runs needs such a scope, so it is refused
  when every create line of the name stands in a scope that never runs; the
  create lines alone record nothing and only draw the warning of the scope that
  holds them.  The update form is additive, like 'log-record': it may repeat
  freely, and every update line whose condition holds records one measurement of
  its own; registration itself produces no data point, and a create line whose
  condition does not hold registers nothing.

  The update form supplies no value of its own, only attributes and an optional
  condition; to record a different quantity, define a separate instrument.  To
  feed a computed or externally-set value, set the create form's 'value' to a
  'var(...)' fetch and assign that variable with 'set-var'.

  For example:
    instrument cnt_int  "my_counter" desc "Counter" value int(1)
    instrument hist_int "my_hist" aggr exp_histogram desc "Latency" value lat_ns_tot unit "ns"
    instrument hist_int "my_hist2" desc "Latency" value lat_ns_tot unit "ns" bounds "100 1000 10000 100000"
    instrument hist_int "req_bytes" value var(txn.otel_size) unit "By"
    instrument update "my_counter" attr "key1" str("val1")

  Arguments :
    type      - the instrument type (see list above)
    name      - the name of the instrument
    aggr      - optional aggregation type (see list above)
    desc      - optional human-readable description of the instrument
    unit      - optional unit string for the instrument
    value     - sample expression providing the measurement value
    bounds    - optional histogram bucket boundaries (hist_int only)
    attr      - attribute key and sample expression (update form only)
    condition - an optional ACL-based 'if' or 'unless' condition


log-record <severity> [id <integer> event <name>] [time [<unit>] <sample>] [span <ref>] [attr <key> <sample>] ... <sample> ... [{ if | unless } <condition>]
  This keyword emits an OpenTelemetry log record within the scope.  The first
  argument is a required severity level.  Optional keywords follow in any order,
  each of them once except 'attr', before the trailing sample expressions that
  form the log record body:

    id <integer>           - numeric event identifier
    event <name>           - event name string
    time [<unit>] <sample> - override the log event timestamp at runtime
    span <ref>             - associate the log record with an open span
    attr <key> <sample>    - add an attribute evaluated at runtime (repeatable)

  The 'id' and 'event' keywords must be specified together or both omitted; any
  other combination is a configuration error.  The identifier must be at least
  1; a zero is reserved as the internal no-id marker and is rejected.

  The 'time' keyword takes an optional unit (one of 's' (the default), 'ms',
  'us', or 'ns') followed by a single HAProxy sample expression.  The value
  is interpreted as a count since the Unix epoch in the chosen unit, and the
  sample is evaluated at runtime and must yield an integer (or a string that
  is convertible to one).  The value sets the event timestamp; when evaluation
  or numeric conversion fails, the event timestamp defaults to the wall-clock.
  The observed timestamp always records when the filter emitted the record.

  The remaining arguments at the end are sample fetch expressions.  A single
  sample preserves its native type; multiple samples are concatenated as a
  string.  Alternatively, the body may be a HAProxy log-format string wrapped
  in '%[ ... ]'; the outer wrapper is stripped before parsing, and the inner
  string may then contain log-format aliases such as %ci, %t or %ft.

  Supported severity levels follow the OpenTelemetry specification:
    trace, trace2, trace3, trace4, debug, debug2, debug3, debug4,
    info, info2, info3, info4, warn, warn2, warn3, warn4,
    error, error2, error3, error4, fatal, fatal2, fatal3, fatal4

  The log record is only emitted when the logger is enabled for the configured
  severity (controlled by the 'min_severity' option in the YAML logs signal
  configuration).  If a 'span' reference is given but the named span is not
  found at runtime, the log record is emitted without span correlation.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  covers the whole record; when it does not pass at runtime, the record is
  skipped entirely, with neither its body nor its attributes evaluated.  The
  keyword may be repeated to emit several records from one scope.

  For example:
    log-record info str("heartbeat")
    log-record info id 1001 event "http-request" span "Frontend HTTP request" attr "http.method" method method url
    log-record trace id 1000 event "session-start" span "Client session" attr "src_ip" src src str(":") src_port
    log-record warn id 1018 event "server-unavailable" str("503 Service Unavailable")
    log-record info id 1002 event "server-session-end" "%[%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts]"
    log-record info time us date(0,us) str("microsecond-accurate event")
    log-record info time ns hdr(x-event-time-ns) str("client-supplied timestamp")

  Arguments :
    severity  - the log severity level (see list above)
    id        - optional numeric event identifier
    event     - optional event name
    time      - optional explicit timestamp (with optional unit selector)
    span      - optional name of an open span to associate with
    attr      - optional attribute key-value pairs (repeatable)
    sample    - sample fetch expression(s) forming the log record body
    condition - an optional ACL-based 'if' or 'unless' condition


link { <ref> ... | <ref> attr <key> <sample> ... } [{ if | unless } <condition>]
  This keyword adds span links to the currently active span.  A span link
  represents a causal relationship to another span without establishing a
  parent-child hierarchy.  Links are useful for connecting spans across
  different traces or for associating related spans within the same trace.

  In the first form, multiple span or context names can be specified in one
  line.  Each name is resolved at runtime by searching for an active span or
  an extracted context with that name.  If a referenced span or context cannot
  be found, the link is silently skipped.

  In the second form, a single link reference is followed by the 'attr' keyword
  and one or more <key> <sample> attribute pairs, which attach attributes to
  that one link.  Each pair gives a key and a single sample expression as its
  value, and the keyword may stand again before each pair; see the 'attribute'
  keyword description for the data type conversion table.

  The attributes belong to that one link, so an 'attr' written after a list of
  references is rejected.  Because the keyword opens the form and separates the
  pairs, neither a link target after the first one nor an attribute key can be
  the literal word 'attr'.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  guards the link; it is added only when the condition holds at request time.
  On a line naming several references, the condition applies to each of them.
  A link target cannot be the literal word 'if' or 'unless'; that word always
  starts the condition.

  The keyword may be repeated freely, with or without a condition: every line
  whose condition holds adds its links, so naming the same target on several
  lines adds one link per line.  A target repeated within a single line is a
  configuration error, and so is a repeated attribute key.

  For example:
    link "Span A" "Span B"
    link "inbound" attr "link.detail" str("follows-from")
    link "Span A" "Span B" unless { path_beg /health }

  Arguments :
    ref       - the name of a span or span context to link to
    key       - attribute key attached to a single link
    sample    - sample expression giving the attribute value
    condition - an optional ACL-based 'if' or 'unless' condition


otel-event <name> [{ if | unless } <condition>]
  Set the event that triggers the 'otel-scope' to which it is assigned.  Only
  one 'otel-event' may be defined per scope; a second is a configuration error.
  Optionally, it can be followed by an ACL-based condition, in which case it
  will only be evaluated if the condition is true.  A scope with no 'otel-event'
  is run by an 'otel-group' action instead; when no group holds it, it never
  runs and a warning says so at load.

  The event also fixes where the scope's sample fetches and ACLs are evaluated.
  A fetch that reads data HAProxy does not hold at that point, a response header
  at a request event for instance, yields nothing at run time, so the check
  reports it as a warning, in a condition as well as in a value; a scope that an
  'otel-group' action runs is checked against the rule set holding the action
  the same way.

  ACL-based conditions are executed in the context of a stream that processes
  the client and server connections.  To configure and use the ACL, see section
  7 of the HAProxy Configuration Manual.  On this and every other keyword that
  takes a condition, a bare 'if' or 'unless' with no ACL terms following it is
  rejected.  They are likewise rejected as a span or context reference and as
  the name of a span, an otel-scope, an otel-group or an extract context.

  Arguments :
    name      - the event name
    condition - a standard ACL-based condition

  Supported events are (the table gives the names of the events in the OTel
  filter and the corresponding equivalent in the SPOE filter):

    -------------------------------------|------------------------------
      the OTel filter                    |  the SPOE filter
    -------------------------------------|------------------------------
      on-stream-start                    |  -
      on-stream-stop                     |  -
      on-idle-timeout                    |  -
      on-backend-set                     |  -
    -------------------------------------|------------------------------
      on-client-session-start            |  on-client-session
      on-frontend-tcp-request            |  on-frontend-tcp-request
      on-http-wait-request               |  -
      on-http-body-request               |  -
      on-frontend-http-request           |  on-frontend-http-request
      on-switching-rules-request         |  -
      on-backend-tcp-request             |  on-backend-tcp-request
      on-backend-http-request            |  on-backend-http-request
      on-http-tarpit-request             |  -
      on-process-server-rules-request    |  -
      on-http-process-request            |  -
      on-tcp-rdp-cookie-request          |  -
      on-process-sticking-rules-request  |  -
      on-http-headers-request            |  -
      on-http-end-request                |  -
      on-client-session-end              |  -
      on-server-unavailable              |  -
    -------------------------------------|------------------------------
      on-server-session-start            |  on-server-session
      on-tcp-response                    |  on-tcp-response
      on-http-wait-response              |  -
      on-process-store-rules-response    |  -
      on-http-response                   |  on-http-response
      on-http-headers-response           |  -
      on-http-end-response               |  -
      on-http-reply                      |  -
      on-server-session-end              |  -
    -------------------------------------|------------------------------

  The next table shows the filter callback that raises each event.  Events bound
  to a channel analyzer fire from channel_pre_analyze (before the analyzer runs)
  or channel_post_analyze (after it finishes).  The rest fire from a lifecycle
  or data-filtering callback and are not tied to an analyzer:

    -------------------------------------|------------------------------
      the OTel filter event              |  raising filter callback
    -------------------------------------|------------------------------
      on-stream-start                    |  stream_start
      on-stream-stop                     |  stream_stop
      on-idle-timeout                    |  check_timeouts
      on-backend-set                     |  stream_set_backend
    -------------------------------------|------------------------------
      on-client-session-start            |  channel_start_analyze
      on-frontend-tcp-request            |  channel_pre_analyze
      on-http-wait-request               |  channel_post_analyze
      on-http-body-request               |  channel_pre_analyze
      on-frontend-http-request           |  channel_pre_analyze
      on-switching-rules-request         |  channel_pre_analyze
      on-backend-tcp-request             |  channel_pre_analyze
      on-backend-http-request            |  channel_pre_analyze
      on-http-tarpit-request             |  channel_pre_analyze
      on-process-server-rules-request    |  channel_pre_analyze
      on-http-process-request            |  channel_pre_analyze
      on-tcp-rdp-cookie-request          |  channel_pre_analyze
      on-process-sticking-rules-request  |  channel_pre_analyze
      on-http-headers-request            |  http_headers
      on-http-end-request                |  http_end
      on-client-session-end              |  channel_end_analyze
      on-server-unavailable              |  channel_end_analyze
    -------------------------------------|------------------------------
      on-server-session-start            |  channel_start_analyze
      on-tcp-response                    |  channel_pre_analyze
      on-http-wait-response              |  channel_post_analyze
      on-process-store-rules-response    |  channel_pre_analyze
      on-http-response                   |  channel_pre_analyze
      on-http-headers-response           |  http_headers
      on-http-end-response               |  http_end
      on-http-reply                      |  http_reply (never invoked)
      on-server-session-end              |  channel_end_analyze
    -------------------------------------|------------------------------

  --- Proxy mode (HTTP and TCP) ---

  The filter attaches to both HTTP-mode and TCP-mode proxies.  On an HTTP proxy
  it runs HTX-level analysis, so every event above is reachable.  A TCP proxy
  carries no HTTP message and runs no HTTP analysis, so the HTTP-phase events
  never fire on one: a scope bound to such an event on a non-HTTP proxy is inert
  and is reported as a warning at config load.  These can fire on a TCP-mode
  proxy:

    stream lifecycle:
      on-stream-start
      on-stream-stop
      on-idle-timeout
      on-backend-set

    request channel:
      on-client-session-start
      on-frontend-tcp-request
      on-switching-rules-request
      on-backend-tcp-request
      on-process-server-rules-request
      on-tcp-rdp-cookie-request
      on-process-sticking-rules-request
      on-client-session-end
      on-server-unavailable

    response channel:
      on-server-session-start
      on-tcp-response
      on-process-store-rules-response
      on-server-session-end

  On an HTTP proxy the filter enables the analysers of its events itself, so
  each analyser event fires on every request.  On a raw TCP stream it cannot,
  as an analyser that nothing retires would stall the connection, so there an
  analyser event fires only when the proxy's own rules schedule the analyser:

    tcp-request content rules   on-frontend-tcp-request, on-backend-tcp-request
    use-server rules            on-process-server-rules-request
    persist rdp-cookie          on-tcp-rdp-cookie-request
    stick rules                 on-process-sticking-rules-request,
                                on-process-store-rules-response
    tcp-response content rules  on-tcp-response

  The lifecycle and session events fire on every connection.

  The remaining events carry HTTP message data and fire only on an HTTP-mode
  proxy:

    request channel:
      on-http-wait-request
      on-http-body-request
      on-frontend-http-request
      on-backend-http-request
      on-http-tarpit-request
      on-http-process-request
      on-http-headers-request
      on-http-end-request

    response channel:
      on-http-wait-response
      on-http-response
      on-http-headers-response
      on-http-end-response
      on-http-reply

  --- Filter placement (frontend, backend and listen) ---

  The filter line may stand in a frontend, a backend or a listen section, and
  the events a scope sees depend on where.  A filter of a frontend joins the
  stream at its creation and sees every event.  A filter of a backend section
  joins it at backend selection: HAProxy never runs the stream-start callback
  for it, the frontend-phase analysers have run by then, and HAProxy releases
  it as soon as both channels have ended their analysis, before the stream-stop
  callback and on every path.  These events never fire for such a filter, and a
  scope bound to one of them is reported as a warning at config load:

    on-stream-start
    on-frontend-tcp-request
    on-http-wait-request
    on-frontend-http-request
    on-switching-rules-request
    on-stream-stop

  The on-http-body-request event fires for such a filter only when the backend
  itself asks for the body, with 'option http-buffer-request' or with 'balance
  url_param', and the frontend did not; a scope bound to it in a backend that
  asks for neither draws the same warning.

  The on-client-session-end and on-server-session-end events fire as usual, both
  ends of the analysis coming before the release, and an aborted stream reaches
  them too, since HAProxy keeps the filter analysers on every abort path.  The
  idle timer is armed when the filter is attached, so on-idle-timeout fires for
  it from that point on.

  A listen proxy serves its own connections as frontend and backend at once,
  and HAProxy skips the backend-phase request analysers when the frontend is
  the backend.  There on-backend-tcp-request and on-backend-http-request fire
  only for a stream switched to another backend or routed in from a different
  frontend; a scope bound to one of them is told so by a warning at config load.
  A parent chain that runs through one of these events therefore breaks on a
  listen proxy, and with hard errors on, the filter is disabled for the stream.

  --- Stream lifecycle events (not tied to a channel analyzer) ---

  The on-stream-start and on-stream-stop events fire from the stream_start and
  stream_stop filter callbacks respectively, before any channel processing
  begins and after all channel processing ends.  No channel is available at
  that point, so context injection/extraction via HTTP headers cannot be used
  in scopes bound to these events.  Sample fetches in these scopes are not
  direction-constrained.

  The on-idle-timeout event fires periodically when the stream has no data
  transfer activity.  It requires the 'idle-timeout' keyword to set the
  interval.  This event is useful for heartbeat spans, idle-time metrics, and
  idle-time log records.  It fires from the check_timeouts filter callback
  using HAProxy's tick-based timer infrastructure.

  The timer is driven by the stream task's own expiry, not the request channel's
  analyse_exp, which analysers overwrite freely (the tarpit analyser, for one,
  assigns its own timeout there).  It therefore keeps firing at its interval
  even while an analyser holds the request, such as during a tarpit timeout, and
  the held request still completes.

  The on-backend-set event fires from the stream_set_backend filter callback
  when a backend is assigned to the stream.  It also fires when the stream stays
  on the same proxy, ie when the frontend and the backend are identical.


  --- Request channel events ---

  Analyzer events (tied to AN_REQ_* bits):

  The on-frontend-tcp-request event fires during frontend TCP content inspection
  (AN_REQ_INSPECT_FE).

  The on-http-wait-request event fires after the complete HTTP request has been
  received (AN_REQ_WAIT_HTTP).  This is a post-analyzer event.

  The on-http-body-request event fires when the request body analyzer is reached
  (AN_REQ_HTTP_BODY).  Unlike on-http-wait-request, it is a pre-analyzer event:
  it fires before the analyzer has buffered the body, so the request body is
  not yet available -- body fetches such as req.body and req.body_len evaluate
  to empty or zero.  Treat it as a marker for the start of body processing, not
  a place to inspect the body; the body is buffered only after this analyzer
  completes, and only when the proxy is configured to hold it (for example with
  'option http-buffer-request').

  The on-frontend-http-request event fires during frontend HTTP request
  processing: header rules, monitoring, statistics and redirects
  (AN_REQ_HTTP_PROCESS_FE).

  The on-switching-rules-request event fires when backend switching rules are
  evaluated (AN_REQ_SWITCHING_RULES).

  The on-backend-tcp-request event fires during backend TCP content inspection
  (AN_REQ_INSPECT_BE).

  The on-backend-http-request event fires during backend HTTP request processing
  (AN_REQ_HTTP_PROCESS_BE).

  The on-http-tarpit-request event fires when a request is held by the tarpit
  analyzer until the configured delay elapses (AN_REQ_HTTP_TARPIT).  Unlike the
  other request events, it is reached only for requests that a tarpit rule has
  marked, never for ordinary traffic.

  The on-process-server-rules-request event fires when use-server rules are
  evaluated (AN_REQ_SRV_RULES).

  The on-http-process-request event fires during inner HTTP request processing
  (AN_REQ_HTTP_INNER).

  The on-tcp-rdp-cookie-request event fires when RDP cookie persistence is
  evaluated (AN_REQ_PRST_RDP_COOKIE).

  The on-process-sticking-rules-request event fires when stick-table persistence
  matching rules are evaluated (AN_REQ_STICKING_RULES).

  Non-analyzer events (not tied to AN_REQ_* bits):

  The on-client-session-start event fires when the request channel analysis
  begins.  It corresponds to the start of a new client session.

  The on-http-headers-request event fires from the http_headers filter callback
  after all HTTP request headers have been parsed and analyzed.

  The on-http-end-request event fires from the http_end filter callback when all
  HTTP request data has been processed and forwarded.

  The on-client-session-end event fires when the request channel analysis ends.

  The on-server-unavailable event fires when the response channel never started
  its analysis, that is when no server was reached: HAProxy answered in its
  place, as with the 503 it returns when no connection could be established, or
  the stream ended before a connection.  It stands in for the response events
  then.  The filter decides this on whichever channel ends its analysis last,
  since a server connected after the request channel has ended still starts the
  response analysis; the event keeps the request channel it is bound to.


  --- Response channel events ---

  Analyzer events (tied to AN_RES_* bits):

  The on-tcp-response event fires during TCP response content inspection
  (AN_RES_INSPECT).

  The on-http-wait-response event fires after the complete HTTP response has
  been received (AN_RES_WAIT_HTTP).  This is a post-analyzer event.

  The on-process-store-rules-response event fires when stick-table store rules
  are evaluated (AN_RES_STORE_RULES).

  The on-http-response event fires during backend HTTP response processing
  (AN_RES_HTTP_PROCESS_BE).

  Non-analyzer events (not tied to AN_RES_* bits):

  The on-server-session-start event fires when the response channel analysis
  begins, after a server connection has been established.

  The on-http-headers-response event fires from the http_headers filter callback
  after all HTTP response headers have been parsed and analyzed.

  The on-http-end-response event fires from the http_end filter callback when
  all HTTP response data has been processed and forwarded.

  The on-http-reply event is meant to fire from the http_reply filter callback
  when HAProxy generates an internal reply (error page, deny response, redirect)
  on the response channel.  In practice it never fires: HAProxy has not invoked
  the http_reply filter callback since version 2.2-dev8 (2020), when its HTTP
  error handling was reworked to build replies through the http_reply_message()
  helper and the 'struct http_reply' type.  The flt_http_reply() dispatcher
  still exists in the filter API but nothing calls it, so the filter callback
  above never runs and the event never triggers.  To trace internally generated
  replies, attach the scope through an 'http-after-response otel-group' action,
  which runs on every response, including the ones that HAProxy synthesises.

  The on-server-session-end event fires when the response channel ends its own
  analysis.  HAProxy runs that end on both channels whatever the stream did, so
  the event also fires for a stream that never reached a server and therefore
  never saw on-server-session-start.


otel-stop [{ if | unless } <condition>]
  Stop the OpenTelemetry filter for the current connection.  When this keyword
  is reached, every span and span context still open on the stream is finished
  and the filter is disabled for the rest of that connection: no further scopes
  run for it, whether triggered by a later event or by an 'otel-group' action.
  It is the per-connection counterpart to the global 'flt-otel disable' CLI
  command that disables the filter for every connection at once.

  The stop is evaluated after the rest of the scope has run, so a stopping scope
  still emits the spans, log records and metrics it defines before tracing ends.

  Without a condition the directive always fires.  An optional 'if' or 'unless'
  condition, using the same syntax as the 'otel-event' keyword, makes the stop
  conditional; the filter is left active when the condition is not met.  The
  keyword may be repeated, one line per condition, and the stop fires when the
  condition of any line holds.  The bare form without a condition sets the stop
  definitively, so it must be the last line: any 'otel-stop' that follows it is
  a configuration error, since it could not change anything any more.

  Arguments :
    condition - an optional ACL-based 'if' or 'unless' condition

  Example :
    otel-scope skip_healthchecks
        otel-event on-client-session-start
        otel-stop if { path_beg /health }


idle-timeout <time>
  Set the idle timeout interval for a scope bound to the 'on-idle-timeout'
  event.  The timer fires periodically at the given interval when the stream
  is idle.  This keyword is mandatory for scopes using the 'on-idle-timeout'
  event and cannot be used with any other event, nor with a scope that carries
  no 'otel-event' at all.  Only one 'idle-timeout' may be defined per scope; a
  second is a configuration error.

  The <time> argument accepts the standard HAProxy time format: a number
  followed by a unit suffix (us, ms, s, m, h, d).  A value of zero is not
  permitted.

  Arguments :
    time - the idle timeout interval (e.g. 5s, 500ms, 1m)

  Example :
    scopes on_idle_timeout
    ..
    otel-scope on_idle_timeout
        idle-timeout 5s
        span "heartbeat" root
            attribute "idle.elapsed" str("idle-check")
        instrument cnt_int "idle.count" value int(1)
        log-record info str("heartbeat")
        otel-event on-idle-timeout


span <name> [parent <ref>] [link <ref>] [root] [kind <kind>]
  Creating a new span (or referencing an already opened one).  If a new span is
  created, it can have a parent reference to another span or context, an inline
  link to another span, or carry the 'root' mark.  Without a parent reference
  the span starts a trace of its own whether it carries the mark or not, so
  the two are the same kind of span.  Only one span name may carry the mark: a
  second name marked 'root' is a configuration error, while the same name marked
  in another otel-scope is the usual alternative creation, and the parent of
  a marked span may be an extracted context but never another span.  Leaving
  every span unmarked is not an error, only a warning at load, and only the
  spans of the scopes that run count towards it.  A 'parent' the stream does
  not hold is a runtime error and the span is not created; an inline 'link'
  that does not resolve is left out.  A span may not be called '*', '*req*'
  or '*res*', the wildcard names the 'finish' keyword keeps for whatever the
  stream holds.

  The parent reference is set using the 'parent' keyword followed by the name
  of an existing span or extracted context.  A span cannot be its own parent,
  since the reference is looked up before the span exists.  An inline link is
  set using the 'link' keyword followed by a span or context name.  The span is
  marked as the root span using the 'root' keyword.  The 'kind' keyword sets the
  OTel span kind, one of internal, server (the default), client, producer or
  consumer.

  Each 'span' opens a sub-context within the scope: the 'link', 'attribute',
  'event', 'baggage', 'inject', 'status' and 'exception' keywords that follow
  it apply to that span -- the 'currently active span' -- until the next 'span'
  or the end of the scope.  A later bare 'span <name>' naming an already created
  span makes it the active one again, so further operations can be attached to
  it; repeating the definition with arguments in the same scope is an error.
  Another otel-scope may carry a defining line of the same name, which the
  parser accepts because it cannot know which scope runs first: at run time
  exactly one line creates the span, and a defining line of another scope that
  finds it already created is an error, not a line without effect.

  For example:
    span "HAProxy session" root
    span "Client session" parent "HAProxy session"
    span "HTTP request" parent "TCP request" link "HAProxy session"
    span "Client session" parent "otel_ctx_1"

  A 'span' declaration takes one inline link; a second one on the line is a
  configuration error, like a repeated 'parent', 'root' or 'kind'.  For multiple
  links, use the standalone 'link' keyword described above.

  Arguments :
    name   - the name of the span being created or referenced
             (operation name)
    parent - 'parent <ref>' references an existing span or extracted
             context as the span's parent
    link   - 'link <ref>' adds an inline link to another span or context
    root   - marks the span as the root span; no effect at run time
    kind   - 'kind <kind>' sets the span kind: internal, server (the
             default), client, producer or consumer


status <code> [<sample> ...] [{ if | unless } <condition>]
  This keyword sets the status for the currently active span.  The status
  indicates the outcome of the operation represented by the span.

  The status code is one of the following predefined values:
    - ignore : do not set any status (default)
    - unset  : explicitly mark status as unset
    - ok     : the operation completed successfully
    - error  : the operation resulted in an error

  An optional description can follow the status code, consisting of one or more
  sample expressions whose values are concatenated as a string.  A description
  is meaningful only with the 'error' status, as required by the OpenTelemetry
  specification; the OTLP exporter sets the status message for 'error' and omits
  it for 'ok' and 'unset', so a description on those codes is accepted but not
  exported.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  decides whether the status applies.  The keyword may be given more than once
  per span; the lines are evaluated at runtime in configuration order, and the
  first one whose condition holds sets the span status, the remaining lines
  being skipped.  A line without a condition always matches, so an unconditional
  'status' is the default and comes last, and a line after it is a configuration
  error.  The same first-match scheme also governs 'attribute', 'baggage', the
  variable keywords and the create form of 'instrument'; 'link', 'exception',
  'log-record' and an 'instrument' update apply every line whose condition holds
  instead, and so does 'event', though the value of a repeated (name, key) pair
  is first-match again.

  For example:
    status "ok"
    status "error" str("http.status_code: ") status
    status "error" str("server error") if { status ge 500 }

  Arguments :
    code      - the status code (ignore, unset, ok, error)
    sample    - optional sample expression(s) for the status description
    condition - an optional ACL-based 'if' or 'unless' condition


exception <type> [message <sample> ...] [attr <key> <sample> ...] [{ if | unless } <condition>]
  This keyword records an exception on the currently active span.  It is emitted
  via the OpenTelemetry record_exception() operation as a span event named
  'exception' that follows the semantic conventions.  HAProxy has no call stack,
  so 'exception.stacktrace' is never set.

  The first argument is the exception type, a literal string that is stored as
  the 'exception.type' attribute.  The optional 'message' keyword introduces
  sample expressions whose values are concatenated into 'exception.message'.
  The repeatable 'attr <key> <sample>' clause adds further attributes to the
  event.  A second 'message' clause is a configuration error, as is a repeated
  key.

  Recording an exception does not change the span status, so pair it with a
  'status error' line when the exception marks the span as failed.  An optional
  trailing 'if'/'unless' condition (same syntax as 'otel-event') covers the
  record.  The keyword may be repeated to record several exceptions on one span.
  The lines are independent: every line whose condition holds records its own
  exception event, the same type included.

  For example:
    exception upstream_error message str("HTTP ") status
    exception bad_gateway attr "http.route" path if { status ge 500 }

  Arguments :
    type      - the exception type, stored as 'exception.type'
    message   - sample expression(s) concatenated into 'exception.message'
    key       - the attribute key for an 'attr' clause
    sample    - sample expression(s) for the message or an attribute value
    condition - an optional ACL-based 'if' or 'unless' condition


set-var <var-name> <sample> ... [{ if | unless } <condition>]
  Set a HAProxy variable to the string value of a sample expression or a
  log-format, evaluated when the scope's event fires.  The value grammar matches
  the 'attribute' keyword: bare sample fetches are concatenated and a '%[ ... ]'
  wrapper marks a HAProxy log-format string.  The result is always stored as a
  string.  The variable name includes its scope (proc, sess, txn, req or res)
  and is registered at parse time.  Assignments run before the scope's spans, so
  span attributes, instruments and log records can read the new value through
  'var(...)'.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  decides whether the variable is set.  The keyword may be repeated, one line
  per condition: the lines of one variable are tried in configuration order,
  the first whose condition holds sets that variable, while a line without a
  condition, placed last, acts as the variable's default.

  For example:
    set-var txn.otel_method method
    set-var txn.otel_path str("path=") path
    set-var sess.otel_tid var(sess.otel.uuid)
    set-var txn.otel_internal str("1") if { src 127.0.0.1 }

  Arguments :
    var-name  - the HAProxy variable name (including its scope)
    sample    - sample expression(s) or log-format for the value
    condition - an optional ACL-based 'if' or 'unless' condition


set-var-ctx <var-name> <ref> <field> [{ if | unless } <condition>]
  Set a HAProxy variable from a field of a referenced span or span context,
  evaluated when the scope's event fires.  <ref> names a span or a context
  that some otel-scope defines, this one or another; it is resolved against
  the active spans first and then the extracted contexts, and an unresolved
  reference leaves the variable unchanged.  Because these assignments run after
  the scope's spans, a span created in the same scope may be referenced.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  decides whether the variable is set.  The keyword may be repeated, one line
  per condition: the lines of one variable are tried in configuration order,
  the first whose condition holds sets that variable, while a line without a
  condition, placed last, acts as the variable's default.

  Supported fields:
    trace-id          - trace id, lowercase hex
    span-id           - span id, lowercase hex
    trace-flags       - W3C trace-flags byte, lowercase hex
    traceparent       - full W3C "00-<trace>-<span>-<flags>" string
    tracestate        - full tracestate header (context references only)
    tracestate(<key>) - a single tracestate entry (context references only)
    baggage           - the whole baggage carrier (all entries)
    baggage(<key>)    - a single baggage entry value
    sampled           - "1" if the sampled flag is set, else "0"
    valid             - "1" if the context is valid, else "0"
    remote            - "1" if the context is remote, else "0"

  For 'baggage(<key>)', a span reference reads the value set on that span, while
  a context reference reads the named entry from the inbound carrier.  Keyless
  'baggage' returns the whole carrier instead: the inbound carrier for a context
  reference, or the serialized outbound baggage for a span reference.  The
  'tracestate' fields are available only when <ref> names a context.

  A key written between the parentheses follows the rule of its W3C carrier,
  and the two differ.  A baggage key is an HTTP token: letters, digits and the
  punctuation "!#$%&'*+-.^_`|~", at most 4096 characters of them.  A tracestate
  key holds at most 256 lower-case letters, digits, '_', '-', '*' and '/' and
  begins with a letter, or names a tenant id of at most 241 characters and a
  system id of at most 14 around a single '@', the tenant id opening with a
  letter or a digit and the system id with a letter.  No carrier holds another
  key, so one written here is refused.

  For example:
    extract "inbound" use-headers
    span "HAProxy session" root
    set-var-ctx txn.in_traceid  "inbound" trace-id
    set-var-ctx txn.traceparent "HAProxy session" traceparent
    set-var-ctx txn.user        "inbound" baggage(userId)
    set-var-ctx txn.local_tid   "HAProxy session" trace-id if { src 127.0.0.1 }

  Arguments :
    var-name  - the HAProxy variable name (including its scope)
    ref       - the referenced span or context name
    field     - the span/context field to read (see list above)
    condition - an optional ACL-based 'if' or 'unless' condition


unset-var <var-name> ... [{ if | unless } <condition>]
  Remove one or more HAProxy variables when the scope's event fires.  A variable
  that is not currently set is silently ignored.  Removal runs after the scope's
  spans, instruments and log records, so all of them still read the variables;
  only 'finish' and 'otel-stop' run afterwards.

  An optional trailing 'if'/'unless' condition (same syntax as 'otel-event')
  covers the directive; when it does not pass, none of the listed variables
  are removed.  The keyword may be repeated within one scope, with one line per
  condition: the lines of one variable are tried in configuration order, the
  first whose condition holds removes it, and the line without a condition,
  placed last, acts as the variable's default.  A variable named twice on one
  line is a configuration error.

  For example:
    unset-var txn.otel_scratch txn.otel_tmp
    unset-var txn.otel_scratch if { src 127.0.0.1 }

  Arguments :
    var-name  - the HAProxy variable name(s) to remove (including scope)
    condition - an optional ACL-based 'if' or 'unless' condition


4.4. "otel-group" section
--------------------------

This section defines a group of OTel scopes that is not triggered by an event
but by HAProxy rules in the configuration file -- specifically 'http-request',
'http-response', 'http-after-response', 'tcp-request content' and 'tcp-response
content'.

The action keyword used in these rules is 'otel-group', and it takes the filter
id and the group name as arguments:

  http-response otel-group <filter-id> <group-name> [{ if | unless } ...]

Both names are resolved when the configuration is checked: the filter id has
to name an OTel filter of that proxy and the group has to be named on a 'groups'
line of the otel-instrumentation section that filter reads, or HAProxy refuses
to start.  A group section those lines leave out is an error on its own.  The
scopes of the group are checked against that rule set as well: a sample fetch or
a condition that cannot be evaluated there is reported as a warning at load.


otel-group <name>
  Creates a new OTel group definition named <name>.  The name is unique within
  the top-level OTel scope, so a second group of that name is a configuration
  error, while another top-level scope may reuse it.  The section must hold at
  least one 'scopes' line; a group with none is an error.  A 'groups' line of
  the otel-instrumentation section must name the group; one those lines leave
  out is an error as well.

  Arguments :
    name - the name of the OTel group


  The following keywords are supported in this section:
    - scopes


scopes <name> ...
  'otel-scope' sections that are part of the specified group are defined.  If
  the mentioned 'otel-scope' sections are used only in some OTel group, they
  do not have to have defined events.  Several 'otel-scope' sections can be
  specified in one line and the keyword may be repeated.  Every name must be
  a defined section and is unique in the group; a repeat is a configuration
  error.  A trailing 'if'/'unless' condition is not accepted.

  Arguments :
    name - the name of the 'otel-scope' section


In addition to the 'otel-group' action, the filter registers a sample fetch
usable in these rules and in any ACL:

  otel.context(<name>) : boolean

It returns true when the span context named <name> was extracted on the current
stream with the 'extract' keyword and is valid.  When none was extracted, the
fetch produces no sample, so an ACL using '-m found' reads as false, letting
a configuration act directly on whether an upstream context was propagated,
without first copying a context field into a variable with 'set-var-ctx'.

  Arguments :
    name - the name of the span context to check

The filter also exposes the raw payload byte counts accumulated on the stream:

  otel.bytes_in  : integer
  otel.bytes_out : integer

'otel.bytes_in' counts the payload on the request channel (client to HAProxy)
and 'otel.bytes_out' the payload on the response channel (HAProxy to client).
Both grow as data is forwarded, so they are best read from a close scope --
on-client-session-end or on-server-session-end -- where they hold the final
connection totals.  They are populated in TCP mode; on an HTTP proxy they read
0, because an HTX buffer carries protocol framing rather than raw payload.


5. Examples
------------

Several examples of the OTel filter configuration can be found in the test
directory.  A brief description of the prepared configurations follows:

cmp   - a configuration made for comparison purposes with other tracing
        implementations.

sa    - a standalone configuration that exercises most of the events (their
        exact coverage is listed in the test/README-sa file).  It demonstrates
        spans, attributes, events, links, baggage, status and other features,
        and serves as the reference for the speed tests.

full  - a configuration that extends 'sa' with the remaining lifecycle events;
        it covers every filter event except 'on-http-tarpit-request'.

ctx   - a configuration similar to 'sa', with the difference that the spans are
        opened using extracted span contexts as references instead of direct
        parent span names.  This demonstrates the inject/extract context
        propagation mechanism using HAProxy variables.

fe be - a more complex example of the OTel filter configuration that uses two
        cascaded HAProxy services (frontend and backend).  The span context
        between HAProxy processes is transmitted via the HTTP header using
        inject/extract.

tcp   - a TCP-mode (non-HTTP) configuration that traces the raw connection with
        a single session span, recording one event per TCP hook and counting the
        forwarded payload via the 'otel.bytes_in' and 'otel.bytes_out' sample
        fetches.  It exercises every event that fires on a TCP-mode proxy.

updown - an up-down counter configuration in which a 'udcnt_int' instrument
         tracks the number of active client sessions: a signed delta carried
         through a session variable raises the counter by one when a client
         session starts and lowers it by one when the session ends.  A second
         counter tallies the total sessions started, and the metrics are read
         from an ostream exporter file after a clean shutdown.

err   - a configuration that exercises the runtime error-reporting path: its
        response scope parents a span under an 'orphan span' created only by the
        on-server-unavailable scope, so span creation fails on every response
        and the rate-limited error and warning logging runs on ordinary traffic.
        The counters are read with 'flt-otel status'.

empty - an empty configuration in which the OTel filter is initialized but no
        event is triggered.  It is not very usable, except to check the behavior
        of the OTel filter in the case of a similar configuration.


The OTel filter does not use tracer plugins.  Instead, telemetry data is
exported using the OpenTelemetry protocol (OTLP) directly to any compatible
backend.  The backend is configured through the YAML configuration file
specified by the 'config' keyword.

In order to be able to collect and view trace data we need an OpenTelemetry
compatible backend.  There are many options available, including:

  - Jaeger  : https://www.jaegertracing.io/
  - Grafana : https://grafana.com/oss/tempo/
  - Zipkin  : https://zipkin.io/
  - SigNoz  : https://signoz.io/
  - Datadog : https://www.datadoghq.com/

For quick testing, a simple setup using the OpenTelemetry Collector and Jaeger
can be started with Docker:

  # docker run -d --name jaeger -p 4317:4317 -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest

This starts Jaeger with OTLP/gRPC on port 4317, OTLP/HTTP on port 4318 and the
web UI on port 16686.  If we want to use that container later, it can be started
and stopped using the 'docker container start/stop' commands.

The test configurations use a YAML file that defines an OTLP/HTTP exporter
sending data to localhost:4318.  A typical minimal YAML configuration looks like
this:

  --- otel.yml --------------------------------------------------------
  exporters:
    my_exporter:
      type:     otlp_http
      endpoint: "http://localhost:4318/v1/traces"

  samplers:
    my_sampler:
      type: always_on

  processors:
    my_processor:
      type: batch

  providers:
    my_provider:
      resources:
        - service.name: "haproxy"

  signals:
    traces:
      default:
        scope_name: "HAProxy OTel"
        exporters:  my_exporter
        samplers:   my_sampler
        processors: my_processor
        providers:  my_provider
  ---------------------------------------------------------------------

In order to use any of the configurations from the test directory, we can run
one of the pre-configured scripts (from within that directory, because they
resolve the HAProxy binary and their configuration files relative to it):

  % ./run-sa.sh
  % ./run-full.sh
  % ./run-ctx.sh
  % ./run-cmp.sh
  % ./run-fe-be.sh
  % ./run-tcp.sh
  % ./run-updown.sh
  % ./run-err.sh
  % ./run-empty.sh
  % ./run-optctx.sh

The parser test runs the same way but starts no proxy.  It checks a series of
generated configuration files with 'haproxy -c' against the keyword rules of
section 4.5 of README-implementation and reports the outcome of every case;
further cases cover the arguments a keyword accepts and the checks made once
the whole configuration is known.  See test/README-parser.

  % ./run-parser.sh


5.1. Benchmarking results
--------------------------

To check the performance impact of the OTel filter, test configurations located
in the test directory have been benchmarked.  The test results (with the names
README-speed-xxx, where xxx is the name of the configuration being tested) are
also in the test directory.

Testing was done with the wrk utility using 8 threads, 256 connections and a
5-minute test duration, with HAProxy's haterm dummy HTTP server acting as the
origin server.  Detailed results and methodology are documented in the file
test/README-test-speed.

Below is a summary of the 'sa' (standalone) configuration results, the heaviest
of the benchmarked scenarios and thus the worst-case figures:

  ---------------------------------------------------------------
   rate-limit     req/s     avg latency     overhead
  ---------------------------------------------------------------
      100.0%     68,686         3.73 ms        77.3%
       50.0%    107,213         2.44 ms        64.5%
       25.0%    149,747         1.84 ms        50.4%
       10.0%    205,090         1.42 ms        32.1%
        2.5%    266,251         1.01 ms        11.9%
     disabled   298,250         0.88 ms         1.3%
     off        302,051         0.87 ms     baseline
  ---------------------------------------------------------------

The 'off' baseline is measured with the 'filter opentelemetry' directive
commented out, so the filter is not loaded at all.  The 'disabled' level has
the filter loaded but disabled via 'option disabled', so the initialization
overhead is included but no events are processed.

As the table shows, the overhead is substantial under full-load tracing and
falls with the rate limit: about 50% at a 25% rate limit, 32% at 10% and 12%
at 2.5%.  These are saturation figures measured at the capacity ceiling of the
machine; a deployment running below that ceiling mainly pays the per-request
latency cost instead.

The sa configuration is itself a deliberately signal-heavy stress test; any
realistic production configuration produces far less telemetry per stream and
shows a correspondingly smaller overhead at any rate limit.


6. OTel CLI
------------

Via the HAProxy CLI interface we can find out the current status of the OTel
filter and change several of its settings.

All supported CLI commands can be found in the following way, using the socat
utility with the assumption that the HAProxy CLI socket path is set to
/tmp/haproxy.sock (of course, instead of socat, nc or other utility can be used
with a change in arguments when running the same):

  % echo "help" | socat - UNIX-CONNECT:/tmp/haproxy.sock | grep flt-otel
  --- command output ----------
  flt-otel debug [level]                  : set the OTEL filter debug level (default: get current debug level)
  flt-otel disable [@<filter>]            : disable the OTEL filter
  flt-otel enable [@<filter>]             : enable the OTEL filter
  flt-otel soft-errors [@<filter>]        : disable hard-errors mode
  flt-otel hard-errors [@<filter>]        : enable hard-errors mode
  flt-otel reset-errors [@<filter>]       : reset runtime-error counters
  flt-otel logging [@<filter>] [state]    : set logging state (default: get current logging state)
  flt-otel noflush [@<filter>] [state]    : set noflush mode (default: get current noflush mode)
  flt-otel rate [@<filter>] [value]       : set the rate limit (default: get current rate value)
  flt-otel status [@<filter>]             : show the OTEL filter status
  flt-otel flush [@<filter>]              : force-export buffered telemetry now
  flt-otel instruments [@<filter>]        : show configured metric instruments
  flt-otel scopes [@<filter>]             : show configured scopes and groups
  --- command output ----------

'flt-otel debug' can only be used in case the OTel filter is compiled with the
debug mode enabled.  When invoked without arguments, these commands display the
current value of the respective setting.

By default, the commands operate on all configured OTel filter instances at
once.  Every command except 'debug', whose setting is process-global, takes
an optional "@<filter>" token right after the subcommand keyword that limits
it to the single instance whose filter id matches <filter>; the id is the one
set with the 'id' keyword on the filter line, and it is unique across all the
proxies ("otel-filter" by default).  For 'logging' and 'rate', the state or
value argument then follows the token, mirroring the "[@<ver>]" selector of
the HAProxy 'show map' family of commands.  A surplus argument, like a target
given without its leading '@', and a token that names an unknown filter are
both rejected:

  % echo "flt-otel rate @my-filter 25.5" | socat - UNIX-CONNECT:/tmp/haproxy.sock
  --- command output ----------
  flt-otel : filter my-filter rate limit set to 25.50
  --- command output ----------

The 'flt-otel status' command prints a report for each configured filter
instance.  A typical response looks like this (the 'debug level' line and
the 'counters' block appear only in a debug build):

  % echo "flt-otel status" | socat - UNIX-CONNECT:/tmp/haproxy.sock
  --- command output ----------
   opentelemetry filter status
  ------------------------------------------------------------------------------
     library:         C++ 1.26.0, C wrapper 3.4.0-1028
     debug level:     0x077f
     export pipeline:
       signal   queued  dropped  exported  failed  last export
       traces   0/2048        0   2400/12     0/0  120 ms
       logs     0/2048        0     300/3     0/0  1500 ms
       metrics  -             -       -/5     -/0  9800 ms
     sdk diagnostics: 0

     proxy fe_main, filter otel
       configuration: otel.cfg
       groups/scopes: 2/29

       instrumentation otel-instr
         configuration: otel.yml
         tracer:        active
         meter:         active
         logger:        active
         rate limit:    100.00 %
         hard errors:   no
         disabled:      no
         require ctx:   no
         noflush:       no
         logging:       enabled
         runtime err:   hard 0, soft 0
         suppressed:    pending 0, total 0
         idle timeout:  1000 ms
         analyzers:     01e03fbe

       counters:
         attached: run 0, rate-limit 0, disabled 0, error 0
         disabled: scope 0, hard-error 0
  --- command output ----------

Most lines are self-explanatory.  Per signal, the 'export pipeline' table shows
the batch-processor state: 'queued' is the live queue depth over the configured
capacity, 'dropped' the records dropped because the queue was full, 'exported'
and 'failed' the Export() outcomes as records/calls -- the records moved and the
Export() calls that carried them -- and 'last export' the age of the most recent
successful export.  The call counts reflect the Export() return value: a real
delivery result for synchronous exporters such as ostream and otlp_file, but
only acceptance for the async OTLP/HTTP and OTLP/gRPC exporters, which hand the
batch to a background client and report a connection failure only later.  Those
failures instead appear in 'sdk diagnostics', the internal-message count the SDK
reported; for an OTLP exporter a rising 'sdk diagnostics' with a stale 'last
export' is the signature of an unreachable collector.  Metrics use a periodic
reader rather than a queue, so their 'queued' and 'dropped' read '-', and the
records side of 'exported' and 'failed' reads '-' too because the reader has no
per-record count; the call counts and 'last export' track the periodic push,
which runs on its configured interval whether or not any instrument recorded
data, so the 'exported' count climbs even when no instruments are configured.

'idle timeout' is the smallest scope idle timeout in milliseconds (0 means
off), and 'analyzers' is the channel analyzer bitmask the filter registered.
The 'counters' block tallies how streams attached -- run, rate-limited,
disabled or errored -- and how many later had tracing disabled, by a scope
decision ('otel-stop' or the 'require-context' option) or by a hard error.

'runtime err' reports the data-path errors the filter met while processing
streams: 'hard' counts the hard-error episodes that disabled the filter for a
stream, 'soft' the errors that were swallowed so processing could continue.
The 'suppressed' line below it tracks the log lines the rate limiter held back:
'pending' counts the lines held back since the last emitted line, which the next
emission reports as a "(N more suppressed)" suffix and zeros, while 'total' is
the lifetime count and it only ever climbs.  Both error kinds are sent to the
per-instance 'log' target -- hard errors at level err, soft errors at level
warning -- but each emission is edge-triggered and rate-limited per instance,
so a fault recurring on every stream yields only a couple of lines per window
while these counters keep the true tally.  'option dontlog-normal' keeps the
error lines but drops the warnings, and 'flt-otel logging off' silences both;
the counters climb either way.  An error text may quote a sample value, so its
non-printable bytes reach the log escaped in the '#XX' form and its length is
bounded.

'flt-otel flush' forces every filter instance to export its buffered telemetry
at once, calling force_flush on the active tracer, meter and logger.  All of
these calls share a single five-second budget, so an exporter that does not
respond cannot stall the command for the sum of per-call timeouts; a call that
finds the budget spent is skipped, and its data stays buffered for the regular
periodic export.  It requires admin access.

'flt-otel noflush' arms or disarms the noflush mode at runtime, with the same
meaning as the 'option noflush' configuration keyword: when a collector is known
to be down before a planned stop or reload, 'flt-otel noflush on' makes the
teardown drop the buffered telemetry at once instead of waiting for the export
timeouts, without any configuration change.  Without an argument the command
reports the current mode.  Setting it requires admin access.

'flt-otel reset-errors' clears the 'runtime err' counters and the runtime-log
rate-limiter state for every filter instance, so a fresh fault is reported again
instead of folding into the running suppressed tally; only the lifetime 'total'
is retained.  It requires admin access.

'flt-otel instruments' lists the metric instruments declared in each scope.  A
create-form instrument is shown with its type, unit and creation state -- 'not
created' before first use, 'created (idx N)' once the meter has built it (which
happens lazily, on the first record), 'pending' while another thread builds it,
or 'failed' when the creation kept failing and was given up.  An update-form
instrument has no unit or creation state of its own, so only its name and the
'update' type are shown:

  % echo "flt-otel instruments" | socat - UNIX-CONNECT:/tmp/haproxy.sock
  --- command output ----------
   opentelemetry filter instruments
  ------------------------------------------------------------------------------

     proxy otel-test-full-frontend, filter otel-test-full
       scope                  instrument               type       unit          state
       on_stream_start        haproxy.sessions.active  udcnt_int  {session}     created (idx 0)
       on_stream_start        haproxy.fe.connections   gauge_int  {connection}  created (idx 1)
       [...]
       frontend_http_request  haproxy.http.latency     update
  --- command output ----------

A filter that has no instruments configured is shown as '(no instruments)'.

'flt-otel scopes' lists the configured scopes, each with its bound event and
whether the configuration uses it, then the groups with their scope count.  A
debug build appends an 'events' section that reports how often each event was
dispatched; events not dispatched so far are left out:

  % echo "flt-otel scopes" | socat - UNIX-CONNECT:/tmp/haproxy.sock
  --- command output ----------
   opentelemetry filter scopes
  ------------------------------------------------------------------------------

     proxy otel-test-full-frontend, filter otel-test-full
       scopes:
         scope            event            used
         on_stream_start  on-stream-start  yes
         [...]
       groups:
         group                      scopes
         http_response_group        2
         http_after_response_group  1
       events:
         event            dispatched
         on-stream-start  3
         [...]
  --- command output ----------

Each section is shown as '(no scopes)', '(no groups)' or '(no events)' when it
has no rows.


7. Known bugs and limitations
-------------------------------

The name of the span context definition can contain only letters, numbers and
the characters '_', '.' and '-'.  Also, all uppercase letters in the name are
converted to lowercase.  The character '-' is converted internally to the 'D'
character; since a HAProxy variable is generated from that name, this should
be taken into account if we want to use the variable somewhere in the HAProxy
configuration.  The above mentioned span context is used in the 'inject' and
'extract' keywords.

An inline span link (using the 'link' keyword within a 'span' declaration) is
given once per span declaration: a second inline link on the same line is a
configuration error.  For multiple links, use the standalone 'link' keyword
instead.

Let's look a little at the example test/fe-be (configurations are in the
test/fe and test/be directories, 'fe' is here the abbreviation for frontend and
'be' for backend).  In case we have the 'rate-limit' set to a value less than
100.0, then distributed tracing will not be started with each new HTTP request.
It also means that the span context will not be delivered (via the HTTP header)
to the backend HAProxy process.  Set 'option require-context' on the backend,
leaving its 'rate-limit' at 100.0, so that it continues every trace propagated
to it and produces no telemetry for a request that carries none, rather than
starting a new, disconnected trace.