[DSM] Add technology pages for DSM documentation

config/_default/menus/main.en.yaml

@@ -4355,26 +4355,31 @@ menu:
       identifier: data_streams
       parent: apm_heading
       weight: 50000
+    - name: Setup
+      url: data_streams/setup
+      identifier: data_streams_setup
+      parent: data_streams
+      weight: 1
     - name: Schema Tracking
       url: data_streams/schema_tracking
       identifier: data_streams_schema_tracking
       parent: data_streams
-      weight: 1
+      weight: 2
     - name: Live Messages
       url: data_streams/live_messages
       identifier: data_streams_live_messages
       parent: data_streams
-      weight: 2
+      weight: 3
     - name: Data Pipeline Lineage
       url: data_streams/data_pipeline_lineage
       identifier: data_streams_pipeline_lineage
       parent: data_streams
-      weight: 3
+      weight: 4
     - name: Guide
       url: data_streams/guide
       identifier: data_streams_guide
       parent: data_streams
-      weight: 4
+      weight: 5
     - name: Data Jobs Monitoring
       url: data_jobs/
       pre: data-jobs-monitoring

content/en/data_streams/_index.md

@@ -43,14 +43,34 @@ Data Streams Monitoring provides a standardized method for teams to understand a
 * Pinpoint faulty producers, consumers or queues, then pivot to related logs or clusters to troubleshoot faster.
 * Prevent cascading delays by equipping service owners to stop backed up events from overwhelming downstream services.
 
-## Setup
+### Supported languages and technologies
 
 Data Streams Monitoring instruments Kafka _clients_ (consumers/producers). If you can instrument your client infrastructure, you can use Data Streams Monitoring.
 
-For installation instructions and lists of supported technologies, choose your language:
+|   | Java | Python | .NET | Node.js | Go |
+| - | ---- | ------ | ---- | ------- | -- |
+| Apache Kafka | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} |
+| Amazon Kinesis | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | |
+| Amazon SNS | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | |
+| Amazon SQS | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | |
+| Azure Service Bus | | | {{< X >}} | | |
+| Google Pub/Sub | {{< X >}} | | | {{< X >}} | |
+| IBM MQ | | | {{< X >}} | | |
+| RabbitMQ | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | |
+
+Data Streams Monitoring requires minimum Datadog tracer versions. See each setup page for details.
+
+## Setup
+
+### By language
 
 {{< partial name="data_streams/setup-languages.html" >}}
 
+
+### By technology
+
+{{< partial name="data_streams/setup-technologies.html" >}}
+
 <br/>
 
 ## Explore Data Streams Monitoring
@@ -86,7 +106,7 @@ Alternatively, click a service to open a detailed side panel and view the **Path
 
 Slowdowns caused by high consumer lag or stale messages can lead to cascading failures and increase downtime. With out-of-the-box alerts, you can pinpoint where bottlenecks occur in your pipelines and respond to them right away. For supplementary metrics, Datadog provides additional integrations for message queue technologies like [Kafka][4] and [SQS][5].
 
-Through Data Stream Monitoring's out-of-the-box monitor templates, you can setup monitors on metrics like consumer lag, throughput, and latency in one click. 
+Through Data Stream Monitoring's out-of-the-box monitor templates, you can setup monitors on metrics like consumer lag, throughput, and latency in one click.
 
 {{< img src="data_streams/add_monitors_and_synthetic_tests.png" alt="Datadog Data Streams Monitoring Monitor Templates" style="width:100%;" caption="Click 'Add Monitors and Synthetic Tests' to view monitor templates" >}}
 
@@ -98,7 +118,7 @@ Click on the **Throughput** tab on any service or queue in Data Streams Monitori
 
 By filtering to a single Kafka, RabbitMQ, or Amazon SQS cluster, you can detect changes in incoming or outgoing traffic for all detected topics or queues running on that cluster:
 
-### Quickly pivot to identify root causes in infrastructure, logs, or traces 
+### Quickly pivot to identify root causes in infrastructure, logs, or traces
 
 Datadog automatically links the infrastructure powering your services and related logs through [Unified Service Tagging][3], so you can easily localize bottlenecks. Click the **Infra**, **Logs** or **Traces** tabs to further troubleshoot why pathway latency or consumer lag has increased.
 

content/en/data_streams/setup/_index.md

@@ -0,0 +1,12 @@
+---
+title: Setup Data Streams Monitoring
+---
+
+### By language
+
+{{< partial name="data_streams/setup-languages.html" >}}
+
+
+### By technology
+
+{{< partial name="data_streams/setup-technologies.html" >}}

content/en/data_streams/dotnet.md → content/en/data_streams/setup/language/dotnet.md

@@ -4,6 +4,8 @@ further_reading:
   - link: 'https://www.datadoghq.com/blog/confluent-connector-dsm-autodiscovery/'
     tag: 'Blog'
     text: 'Autodiscover Confluent Cloud connectors and easily monitor performance in Data Streams Monitoring'
+aliases:
+  - /data_streams/dotnet
 ---
 
 ### Prerequisites
@@ -32,35 +34,26 @@ environment:
   - DD_DATA_STREAMS_ENABLED: "true"
 ```
 
-### Monitoring Kafka Pipelines
-Data Streams Monitoring uses message headers to propagate context through Kafka streams. If `log.message.format.version` is set in the Kafka broker configuration, it must be set to `0.11.0.0` or higher. Data Streams Monitoring is not supported for versions lower than this.
+{{% data_streams/monitoring-kafka-pipelines %}}
 
-### Monitoring SQS pipelines
-Data Streams Monitoring uses one [message attribute][2] to track a message's path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.
+{{% data_streams/monitoring-sqs-pipelines %}}
 
-{{% data-streams-monitoring/monitoring-rabbitmq-pipelines %}}
+{{% data_streams/monitoring-rabbitmq-pipelines %}}
 
-### Monitoring SNS-to-SQS pipelines
-To monitor a data pipeline where Amazon SNS talks directly to Amazon SQS, you must enable [Amazon SNS raw message delivery][12].
+{{% data_streams/monitoring-sns-to-sqs-pipelines %}}
 
-### Monitoring Azure Service Bus
-
-Setting up Data Streams Monitoring for Azure Service Bus applications requires additional configuration for the instrumented application.
-
-1. Either set the environment variable `AZURE_EXPERIMENTAL_ENABLE_ACTIVITY_SOURCE` to `true`, or in your application code set the `Azure.Experimental.EnableActivitySource` context switch to `true`. This instructs the Azure Service Bus library to generate tracing information. See [Azure SDK documentation][11] for more details.
-2. Set the `DD_TRACE_OTEL_ENABLED` environment variable to `true`. This instructs the .NET auto-instrumentation to listen to the tracing information generated by the Azure Service Bus Library and enables the inject and extract operations required for Data Streams Monitoring.
+{{% data_streams/monitoring-azure-service-bus %}}
 
 ### Monitoring connectors
 
 #### Confluent Cloud connectors
-{{% dsm_confluent_connectors %}}
+{{% data_streams/dsm-confluent-connectors %}}
 
 ## Further reading
 
 {{< partial name="whats-next/whats-next.html" >}}
 
 [1]: /agent
-[2]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html
 [3]: https://www.nuget.org/packages/Confluent.Kafka
 [4]: https://www.nuget.org/packages/RabbitMQ.Client
 [5]: https://www.nuget.org/packages/AWSSDK.SQS
@@ -70,4 +63,3 @@ Setting up Data Streams Monitoring for Azure Service Bus applications requires a
 [9]: #monitoring-azure-service-bus
 [10]: https://www.nuget.org/packages/Azure.Messaging.ServiceBus
 [11]: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/samples/Diagnostics.md#enabling-experimental-tracing-features
-[12]: https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html

content/en/data_streams/go.md → content/en/data_streams/setup/language/go.md

@@ -4,6 +4,8 @@ further_reading:
     - link: 'https://www.datadoghq.com/blog/confluent-connector-dsm-autodiscovery/'
       tag: 'Blog'
       text: 'Autodiscover Confluent Cloud connectors and easily monitor performance in Data Streams Monitoring'
+aliases:
+  - /data_streams/go
 ---
 
 The following instrumentation types are available:
@@ -27,10 +29,9 @@ To start with Data Streams Monitoring, you need recent versions of the Datadog A
 
 ### Installation
 
-### Monitoring Kafka Pipelines
-Data Streams Monitoring uses message headers to propagate context through Kafka streams. If `log.message.format.version` is set in the Kafka broker configuration, it must be set to `0.11.0.0` or higher. Data Streams Monitoring is not supported for versions lower than this.
+{{% data_streams/monitoring-kafka-pipelines %}}
 
-{{% data-streams-monitoring/monitoring-rabbitmq-pipelines %}}
+{{% data_streams/monitoring-rabbitmq-pipelines %}}
 
 #### Automatic Instrumentation
 
@@ -128,15 +129,15 @@ if ok {
 ### Monitoring connectors
 
 #### Confluent Cloud connectors
-{{% dsm_confluent_connectors %}}
+{{% data_streams/dsm-confluent-connectors %}}
 
 ## Further reading
 
 {{< partial name="whats-next/whats-next.html" >}}
 
 [1]: /agent/
 [2]: https://github.com/DataDog/dd-trace-go
-[3]: https://docs.datadoghq.com/tracing/trace_collection/library_config/go/
+[3]: /tracing/trace_collection/library_config/go/
 [4]: https://datadoghq.dev/orchestrion/
 [5]: https://datadoghq.dev/orchestrion/docs/getting-started/
 [6]: https://github.com/DataDog/dd-trace-go/blob/main/datastreams/propagation.go#L37

content/en/data_streams/java.md → content/en/data_streams/setup/language/java.md

@@ -10,6 +10,8 @@ further_reading:
     - link: 'https://www.datadoghq.com/blog/confluent-connector-dsm-autodiscovery/'
       tag: 'Blog'
       text: 'Autodiscover Confluent Cloud connectors and easily monitor performance in Data Streams Monitoring'
+aliases:
+  - /data_streams/java
 ---
 
 ### Prerequisites
@@ -70,10 +72,9 @@ Use Datadog's Java tracer, [`dd-trace-java`][6], to collect information from you
 1. [Add the `dd-java-agent.jar` file][7] to your Kafka Connect workers. Ensure that you are using `dd-trace-java` [v1.44+][8].
 1. Modify your Java options to include the Datadog Java tracer on your worker nodes. For example, on Strimzi, modify `STRIMZI_JAVA_OPTS` to add `-javaagent:/path/to/dd-java-agent.jar`.
 
-### Monitoring SQS pipelines
-Data Streams Monitoring uses one [message attribute][3] to track a message's path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.
+{{% data_streams/monitoring-sqs-pipelines %}}
 
-{{% data-streams-monitoring/monitoring-rabbitmq-pipelines %}}
+{{% data_streams/monitoring-rabbitmq-pipelines %}}
 
 ### Monitoring SNS-to-SQS pipelines
 To monitor a data pipeline where Amazon SNS talks directly to Amazon SQS, you must perform the following additional configuration steps:
@@ -100,16 +101,15 @@ Enable [Amazon SNS raw message delivery][1].
 {{% /tab %}}
 {{< /tabs >}}
 
-### Monitoring Kinesis pipelines
-There are no message attributes in Kinesis to propagate context and track a message's full path through a Kinesis stream. As a result, Data Streams Monitoring's end-to-end latency metrics are approximated based on summing latency on segments of a message's path, from the producing service through a Kinesis Stream, to a consumer service. Throughput metrics are based on segments from the producing service through a Kinesis Stream, to the consumer service. The full topology of data streams can still be visualized through instrumenting services.
+{{% data_streams/monitoring-kinesis-pipelines %}}
 
 ### Manual instrumentation
 Data Streams Monitoring propagates context through message headers. If you are using a message queue technology that is not supported by DSM, a technology without headers (such as Kinesis), or Lambdas, use [manual instrumentation to set up DSM][5].
 
 ### Monitoring connectors
 
 #### Confluent Cloud connectors
-{{% dsm_confluent_connectors %}}
+{{% data_streams/dsm-confluent-connectors %}}
 
 #### Self-hosted Kafka connectors
 
@@ -123,7 +123,6 @@ Data Streams Monitoring can collect information from your self-hosted Kafka conn
 
 [1]: /agent
 [2]: /tracing/trace_collection/dd_libraries/java/
-[3]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html
 [4]: /agent/remote_config/?tab=configurationyamlfile#enabling-remote-configuration
 [5]: /data_streams/manual_instrumentation/?tab=java
 [6]: https://github.com/DataDog/dd-trace-java

content/en/data_streams/nodejs.md → content/en/data_streams/setup/language/nodejs.md

@@ -10,6 +10,8 @@ further_reading:
     - link: 'https://www.datadoghq.com/blog/confluent-connector-dsm-autodiscovery/'
       tag: 'Blog'
       text: 'Autodiscover Confluent Cloud connectors and easily monitor performance in Data Streams Monitoring'
+aliases:
+  - /data_streams/nodejs
 ---
 ### Prerequisites
 
@@ -36,27 +38,23 @@ environment:
   - DD_DATA_STREAMS_ENABLED: "true"
 ```
 
-### Monitoring Kafka Pipelines
-Data Streams Monitoring uses message headers to propagate context through Kafka streams. If `log.message.format.version` is set in the Kafka broker configuration, it must be set to `0.11.0.0` or higher. Data Streams Monitoring is not supported for versions lower than this.
+{{% data_streams/monitoring-kafka-pipelines %}}
 
-### Monitoring SQS pipelines
-Data Streams Monitoring uses one [message attribute][4] to track a message's path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.
+{{% data_streams/monitoring-sqs-pipelines %}}
 
-{{% data-streams-monitoring/monitoring-rabbitmq-pipelines %}}
+{{% data_streams/monitoring-rabbitmq-pipelines %}}
 
-### Monitoring SNS-to-SQS pipelines
-To monitor a data pipeline where Amazon SNS talks directly to Amazon SQS, you must enable [Amazon SNS raw message delivery][8].
+{{% data_streams/monitoring-sns-to-sqs-pipelines %}}
 
-### Monitoring Kinesis pipelines
-There are no message attributes in Kinesis to propagate context and track a message's full path through a Kinesis stream. As a result, Data Streams Monitoring's end-to-end latency metrics are approximated based on summing latency on segments of a message's path, from the producing service through a Kinesis Stream, to a consumer service. Throughput metrics are based on segments from the producing service through a Kinesis Stream, to the consumer service. The full topology of data streams can still be visualized through instrumenting services.
+{{% data_streams/monitoring-kinesis-pipelines %}}
 
 ### Manual instrumentation
 Data Streams Monitoring propagates context through message headers. If you are using a message queue technology that is not supported by DSM, a technology without headers (such as Kinesis), or Lambdas, use [manual instrumentation to set up DSM][7].
 
 ### Monitoring connectors
 
 #### Confluent Cloud connectors
-{{% dsm_confluent_connectors %}}
+{{% data_streams/dsm-confluent-connectors %}}
 
 ## Further reading
 
@@ -65,8 +63,6 @@ Data Streams Monitoring propagates context through message headers. If you are u
 [1]: /agent
 [2]: /tracing/trace_collection/dd_libraries/nodejs
 [3]: https://pypi.org/project/confluent-kafka/
-[4]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html
 [5]: https://www.npmjs.com/package/amqplib
 [6]: https://www.npmjs.com/package/rhea
 [7]: /data_streams/manual_instrumentation/?tab=nodejs
-[8]: https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html

content/en/data_streams/python.md → content/en/data_streams/setup/language/python.md

@@ -10,6 +10,8 @@ further_reading:
     - link: 'https://www.datadoghq.com/blog/confluent-connector-dsm-autodiscovery/'
       tag: 'Blog'
       text: 'Autodiscover Confluent Cloud connectors and easily monitor performance in Data Streams Monitoring'
+aliases:
+  - /data_streams/python
 ---
 
 ### Prerequisites
@@ -36,27 +38,23 @@ environment:
   - DD_DATA_STREAMS_ENABLED: "true"
 ```
 
-### Monitoring Kafka Pipelines
-Data Streams Monitoring uses message headers to propagate context through Kafka streams. If `log.message.format.version` is set in the Kafka broker configuration, it must be set to `0.11.0.0` or higher. Data Streams Monitoring is not supported for versions lower than this.
+{{% data_streams/monitoring-kafka-pipelines %}}
 
-### Monitoring SQS Pipelines
-Data Streams Monitoring uses one [message attribute][4] to track a message's path through an SQS queue. As Amazon SQS has a maximum limit of 10 message attributes allowed per message, all messages streamed through the data pipelines must have 9 or fewer message attributes set, allowing the remaining attribute for Data Streams Monitoring.
+{{% data_streams/monitoring-sqs-pipelines %}}
 
-{{% data-streams-monitoring/monitoring-rabbitmq-pipelines %}}
+{{% data_streams/monitoring-rabbitmq-pipelines %}}
 
-### Monitoring Kinesis pipelines
-There are no message attributes in Kinesis to propagate context and track a message's full path through a Kinesis stream. As a result, Data Streams Monitoring's end-to-end latency metrics are approximated based on summing latency on segments of a message's path, from the producing service through a Kinesis Stream, to a consumer service. Throughput metrics are based on segments from the producing service through a Kinesis Stream, to the consumer service. The full topology of data streams can still be visualized through instrumenting services.
+{{% data_streams/monitoring-kinesis-pipelines %}}
 
-### Monitoring SNS-to-SQS pipelines
-To monitor a data pipeline where Amazon SNS talks directly to Amazon SQS, you must enable [Amazon SNS raw message delivery][7].
+{{% data_streams/monitoring-sns-to-sqs-pipelines %}}
 
 ### Manual instrumentation
 Data Streams Monitoring propagates context through message headers. If you are using a message queue technology that is not supported by DSM, a technology without headers (such as Kinesis), or Lambdas, use [manual instrumentation to set up DSM][6].
 
 ### Monitoring connectors
 
 #### Confluent Cloud connectors
-{{% dsm_confluent_connectors %}}
+{{% data_streams/dsm-confluent-connectors %}}
 
 ## Further reading
 
@@ -65,7 +63,5 @@ Data Streams Monitoring propagates context through message headers. If you are u
 [1]: /agent
 [2]: /tracing/trace_collection/dd_libraries/python
 [3]: https://pypi.org/project/confluent-kafka/
-[4]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html
 [5]: https://pypi.org/project/kombu/
 [6]: /data_streams/manual_instrumentation/?tab=python
-[7]: https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html

content/en/data_streams/setup/technologies/azure_service_bus.md

@@ -0,0 +1,34 @@
+---
+title: Data Streams Monitoring for Azure Service Bus 
+---
+
+### Prerequisites
+
+* [Datadog Agent v7.34.0 or later][1]
+
+<table>
+  <thead>
+    <tr>
+      <th>Language</th>
+      <th>Library</th>
+      <th>Minimal tracer version</th>
+      <th>Recommended tracer version</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><a href="/data_streams/dotnet">.NET</a></td>
+      <td><a href="https://www.nuget.org/packages/Azure.Messaging.ServiceBus">Azure.Messaging.ServiceBus</a></td>
+      <td>2.53.0 </td>
+      <td>2.53.0 or later </td>
+    </tr>
+  </tbody>
+</table>
+
+### Setting up Data Streams Monitoring
+See setup instructions for [.NET][2].
+
+{{% data_streams/monitoring-azure-service-bus %}}
+
+[1]: /agent
+[2]: /data_streams/setup/language/dotnet