Webinar
Understanding the Meta, Comcast, and LinkedIn Outages

Product Updates

Send ThousandEyes Data to Any Observability Backend Using OpenTelemetry

By Antonio Jimenez Martinez
| | 11 min read

Summary

We show you how easy it is to export your ThousandEyes test data into any observability backends using OpenTelemetry.


The widespread industry adoption of OpenTelemetry (OTel) presented an opportunity for ThousandEyes to build a system that greatly simplifies integration with OTel-compliant sources. The end result was ThousandEyes for OpenTelemetry, a pipeline for customers to correlate ThousandEyes data within their choice observability platforms. With ThousandEyes for OpenTelemetry, teams can gain powerful insights and visualizations across diverse data sets, facilitating quicker problem-solving and faster service restoration. 

In this blog post, we will guide you through three easy steps to sending your ThousandEyes data to any observability backend using OpenTelemetry. But before you get started, you’ll need an active ThousandEyes account. If you’re not a current customer, you can create a 15-day free trial account and follow along.

ThousandEyes data is propagated to other observability backends through the OpenTelemetry Collector using the OTLP exporter, which you can configure to send data using HTTP or GRPC protocol. You’ll be able to select the appropriate configuration when creating the integration.

This tutorial is API-based, so you will need an API token to perform the API request. You can find more information in the ThousandEyes Developer Reference.


Step 1: Create a ThousandEyes Network Test

Before we can send ThousandEyes metrics over OTel, we need to capture some test data. So the first thing we need to do is create a ThousandEyes network test, as shown below:

curl -L 'https://api.thousandeyes.com/v6/tests/agent-to-server/new' -H 'Content-Type: application/json' -H 'Authorization: Bearer $BEARER_TOKEN' -d '{
        "interval": 60,
        "agents": [
          {"agentId": 45}
        ],
        "testName": "OpenTelemetry Test",
        "server": "www.google.com",
        "port": 80,
        "alertsEnabled": 0
      }'

The response will contain a “testId,” which you must provide in the next step. In our case, the testId is 281474976717575.


Step 2: Create a Tag

In this next step, we will create a tag and associate it with our test. 

The ThousandEyes Tags API provides a tagging system with key/value pairs. It allows you to tag assets within the ThousandEyes platform (such as agents, tests, or alert rules) with meaningful metadata.

Begin by creating a tag, as shown below.

curl -i -XPOST https://api.thousandeyes.com/v7/tags -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
        "key": "OpenTelemetryTestKey",
        "value": "OpenTelemetryTestValue",
        "objectType": "test",
        "accessType": "all"
      }'

The response will contain a “tagId,” which you must provide in the next step. For example, 17f6cccd-cc43-450d-925d-04b400847e82.

Then, associate the test with a tag by entering the following command.

curl -i -XPOST https://api.thousandeyes.com/v7/tags/17f6cccd-cc43-450d-925d-04b400847e82/assign -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "assignments": [
    {
      "id": "281474976717575",
      "type": "test"
    }
  ]
}'

Step 3: Create an Integration

Now that we have some data available, we can create the streaming integration by providing a target endpoint, the tag key and value (from the previous step), and the protocol that we want to use (HTTP or GRPC). The target endpoint will depend on the observability backend you use. Let’s walk through a few integrations with different observability vendors.

Splunk Observability

Splunk only allows receiving data using HTTP protocol. Create an HTTP stream integration by entering the following command.

curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "tagMatch": [{
    "key": "OpenTelemetryTestKey",
    "value": "OpenTelemetryTestValue", 
    "objectType": "test"
  }],
  "endpointType": "http",
  "streamEndpointUrl": "https://ingest.eu0.signalfx.com:443/v2/datapoint/otlp",
  "customHeaders" : {
    "X-SF-Token": "TOKEN",
    "Content-Type": "application/x-protobuf"
  }
}'

In Splunk, you can confirm that the data has been received, as shown in Figure 1.

Screenshot of Splunk platform ingesting ThousandEyes network latency data.
Figure 1. Splunk platform ingesting ThousandEyes network latency data.

The endpoint to send the data to Splunk is “https://ingest.{REALM}.signalfx.com/v2/datapoint/otlp”. “Realm” is the region of your observability instance. You can find more information about Realm at https://docs.splunk.com/Splexicon:Realm.

For more information on sending metrics to Splunk using OpenTelemetry, visit the Splunk Developer Guide.

Honeycomb

Honeycomb allows the receiving of data using HTTP and GRPC protocols. To create the GRPC integration, enter the following command.

 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "tagMatch": [{
    "key": "OpenTelemetryTestKey",
    "value": "OpenTelemetryTestValue", 
    "objectType": "test"
  }],
  "endpointType": "grpc",
  "streamEndpointUrl": "https://api.honeycomb.io:443",
  "customHeaders" : {
    "x-honeycomb-team": "TOKEN",
    "x-honeycomb-dataset": "OpenTelemetryTestGrpc"
  }
}'

In Honeycomb, you can confirm that the platform received the data, as shown in Figure 2 and Figure 3.

Screenshot of Honeycomb platform ingesting ThousandEyes data.
Figure 2. Honeycomb platform ingesting ThousandEyes data.
Screenshot of Honeycomb platform ingesting ThousandEyes raw data.
Figure 3. Honeycomb platform ingesting ThousandEyes raw data.

You can find information about how to access the Honeycomb token at the following link: https://docs.honeycomb.io/working-with-your-data/settings/api-keys/

For more information on sending metrics to Honeycomb using OpenTelemetry, visit https://docs.honeycomb.io/getting-data-in/otel-collector/.

Signoz

Signoz allows the receiving of data using HTTP and GRPC protocols. To create the GRPC integration, enter the following command.

 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "tagMatch": [{
    "key": "OpenTelemetryTestKey",
    "value": "OpenTelemetryTestValue", 
    "objectType": "test"
  }],
  "endpointType": "grpc",
  "streamEndpointUrl": "https://ingest-eu.signoz.io:4317",
  "customHeaders" : {
    "signoz-access-token": "SIGNOZ_INGESTION_KEY"
  }
}'

The endpoint to send the data to Signoz is “https://ingest-<region>.signoz.io”. The regions could be “eu”, “us” and “in”. For the SIGNOZ_INGESTION_KEY, the API token is provided by Signoz. You can find your ingestion key in the SigNoz cloud account details sent to your email. You can also find more information at https://signoz.io/docs/userguide/logs

In Signoz, you can confirm that the platform has received the data, as shown in Figure 4.

Screenshot of Signoz platform ingesting ThousandEyes network latency data.
Figure 4. Signoz platform ingesting ThousandEyes network latency data.

Grafana

Grafana allows the receiving of data using HTTP protocol. To create an HTTP stream integration, enter the following command.

 curl -v -XPOST https://api.thousandeyes.com/v7/stream -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER_TOKEN" -d '{
  "type": "opentelemetry",
  "tagMatch": [{
    "key": "OpenTelemetryTestKey",
    "value": "OpenTelemetryTestValue", 
    "objectType": "test"
  }],
  "endpointType": "http",
  "streamEndpointUrl": "https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/metrics",
  "customHeaders" : {
    "Authorization": "Basic $(echo -n $GRAFANA_INSTANCE_ID:$GRAFANA_CLOUD_ACCESS_POLICY_TOKEN | base64)"
  }
}'

In Grafana, you can confirm that the platform has received the data, as shown in Figure 5.

Screenshot of Grafana platform ingesting ThousandEyes network test data.
Figure 5. Grafana platform ingesting ThousandEyes network test data.
Screenshot of Grafana platform ingesting ThousandEyes network latency data.
Figure 6. Grafana platform ingesting ThousandEyes network latency data.

For more information on sending metrics to Grafana using OpenTelemetry, visit: https://grafana.com/docs/grafana-cloud/monitor-infrastructure/otlp/send-data-otlp/.


With its support for the OpenTelemetry format, ThousandEyes has opened the doors for more meaningful insights across the observability space. So whether you’re using Cisco AppDynamics, Grafana, Splunk, or one of many options on the market today, your favorite OTel-compliant platform can now consume ThousandEyes metrics.

Setup an integration today and start exporting ThousandEyes data into your favorite observability backends. And if you have any questions, feel free to contact us.


Subscribe to the ThousandEyes Blog

Stay connected with blog updates and outage reports delivered while they're still fresh.

Upgrade your browser to view our website properly.

Please download the latest version of Chrome, Firefox or Microsoft Edge.

More detail