APM setup#

Application performance monitoring in ObserveIQ is built on OpenTelemetry. There is no proprietary agent, which means the instrumentation you add here is portable and is not something you have to unpick if you ever move.

What you get once instrumented#

• A service map drawn automatically from the calls your services actually make.
• Request rate, error rate and response time per service and per endpoint.
• Trace search with a waterfall view showing where the time went in a single request.
• Traces linked to logs, so a slow operation opens the log lines it produced.
• Comparison against normal, judging response time against the same hour in previous weeks.

Common configuration#

Every language uses the same three settings.

Add OTEL_RESOURCE_ATTRIBUTES to separate environments without creating separate services:

export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=1.4.2"

Node.js#

npm install --save @opentelemetry/api @opentelemetry/auto-instrumentations-node

export OTEL_SERVICE_NAME="checkout-api"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://app.observeiq.io/v1"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OBSERVEIQ_API_KEY"

node --require @opentelemetry/auto-instrumentations-node/register app.js

Java#

Download the agent and attach it with -javaagent. No code changes are required.

curl -sLO https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

java -javaagent:./opentelemetry-javaagent.jar \
     -Dotel.service.name=orders-service \
     -Dotel.exporter.otlp.endpoint=https://app.observeiq.io/v1 \
     -Dotel.exporter.otlp.headers=Authorization=Bearer\ $OBSERVEIQ_API_KEY \
     -jar your-application.jar

Python#

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install

export OTEL_SERVICE_NAME="billing-worker"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://app.observeiq.io/v1"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OBSERVEIQ_API_KEY"

opentelemetry-instrument python app.py

.NET#

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("payments-api"))
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter());

Then set the endpoint and headers as environment variables, as above.

Go#

Go requires a small amount of code because it has no runtime agent.

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/trace"
)

exp, err := otlptracehttp.New(ctx)          // reads OTEL_EXPORTER_OTLP_* from the environment
if err != nil { log.Fatal(err) }
tp := trace.NewTracerProvider(trace.WithBatcher(exp))
otel.SetTracerProvider(tp)
defer tp.Shutdown(ctx)

PHP and Ruby#

Both follow the same pattern: install the OpenTelemetry SDK and auto-instrumentation package for the framework, then set the three environment variables above. For PHP, enable the opentelemetry extension and set OTEL_PHP_AUTOLOAD_ENABLED=true.

AWS Lambda#

Lambda cannot use a normal background exporter, because the runtime is frozen between invocations. Use the upstream OpenTelemetry Lambda layers instead.

1. Add two layers to the function: the language instrumentation layer and the collector layer.
2. Set these environment variables:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
OTEL_SERVICE_NAME=your-function-name
OTEL_EXPORTER_OTLP_ENDPOINT=https://app.observeiq.io/v1
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer obs_your_key_here

1. Set the function's tracing mode to PassThrough, so it does not also send to X-Ray.

Cold starts, initialisation time, billed duration and memory use are collected separately from the function's log group, because they are not available as CloudWatch metrics.

Verifying it works#

Send a few requests through the application, then open APM. If nothing appears after two minutes:

• Check the application logs for OpenTelemetry export errors. A 401 means the API key is wrong or expired; a connection error means the endpoint is unreachable from your network.
• Confirm the endpoint ends in /v1 and that the header is Authorization=Bearer obs_....
• Confirm the service actually received traffic. A service with no requests produces no spans.