Skip to content

Evolve observability documentation #797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 26, 2025
Merged

Evolve observability documentation #797

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
481ca7c
Evolve observability documentation
dandavison Mar 25, 2025
2c63456
Code review suggestions
dandavison Mar 25, 2025
b636d3a
npx doctoc README.md
dandavison Mar 25, 2025
832e30d
Use features@main
dandavison Mar 26, 2025
434a314
Update tests to expect uuid version 7 from server
dandavison Mar 26, 2025
5ef02ea
Skip failing tests
dandavison Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ informal introduction to the features and their implementation.
- [Worker Shutdown](#worker-shutdown)
- [Testing](#testing-1)
- [Workflow Replay](#workflow-replay)
- [OpenTelemetry Support](#opentelemetry-support)
- [Observability](#observability)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May need to run npx doctoc README.md, unsure

- [Protobuf 3.x vs 4.x](#protobuf-3x-vs-4x)
- [Known Compatibility Issues](#known-compatibility-issues)
- [gevent Patching](#gevent-patching)
Expand Down Expand Up @@ -1344,10 +1344,31 @@ async def check_past_histories(my_client: Client):
)
```

### OpenTelemetry Support
### Observability

OpenTelemetry support requires the optional `opentelemetry` dependencies which are part of the `opentelemetry` extra.
When using `pip`, running
See https://github.com/temporalio/samples-python/tree/main/open_telemetry for a sample demonstrating collection of
metrics and tracing data emitted by the SDK.

#### Metrics

The SDK emits various metrics by default: see https://docs.temporal.io/references/sdk-metrics. By default, these are
emitted with attributes `namespace`, `task_queue`, `workflow_type` / `activity_type`, and
`service_name=temporal-core-sdk`. To emit additional attributes with all metrics, pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not accurate. We emit lots of client side metrics, worker metrics, etc that don't have some (or all) of these attributes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, deleted

[global_tags](https://python.temporal.io/temporalio.runtime.TelemetryConfig.html#global_tags) when creating the
[TelemetryConfig](https://python.temporal.io/temporalio.runtime.TelemetryConfig.html).

For emitting custom metrics, the SDK makes a metric meter available:
- In Workflow code, use https://python.temporal.io/temporalio.workflow.html#metric_meter
- In Activity code, use https://python.temporal.io/temporalio.activity.html#metric_meter
- In normal application code, use https://python.temporal.io/temporalio.runtime.Runtime.html#metric_meter
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a bit confused, where does normal application code exist if not in a workflow or activity?

Copy link
Member

@cretz cretz Mar 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where they use the client in non-activity situations, e.g. a client interceptor or anywhere they use the client (but really they probably should use their favorite metrics client/system instead of using Temporal's, but meh)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I describe it to people is that a codebase using Temporal has 3 distinct categories of code that involve Temporal: workflow code, activity code, and "normal application code" that is using a Temporal client to start and interact with workflows, issue other queries, create schedules etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks both, helpful :)


The attributes emitted by these default to `namespace`, `task_queue`, and `workflow_type`/`activity_type`; use
`with_additional_attributes` to create a meter emitting additional attributes.

#### Tracing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call this "OpenTelemetry Tracing" both for discoverability and so users don't get confused and think we have general purpose or multi-vendor tracing (like some SDKs do)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


Tracing support requires the optional `opentelemetry` dependencies which are part of the `opentelemetry` extra. When
using `pip`, running

pip install 'temporalio[opentelemetry]'

Expand Down