Why are downstream gRPC errors about ingestion-internal appearing?
When running ITRS Analytics apps such as the Notifications app and the Capacity Planner app, you may see frequent warning messages in your app logs about failed gRPC exports to the ingestion-internal service. These warnings indicate that apps are attempting to send telemetry data to a service that is not accessible.
The logs of the app pods display warnings similar to the following:
WARN 1 --- [ternal:port/...] i.o.exporter.internal.grpc.GrpcExporter : Failed to export spans. Server responded with gRPC status code 2. Error message: ingestion-internal: Name or service not known
or
[OkHttp http://ingestion-internal:port/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: ingestion-internal: Name or service not known
What causes this issue? Copied
Starting with ITRS Analytics 2.16.x, the traces ingestion service is enabled by default in the platform configuration:
ingestion:
traces:
enabled: true
This means the platform is configured to accept trace data. However, the actual collection of traces from platform workloads is not enabled by default:
collection:
traces:
enabled: false
When trace collection is disabled, apps are still configured to export trace data to the ingestion-internal service, but the service is not running or accessible. This mismatch causes the gRPC connection failures and the resulting warning messages.
Do I need to install the Traces app to fix this? Copied
No, you do not need to install the Traces app to resolve these error messages. The traces ingestion service is part of the core platform configuration and does not require a separate Traces app installation.
How can I resolve these errors? Copied
To eliminate these warning messages, you need to explicitly enable trace collection in your platform configuration. This ensures that the ingestion-internal service is properly deployed and accessible to apps attempting to export trace data.
Update your configuration to enable trace collection:
collection:
traces:
enabled: true
After applying this configuration change, the ingestion-internal service will be available, and apps will be able to successfully export their trace data. The warning messages should no longer appear in your logs.
Note
Enabling trace collection will deploy the necessary infrastructure to collect and process trace data from your platform workloads. If you do not need trace data, you can alternatively disable trace exports at the app level, though enabling collection is the recommended approach for a complete observability solution.