The issue was due to a combination of hitting our IP address limit for our Kubernetes deployment and a bad version number for our Datadog Java Agent. Datadog Java Agent is 1.46.0 on Maven but that version doesn't not exist on Google Cloud Console Artifact Registry. The incorrect version prevented new pods from starting up and the ip address allocation prevented proper visibility into this issue.
Followups include: * Adding a new subnet to increase the number of IPs * Add a CI check to make sure the JAR artifact exists for the Datadog Agent Library we need
Posted Feb 05, 2025 - 04:19 PST
Monitoring
A fix has been implemented and we are monitoring the results.