AI CMO Engine — Bedrock & Search API Quota Exhaustion
ResolvedRate limiting, exponential backoff, and daily quota checks deployed. Campaign orchestrator now gracefully pauses when API quotas are exhausted instead of repeated failed requests. SerpAPI and Bedrock quotas reset overnight.
Root cause: AWS Bedrock daily token quota exceeded (ThrottlingException), SerpAPI search quota exhausted, and Hunter.io rate limits hit. All campaign discovery and AI-driven orchestration returning 429 errors. Service is running but campaigns produce zero leads.
AI CMO Engine reported as outage on status page. Investigating campaign orchestrator failures and API connectivity.
CORS Security Hardening — Gateway Configuration Update
ResolvedCORS origin whitelist deployed to both dev and production gateways. Security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, HSTS) added. Production route configuration synchronized with all service endpoints.
Scheduled security maintenance: replacing permissive CORS configuration with strict origin whitelist for *.truthlocks.com domains.
Platform-Wide Service Outage
ResolvedThe internal network routing and JWT authentication issues have been fully resolved. All services are reporting healthy responses and the platform is fully operational.
A fix has been rolled out applying the correct internal DNS health-check endpoints and enabling internal ALB routing for the API gateway. We are monitoring the recovery.
We identified missing internal configuration URLs and a misconfigured database security rule that caused services to crash. Applying patches and re-deploying.
We are investigating widespread 503 Service Unavailable errors affecting the API Gateway and internal service-to-service communication.