Self-hosted observability with transactional incident notifications

Architecture

Applications emit traces, metrics, and structured logs through OpenTelemetry instrumentation. Prometheus scrapes exposed application and infrastructure metric endpoints. OpenSearch receives and retains structured logs through a separately deployed log-ingestion path. Grafana provides dashboards over configured telemetry data sources. Sentry receives application error and performance telemetry according to its application integration configuration. PostgreSQL stores incident records, alert receipts, notification policies, acknowledgement state, and transactional-outbox rows. An authenticated alert-ingress route validates alert source authenticity, deduplicates alert state, and transactionally creates or updates incidents and notification outbox rows. A continuously running or scheduled dispatcher claims pending outbox rows, sends SMS notifications through Twilio, and records delivery attempts; a separate execution path recovers expired sending leases. Alert evaluation, alert ingress, log ingestion, incident routing, on-call policy, dispatcher scheduling or supervision, and provider reconciliation are project-specific implementation and deployment work rather than verified product integrations.

Implementation sequence

• Define measurable health endpoints and service-level signals: readiness, liveness, dependency checks, request latency, error rate, worker backlog, outbox age, database connectivity, and capacity indicators. Ensure health endpoints do not expose secrets or detailed dependency topology publicly.
• Instrument applications with OpenTelemetry. Apply consistent service names, environment labels, trace and request correlation identifiers, and structured log fields. Establish field-level rules that exclude credentials, access tokens, session identifiers, message bodies, and unnecessary personal data.
• Deploy Prometheus and configure authenticated or network-restricted scraping of approved metric endpoints. Create recording and alert rules for sustained availability, latency, error-rate, backlog, failed-job, and telemetry-pipeline conditions. Alert rule thresholds and evaluation windows must be validated against actual service behavior.
• Deploy an authenticated alert-ingress route between alert evaluation and PostgreSQL. Before trusting webhook fields, verify the alert source's authenticity against the raw request using the configured verification method; reject or quarantine unverified requests without changing incident, receipt, or outbox state.
• Persist each verified alert receipt with a stable source event identity and payload evidence sufficient for audit. Atomically claim or deduplicate the receipt and, in the same PostgreSQL transaction, create or update the incident's alert state and create any required notification outbox row. Commit completion only with those state changes; duplicate delivery or a crash before commit must remain safely retryable without creating an additional incident transition or notification.
• Deploy OpenSearch and a separately configured log-forwarding/ingestion path. Send structured logs with timestamps, severity, service, environment, request or trace correlation ID, and incident ID where available. Define index lifecycle, retention, access controls, and a procedure for ingestion failure.
• Configure Grafana data sources and dashboards for service health, release correlation, worker/outbox backlog, notification outcomes, alert-ingress failures, and incident acknowledgement status. Restrict dashboard editing and sensitive log access by role.
• Configure Sentry application integration for exception capture and approved performance telemetry. Set ownership, issue-triage workflow, release/environment tags, and data-scrubbing rules before sending production events.
• Create PostgreSQL tables for incidents, alert receipts, escalation policies, notification recipients, acknowledgement events, notification attempts, and an outbox. Insert each business event requiring a notification and its outbox record in the same database transaction.
• Provide an operational execution path for both outbox dispatch and stale-lease recovery, such as supervised long-running workers or scheduled workers. Monitor executor availability, failed runs, backlog, and lease age; when unavailable, leave records recoverable in PostgreSQL and alert operators rather than marking notifications completed or discarded.
• Use explicit outbox states: pending, sending, sent, unknown, and error. A worker atomically claims eligible pending rows, assigns an attempt identifier and lease, changes them to sending, and records every request, response, or failure in an append-only attempt history. Recover stale sending records through controlled lease expiry; do not silently discard them or mark them sent after a crash.
• For a successful provider response, persist the provider message identifier when returned and move the row to sent only after the database update succeeds. For validation failures or non-retryable configuration failures, retain the error context and mark error for operator review. Use bounded retry scheduling for failures considered retryable by project policy.
• If a provider timeout occurs after a request may have been transmitted, set the outbox row to unknown rather than failed. Reconcile unknown rows by provider message ID or provider-supported idempotency or delivery-event facilities only if those capabilities have been separately verified for the chosen integration. If no lookup or deduplication mechanism is available, retrying is at-least-once delivery and can create duplicate SMS messages.
• Implement an operator queue for unknown and error notification records. Operators should inspect persisted request metadata, any available provider identifier or delivery evidence, the incident status, and recipient acknowledgement before manually resending, cancelling, or closing a record. Preserve the decision and actor in the audit trail.
• Define incident severity, ownership, acknowledgement expectations, escalation timing, quiet-hour policy, and a runbook for Prometheus, alert ingress, OpenSearch, Grafana, Sentry, database, dispatcher/recovery execution, and notification-delivery failures. Test alerting, acknowledgement, escalation, outbox recovery, and restoration procedures in a non-production environment.
• Future deployment work includes capacity sizing, network placement, TLS termination, secrets distribution, telemetry collectors or log shippers, alert-routing and ingress infrastructure, worker supervision or scheduling, high availability decisions, backup scheduling, restore tests, and on-call operational ownership.

Security and operations

• Keep Twilio credentials, alert-ingress verification material, Sentry credentials, database credentials, and telemetry endpoint credentials in a managed secret mechanism; never place them in source control, dashboard variables, client-side code, or logs.
• Apply least-privilege database roles. The alert-ingress service and notification worker should have only the database permissions needed for their receipt, incident, outbox, and attempt records, and no general administrative role.
• Protect metrics endpoints, alert ingress, Grafana, OpenSearch, Sentry, and PostgreSQL with authenticated access, network segmentation, and encrypted transport. Health endpoints intended for load balancers should expose only the minimum safe response.
• Treat alert payloads, logs, traces, error reports, incident descriptions, phone numbers, and provider message identifiers as potentially sensitive data. Redact sensitive fields before export, restrict search access, and define retention and deletion requirements.
• Back up PostgreSQL incident, alert-receipt, and outbox data, including audit history needed to resolve duplicate alerts and ambiguous sends. Encrypt backups, restrict restore access, define retention, and periodically perform documented restore tests.
• Back up OpenSearch configuration and any log data required by retention or investigation policy. Log availability and durability depend on the selected deployment topology and ingestion design.
• Do not use SMS alone as proof that an incident was received or acknowledged. Record acknowledgement through an authenticated operational workflow and retain the associated audit event.

Limitations

• The catalogue does not include a dedicated alert-routing or paging product. Prometheus alert evaluation, authenticated alert ingress, alert-state deduplication, and delivery to the application notification worker require project-specific deployment and integration work; this design does not assert that such routing is configured.
• The catalogue does not establish Twilio idempotency, message lookup, delivery-event, or reconciliation capabilities. Therefore, a timeout after transmission must remain unknown until independently verified evidence is available; where reconciliation or deduplication is unavailable, the notification channel is at-least-once and recipients may receive duplicates.
• OpenTelemetry instrumentation alone does not deploy a collector, storage backend, or log shipper. Those runtime components and their configuration are external deployment prerequisites.
• OpenSearch is a log-search platform, but log retention, availability, access isolation, ingestion throughput, and backup behavior depend on the selected self-hosted or managed deployment and project configuration.
• This stack provides observability building blocks, not a completed incident-management process. On-call staffing, contact data maintenance, escalation authority, worker operation, runbooks, and exercises remain operational responsibilities.
• No pricing, message throughput, retention limit, recovery objective, or provider service-level claim is made because the brief provides no dated evidence for them.

Alternatives to consider

• Use SendGrid (13), Postmark (12), or Resend (11) for transactional email as a separate notification channel if email escalation is required; each would require its own outbox delivery and reconciliation design.
• Use RabbitMQ (91) or Temporal (92) if the project later needs more extensive asynchronous orchestration, but PostgreSQL-backed outbox processing is the lower-complexity starting point for notification dispatch.
• Use Amazon S3 (51), Backblaze B2 (53), Wasabi Hot Cloud Storage (95), Cloudflare R2 (52), or MinIO (94) for separate backup storage after independently assessing retention, encryption, restore, and access requirements.
• A managed or self-hosted deployment choice for Grafana, Prometheus, OpenSearch, Sentry, and PostgreSQL should be made based on operational ownership and compliance requirements, which are not specified in the brief.