Notification service

What is runningCopy Anchor Link

pagermate-notifications.service supervises one dispatcher and 10 delivery workers by default. It runs as www-data, starts on boot, restarts failed child processes, and stops its children on shutdown. Postfix is a separate standard mail transport dependency; operators manage the PagerMate dispatcher and worker pool through one unit.

The dispatcher consumes committed incident events from the shared database, covering both GUI-created and public API incidents. It records processed event IDs rather than assuming that database IDs commit in order. Transactional, uniquely keyed delivery rows form the durable queue; this does not use Laravel’s generic jobs table or require Redis, Supervisor, a cron entry, or queue:work.

Workers claim queue rows with database FOR UPDATE SKIP LOCKED locks. Different workers can send different incidents concurrently. An incident lock serialises notifications with acknowledgement and resolution. Jobs survive service restarts; a failed worker’s uncommitted claim rolls back.

How an incident is alertedCopy Anchor Link

1. A new triggered incident starts a run from its active service’s enabled escalation policy. An explicitly assigned enabled responder can receive an initial alert as well. Without a policy, an explicit assignee receives one initial level; with neither, routing is recorded as blocked.
2. The run stores a snapshot of the ordered levels, response windows, and repeat count. The first level is due immediately; the dispatcher normally checks once per second.
3. Each level resolves its user and schedule targets. Schedule rotation uses the existing local-time and daylight-saving rules, current overrides, and vacation intervals. Disabled, cross-organisation, or unavailable responders are excluded. A vacation creates a gap; the next rotation member is not silently substituted.
4. Duplicate recipients within the same level/channel receive one queued delivery. Workers recheck incident state, availability, schedule selection, current contact address, and preferences before submission. If an override or handoff changes while a delivery is queued, current recipients are queued instead.
5. After each level’s response window, the next level is processed while the incident remains triggered. A repeat count of two means three full passes. Gaps and delivery failures do not prevent later configured levels from running. After the last response window, the run becomes exhausted; it does not resolve the incident.
6. Acknowledgement and resolution stop pending alert delivery and escalation. Reopen and unacknowledge start a fresh run. Repeated grouped API reports add incident history without restarting the run. Resolution emails are opt-in and target responders whose alerts from the latest run were accepted by a transport.

Disabling the policy or service, or changing the service’s policy, stops a running route. Other policy edits apply to subsequent runs; existing rules stay snapshotted. Schedule and preference changes are evaluated dynamically. Fix a blocked route and explicitly restart it when needed.

Email preferences and transportCopy Anchor Link

Only email is implemented. The notification email overrides the account email when present. Email-enabled and event preferences are checked before sending. Assignment and escalation default on; resolution defaults off. Enabled agents on vacation do not receive incident email. SMS/voice checkboxes do not activate unimplemented providers.

The application submits to Postfix at 127.0.0.1:25 with sender alerts@pagermate.com. Postfix listens only on loopback and sends outbound mail. The concise Notification sent timeline entry means transport acceptance, not an external inbox receipt. Routine routing starts, processed levels, and exhausted-run diagnostics are hidden from the incident timeline. Notification results do not expose queue IDs or transport metadata; full records remain available for server-side troubleshooting. Postfix handles retries after accepting a message; inspect its queue and logs for remote deferrals or bounces. The deployment test used a local mailbox. A later user-requested external test confirmed acceptance by the destination mail server; this is not an inbox or read receipt.

Before transport acceptance, a failed delivery retries up to five attempts with delays of 15, 60, 180, and 600 seconds between attempts. After the fifth failure it remains in the database as failed, with the exception type and a timeline event. Acknowledged, resolved, disabled, stale, or opted-out deliveries are skipped/cancelled instead of sent. Escalation timing continues independently of retry timing.

Delivery is at least once: a crash after SMTP acceptance but before the database commits can cause a duplicate. Stable message IDs and unique queue keys reduce duplicates but cannot guarantee exactly-once inbox delivery. Acknowledgement can wait briefly for a bounded in-flight SMTP submission, and accepted messages cannot be recalled.

Workers on multiple serversCopy Anchor Link

Additional servers can run the same PHP supervisor and workers against the same writable primary database. Deploy the same application version, environment-backed configuration and channel providers on each host. The current unit uses /var/www/web; provision that path or adjust the unit and supervisor paths together. Each host has its own configured worker count and local singleton lock. The database queue coordinates work across hosts, rather than the host-local lock file.

Configure secure database connectivity and a working mail transport on each server. The current 127.0.0.1:25 configuration requires a local Postfix instance on every worker host, or a change to a shared SMTP provider. Mail already accepted by a local Postfix belongs to that host’s mail queue, so worker redundancy alone does not make accepted mail resilient to losing that host’s disk.

Database row locks, incident/run locks, processed-event tracking, and unique delivery keys protect against concurrent queue claims and duplicate level execution. The SMTP-acceptance/database-commit crash window still permits duplicate email, as described above. A message already being submitted cannot be recalled by acknowledgement. Multi-host operation has not been deployment-tested; validate failover and load before relying on it in production.

TimezonesCopy Anchor Link

The Laravel application uses UTC for queue deadlines, event timestamps and absolute vacation/override boundaries. Schedule rotations use each schedule’s configured timezone and local-calendar handoff rules, including daylight-saving changes. Workspace display dates use the organisation timezone. The notification resolver reuses the same schedule implementation as the GUI rather than calculating a second rotation.

Keep every worker clock synchronised with NTP and database timestamps in UTC. Matching wall-clock time matters for due checks on different servers even though their operating-system timezone does not determine a schedule’s rotation.

Start, stop, restart, and logsCopy Anchor Link

sudo systemctl status pagermate-notifications
sudo systemctl start pagermate-notifications
sudo systemctl stop pagermate-notifications
sudo systemctl restart pagermate-notifications
sudo journalctl -u pagermate-notifications -f

Restart the service after application code, channel registration, or environment changes. Stopping does not delete queued work. ALERTING_ENABLED=false disables dispatch and delivery after a service restart; the systemd unit itself can remain running.

Change the worker countCopy Anchor Link

Edit /etc/default/pagermate-notifications:

PAGERMATE_NOTIFICATION_WORKERS=10

Set 1–100 workers, then run:

sudo systemctl restart pagermate-notifications

The dispatcher is additional to the configured delivery-worker count. The supervisor and workers all run in PHP; the service requires PHP CLI with pcntl and proc_open, with no Python runtime dependency. The supervisor is /var/www/web/deploy/notification-supervisor.php; the installed unit is /etc/systemd/system/pagermate-notifications.service.

Queue status and recoveryCopy Anchor Link

Run from /var/www/web:

php artisan notifications:status

This shows enabled state, the dispatcher heartbeat, queue/run counts, and recent failed delivery IDs. A settings API availability flag requires a recent dispatcher heartbeat and a real mail transport; it does not prove remote mailbox delivery.

After fixing the underlying transport problem, retry a specific failed delivery:

php artisan notifications:status --retry=DELIVERY_ID

The worker rechecks current incident state, run state, and preferences before sending; retrying an acknowledged incident or stopped/exhausted run does not force an alert. Restart the route explicitly when a new run is needed. For a still-triggered incident whose policy needs to restart, or an older incident that predates service activation:

php artisan notifications:status --enroll=INCIDENT_DATABASE_ID

Use the numeric database ID, not the public INC-... number. This explicitly starts a fresh run and can send another initial notification. Existing historical incidents were not automatically replayed at installation, to avoid an alert flood. New events are handled normally. Notifications are visible in the incident timeline.

For mail after Postfix acceptance:

sudo systemctl status postfix
sudo postqueue -p
sudo journalctl -u postfix -f

Local-only end-to-end testCopy Anchor Link

cd /var/www/web
php artisan notifications:test-local
# Also exercise public API ingestion, deduplication, acknowledge, and resolve:
php artisan notifications:test-local --via-api

This creates temporary test records in a separate test organisation, calls the actual GUI incident API, waits for the running service to submit email, and acknowledges/resolves through the API. It cleans up its test application records. It never chooses an external address: the destination is pagermate-alert-test@localhost.localdomain.

The Postfix test message remains in /var/mail/pagermate-alert-test for inspection with appropriate server permissions. A consumed public incident number is not reused. The command checks transport acceptance; inspect the local mailbox or Postfix log for final local delivery.

Add another channelCopy Anchor Link

Channel implementations live in /var/www/web/app/Services/Alerting. Implement NotificationChannel:

interface NotificationChannel
{
    public function eligible(object $user, object $preferences, string $kind): bool;
    public function send(object $incident, object $user, object $preferences, object $delivery): void;
}

eligible checks the channel’s address/contact flag and the assignment, escalation, or resolution preference. send must return only after the real provider accepts the message, and throw on failure. Use a bounded network timeout and the delivery’s dedup_key as the provider’s idempotency key when supported. Keep provider credentials in environment-backed configuration; do not store them in queue payloads or logs.

Register the tested implementation in /var/www/web/config/alerting.php:

'channels' => [
    'mail' => App\Services\Alerting\MailChannel::class,
    // Add only after implementing and configuring a real provider:
    // 'sms' => App\Services\Alerting\SmsChannel::class,
],

The existing engine handles selection, vacations, overrides, escalation, retries, locks, event history, and one delivery row per recipient/channel. New registrations apply when levels are queued; they do not retroactively add channels to already queued deliveries. Keep transport-specific code in the channel class, test preferences/failures/idempotency, clear a config cache if one is used, and restart pagermate-notifications. Add accurate per-channel availability to the settings API when exposing a new channel in the GUI.

Incident links while the GUI is localCopy Anchor Link

The current GUI runs locally on the operator’s Mac, so email contains the public incident number and asks the responder to find it in their workspace. No non-working server incident link is included. Once the GUI is deployed, set ALERTING_INCIDENT_URL in /var/www/web/.env to its real incident URL template, using {id} for the database ID or {number} for the public INC-... identifier. Clear any config cache and restart the service. Leave the value empty to keep incident-number-only emails.

Source and configurationCopy Anchor Link

• Engine: /var/www/web/app/Services/Alerting/NotificationEngine.php
• Coverage and preferences: /var/www/web/app/Services/Alerting/RecipientResolver.php
• Mail adapter: /var/www/web/app/Services/Alerting/MailChannel.php
• Channels and retry defaults: /var/www/web/config/alerting.php
• Mail transport: /var/www/web/config/mail.php, /var/www/web/.env, /etc/postfix/main.cf
• Durable tables: notification_engine, notification_events, notification_runs, notification_deliveries

The engine uses the same database as the GUI and public API. Do not truncate the event ledger or edit queue records to imitate delivery; use the recovery commands and retain history for investigation.