Hi,

Thanks for ntfy-alertmanager — we use it in a homelab monitoring stack.

We hit a reliability issue and wanted to report it (with a suggested,
opt-in fix).

Summary
-------
When ntfy-alertmanager fails to publish a notification to ntfy (e.g. ntfy is
briefly down / DNS fails / connection refused), the webhook handler logs the
error but still returns HTTP 200 to Alertmanager. Alertmanager therefore
considers the notification delivered and does not retry, so the alert is
silently lost until the next repeat_interval (or forever, if it resolves
before then).

Details
-------
In handleWebhooks (main.go), the publish error is only logged; the handler
then returns with the default 200 status:

err := br.publish(n, topicParam)
if err != nil {
logger.Error("Failed to publish notification",
slog.String("error", err.Error()))
}
// no http.Error(...) -> response stays 200 OK

Alertmanager only retries a webhook notification on a non-2xx (specifically
5xx) response. Because the bridge always answers 200, its built-in retry never
engages.

Impact
------
For critical alerts this means a transient ntfy outage can drop the alert
entirely. Observed in production: the ntfy container was restarted for a few
seconds; the alert Alertmanager tried to deliver during that window was lost,
because the bridge returned 200 despite the failed publish.

Steps to reproduce
------------------
1. Point ntfy at an unreachable server (or stop ntfy).
2. Send a firing alert to Alertmanager that routes to the ntfy webhook
receiver (or POST an Alertmanager webhook payload directly to the bridge).
3. Observe: the bridge logs "Failed to publish notification", but the HTTP
response is 200.
4. Alertmanager records the notification as sent and never retries -> alert
lost.

Suggested fix
-------------
Return a 5xx when publishing fails, so Alertmanager's existing retry/backoff
takes over. To avoid changing behaviour for users who rely on fire-and-forget,
this could be opt-in via a config directive, e.g.:

ntfy {
...
return-errors true # respond 5xx on publish failure so AM retries
}

Successful notifications are already de-duplicated via the cache
(cache.Contains(fingerprint, status)), so an Alertmanager retry is idempotent
— only the failed notification would be re-sent.

I'm happy to send a patch if you'd accept this (opt-in, defaulting to the
current behaviour).

Version
-------
xenrox/ntfy-alertmanager 1.0.0 (behaviour also present on current master).

Thanks,
Mirko

On Fri Jul 31, 2026 at 9:32 PM CEST, admin admin wrote:

Hey thanks for the report, this should indeed be fixed.

> this could be opt-in via a config directive, e.g.:
>
> ntfy {
> ...
> return-errors true # respond 5xx on publish failure so AM retries
> }
>
> Successful notifications are already de-duplicated via the cache
> (cache.Contains(fingerprint, status)), so an Alertmanager retry is idempotent
> — only the failed notification would be re-sent.
>
> I'm happy to send a patch if you'd accept this (opt-in, defaulting to the
> current behaviour).

I would appreciate you sending a patch. I think it is fine though to default
to the new behaviour instead of the current one, because a retry is the sensible
default if the alert failed.

When publishing to ntfy failed, the webhook handler only logged the error
and still responded with 200. Alertmanager treats that as a successful
delivery and never retries, so an alert can be lost silently while ntfy is
briefly unavailable.

Respond with 500 instead, so that Alertmanager's retry mechanism re-delivers
the notification. Notifications that were published successfully are cached
and skipped on the retry, so re-delivery does not duplicate them.
---

Thanks for the quick reply. As discussed, this makes the new behaviour the
default instead of hiding it behind a config option. Built and vetted against
current master (9564841).

 main.go | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/main.go b/main.go
index ddeac5a..ad07458 100644
--- a/main.go
+++ b/main.go
@@ -560,11 +560,13 @@ func (br *bridge) handleWebhooks(w http.ResponseWriter, r *http.Request) {
 	logger.Debug("Received alert",
 		slog.Any("payload", event))
 
+	publishFailed := false
 	if br.cfg.AlertMode == config.Single {
 		notifications := br.singleAlertNotifications(ctx, &event)
 		for _, n := range notifications {
 			err := br.publish(n, topicParam)
 			if err != nil {
+				publishFailed = true
 				logger.Error("Failed to publish notification",
 					slog.String("error", err.Error()))
 			} else {
@@ -579,10 +581,19 @@ func (br *bridge) handleWebhooks(w http.ResponseWriter, r *http.Request) {
 		notification := br.multiAlertNotification(&event)
 		err := br.publish(notification, topicParam)
 		if err != nil {
+			publishFailed = true
 			logger.Error("Failed to publish notification",
 				slog.String("error", err.Error()))
 		}
 	}
+
+	// Report publish failures to Alertmanager so that its retry mechanism can
+	// re-deliver the notification. Notifications that were published
+	// successfully are cached and skipped on the retry, so re-delivery does not
+	// duplicate them.
+	if publishFailed {
+		http.Error(w, "Failed to publish notification", http.StatusInternalServerError)
+	}
 }
 
 func (br *bridge) handleHealth(w http.ResponseWriter, _ *http.Request) {
-- 
2.55.0.windows.3

ntfy-alertmanager/patches/.build.yml: SUCCESS in 51s

[main: Return 500 when publishing a notification fails][0] from [admin, admin][1]

[0]: https://lists.xenrox.net/~xenrox/public-inbox/patches/37
[1]: admin@seipel-web.de

✓ #405 SUCCESS ntfy-alertmanager/patches/.build.yml https://builds.xenrox.net/~xenrox/job/405

Thanks for the patch!

I have applied it and published a new release.