Fixing TrueNAS 404 Errors In Docker-borg-client

by Editorial Team 48 views
Iklan Headers

Hey everyone, let's dive into a common snag many of you might be hitting when using docker-borg-client with TrueNAS. Specifically, we're talking about those pesky 404 errors popping up when the system tries to send notifications. Don't worry, we'll break down the issue, why it's happening, and what we can do to fix it. This is a crucial topic for anyone relying on consistent backup notifications, so pay attention! We'll cover everything from the root cause to potential solutions. Let's get started!

The Problem: TrueNAS Notifications Failing with 404 Errors

So, what's the deal? If you're seeing errors like TrueNAS notification failed (HTTP 404): 404: Not Found in your logs after a backup, you're in the right place. This 404 error means the TrueNAS API endpoint the script is trying to reach doesn't exist anymore (or never did in the first place, depending on the TrueNAS version). This is a frustrating situation, especially when you depend on these notifications to keep tabs on your backups. This can be particularly problematic because you might not realize your backups aren't working if your notification system isn't functioning correctly. Let's dig deeper into the problem.

Root Cause Analysis

The root of this problem lies in the docker-borg-client's notification script. The script currently attempts to send notifications via a POST request to the /api/v2.0/alert/oneshot/create endpoint. The main problem is that this API endpoint simply doesn't exist in the TrueNAS API documentation. This discrepancy is the key to understanding the error. The script, specifically in scripts/notify.sh (line 73), is using the following command:

curl -s -w "\n%{http_code}" $curl_opts \
    -X POST "${api_url}/alert/oneshot/create" \
    -H "Authorization: Bearer ${api_key}" \
    -H "Content-Type: application/json" \
    -d "$json_payload"

This script tries to reach a non-existent endpoint. We can check the TrueNAS API documentation to confirm this, which is typically found at https://[your-truenas-ip]/api/docs/current/api_methods_alert.html. You'll notice that the available alert methods are different: alert.dismiss, alert.list, alert.list_categories, alert.list_policies, and alert.restore. There is no alert.oneshot.create or alert.create endpoint. This discrepancy causes the 404 error, because the server can't find the requested resource.

Impact of the Error

While the 404 error is certainly annoying, it's thankfully non-fatal. Backups themselves are not affected. The notify.sh script exits with a code of 0 regardless, so the backup process continues as normal. However, these errors do create extra noise in the container logs, making it harder to spot actual problems. It can lead to you missing other crucial issues amidst the clutter. Therefore, although not critical, the error should be resolved.

Affected Environment

This issue primarily impacts users of docker-borg-client with TrueNAS SCALE. The exact version of TrueNAS SCALE where the issue first appeared is still being determined and will be clarified after gathering user input. Also, if you have NOTIFY_TRUENAS_ENABLED=true in your configuration, you're likely to see these errors.

Workarounds and Solutions

So, what can we do to mitigate this issue? Let's explore some options, ranging from simple workarounds to more comprehensive solutions. This issue has several potential resolutions.

Current Workaround: Disable TrueNAS Notifications

The simplest workaround is to disable TrueNAS notifications altogether. This can be done in two ways. First, you can set NOTIFY_TRUENAS_ENABLED=false. Second, you can simply avoid setting the NOTIFY_TRUENAS_ENABLED variable, because it defaults to false. This stops the script from trying to send notifications to the non-existent endpoint, thereby eliminating the 404 errors. This prevents the error messages from cluttering your logs.

Proposed Solutions

We have several potential solutions to the core issue. Each has its pros and cons, but all aim to provide a more robust and reliable notification system.

Option 1: Research TrueNAS API Changes

This option involves a deep dive into the TrueNAS API to understand how it has evolved. We need to identify which TrueNAS SCALE versions are affected by this issue. If we can find a replacement endpoint for newer versions of TrueNAS, we could update the notify.sh script to use that correct endpoint. This would require thorough testing and documentation to ensure compatibility across different TrueNAS versions. This would allow the system to send notifications to the updated endpoint.

Option 2: Remove TrueNAS-Specific Notifications

Another approach is to completely remove the TrueNAS-specific notification functionality. In this case, we would remove the NOTIFY_TRUENAS_* environment variables and instead, instruct users to configure TrueNAS's built-in alert rules to monitor container logs. TrueNAS already has the ability to alert on container status and logs via its native alerting features. This approach simplifies the docker-borg-client setup but relies on the user configuring the alerts within TrueNAS. This would make the setup more generic but require users to manually configure their own alerts.

Option 3: Implement Generic Webhook Notifications

This is potentially the most versatile solution. Instead of relying on a TrueNAS-specific API, we could replace the TrueNAS integration with generic webhook support. This would allow users to send notifications to various services like Discord, Slack, Gotify, ntfy, or even custom webhooks. This solution offers greater portability, working beyond just TrueNAS environments. For example, users could configure notifications using variables like:

NOTIFY_WEBHOOK_URL=https://discord.com/api/webhooks/...
NOTIFY_WEBHOOK_FORMAT=discord  # or slack, generic, etc.

This design allows for greater flexibility. Webhooks would cover a wider range of notification options. This approach is far more flexible.

Option 4: Multiple Notification Backends

This approach combines the strengths of the previous options. It keeps the TrueNAS support (if a valid endpoint is found), adds webhook support, and potentially email support (using SMTP). This gives users the flexibility to choose their preferred method. Each user could decide which notification method to use. Having multiple backends makes the notification system robust and adaptable.

Community Input and Questions

We'd love your input, guys! Your feedback will help shape the best solution. We need community input to make the best decisions. Let's make this notification system even better.

Questions for You

To help us understand the scope of the problem and prioritize the best solution, please answer the following questions:

  1. What TrueNAS SCALE version are you running? This will help us identify which versions are affected. This will allow us to find the root cause.
  2. Do you actually need/use TrueNAS notifications, or is this feature unused? Knowing whether users rely on this feature will help us prioritize its fix or replacement. It will inform us how important the notifications are.
  3. Would you prefer:
    • Fixed TrueNAS integration (if an endpoint exists)?
    • Generic webhook notifications (Discord, Slack, etc.)?
    • Email notifications?
    • Just disable this feature entirely?

Your answers will guide the development process. Please share your thoughts! Your insights are very valuable.

References

Here are some helpful links to learn more:

Reproduction Steps

If you're encountering this issue, here's how to reproduce it:

  1. Deploy the container with TrueNAS notifications enabled:

    NOTIFY_TRUENAS_ENABLED=true
NOTIFY_TRUENAS_API_URL=https://192.168.x.x/api/v2.0
NOTIFY_TRUENAS_API_KEY=your-api-key

  2. Run a backup (this will trigger a notification).

  3. Check the logs – you'll see the 404 error.

Remember, this is a discussion. The backups should still work. Let us know what you think!