Borgmatic cronjob failed, manual working well

IdrisK

Hi,
i had setup borgmatic according to following documentation:
https://docs.mailcow.email/de/third_party/borgmatic/third_party-borgmatic/#erstellen-sie-dataconfborgmaticetcconfigyaml

I am using as targe hetzner storage box. By executing the backup manual its works, when it is executed by crontab its failed. Do someone know what could be the issue ?

When i am executing following command, i get a sucess backup into the logfiles:

root@mail:/opt/mailcow-dockerized: docker compose exec borgmatic-mailcow /usr/local/bin/borgmatic –log-file /etc/borgmatic.d/borgmatic.log –stats -v 0 2>&1
<
[2024-07-19 17:23:30,372] INFO: Sent Telegram notification.
[2024-07-19 17:23:30,373] INFO: /etc/borgmatic.d/config.yaml: Running command for pre-backup hook
[2024-07-19 17:23:30,375] WARNING: Starting a backup job.
[2024-07-19 17:23:30,375] INFO: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Creating archive
[2024-07-19 17:23:30,376] INFO: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Dumping MySQL databases
[2024-07-19 17:23:30,388] WARNING: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Ignoring configured “read_special” value of false, as true is needed for database hooks.
[2024-07-19 17:23:32,455] ANSWER: Creating archive at “ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup::borgmatic-2024-07-19T17:23:31”
[2024-07-19 17:23:33,278] ANSWER: ——————————————————————————
[2024-07-19 17:23:33,278] ANSWER: Repository: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup
[2024-07-19 17:23:33,280] ANSWER: Archive name: borgmatic-2024-07-19T17:23:31
[2024-07-19 17:23:33,280] ANSWER: Archive fingerprint: 5aff0bfbb3028dd323fd5f7af8f35e331aab72e15d64e49bb4e7bcdd9b52a14c
[2024-07-19 17:23:33,280] ANSWER: Time (start): Fri, 2024-07-19 17:23:32
[2024-07-19 17:23:33,280] ANSWER: Time (end): Fri, 2024-07-19 17:23:33
[2024-07-19 17:23:33,281] ANSWER: Duration: 0.73 seconds
[2024-07-19 17:23:33,281] ANSWER: Number of files: 408
[2024-07-19 17:23:33,281] ANSWER: Utilization of max. archive size: 0%
[2024-07-19 17:23:33,281] ANSWER: ——————————————————————————
[2024-07-19 17:23:33,281] ANSWER: Original size Compressed size Deduplicated size
[2024-07-19 17:23:33,281] ANSWER: This archive: 111.05 MB 23.52 MB 2.24 MB
[2024-07-19 17:23:33,282] ANSWER: All archives: 222.10 MB 47.04 MB 23.33 MB
[2024-07-19 17:23:33,282] ANSWER: Unique chunks Total chunks
[2024-07-19 17:23:33,282] ANSWER: Chunk index: 406 822
[2024-07-19 17:23:33,282] ANSWER: ——————————————————————————
[2024-07-19 17:23:33,368] INFO: /etc/borgmatic.d/config.yaml: Running command for post-backup hook
[2024-07-19 17:23:33,369] WARNING: Backup created.
[2024-07-19 17:23:33,370] INFO: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Pruning archives
[2024-07-19 17:23:34,748] ANSWER: ——————————————————————————
[2024-07-19 17:23:34,751] ANSWER: Original size Compressed size Deduplicated size
[2024-07-19 17:23:34,751] ANSWER: Deleted data: 0 B 0 B 0 B
[2024-07-19 17:23:34,752] ANSWER: All archives: 222.10 MB 47.04 MB 23.33 MB
[2024-07-19 17:23:34,833] ANSWER: Unique chunks Total chunks
[2024-07-19 17:23:34,834] ANSWER: Chunk index: 406 822
[2024-07-19 17:23:34,834] ANSWER: ——————————————————————————
[2024-07-19 17:23:34,836] INFO: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Compacting segments
[2024-07-19 17:23:35,873] INFO: Remote: compaction freed about 1.26 kB repository space.
[2024-07-19 17:23:35,962] INFO: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Running consistency checks
[2024-07-19 17:23:37,395] INFO: Skipping repository check due to configured frequency; 29 days, 23:11:28.845139 until next check (use –force to check anyway)
[2024-07-19 17:23:37,395] INFO: Skipping archives check due to configured frequency; 29 days, 23:11:28.844826 until next check (use –force to check anyway)
[2024-07-19 17:23:37,396] INFO: /etc/borgmatic.d/config.yaml: Pinging Apprise services: tgram
[2024-07-19 17:23:37,497] INFO: Sent Telegram notification.
[2024-07-19 17:23:37,498] WARNING:
[2024-07-19 17:23:37,499] WARNING: summary:
[2024-07-19 17:23:29,490] WARNING: /etc/borgmatic.d/config.yaml: The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.
[2024-07-19 17:23:37,498] INFO: /etc/borgmatic.d/config.yaml: Successfully ran configuration file
`

When the cronjob is executed i get following information into the log-file:

[2024-07-19 17:10:02,033] INFO: /etc/borgmatic.d/config.yaml: Pinging Apprise services: tgram [2024-07-19 17:10:02,566] INFO: Sent Telegram notification. [2024-07-19 17:10:02,567] INFO: /etc/borgmatic.d/config.yaml: Running command for pre-backup hook [2024-07-19 17:10:02,570] WARNING: Starting a backup job. [2024-07-19 17:10:02,571] INFO: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Creating archive [2024-07-19 17:10:02,571] INFO: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Dumping MySQL databases [2024-07-19 17:10:02,586] WARNING: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Ignoring configured "read_special" value of false, as true is needed for database hooks. **[2024-07-19 17:10:04,705] ANSWER: passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND or via BORG_PASSPHRASE_FD is incorrect.** [2024-07-19 17:10:04,756] CRITICAL: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Error running actions for repository [2024-07-19 17:10:04,757] CRITICAL: Command 'borg create --exclude-from /tmp/tmpzea878_z --exclude-caches --compression auto,zstd --one-file-system --read-special --info --stats ssh://u413845-sub2@u413845.your-storagebox.de/./backup::{hostname}-{now} /etc/borgmatic.d/config.yaml /mnt/source/crypt /mnt/source/postfix /mnt/source/redis /mnt/source/rspamd /mnt/source/vmail /root/.borgmatic' returned non-zero exit status 52. [2024-07-19 17:10:04,771] INFO: /etc/borgmatic.d/config.yaml: Running command for on-error hook [2024-07-19 17:10:04,774] ERROR: Error while creating a backup. [2024-07-19 17:10:04,774] INFO: /etc/borgmatic.d/config.yaml: Pinging Apprise services: tgram [2024-07-19 17:10:04,977] INFO: Sent Telegram notification. [2024-07-19 17:10:04,979] CRITICAL: /etc/borgmatic.d/config.yaml: An error occurred [2024-07-19 17:10:04,979] CRITICAL: [2024-07-19 17:10:04,980] CRITICAL: summary: [2024-07-19 17:10:01,518] WARNING: /etc/borgmatic.d/config.yaml: The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release. [2024-07-19 17:10:04,979] CRITICAL: /etc/borgmatic.d/config.yaml: An error occurred [2024-07-19 17:10:04,756] CRITICAL: ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup: Error running actions for repository [2024-07-19 17:10:04,757] CRITICAL: passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND or via BORG_PASSPHRASE_FD is incorrect. [2024-07-19 17:10:04,757] CRITICAL: Command 'borg create --exclude-from /tmp/tmpzea878_z --exclude-caches --compression auto,zstd --one-file-system --read-special --info --stats ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup::{hostname}-{now} /etc/borgmatic.d/config.yaml /mnt/source/crypt /mnt/source/postfix /mnt/source/redis /mnt/source/rspamd /mnt/source/vmail /root/.borgmatic' returned non-zero exit status 52.

Here my Setup:
docker-compose.override.yaml
`

IK 2024

https://docs.mailcow.email/de/third_party/borgmatic/third_party-borgmatic/#einfuhrung

services:
borgmatic-mailcow:
image: ghcr.io/borgmatic-collective/borgmatic
hostname: borgmatic
restart: always
dns: ${IPV4_NETWORK:-172.22.1}.254
volumes:
- vmail-vol-1:/mnt/source/vmail:ro
- crypt-vol-1:/mnt/source/crypt:ro
- redis-vol-1:/mnt/source/redis:ro
- rspamd-vol-1:/mnt/source/rspamd:ro
- postfix-vol-1:/mnt/source/postfix:ro
- mysql-socket-vol-1:/var/run/mysqld/
- borg-config-vol-1:/root/.config/borg
- borg-cache-vol-1:/root/.cache/borg
- ./data/conf/borgmatic/etc:/etc/borgmatic.d:Z
- ./data/conf/borgmatic/ssh:/root/.ssh:Z
environment:
- TZ=${TZ}
- BORG_PASSPHRASE=mysecret
networks:
mailcow-network:
aliases:
- borgmatic

volumes:
borg-cache-vol-1:
borg-config-vol-1:`

/opt/mailcow-dockerized/data/conf/borgmatic/etc/config.yml
`# IK 2024
#https://docs.mailcow.email/de/third_party/borgmatic/third_party-borgmatic/#erstellen-sie-dataconfborgmaticetcconfigyaml
source_directories:
- /mnt/source/vmail
- /mnt/source/crypt
- /mnt/source/redis
- /mnt/source/rspamd
- /mnt/source/postfix

one_file_system: true
repositories:
- ssh://uXXXXX-sub2@uXXXXX.your-storagebox.de/./backup

exclude_patterns:
- ‘/mnt/source/postfix/public/’
- ‘/mnt/source/postfix/private/’
- ‘/mnt/source/rspamd/rspamd.sock’

exclude_caches: true

ssh_command: ssh -p 23
relocated_repo_access_is_ok: true
compression: auto,zstd
archive_name_format: ‘{hostname}-{now}’

keep_daily: 7
keep_weekly: 4
keep_monthly: 12
keep_yearly: 2

mysql_databases:
- name: mailcow
username: mailcow
password: mysecretpw
options: –default-character-set=utf8mb4

#borgmatic-collective/docker-borgmatic?tab=readme-ov-file
#caronc/apprise#productivity-based-notifications
before_backup:
- echo “Starting a backup job.”
after_backup:
- echo “Backup created.”
on_error:
- echo “Error while creating a backup.”

apprise:
states:
- start
- finish
- fail

services:
    - url: tgram://mysecredtoken/mysecretchatid
      label: tgram
start:
    title: ⚙️ Started
    body: Starting backup process.

finish:
    title: ✅ SUCCESS
    body: Backups successfully made.

fail:
    title: ❌ FAILED
    body: Your backups have failed.`

/opt/mailcow-dockerized/data/conf/borgmatic/etc/crontab.txt
`#IK 2024

https://docs.mailcow.email/de/third_party/borgmatic/third_party-borgmatic/#erstellen-sie-dataconfborgmaticetcconfigyaml

10 * * * * PATH=$PATH:/usr/bin:/usr/local/bin /usr/local/bin/borgmatic –log-file /etc/borgmatic.d/borgmatic.log –stats -v 0 2>&1`

DocFraggle

ANSWER: passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND or via BORG_PASSPHRASE_FD is incorrect

That seems to be the problem. I’m not familiar with Borgmatic, but I guess you have to supply the password somehow when running via cronjob

IdrisK

DocFraggle thank you, yes I saw also this error with the BORG_PHASEPHRASE, but I can not find any issue in the configuration.

I checked/copy-paste the Mailcow docs and the borgmatic docs with examples of docker-compose files and config file.

The BORG_PASSPHRASE is only configured in the docker-compose file.

Do someone have an running borgmatic and share the config and the docker-compose file?

hugalafutro

The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.

is the issue imho, i remember having to change all my borgmatic configs everywhere few months back

Change your repositories in data/conf/borgmatic/etc/config.yaml like so (this is copied from my working mailcow config):

repositories:
  - path: ssh://borgwarehouse@[REDACTED]:2223/./[REDACTED]
    label: borgwarehouse

and rebuild the stack. (i don’t think the label is mandatory)

my working config:

service from ‘docker-compose.override.yml’

my ‘data/conf/borgmatic/etc/config.yaml’:

source_directories:
    - /mnt/source/vmail
    - /mnt/source/crypt
    - /mnt/source/redis
    - /mnt/source/rspamd
    - /mnt/source/postfix

repositories:
  - path: ssh://borgwarehouse@[REDACTED]:2223/./[REDACTED]
    label: borgwarehouse

exclude_patterns:
    - '/mnt/source/postfix/public/'
    - '/mnt/source/postfix/private/'
    - '/mnt/source/rspamd/rspamd.sock'

compression: auto,lz4

checks:
  - name: repository
    frequency: 1 week
  - name: archives
    frequency: 1 week

keep_hourly: 24
keep_daily: 7
keep_weekly: 4
keep_monthly: 6

before_backup:
  - echo "Starting a backup job."
after_backup:
  - echo "Backup created."
on_error:
  - echo "Error while creating a backup."

mysql_databases:
    - name: mailcow
      username: mailcow
      password: [REDACTED]
      options: --default-character-set=utf8mb4

i dunno why the docker-compose part didn’t post but here it is:

  borgmatic-mailcow:
    image: b3vis/borgmatic
    hostname: mailcow
    restart: always
    dns: ${IPV4_NETWORK:-172.22.1}.254
    volumes:
      - vmail-vol-1:/mnt/source/vmail:ro
      - crypt-vol-1:/mnt/source/crypt:ro
      - redis-vol-1:/mnt/source/redis:ro,z
      - rspamd-vol-1:/mnt/source/rspamd:ro,z
      - postfix-vol-1:/mnt/source/postfix:ro,z
      - mysql-socket-vol-1:/var/run/mysqld/:z
      - borg-config-vol-1:/root/.config/borg:Z
      - borg-cache-vol-1:/root/.cache/borg:Z
      - ./data/conf/borgmatic/etc:/etc/borgmatic.d:Z
      - ./data/conf/borgmatic/ssh:/root/.ssh:Z
    environment:
      - TZ=${TZ}
      - BORG_PASSPHRASE=[REDACTED]
    networks:
      mailcow-network:
        aliases:
          - borgmatic

volumes:
  borg-cache-vol-1:
  borg-config-vol-1:

IdrisK

hugalafutro Hi, thank you very much for reply and your examples. I change the config file according you comment.
This warning here:
The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.
is not more present after change the config.yml.

But my initial issue with following error by exucutig the cronjob is unfortunatly present and the backup failed again:
ANSWER: passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND or via BORG_PASSPHRASE_FD is incorrect

hugalafutro

try this as your data/conf/borgmatic/etc/crontab.txt (that’s what I have there and “it just works”):
0 * * * * PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1 and restart the borgmatic container, maybe the logfile you have defined there cannot be created or some permission issue.

alternatively enter the shell of the borgmatic container and run your command from the crontab manually and see the output, it might give you more info

I’m not sure why it complains about the borg passphrase, that is defined in the docker-compose as environment variable, and manual backup wouldn’t work either if it wasn’t defined.

just to make 100% sure, if you run docker compose exec borgmatic-mailcow bash -c 'echo "${BORG_PASSPHRASE}"' it returns your borg passphrase as you defined it in docker-compose.override.yml ?

IdrisK

hugalafutro Hi, thanks alot for the hints, i solved the issue.

the main problem was, that i dont know why the random generated Phrase is not working:
BORG_PASSPHRASE=ASLOHZVBIR*U1JV612D*1M!!K^WQM6O2$74PVC1SV1E3#1MO!OVHEM0KIIJ4$P@V

By rebuildig the container (restart is not enogh for changes of the BORG_PASSPHRASE) i got following hint:
root@mail:/opt/mailcow-dockerized# docker compose exec borgmatic-mailcow bash -c 'echo "${BORG_PASSPHRASE}"' WARN[0000] The "P" variable is not set. Defaulting to a blank string. ASLOHZVBIR*U1JV612D*1M!!K^WQM6O2$74PVC1SV1E31MO!OVHEM0KIIJ4@V

I changed the BORG_PASSPHRASE to less complex phrase without special-charactars and now its working well.

Is this normal, that a coplex phrase is not accepted? I did not found any rules inside Borg or Borgmatic documentation.

DocFraggle

I guess the $ was the problem

IdrisK

you are right. and also the #

accolon

IdrisK

Exactly. In YAML, # starts a comment. You could probably use a quoted string to make this work, i.e. BORG_PASSPHRASE='your_pw_with_special_characters'.