Borgmatic cronjob fails but manually works

semaf

Hey,

I have set up borgmatic according to the documentation. Folder /data/conf/borgmatic/ssh has the private, public keys files id_ed25519 and id_ed25519.pub with chmod 0600.

The folder /data/conf/borgmatic/etc has the files config.yaml and crontab.txt

If I run the manual archiving with docker compose exec borgmatic-mailcow borgmatic -v 2, everything is fine after asking the ssh password.

If the cronjob is running, than it will not do the job. I cant find out what the reason is. Would be happy if somebody can read out from the logs. I can see only the part with permission denied and guess the keys are the reason.

Crontab file exists, using it
time="2024-02-12T23:46:26+01:00" level=info msg="read crontab: /etc/borgmatic.d/crontab.txt"
time="2024-02-12T23:46:26+01:00" level=info msg="crontab is valid"
The variable SUPERCRONIC_EXTRA_FLAGS is empty, starting normally
time="2024-02-12T23:46:26+01:00" level=info msg="read crontab: /etc/borgmatic.d/crontab.txt"
time="2024-02-13T00:02:00+01:00" level=info msg=starting iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Remote: Permission denied, please try again." channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Remote: Permission denied, please try again." channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Connection closed by remote host. Is borg working on the server?" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="rsync: Error running actions for repository" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Command 'borg create --exclude-from /tmp/tmpewpwc6y2 --one-file-system --read-special --stats ssh://xmxmxmx@xmxmxmx.your-storagebox.de:23/./mcsemafat::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic.d/config.yaml /mnt/source /root/.borgmatic' returned non-zero exit status 2." channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="/etc/borgmatic.d/config.yaml: An error occurred" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="summary:" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="/etc/borgmatic.d/config.yaml: An error occurred" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="rsync: Error running actions for repository" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Remote: Permission denied, please try again." channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Remote: Permission denied, please try again." channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Connection closed by remote host. Is borg working on the server?" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Command 'borg create --exclude-from /tmp/tmpewpwc6y2 --one-file-system --read-special --stats ssh://xmxmxmx@xmxmxmx.your-storagebox.de:23/./mcsemafat::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic.d/config.yaml /mnt/source /root/.borgmatic' returned non-zero exit status 2." channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=info msg="Need some help? https://torsion.org/borgmatic/#issues" channel=stdout iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"
time="2024-02-13T00:02:05+01:00" level=error msg="error running command: exit status 1" iteration=0 job.command="PATH=$PATH:/usr/local/bin /usr/local/bin/borgmatic --stats -v 0 2>&1" job.position=0 job.schedule="2 * * * *"

I am using Hetzner to backup and Hetzner has documented to add the ssh keys to the enviroment with $ export BORG_RSH='ssh -i /home/userXY/.ssh/id_ed25519'.
Would it be correct to add it and if yes is that correct $ export BORG_RSH='ssh -i /home/userXY/.ssh/id_ed25519 ?

semaf

Have tested as root and has still getting “permission denied”.

I have tested also to export the key with following error:

root@mc:/opt/mailcow-dockerized# docker compose exec borgmatic-mailcow borg key export --paper xmxmxmx@xmxmxmx.your-storagebox.de:mailcow
Enter passphrase for key '/root/.ssh/id_ed25519':
xmxmxmx@xmxmxmx.your-storagebox.de's password:
Remote: exec request failed on channel 0
Connection closed by remote host. Is borg working on the server?

SSH keys added to Hetzner. SSH file permissions according https://superuser.com/a/215506/537216

semaf

As next what I have tried is ssh into docker and run the command borgmatic --stats -v 0 2>&1. Password for key file has been asked. This has worked and backup has been created. But still the issue with “permission denied” like in the first post log is existing.

DocFraggle

Please check the docs, you have to create the ssh key at data/conf/borgmatic/ssh, that’s a directory inside your mailcow directory!

If you put it in /data/conf/borgmatic/ssh it won’t work

If the path you wrote above was just a typo, I guess the ssh key password is the problem. How exactly should the cronjob enter a password? 😉
Create a key without a password

semaf

DocFraggle If the path you wrote above was just a typo, I guess the ssh key password is the problem. How exactly should the cronjob enter a password? 😉

I could try it for the next years but this was the key. I never thought about it 🙂 Thank you very much for the hint. Maybe this should be set to the documents just not to miss it.