Hi, there.

I’ve just setup mailcow server, over 2 week ago.

My installation not generally standart, because I’ve used s3sf for vmail-vol-1 volume and for sogo-userdata-backup-vol-1 volume.

Because I expect more than 10 TB data.

I installed the s3fs package (apt install s3fs). And setup /etc/fstab file.

#S3FS
s3fs#mail-data /mnt/mail-data fuse _netdev,allow_other,use_path_request_style,url=https://s3.wasabisys.com,passwd_file=/etc/passwd-s3fs 0 0

Next, I maked docker-compose.override.yml file

services:

    dovecot-mailcow:
      volumes:
        - /mnt/mail-data/vmail-vol-1:/var/vmail

    sogo-mailcow:
      volumes:
        - /mnt/mail-data/sogo-userdata-backup-vol-1:/sogo_backup

volumes:
  vmail-vol-1:
    external: true
  sogo-userdata-backup-vol-1:
    external: true

And make common setup by instruction from site .

I’ve catch error, emails is not dragged between directories.
I link this to the fact that moving objects to S3 storage is not possible.

How to catch extended logs?

Advice me please, how to resolve this situation.

P.S.: In SOGo I’ve catch the same error


are you sure the file permissions are correct on your S3 volume? on the docker host it should show rw for userid 5000

    Have something to say?

    Join the community by quickly registering to participate in this discussion. We'd like to see you joining our great moo-community!

    esackbauer
    On the docker host

    root@mailcow:/mnt/mail-data# ls -ldh vmail-vol-1/
    drwxr-xr-x 1 5000 5000 0 Oct 25 18:53 vmail-vol-1/
    root@mailcow:/mnt/mail-data# namei -l vmail-vol-1/
    f: vmail-vol-1/
    drwxr-xr-x 5000 5000 vmail-vol-1

    Inside container mailcowdockerized-dovecot-mailcow-1

    root@mailcow:/mnt/mail-data# docker exec -it mailcowdockerized-dovecot-mailcow-1 sh
    / #  ls -ldh /var/vmail
    drwxr-xr-x 1 vmail vmail 0 Oct 25 18:53 /var/vmail
    / #  ls -ldh /var/vmail
    drwxr-xr-x 1 vmail vmail 0 Oct 25 18:53 /var/vmail

    Also try to create and move file inside container: all looks good.

    /var/vmail # mkdir test1
    /var/vmail # mkdir test2
    /var/vmail # touch test1/file
    /var/vmail # mv /var/vmail/test1/file /var/vmail/test2/ 
    /var/vmail # cd /var/vmail/test2/
    /var/vmail/test2 # ls
    file

    Also try to create and move file inside container: all looks good.

    /var/vmail # mkdir test1
    /var/vmail # mkdir test2
    /var/vmail # touch test1/file
    /var/vmail # mv /var/vmail/test1/file /var/vmail/test2/ 
    /var/vmail # cd /var/vmail/test2/
    /var/vmail/test2 # ls
    file

    esackbauer

    Verify umask on host . Looks good.

    root@mailcow:~# umask
    0022

    Next I cheked logs from mailcowdockerized-dovecot-mailcow-1 container .
    There are a lot of logs ( 50 MB ) . I reproduced the problem. And find сorrect log.

    docker logs mailcowdockerized-dovecot-mailcow-1 > /root/mailcowdockerized-dovecot-mailcow-1.log 
    tail -100 /root/mailcowdockerized-dovecot-mailcow-1.log

    I finded this record

    Nov 14 10:36:39 94878ace0a9b dovecot: imap(andrsharov@company.org)<269061><iSo+IdominjCQ5ji>: Error: Mailbox INBOX: link(/var/vmail/company.org/andrsharov/Maildir/cur/1731567647.M313866P267932.94878ace0a9b,S=3623,W=3702:2,Sa, /var/vmail/company.org/andrsharov/Maildir/.Archive/tmp/1731569798.M878791P269061.94878ace0a9b) failed: Not supported

    Seems then you are out of luck. I am not too deep into Linux file systems, but it seems mailcow uses a feature which this file system is not providing.

    Seems you are right

    root@mailcow:~# docker exec -it mailcowdockerized-dovecot-mailcow-1 sh
    cd /var/vmail 
    /var/vmail # mkdir test1
    /var/vmail # mkdir test2
    
    touch /var/vmail/test1/file
    
    /var/vmail # ls -s /var/vmail/test1/file /var/vmail/test2/file
    ls: cannot access '/var/vmail/test2/file': No such file or directory
    1 /var/vmail/test1/file

    fsspec/s3fs239

    Based on the situation, can you give advice?

    I now have 2.5 TB letters from 405 mailboxes over 10-12 years .

    I expect to increase volume to 10 TB in the next 1.5 years.

    Our users definetely need deep search over all time.

    Can you suggest an architectural solution to how to store this amount of mail?

    Rent a decent hosting with sufficient storage, and use the default file system of e.g. Debian.

    DocFraggle

    Tariffs are available on the Wasabi website Pricing Landing

    6.99$/1 TB/month

    This is most cheapest S3 Storage

    Right now , I’am use some about 150GB , but I should pay for a 1TB minimum.

    Sync from gmail was started perfectly , but after 100GB transferred data (for a 1 week) , the speed dropped catastrophically mailcow community Icon imapsync. Host2 folder INBOX: Could not select: [SERVERBUG]

    [unknown]

    Right now we are using dedicated server: Intel Core i7-8700 3.2 ГГц , 64 ГБ DDR4 NON-ECC ,
    2 × 480 ГБ SSD SATA . For a 59$ per month.

    Proxmox 8 with Virtual Machine mailcow : 8 cores, 32 GB, 120 GB qcow2 disk.

    Gmail want for 6$/month per user * 405 users = 2450 $/per month

    Gmail give for this money 30GB * 405 users = 12,150TB.

    Mailcow : 59$ per month for server + 3TB * 7 $ for S3 Buckets = 80$ per/month

    The difference is obvious .

    From en.wikipedia.org Icon Wikipedia

    :
    “There are various User Mode File System (FUSE)–based file systems for Unix-like operating systems (for example, Linux) that can be used to mount an S3 bucket as a file system. The semantics of the Amazon S3 file system are not that of a POSIX file system, so the file system may not behave entirely as expected.”

    Even if you get the basics working, you might end up in having problems with full text search (feature missing, very slow indexing, slow retrieving, crashing search engine etc)
    As long as you do not have a tiered storage for mails inbuilt into mailcow, I can only say avoid object storage like S3 at all costs.
    Maybe use docs.mailcow.email Icon mailpiler

    on cheap storage for older mails and archiving, this might support slower storage better.
    Edit: just read that mailpiler is only supporting S3 in its paid mailpiler version.

      esackbauer

      Right now I will be work with this installation, until all mail will be imported.

      Our staff doesn’t like the inability to drag and drop emails, but it’s not a big deal.

      We don’t use SOGo web interface (we use it only for filters) , at the most PC ThunderBird are installed. And all work well.

      Of course we can rent server with 12TB HDD ( 2 * 12TB for raid1) . But this does not solve the scalability problem.

      You could also reach out to servercow.de for an offer.

      13 days later

      Decided to move all to regular disks .

      No one is typing