I have 110 mailboxes and 110 syncjobs defined. There is 4 syncjobs not working: * 3 of them have this status EXIT_CONNECTION_FAILURE_HOST2 and share the same problem in the log "Host2: sslcheck did not detected open ssl port 993. Will use standard 143 port." * the last one has no exit status The strange thing is the fact that it is working for the other mailbaoxes, and for these one, some messages are imported but it fails, restart, reimport some messages, fail again and restart. [upl-image-preview url=https://community.mailcow.email/assets/files/2024-09-25/1727243689-101635-capture-decran-du-2024-09-25-13-41-14.png]

@"DocFraggle"#p18628 I, it is a question of course we thought about. Right now we have: * some functions using the api. The goal is to generate from our own tables the mailboxes and the syncjobs: we executed these functions on 3 different servers and it saved us a lot of time. Next version of mailcow I will check the api to confirm it is ok, or need changes * some functions using the api, from postgres, to present the result as tables (you know, relations between tables is convenient), but I don't want to use them because * with foreign data wrapper in postgres I can request "directly" MySQL database. If the tables change, I drop the schema and import it again : 2 sql commands.

Syncjobs working... Not all the time!

renaud

I have 110 mailboxes and 110 syncjobs defined. There is 4 syncjobs not working:

3 of them have this status EXIT_CONNECTION_FAILURE_HOST2 and share the same problem in the log “Host2: sslcheck did not detected open ssl port 993. Will use standard 143 port.”
the last one has no exit status

The strange thing is the fact that it is working for the other mailbaoxes, and for these one, some messages are imported but it fails, restart, reimport some messages, fail again and restart.

DocFraggle

Did you check the remote server’s logs for possible errors? That’s if you can access them, of course.

Nice Grafana board btw, did you build it yourself? I’ve been too lazy to build one…

renaud

The problem is not the remote access but the access on the “localhost”

Host2: connecting and login on host2 [localhost] port [143] with user [mat@xxx.org*qsdqsd@mailcow.local]
Host2 failure: can not open imap connection on host2

About Grafana, I am managing observability for the company and I want one pair of googles for everything.

This is why:

first I coded api get to grab information from api, inside Postgres with plpython
I also coded api put to create automatically (from tables) the mailboxes and the syncjobs
then managed to have a Foreign Data Wrapper from MySQL to Postgres in order to have only one datasource for Grafana. I have around 200 FDW in my Postgres and it simplifies ² all the tasks
I got few difficulties to import the MySQL schema but it is doable.

Next I plan to add some alarms to inform us about quotas.

Maybe mailcow dockerized team should think about grafana in the package.

DocFraggle

renaud I also coded api put to create automatically (from tables) the mailboxes and the syncjobs

Are you aware that there is a great API already in place? Just add /api to your mailcow URL and check the Swagger page (or check the one from the mailcow demo page: https://demo.mailcow.email/api/ )

Back to your problem: is it possible that it’s a resource problem? 110 sync jobs are a lot, did you try to disable the other 106 jobs and run only the 4 jobs in question?

DocFraggle

renaud I also coded api put to create automatically (from tables) the mailboxes and the syncjobs

That’s true, but I was relating to this. You can do that quite easily with the mailcow API and won’t run into problems if the mailcow team decides to move tables or rename fields in the future

renaud

DocFraggle

Yes, I already wrote some functions for API requests but for Grafana I prefer to use a unique postgres datasource instead of hundreds datasources.

DocFraggle

I found this post: https://community.mailcow.email/d/303-imapsync-log-file-and-parallel-sync/4

You should be able to see the logs of imapsync, maybe this helps

renaud

Yes, thanks for the link, I will investigate !
There some questions remaining for me:

why the syncjobs are not executed in parallel
where is the list, it takes some times to start, the log is not available during the processes but at the end
I would like to execute / interrupt by myself

I want to add something. At the moment you convert the data from api into tables, you can organize relations and history. What was the status of this point 7 days ago. You don’t have this information with the API just offering a photo of the system right now.

With history you can draw tendencies.

renaud

DocFraggle I, it is a question of course we thought about.

Right now we have:

some functions using the api. The goal is to generate from our own tables the mailboxes and the syncjobs: we executed these functions on 3 different servers and it saved us a lot of time. Next version of mailcow I will check the api to confirm it is ok, or need changes
some functions using the api, from postgres, to present the result as tables (you know, relations between tables is convenient), but I don’t want to use them because
with foreign data wrapper in postgres I can request “directly” MySQL database. If the tables change, I drop the schema and import it again : 2 sql commands.