I am having somewhat of a discussion with the developers/support engineers of the forums software I use.

I am experiencing slow POST when someone replies to a message on my forums. After searching for days, it turns out, when someone replies to a forum post, the forums software waits for the topic notification emails to be send out before completing the POST, which then slows down the reply for about 5 seconds.

So I changed the mailer in the forum email settings to PHP. All goes blazing fast. Then change back to smtp on mailcow and it is slow again. After some searching it turns out that mailcow uses smtp transaction delays for preventing spam and whitelists the sender for some time after the first attempt.

Invison Community support says “get another provider/mailserver” I say: “Mail should be queued. POSTS should not rely on on an external system or network. There is no reason to wait for email to be send.”

My questions:

  • Who is right in this case. What is best practice?
  • How to resolve the issue without getting another provider/mailserver or forum software 😉

For reference my question and their response in the hidden support forums:

Hi,

On our forums we are experiencing extremely slow POST times (> 5sec) on our otherwise crazy fast server.

It took me very long to find the culprit as behaviour varied from topic to topic / time to time.

As I ruled out all possible server issues, I then focused on the actual differences in the topcis and then found out only topics that have followers took long. Most new topcis or topics I posted and then reply to myself did not.

Then I looked into the email settings, tried PHP as mailer (I use normally use SMTP) and then all posts where practically instant.

Turns out when posting messages, the forum software seems to wait for the email notifications being send. Seems to me this should be a separate (background) process or queue, as most SMTP servers have transaction delays due to spam prevention.

SMTP Transaction Delays
As it turns out, one of the more effective ways of stopping spam is by imposing transaction delays during an inbound SMTP dialogue. This is a primitive form of teergrubing, see: Lutz Donnerhacke: Teergrubing FAQ

iks-jena.de
Lutz Donnerhacke: Teergrubing FAQ
Lutz DonnerhackeLutz Donnerhackezur ck

Most spam and nearly all e-mail borne virii are delivered directly to your server by way of specialized SMTP client software, optimized for sending out large amounts of mail in a very short time. Such clients are commonly known as Ratware.

In order to accomplish this task, ratware authors commonly take a few shortcuts that, ahem, “diverge” a bit from the RFC 2821 specification. One of the intrinsic traits of ratware is that it is notoriously impatient, especially with slow-responding mail servers. They may issue the HELO or EHLO command before the server has presented the initial SMTP banner, and/or try to pipeline several SMTP commands before the server has advertised the PIPELINING capability.

That makes SMTP unusable as a mailer.

Could IPS please clarify why the design seems to work as outlined and there is no queue or separate proces? Are you open to change this?

From my point of view you are right, its stupid to wait on POST for an external service.

However you need to configure your rate limits properly, and I guess with some mail providers you can’t do that, hence they did this “workaround”.

So, what are your rate limits? Did you add the forum server to the “forwarding hosts”?

Have something to say?

Join the community by quickly registering to participate in this discussion. We'd like to see you joining our great moo-community!

Hi,

Thanks for responding.
yes, I have the hostname/ip/subnet of the server that runs the forum software in the forwarding hosts of mailcow.

No rate limits set for the specific domain:

And you disabled the spam filter with the forwarding host?

And strange is that the rate limits do not show “disabled” but instead an empty box. With my domain there is “disabled” which appears when I configure 0 msgs/second.

Yes, I did:

So I need to set it to 0? I do not get ‘ disabled’ then.
I think it was set to 0. When I set it to 1, the ‘1’ shows, when setting it to -1, ‘-1’ shows, but when setting it to 0, I get:

well somehow I do….. 😉

16 days later

Hi, I am still struggling with this issue.

I have followed all advise above and on most topics, posting a reply is now speedy, except for when a member is subscribed to a topic that has a non-existing e-mail address in their profile. The forum software then tries to send messages out to subscribed members, until one comes along that is non-existing and wait’s for the mailserver to respond, which takes 7+ seconds.

Also when I mail to a non-existing e-mail address (for example a non-existing hotmail address) from my Mac OS mailclient, using the same e-mail account as used on my forums, I have the same issue. From pressing send, to the mail actually being sent to the receiving mailserver, takes about 7-8 seconds.

Why does this happen and how to solve it?

The supplier of the forum software literally responded:

As this is currently intended, if you would like to see this change, I would advise posting in our Feedback section.
However, please keep in mind that it is abnormal what is happening here with your SMTP server. Injection of mail shouldn’t cause a delay in the application sending it, even if it’s done via background tasks, this could create issues. Especially, if you’re sending a large batch of email notifications. Even in a normal case of say 15 email notifications, that normally isn’t a problem for the background scheduler to get through in a single run. However, you would hit a timeout on a default PHP configuration if each email takes say 2+ seconds. Therefore, you may need 2 or more runs to get through that, which would then delay other tasks as well.

Thanks,

Michel

No one is typing