What are all those global session limits about for Ironport ESA's ?

And why it is a good idea to understand what they are used for ...

For those which prefer to read the Cisco admin manual first to get a basic understanding search for : Chapter: Configuring the Gateway to Receive Email .

The Ironport ESA global session limits are defined under Network \ Listeners and are usually something along those lines when you setup a new virtual device:

You usually have your default listener(s) defined, together with all the required settings. The parts which are usually ignored are all the values in the section"Global Settings".

Let me try to cover this section one by one here to give you a better understanding:

Maximum Concurrent Connections

This value defines the maximum as defined across all connections types and to make your life more challenging is usually depended on the ESA model type you are running on. For a larger virtual appliance it is set to 300, some hardware appliances are set between 50-300.

The number of sessions defined here is shared equally between IPv4 and IPv6 pools. Should you only use IPv4 , as most of us currently do, you will start with 300.

Maximum Concurrent TLS Connections

There are two different system wide TLS concurrency limits configured on the appliance: for inbound TLS you can find the maximum number of concurrent TLS connections in this global section and for outbound TLS you can review the delivery settings via CLI \ deliveryconfig > SETUP command.

The number of sessions defined here is shared as well equally between IPv4 and IPv6 pools. Should you only use IPv4 , as most of us currently do, you will again start with 100.

A TLS connection limit exceeded warning is generated from the appliance if the number of connections exceed beyond the current configuration. Connections after the limit exceeds would be treated as soft bounces and hence they would automatically deliver again once connection is available. This is the only moment where you should carefully increase the values specified above.

Injection Counter Reset Period

Ironports manages an internal session table of all connecting IP addresses. The value specified here defines how often this internal caching table should be reset to avoid performance issues.

On systems which are not heavily used, like internal or outbound systems, a value of 60 Min is highly recommended. For systems dealing with inbound traffic this value should be reduced to 15 Min.

Timeout for unsuccessful inbound connections

This value defines the amount of time the Ironport's will allow an unsuccessful inbound connection to remain open before closing it.

Such a connection can be either a SMTP or an ESMTP connection attempt which get interrupted after the initial session negotiation. The timeout value now defines after how many minutes such a session should be terminated by the gateway with Error "421 - Timed out waiting for successful message injection, disconnecting."

Every incoming connection is initially considered unsuccessful until the SMTP message got completed and the sending system did disconnect. This setting is only available for public listeners.

Total time limit for all inbound connections

This value defines the amount of time the Ironport's will allow an inbound connection to remain open before closing it.

There is a a caveat to this setting. After waiting 80% of this maximum connection time the gateway will issue Error "421 - Exceeded allowable connection time, disconnecting." This is done to preserve system resources on the gateways. This setting is only available for public listeners.

While the default value of 15 Min make sense in most of the cases we have run into issues where external service providers sending large email messages to very large user groups in our environment did suddenly fail. On heavily used systems with lot of security controls and message filters in place the timeout value will interrupt the complete delivery of such emails. In such a case the sending system will most likely retry a successful conversation but will always fail, impacting your CPU's load, availability and user satisfaction.

Maximum size of subject

This value defines the maximum size of the subject allowed.

It was my understanding so far that the length of the subject field is defined by the RFC and is set to 255 characters.

I must assume that this value specifies something else and I hope somebody in the "know" from Cisco reading this blog can shed some light into this. Using my good friend Dr. Google and friend #2 Cisco Admin guide did not provide anything else useful.

Why did I cover those settings today ?

We had a working inbound infrastructure which was performing very well for many years. Suddenly a week ago we had some isolated end user complains that certain newsletters from our parent company using O365 would no longer arrive. Splunking the data we could identify 6 of such emails within the last 4 hours. Checking the SMTP and mail logs we could identify ongoing connection attempts from a given sender which got interrupted by our inbound gateways sometime after 20 recipients, sometime 150, sometime just 5.

Mail Log Extract

Fri Aug 14 05:48:55 2020 Info: MID 166977652 ICID 37850720 From: <joao.baio@xxx.com>

Fri Aug 14 05:53:32 2020 Info: MID 166978484 ICID 37851172 From: <joao.baio@xxx.com>

Fri Aug 14 06:00:06 2020 Info: MID 166979684 ICID 37851723 From: <joao.baio@xxx.com>

Fri Aug 14 06:03:57 2020 Info: MID 166980781 ICID 37852077 From: <joao.baio@xxx.com>

Fri Aug 14 06:20:46 2020 Info: MID 166983969 ICID 37853289 From: <joao.baio@xxx.com>

Fri Aug 14 06:26:49 2020 Info: MID 166984856 ICID 37853638 From: <joao.baio@xxx.com>

Fri Aug 14 06:39:22 2020 Info: MID 166986800 ICID 37854289 From: <joao.baio@xxx.com>

You can see the pattern that the sending system keeps trying to deliver the message every few minutes, the same for all other newsletters in question.

Matching SMTP Log Extract

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 address 40.107.13.42 dns host mail-eopbgr130042.outbound.protection.outlook.com sbrs 3.5

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 >> 220 smtp.fmc-na.com ESMTP

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 << EHLO EUR01-HE1-obe.outbound.protection.outlook.com

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 >> 250-smtp.fmc-na.com\r\n250-8BITMIME\r\n250-SIZE 52428800\r\n250 STARTTLS

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 << STARTTLS

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 >> 220 Go ahead with TLS

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 << EHLO EUR01-HE1-obe.outbound.protection.outlook.com

Fri Aug 14 06:39:10 2020 Info: ICID 37854289 >> 250-smtp.fmc-na.com\r\n250-8BITMIME\r\n250 SIZE 52428800

Fri Aug 14 06:39:11 2020 Info: ICID 37854289 << MAIL FROM:<joao.baio@fmc-ag.com> SIZE=57156

Fri Aug 14 06:39:22 2020 Info: ICID 37854289 >> 250 sender <joao.baio@fmc-ag.com> ok

Fri Aug 14 06:39:22 2020 Info: ICID 37854289 << RCPT TO:<Karen.Butler@fmc-na.com>

Fri Aug 14 06:39:33 2020 Info: ICID 37854289 >> 250 recipient <Karen.Butler@fmc-na.com> ok

Fri Aug 14 06:39:33 2020 Info: ICID 37854289 << RCPT TO:<karen.gregory@fmc-na.com>

Fri Aug 14 06:39:45 2020 Info: ICID 37854289 >> 250 recipient <karen.gregory@fmc-na.com> ok

Fri Aug 14 06:39:45 2020 Info: ICID 37854289 << RCPT TO:<Karen.Reid-Cleugh@fmc-na.com>

Fri Aug 14 06:39:56 2020 Info: ICID 37854289 >> 250 recipient <Karen.Reid-Cleugh@fmc-na.com> ok

Fri Aug 14 06:39:56 2020 Info: ICID 37854289 << RCPT TO:<Karsten.Fischer@fmc-na.com>

Fri Aug 14 06:40:08 2020 Info: ICID 37854289 >> 250 recipient <Karsten.Fischer@fmc-na.com> ok

Fri Aug 14 06:40:08 2020 Info: ICID 37854289 << RCPT TO:<Karyn.Leger@fmc-na.com>

Fri Aug 14 06:40:19 2020 Info: ICID 37854289 >> 250 recipient <Karyn.Leger@fmc-na.com> ok

Fri Aug 14 06:40:19 2020 Info: ICID 37854289 << RCPT TO:<Katherine.Dobbs@fmc-na.com>

Fri Aug 14 06:40:30 2020 Info: ICID 37854289 >> 250 recipient <Katherine.Dobbs@fmc-na.com> ok

Fri Aug 14 06:40:31 2020 Info: ICID 37854289 << RCPT TO:<Kathleen.Boccardi@fmc-na.com>

Fri Aug 14 06:40:42 2020 Info: ICID 37854289 >> 250 recipient <Kathleen.Boccardi@fmc-na.com> ok

Fri Aug 14 06:40:42 2020 Info: ICID 37854289 << RCPT TO:<Kathleen.Dwyer@fmc-na.com>

Fri Aug 14 06:40:54 2020 Info: ICID 37854289 >> 250 recipient <Kathleen.Dwyer@fmc-na.com> ok

Fri Aug 14 06:40:54 2020 Info: ICID 37854289 << RCPT TO:<kathleen.oneil@fmc-na.com>

Fri Aug 14 06:41:05 2020 Info: ICID 37854289 >> 250 recipient <kathleen.oneil@fmc-na.com> ok

Fri Aug 14 06:41:05 2020 Info: ICID 37854289 << RCPT TO:<kedron.elliott@fmc-na.com>

Fri Aug 14 06:41:17 2020 Info: ICID 37854289 >> 250 recipient <kedron.elliott@fmc-na.com> ok

Fri Aug 14 06:41:17 2020 Info: ICID 37854289 << RCPT TO:<Keith.Edwards@fmc-na.com>

Fri Aug 14 06:41:29 2020 Info: ICID 37854289 >> 250 recipient <Keith.Edwards@fmc-na.com> ok

Fri Aug 14 06:41:29 2020 Info: ICID 37854289 << RCPT TO:<Lynn.Jensen1@fmc-na.com>

Fri Aug 14 06:41:40 2020 Info: ICID 37854289 >> 250 recipient <Lynn.Jensen1@fmc-na.com> ok

Fri Aug 14 06:41:40 2020 Info: ICID 37854289 << RCPT TO:<kelley.brant@fmc-na.com>

Fri Aug 14 06:41:52 2020 Info: ICID 37854289 >> 250 recipient <kelley.brant@fmc-na.com> ok

Fri Aug 14 06:41:52 2020 Info: ICID 37854289 << RCPT TO:<Kelly.Wootton@fmc-na.com>

Fri Aug 14 06:42:04 2020 Info: ICID 37854289 >> 250 recipient <Kelly.Wootton@fmc-na.com> ok

Fri Aug 14 06:42:04 2020 Info: ICID 37854289 << RCPT TO:<ken.sauls@fmc-na.com>

Fri Aug 14 06:42:14 2020 Info: ICID 37854289 >> 250 recipient <ken.sauls@fmc-na.com> ok

Fri Aug 14 06:42:15 2020 Info: ICID 37854289 << RCPT TO:<Kenneth.Finnegan@fmc-na.com>

Fri Aug 14 06:42:25 2020 Info: ICID 37854289 >> 250 recipient <Kenneth.Finnegan@fmc-na.com>

Fri Aug 14 06:42:26 2020 Info: ICID 37854289 << RCPT TO:<Kenneth.Heiland@fmc-na.com>

Fri Aug 14 06:42:37 2020 Info: ICID 37854289 >> 250 recipient <Kenneth.Heiland@fmc-na.com> o

Fri Aug 14 06:42:37 2020 Info: ICID 37854289 << RCPT TO:<Kenneth.lakin@fmc-na.com>

Fri Aug 14 06:42:49 2020 Info: ICID 37854289 >> 250 recipient <Kenneth.lakin@fmc-na.com> ok

Fri Aug 14 06:42:49 2020 Info: ICID 37854289 << RCPT TO:<Kent.Wanzek@fmc-na.com>

Fri Aug 14 06:43:01 2020 Info: ICID 37854289 >> 250 recipient <Kent.Wanzek@fmc-na.com> ok

Fri Aug 14 06:43:01 2020 Info: ICID 37854289 << RCPT TO:<Kerissa.Adams@fmc-na.com>

Fri Aug 14 06:43:12 2020 Info: ICID 37854289 >> 250 recipient <Kerissa.Adams@fmc-na.com> ok

Fri Aug 14 06:43:12 2020 Info: ICID 37854289 << RCPT TO:<kevin.hickey@fmc-na.com>

Fri Aug 14 06:43:23 2020 Info: ICID 37854289 >> 250 recipient <kevin.hickey@fmc-na.com> ok

Fri Aug 14 06:43:23 2020 Info: ICID 37854289 << RCPT TO:<Kim.Gool@fmc-na.com>

Fri Aug 14 06:43:34 2020 Info: ICID 37854289 >> 250 recipient <Kim.Gool@fmc-na.com> ok

Fri Aug 14 06:43:34 2020 Info: ICID 37854289 << RCPT TO:<Kim.Sonnen@fmc-na.com>

Fri Aug 14 06:43:46 2020 Info: ICID 37854289 >> 250 recipient <Kim.Sonnen@fmc-na.com> ok

Fri Aug 14 06:43:46 2020 Info: ICID 37854289 << RCPT TO:<Kimberlee.Mangan@fmc-na.com>

Fri Aug 14 06:43:57 2020 Info: ICID 37854289 >> 250 recipient <Kimberlee.Mangan@fmc-na.com>

Fri Aug 14 06:43:57 2020 Info: ICID 37854289 << RCPT TO:<Kimberly.Hadley@fmc-na.com>

Fri Aug 14 06:44:09 2020 Info: ICID 37854289 >> 250 recipient <Kimberly.Hadley@fmc-na.com>

Fri Aug 14 06:44:09 2020 Info: ICID 37854289 << RCPT TO:<Kirsten.Howard@fmc-na.com>

Fri Aug 14 06:44:21 2020 Info: ICID 37854289 >> 250 recipient <Kirsten.Howard@fmc-na.com> ok

Fri Aug 14 06:44:21 2020 Info: ICID 37854289 >> 421 Exceeded allowable connection time, disconnecting.

Fri Aug 14 06:44:21 2020 Info: ICID 37854289 close

The common error message we have been getting was "Error 421 - Exceeded allowable connection tile,disconnecting." for all failed attempts.

What's next ? With my understanding of above error messages and the patterns identified this did not make sense at all. The above message was started at 6:39:10 and was aborted at 06:44:21 which is a little more then 5 Minutes. According to our settings and all know rules this does not make sense.

We started modifying some of the global allowable time settings but this did not make any major impact. Since our configuration was on a cluster level we had to break out two of our servers which we decided to further investigate.

Looking further I came across the following Cisco bug report: (https://quickview.cloudapps.cisco.com/quickview/bug/CSCvu00996)

Since we are running 13.5.1-277 I am not sure if this still applies or not but at least it gave me a hint that the global settings or how they are implemented in 13.5.1-277 are to blame.

We then modified all our inbound gateways to the following global settings:

There was still no change in behavior, at this time we had a SEV 1 call with Cisco TAC open to help us troubleshoot this issue as we kept getting more and more tickets. They have also been at a loss and asked us to set up SMTP injection logs for our senders. Since they are O365 hosted and you need to provide a hostname or an IP address - you have a chance of 1 out of 8400 to find the correct one. This did not help further so we started a package trace on the network level trying to isolate incoming port 25 connections from the O365 trusted IP ranges with a hope to get some more insight. To make sure this was working correctly I went ahead and rebooted one of the Ironport virtual gateways. With about 1 Mio connection attempts per day , per gateway we only had very little time to hope to catch a good data sample using firewalls captures.

About 3 days ago we setup a Splunk alert and dashboard to help us troubleshoot how many of those error messages we would get and what the impact would be on mail delivery time. I have been used to see all our inbound gateways showing up in regular intervals but suddenly the rebooted gateway did no longer trigger. Long story short, we rebooted all other gateways as well and the problem disappeared as fast as it came.

What do I think was happening ? I believe that either the global settings are only applied at boot time, which would make sense to me, or there has been a hidden feature aka bug introduced into the latest ESA 13.5.1-277 software release. We will continue to monitor and report back.

Another email mystery solved...