Troubleshooting problems with remote firewalls
Click here to read more about SpamStopsHere, the e-mail security company that brings you this blog.
Troubleshooting issues where a firewall that you don’t control is blocking your traffic can be tricky.
We recently had a customer report that some outgoing e-mail never arrived at its destination. This customer had three e-mail filter servers assigned to their outgoing e-mail, and we found that one of the three servers was consistently unable to resolve the recipient domain’s mail exchanger.
A domain’s authoritative domain name server is supposed to answer queries from the public on which mail exchanger to send e-mail for that domain, in the form of a DNS MX record. However, if the DNS query results in failure, the outgoing e-mail message will fail delivery. One of our three servers was having problems resolving a specific recipient’s DNS MX records because the recipient’s domain name server was refusing our server’s connections.
We let the sender, our customer, know that the recipient was blocking our DNS queries from one of the three servers, blocking approximately one-third of the e-mail being sent to them from our client.
Our client contacted the recipient, and the recipient’s network administrator indicated that no DNS queries were being blocked. I therefore spent some time trying to confirm our initial conclusion. I restarted the local caching DNS server that the affected server was using, and then did a lookup using dig, a DNS query tool, which worked. Additional cached queries worked after that. I was unable to find a problem with our DNS caching server. The probability that there was a problem on our end seemed slim because we weren’t having problems delivering e-mail to any other domain on the affected server. The recipient did indicate that they had recently made some DNS changes, so I suspected that maybe there had just been an anomalous temporary issue with cache corruption that was resolved by restarting the caching nameserver. I did note that the affected server was running a newer version of BIND for the DNS caching server.
A couple of days later however, the server was having problems resolving the recipient’s domain again, and 0ur client was on the phone again asking why the problem was still occuring. We ended up enabling debugging on BIND and finding this in the results:
too many timeouts, disabling EDNS0
A quick Google search showed this to be a common problem with firewalls that do protocol normalization with outdated protocol specifications. We found that especially Checkpoint firewalls using SmartDefense running software released in 2006 were liable to block the newer CD flag in DNS queries. We remembered that our affected server was running a newer version of BIND, which in fact was using the CD flag on DNS queries since it was security aware. We then verified that the recipient’s DNS server was blocking queries with the CD flag by executing a query using dig like this:
dig @[recipient DNS server] +cdflag recipientdomain MX
This query resulted in a SERVFAIL, apparently from the recipient’s DNS server. A query without the CD flag argument worked fine. When we asked specifically if the recipient had a Checkpoint firewall, and told them of the known problem, they were able to disable the outdated filtering and stop blocking our DNS queries. I’m glad that we were able to help the recipient find the problem, because it was likely not only affecting our e-mail to them, but others e-mailing them or trying to visit their web site from behind security aware DNS caching servers.
We have had similar issues where two customers reported missing specific e-mail messages that had been blocked by their own firewall without their apparent knowledge. The one thing in common with the messages were that they had DomainKeys in the headers, and both customers had Cisco firewalls. We told the two seperate customers that we suspected that they were filtering the messages or doing some type of rate limiting, and they weren’t aware of any. We found later that older versions of Cisco PIX firewalls using an SMTP proxy errored on messages with DomainKeys in the headers, and the customers were using the SMTP proxy feature.
If you’re filtering traffic coming into your network, please be aware of what types of filtering that you’re doing. Although the side being filtered may sometimes be able to convince you that you’re blocking their traffic, you should probably be aware ahead of time.
Thank you for reading my first entry, and welcome to our information technology and security blog.
April 21st, 2008 at 12:57 pm
Solution:
add the following lines in /etc/named.conf or /var/named/chroot/etc/named.conf:
logging {
category lame-servers {null; };
category edns-disabled { null; };
};
Hope to help …
Antonio Carlos de Lima
June 5th, 2008 at 1:55 am
hi, i recently found that my internet connection was on the blink with sudden disconnections even in spite of having established a connection with the primary DNS. the message read something like a “remote firewall was preventing connection with the primary DNS”. could this be a virus or is there some way to fix this problem. would appreciate any help in this regard. am a newbie in hardware and IT matters.
June 6th, 2008 at 5:31 pm
Roop,
Where did you see these messages? When you say that your Internet connection was on the blink, even though you had connection with the primary DNS, do you mean that even though you could resolve a hostname, you couldn’t connect to it? It’s normal for your computer to cache DNS records, so that even if your Internet connection goes down you may be able to resolve the name for awhile. Either way, it sounds more likely that you’re having problems with yout Internet connection than with a virus. I’d recommend contacting your ISP’s technical support.