The recent final ruling made clear the following:

“an e-mail recipient cannot be required to pay a fee, provide information other than his or her e-mail address and opt-out preferences, or take any steps other than sending a reply e-mail message or visiting a single Internet Web page to opt out of receiving future e-mail from a sender;”

We are still seeing many unsubscribe links in commerical e-mail from well known companies that take you to a login page where one must log in to make changes to their e-mail subscriptions. These are now illegal for commercial e-mail.

Note that transactional e-mail messages to your clients where you have an existing ongoing relationship do not require an unsubscribe method at all. The law is very specific on what can be considered transactional information, but in general it is only e-mail regarding a transaction that the recipient initiated. Please consult section 316.3 (c) of the final ruling and your attorney for more information.

It was common for domain name speculators, who purchased domain names just to put up pages full of ads, to become domain name registrars or partner with a shady domain name registrar so that they could take advantage of domain tasting to get a leg up on their competition. This resulted in millions of domain names being unavailable every day while they were being tasted, and made it very difficult for the average web site creator to register a good undeveloped domain name.

It also was common for spammers to do the same thing so that they could use new domain names in their spam every day without having to pay for them. It’s well known that spammers have to register a new domain name for a web site link for each new e-mail campaign, because any previous domain name advertised will be blocked by antispam filters. As a result, spammers really only need to have a domain name for a few days.

The policies that allowed domain tasting were originally created so that domain name registrars could get refunds for domain names that customers had failed to pay for, usually due to typos. It wasn’t long before many domain name registrars found that this five day grace period could also earn them a profit by doing domain tasting themselves or by selling the technology to more shady businesses.

ICANN has finally put an end to this practice by only allowing up to ten percent of the domains that a registrar acquires in a given month to be returned without being paid for. This should still protect registrars from customers that don’t pay but prevent the shady practice of domain tasting.

The new proposal discussed at ICANN’s Paris meeting is not currently available online, so I’m trying not to get my hopes up, as it may result in registrars simply paying a $0.20 fee for domains over the ten percent limit. Even if that is the case, this is a great way to set the spammers back financially, even if it’s only $0.20 per domain.

Update: ICANN’s published minutes indicate that the committee proposal was adopted without amendments, which is great news. As a result, pending a comment and implementation period, no top level domain operator may offer any refund to a registrar for any domain names deleted that exceed ten percent of that registrars new registrations in that month or fifty domains, whichever is greater. The proposal to charge $0.20 on all deleted domains is still pending.

Gomes determined that 46% of the messages in his quarantine were false positives, blocking 20% of the e-mail sent to him by readers of his column. Gomes and his colleagues were understandably upset. Unfortunately this is a common experience when using most antispam products. It’s understandable to have a policy against reviewing e-mail that was identified as spam. After all, Gomes’s employer is paying a few pennies a month for an antispam product to save from paying Gomes hundreds of dollars to read spam. If Gomes is going to read the spam, he might as well read it in his inbox, and not use an antispam product. However, it’s very important to use an accurate antispam product for this type of policy to actually pay off.

Using an inaccurate antispam product can actually result in costing a company more money than not using any antispam product. Since the product isn’t accurate enough, Gomes obviously has to actually read his spam as he did before. The difference is that it likely takes him longer to review the spam because it’s no longer in his inbox. He actually has to go somewhere else to read it. Additionally, he’s more likely to miss the legitimate e-mail messages in the pile of spam simply because he may not take the time to go through his quarantine as regularly as he does his inbox. Most organizations will also spend time tracking down missing e-mail and lose business due to the breakdown in communication that this causes.

“The antispam software at my shop is provided by Postini, and we can assume it’s at least as good as anyone else’s…” writes Gomes. This is perhaps one of the most illuminating things that I read in the article. It shows how many Postini users assume that a historically inaccurate product is likely just as good as any other. In reality, all antispam products are not the same.

It’s unfortunately quite common in the industry for companies to offer free desktop antivirus software or some other item of perceived value if you switch to the company’s antispam product. Don’t fall for the gimick. If your organization needs a desktop antivirus product, then find the best desktop antivirus product and a vendor for it. Your organization doesn’t have to choose the same vendor for both products, and doing so is rarely a good idea unless mediocre results are desired.

It’s also quite common for organizations to hire a new IT manager or a new IT services provider and start getting pressured to switch antispam products. Often its because of a manager’s preference for a vendor, the service provider’s preference for only dealing with a single vendor, or the fact that the kickbacks are better with the “preferred” product. You should consider accepting this change if you actually are getting poor accuracy and support with your current antispam product. However, a good manager or services provider doesn’t cause needless disruption simply due to a personal preference or kickbacks.

Many organizations switch antispam products and discover on their own that they’re not all the same. If your organization has made a move in the wrong direction, don’t be afraid to go back. If the move is in the right direction, don’t stop until you have a positive return on investment. A positive ROI is where money invested in something earns more in gains than the money invested. For example a $25 investment in a wheelbarrow can save some organizations thousands of dollars in labor or medical expenses. While there is an initial investment that costs the organization money, the wheelbarrow immediately starts saving labor expenses once it is put to work and it will usually pay for itself after only a few hours. Many small investments in efficiency will save your organization exponentially more money than they cost.

Don’t just look at the price of your anti-spam product, but rather at the cost. A balanced or positive return is what you’re looking for. An accurate antispam product will save your users from having to review the spam that is identified, give your users trust in the product, and save your users actual time each day that they would have otherwise spent reviewing the spam or tracking down missing e-mail. Since you pay your users salaries, this time saved is where paying for an accurate antispam product actually results in a positive ROI. Not using an antispam product will always result in a loss. Using an inaccurate antispam product will just increase the size of the loss.

If a DNS server receives a query that has an answer that is over 512 bytes in size, the DNS server asks the requestor to resend the request over TCP. This is because of the size and reliability limitations of the UDP protocol.

We have had problems with a few recipient e-mail servers hanging when one of our outgoing filtering e-mail servers would connect to it. One significantly unique thing about our outgoing filters is that they have a DNS PTR record that are well over 512 bytes. I always suspected that perhaps the recipient e-mail servers were looking up our PTR records, but were both behind a firewall that didn’t allow DNS connections over TCP and also failed to timeout the PTR record lookup.

It was just a hunch that I had, but few recipients were willing to work with us to determine the problem, until about two weeks ago. One of our clients was trying to send e-mail to a company, and the company’s e-mail server was hanging before even sending a greeting. After notifying our customer of what was occuring, our customer contacted the network administrator at the company to ask that they work with us to resolve the issue. After the network administrator failed to find a problem, he called and blamed the problem on us. We had two SpamStopsHere technical support personnel working with the network administrator for about an hour when the issue was escalated to me.

I was actually on my way out the door at the end of my shift, but I told my colleagues of my earlier theory and suggested that they have the network adminstrator try looking up the PTR record for our outgoing e-mail server from his e-mail server. I just listened in on the conversation long enough to hear him say that the DNS lookup was timing out, as I instantly knew that my theory was correct. I made sure that my colleagues understood my theory and then I left. Later, I was told that the network administrator, for a very large company, was indeed not allowing DNS queries over TCP. Oddly his defense wasn’t that he didn’t know that DNS also uses TCP, but rather that he was doing it for performance reasons.

Regardless of your reasons, if your firewall policy prohibits DNS queries over TCP, you will find that some things won’t work as expected, but only with a few specific sites. The problems may be so rare that you may even tend to blame the problems on the configuration or network of the sites that you can’t connect to. You may find that you are unable to e-mail Yahoo, if for example a lookup of the MX records for Yahoo results in a long answer such as:

yahoo.com.              7196    IN      MX      1 a.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 b.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 c.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 d.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 e.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 f.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 g.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 h.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 i.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 j.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 k.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 l.mx.mail.yahoo.com.
yahoo.com.              7196    IN      MX      1 m.mx.mail.yahoo.com.

In fact, what will be weird is that you will be unable to e-mail people with highly redundant systems, who have enough MX records to deal with a large enterprise level of traffic. Yet, you’ll have no problem e-mailing your uncle’s business that hosts its e-mail server on a cable modem.

This all goes back to my very first blog entry. If you’re going to do filtering, please at least be aware of what you’re filtering. We “problematic sites” will have hard enough time convincing you to unblock our traffic. We shouldn’t also have to convince you that are blocking our traffic.

We here at SpamStopsHere feel for all of the organizations that had all of their servers located in the affected data center. We luckily have data center redundancy. Even if all of the servers that we have hosted at a single provider had been lost, we would have simply been operating near capacity if it had happened during peak hours. We could have quickly established some new servers at alternate data centers to replace the downed servers if needed.

However, the provider had a whole other problem that did force us to operate at almost 100% capacity for one afternoon. Our 15 servers at the provider’s other data centers that still had power lost their DNS PTR records. A DNS PTR record is a DNS record that resolves an IP address to a hostname. Many recipient e-mail servers would have rejected our e-mail had we attempted to deliver e-mail from these servers, so we decided to take the 15 servers offline when this occured Wednesday night on June 4, 2008. For more information, on this, please see my previous blog article.

In order to start implementing a contingency plan, such as replacing 15 servers, one must first determine how much time and money it will take to implement the contingency and weigh that against the estimated time of resolution of the orginal problem to determine whether one should switch one’s efforts to the contingency plan and scrap the original plan. Unfortunately we had been consistenly provided with widely inaccurate estimates, which had affected our determination on whether to implement contingencies. After 36 hours and many false assurances that the PTR records problem would be resolved shortly, we decided to implement our contingency plans so as not to have another day of peak usage that brought us near peak capacity.

I think that if I’ve learned one thing in this it is that contingency plans should immediately be implemented if the estimated time of resolution is being provided by and is in the hands of a third party. I think the entire industry has likely learned the same thing. It’s likely that many organizations have also learned to weigh the costs of data center redundancy over the costs of downtime. It’s unfortunate that many organizations simply did not consider a disaster plan that involved the loss of a data center and would have gladly paid the cost had they considered the possibility.

This information, Jackson says, was gleaned from a report from KnujOn, which is “No Junk” spelled backwards. KnuJon is an organization that supposedly targets spammer owned web sites. Claims on the KnujOn site say that they’ve shut down over 50 thousand junk mail sites. I don’t see any information on the site that says how they’ve accomplished this. I was also unable to locate any such report on the KnujOn web site. KnujOn does have a report that lists their top 10 worst registrars in terms of spam advertised sites.

One of the registrars listed is Dynamic Dolphin, which the KunjOn site owner links back to a possible owner of Scott Richter, the most infamous self admitted spammer in the spam industry’s history. From the look of their web site, Dynamic Dolphin is using the services of Directi, an Indian company which helps businesses get started with becoming their own domain name registrar through their LogicBoxes brand. KnunOn claims that over 17 percent of all of the domains registered by Dynamic Dolphin have been identified as being advertised in spam.

While the information seems sensational, it’s nothing new, but Jackson reports that ICANN is taking action by putting these companies on alert. We’ll see if that has any real world effects.

When the CAN-SPAM Act was signed by the president in 2003, there were many provisions that still needed to be worked out. As a result, I subscribed to the Federal Register mailing list and every day I would look for any updates to the CAN-SPAM Act to be released. On April 19, 2004 when the Federal Trade Commission finally ruled on the “SEXUALLY-EXPLICIT” tag that all sexually explicit commercial e-mail messages must have to comply with the CAN-SPAM Act, I was probably one of the first ones in the industry to find out about the decision on what this tag would be. The same day,  SpamStopsHere created an optional filter that users could subscribe to for e-mail messages that contained this tag. All users were subscribed to this filter by default.

In January of 2005, the FTC clarified what defines the primary purpose of an e-mail message, providing methods to determine whether an e-mail to someone you have an existing business relationship with is really a transactional or relationship e-mail or whether it should be considered primarily commercial in purpose.

It’s been years since then, and updates to the law have been super slow. I continue to wait for the FTC to release the method that will be used to label e-mail messages with a ”Clear and conspicuous identification that the message is an advertisement or solicitation”. It’s been theorized that the “ADV:” tag will be used, as it was originally propsed by the Internet Engineering Task Force awhile before CAN-SPAM.

On May 12, 2008 we finally got some more rules on implementing the CAN-SPAM Act last week though, and I missed it. I’d stopped reading the Federal Register, giving up on the FTC to make these decisions any time soon. Here it is a week later, and I had to read about it in the news. The four rule provisions are interesting ones:

(1) an e-mail recipient cannot be required to pay a fee, provide information other than his or her e-mail address and opt-out preferences, or take any steps other than sending a reply e-mail message or visiting a single Internet Web page to opt out of receiving future e-mail from a sender; (2) the definition of “sender” was modified to make it easier to determine which of multiple parties advertising in a single e-mail message is responsible for complying with the act’s opt-out requirements; (3) a “sender” of commercial e-mail can include an accurately-registered post office box or private mailbox established under United States Postal Service regulations to satisfy the act’s requirement that a commercial e-mail display a “valid physical postal address”; and (4) a definition of the term “person” was added to clarify that CAN-SPAM’s obligations are not limited to natural persons.

The most significant change I see is the first one. The unsubscribe links that make you log in to edit your e-mail preferences will now be illegal. I always found those completely annoying, and I always recommend against senders using them. It’s also interesting to note that a U.S. Post Office box will be considered a valid mailing address. It seems that some of the early court decisions had actually ruled the other way, and it’s great that the FTC has clarified this. They seem to indicate that PMBs, or Private Mail Boxes are still not allowed.

Another interesting thing in the rulings is related to whether an “invite a friend” or “e-mail a friend” link falls under the requirements of the CAN-SPAM Act. The rules now clarify that if you compensate someone with anything of perceived value in exchange for using the “e-mail a friend” form, or if you even entice use of it, that usage falls under the CAN-SPAM Act. As a result, any e-mail from an “e-mail a friend” form likely needs to include a mailing address and an unsubscribe method, and shouldn’t be generated to those who have previously unsubscribed.

Of course, you should consult your attorney, as I can’t provide legal advice. These are only my recommendations, and I’m still waiting for that label to identify all commercial e-mail. I hope that the FTC decides on that sometime this century.

Have you ever gotten e-mail that has a subject line of “??? ????? ???” or is in a language you cannot understand and wondered why someone would send e-mails in gibberish? Well, the e-mail isn’t in gibberish, it’s just that it’s encoded in a format your computer doesn’t understand.

Character sets allow the e-mail client, Web browser, or really any program that displays text, to know which character the author intended amongst the thousands of characters across all the languages of the world.

Why filter based on them?

If you’ve ever used a voicemail system, you’ve probably run into the “Press one for English, press two for Klingon” etc. Filtering on character set is no different than selecting the language you speak. Why receive e-mails in Russian if you’ve never taken Russian 101? As a business, if you don’t have any customers or colleagues that speak a specific language and your employees don’t anticipate receiving e-mails in any other languages, why even deliver the messages to the inbox? They can’t be understood, and will only waste resources processing, storing and having your users read them.

How do I filter based on character sets?

This will differ based on your particular infrastructure, but all mail servers and most e-mail clients I’ve run across have ways to filter or sort e-mail. You can simply look for the character sets you don’t want and then either move them to a folder for review or reject the message outright. We recommend rejecting the e-mail so that the sender knows that the message did not arrive.

If you happen to have an e-mail gateway or a hosted antispam filter like SpamStopsHere, you can create the rules there saving even more load on your mail server itself. 

One thing to note, this is NOT a spam filter. It is simply a policy that you are putting in place saying that you do not want to receive messages claiming to be in a character set you do not understand. This is no different than some mail server admins blocking .zip files from entering their network to prevent virus outbreaks. Not all .zips have viruses, and not all viruses are in .zips, but it’s common enough that many admins will block them regardless.

The common character sets you may want to look for and the ones we offer pre-built are

Russian/Cyrillic (Codes: windows-1251, iso-8859-5, ibm866, koi8-r)

Chinese (Codes: GB2312, GB2313, hz-gb-2312, big-5, ISO-2022-CN)

Korean (Codes: euc-kr, iso-2022-kr, ks_c_5601)

Hebrew (Codes: windows-1255, iso-8859-8)

Japanese (iso-2022-jp, x-euc, Shift_JIS)

It’s good to note if you’re creating rules for this that both the headers and the subject can include the character set, so you’ll want to watch for them in both places.

For more information on character sets you can see RFC 2045, the Unicode homepage or the Wikipedia article on character encoding.

However, an Associated Press article by Lara Jakes Jordan, entitled “38 charged in international phishing scheme“, reports that some phishing scammers were extradited from Bulgaria to face charges here in U.S. federal court. More information is in the FBI press release.

I’m not sure what kind of signal this is sending to the spammers. If they just concentrate on illegally sending spam for counterfeit drugs we’ll look the other way, but if they start stealing money from those they send the spam to, then they should be afraid? I don’t see a big distinction between stealing the money from a spam victim and having them give their money willingly using deception and misleading advertisements. Although perhaps we’ll start seeing “regular” spammers being targeted by the U.S. justice system as well.

Additionally, perhaps Attorney General Mukasey’s “Law Enforcement Stategy to Combat International Organized Crime” will be targeting international spam gangs, if they’re actually controlled by organized crime, as many anti-spam organizations theorize. When you rule a venture illegal that makes billions annualy, you’re bound to drive it into the hands of organized crime. The CAN-SPAM Act of 2003 and various state laws over the years have driven spam from the hands of scummy entrepreneurs to organized crime. If they were still filming the televsion series “The Sopranos”, I can imagine an episode where Tony Soprano hires some Romanian teens to set up his spam shop.

Either way, stopping phishers and deterring the crime internationally is a step in the right direction at making our inboxes safe and spam free. I hope that this isn’t the only result that we get from the attorney general’s recent organized crime initiative. Although I actually hope that the spammers retire rather than come stay in our jails.

Does the domain name in your e-mail sender address resolve?

After your outgoing e-mail server introduces itself to the recipient e-mail server, it starts its first message by first specifying the sender’s e-mail address with the “MAIL” command for the simple mail transport protocol, also known as SMTP.

MAIL FROM: <sender@example.com>

If the e-mail address contains a domain name, the domain name must be a valid fully-qualified domain name. Which means that it must either have domain name service “MX” records or resolve to an “A” record. Domain name service, also known as DNS is the system that resolves mnemonic names to their numeric Internet addresses.

The most common cause for this not to resolve is for the misconfiguration of the e-mail software used by the person sending the e-mail message, also known as the e-mail client. Sometimes the problem is also on the sending e-mail server. Typos are often mistakenly entered, or someone simply enters their e-mail address as “bob” instead of “bob@example.com”. Although “bob” may be a valid local name for e-mail, the sending e-mail server may add it’s own hostname to the e-mail address so that it reads “bob@internal.smtp.example.com” where “internal.smtp.example.com” is just your e-mail server’s internal name and shouldn’t be used as a fully-qualified domain name when sending e-mail out to the Internet.

You can check whether the domain in your e-mail address has DNS MX records by address using the dig command on UNIX/Linux. Below is an example using the example domain name of of “example.com”.

dig smtp.example.com MX

If you have working MX records, you’ll get a result similar to the following.

ANSWER SECTION:
example.com.           600     IN      MX      0 smtp.example.com.

If you don’t have MX records, you usually won’t get back an “answer”.

To do this query on Windows, you can use the “nslookup” command from a command prompt window:

nslookup -query=MX example.com

If your domain doesn’t have MX records, it must then have an A record.

You can check whether your e-mail server’s domain name resolves to an A record by using the “host” command on UNIX/Linux. Below is an example using the example domain name of “example.com”.

host example.com

If you have a working A record, you’ll get a result similar to the following.

example.com has address 1.2.3.4

If you don’t have an A record, you’ll get back a result similar to the following:

Host example.com not found: 3(NXDOMAIN)

To do this search on Windows, you can use the “nslookup” command from a command prompt window:

nslookup example.com

If you have a working A record, you’ll get back a result similar to the following:

Name: example.com
Address: 1.2.3.4

If you don’t have a working A record, you’ll get back a result like this:

*** UnKnown can't find example.com: Non-existent domain

Section 3.6 of RFC 2821 which is on the standards track to obsolete STD 10 requires that only resolvable, fully-qualified, domain names are permitted when domains are used in SMTP. The same RFC indicates that you can use an IP address instead of a domain name, or that you can leave the domain name off for a local name.

Many recipient e-mail servers will reject your e-mail for policy reasons if you don’t use a domain name in the in the MAIL command that has MX records or an A record, and will often cite an RFC 2821 violation. Often times, this can cause delays if the recipient’s DNS resolver is temporarily unable to verify the domain name, or if the sender’s domain name service is temporarily not accepting queries.

You may need to make some changes in DNS, on your e-mail client, or even change your e-mail server’s name in order to meet this requirement.

Other articles in this series:

DNS problems affecting e-mail delivery : PTR records

DNS problems affecting e-mail delivery: mailserver name