[Update] Debrief – click here for the full debrief
[Update: 16th March]
We’re pleased to report that systems are back to normal and delivery rates averaging at just over 98.4%.
From early Monday afternoon we successfully restored high delivery rates across the system. Our team have worked through the night to sort out any resend requests and enable even more additional delivery channels to provide additional resilience in the future.
Thanks for bearing with us, if you’ve been trying to get an email campaign out and experienced delivery problems we’re really sorry for the inconvenience. We’ll post a full debrief shortly. If you’ve experienced delivery issues over the last few days we’ll be in touch with you shortly to arrange a credit on your account.
[Update: 15th March]
While I can report that most of our delivery troubles have been rectified, sadly despite best efforts over this weekend there is still one major service that is waiting to hear from our ISP on the matter, so we hope to have that ball rolling first thing in the morning.
In order to continue to be open about these kind of problems, I must get a little more technical to explain the situation. We operate a variety of email delivery channels, and group our customers into these depending on their reputation and industry. Even in the world of permission marketing there will always be complaints, often because a list operator chooses to resurrect an old list or sends unexpected content. So if a channel is blacklisted with a particular operator, we will check out the problem, apply for a delisting, and move innocent affected clients quickly to unaffected channels.
In this instance a major spam listing site received 3 complaints, did their homework, and blacklisted our entire main block of IP addresses – unusual. Given that we send millions of emails a day and have already terminated the client in question we had hoped to be delisted promptly, but sadly that has not yet happened.
We have definitely learnt a painful lesson. We have put stricter import checks in place even for established customers, and will move traffic elsewhere quicker if this happens again.
Getting your campaigns re-sent
Most campaigns sent since Friday have suffered a 20% delivery problem. All bounces since then will be reversed and everyone who has already contacted the support team and has asked to have their campaign re-sent to those bounced addresses will have this done as soon as we’re confident of success. We would offer this automatically but we’re aware that some campaigns are very time specific.
Thanks for bearing with us!
[Update: 10am 14th March] Deliverability has improved significantly but we’re still waiting on delisting from one anti-spam service.
Houston, we have a problem.
Some clients may find higher than normal bounce volumes for email campaigns that are dispatched today. This is because one of our long standing clients broke all the rules yesterday: they bought a big list, uploaded it (ignoring the warning notice about permission only addresses), and sent those addresses email they didn’t want.
Predictable results followed: lots of unsubscriptions, lots of complaints. Lots of complaints leads to our email reputation being damaged, so we have delivery problems. We are, of course, working very hard to fix this up straight away. All hands most certainly on deck!
This is exactly the kind of situation from which we all want to be protected against, and we do work hard to stop it happening. In this case though we just can’t believe what this client did. If you feel upset because we have put you in a position where your ethical marketing has been affected because we’ve taken on an unethical client I’ve got every sympathy for you. Bear in mind though that we’re not hiding anything from you here: we acknowledge this is a bump in the road and we’re fixing it right now.
What are we going to do about it?
Needless to say, the guilty party is no longer a client. It may seem crazy for us to throw income away in this day and age, but we believe in what we do, and we do permission marketing.
This does of course mean we will be implementing much stronger checks on imported data. Starting straight away we will be imposing manual data checks on any imports that are more than trivial in size. We’re really sorry to be seen to be policing your actions but in this one case we must take responsibility for ensuring that other clients don’t suffer as the result of a single user’s ignorance.
We hope to restore repuation very soon, but if we can’t, we will also be moving affected clients to our other email delivery routes to avoid this reputation problem. Today may be a little bumpy, but we’ll be back on track soon.