When it Rains it Pours
Just last Saturday, 6 days ago, I posted about our recent big successes. Not only that, but I stated “as a general philosophy I believe it’s a good thing to share your successes AND failures“. Well today I’m going to share our big misadventures that happened yesterday. It was a brutal day! Luckily we’ve surmounted everything that was thrown at us and we’re back up and running at 100%.
What happened? Two major things.
The night beforehand we just noticed that our SSL certificate expired. The exact same thing happened to us last year, and I vowed to myself not to let it happen again. I still can’t believe I missed it, but I did. Of course, again like last year, our website was still completely safe it’s just that our website wasn’t a “trusted” site so we got that horrible warning in the web browsers.Obviously the first thing we did is purchase a new SSL certificate. It was installed by early afternoon the next day. But for about 12 hours our purchase page wasn’t officially “trusted” even though it was completely safe (don’t get me started on how useless and a cash grab I think this market is). Although I can’t know for the sure the impact this had on our sales, I can’t imagine it helped.
Secondly, and with a much bigger effect, a small upgrade was performed on our server the night before which didn’t go as planned and eventually caused our webserver to shutdown right around lunchtime. It took about an hour to figure out what was going on and to fix the issue. So for an hour we were completely down.
But it doesn’t stop there. Once we were back up and running, we were getting seg faults from our server. What this means is that the webserver wasn’t entirely stable and would error out. Something was still wrong. Although it wasn’t completely shutting down the server, something wasn’t right and it needed to be resolved right away. Without getting into the technical details, we basically worked with our hosting provider to resolve this issue over an hour or two. The eventually solution that worked was to rebuild our webserver.
The downside of rebuilding our webserver is many of our settings and customizations were lost. So for about 2-3 hours our main page looked horrible. For example the header and menu items weren’t showing up at all. You could still purchase the software safely and securily, but you couldn’t access the menus to navigate the website.
Before I continue let me just say that Servint, our managed solutions hosting provider, was great and worked with us the whole way. They knew what they were doing, and if it wasn’t for them, well I don’t really want to think about it… Yes they do charge a premium, but they’re worth it. When we really need them, they were there. I’d also like to especially thank Tommy from Servint, he was very knowledge and helpful. He was great. He saved the day for us. Thank you Tommy!
Getting back to our adventure, or should I say misadventure, Tommy eventually figured out what happened and resolved the issue. Of course it didn’t end there. My other site FindYourWeddingDress.com was also down due to some other technical issues. Thankfully they were also able to resolve this pretty quickly. We had a few more issues, but everything seems to be completely back to normal now. What a day!
Before I leave, let me share one last thing that really surprised me. During the few hours we were down, or mostly down, we still managed to generated some sales! I’m still shocked by this now. I expected we’d get absolutely no sales during this time. The site was completely down for an hour, then the site was barely operational. I wouldn’t have bought anything from us during this time. Yet we still managed to get some sales. What does it mean? I don’t know. All I can say is that I’m still shocked by this.
And there you go. Today I’ve just shared with you our most recent failures and struggles. This week we failed to renew our SSL certificate on time. Not only that, but we also struggled to keep our website up and fully operational for a few hours. Thankfully we rose to the situation and we’re fully back up and running now. What a day. When it rains it pours!
· February 8th, 2008 · 10:01 pm · Permalink
Ha, we were bitten by the SSL cert thing a couple months ago. Arg. I decided to play it a bit safer this time and set up a milestone in Basecamp for next year about the same time. I figure if nothing else it will show up on the calendar in advance.
And I’m glad you had better luck with ServInt. I hosted with them for about a year or so and the downtime was *ridiculous*. It was about 4-10 hours every other week… and they offered a me a whole month’s discount! I quickly discovered that they don’t manage their own servers… it seems like they sub-sub-sublease it out to some other group. Ugh.
And the ironic part is that they’re “based” down the street from me in McLean, VA… Ha.
· February 10th, 2008 · 4:58 pm · Permalink
Hi Keith,
I thought we had setup a Calendar notification but I guess not… That or something happened to it. It’s very annoying. And the way SSL certificates work, you don’t want to buy several years ahead since it’s locked to an IP (i.e. one hosting/co-location/computer/etc.).
You’ve also had an interesting comment with Servint. Are we talking about the same company? They have their own data center as far as I know. And we’ve never had any downtime. And when I talked to the technician, it was the person working on the box (I asked him some tougher questions).
I’m interested in your reply, because it obviously worries me, even if we’d have a great experience so far.
· February 10th, 2008 · 5:13 pm · Permalink
Yeah, that’s the worst part about the SSL.
If it’s the ServInt at servint.com (not Server Intellect), I think so…. I’ve just double checked my emails with them:
* I was using them through a reselling until March ’06 when I transfered to them directly.
* In May they occasionally started double billing me (forgot about that one).
* That July they started sending me “payment not received” notices despite simultaneously getting “thanks for your payment” emails.
* In August the intermittent downtimes started on a near weekly basis and on 29 September, I sent this message:
“Why exactly do I bother to pay my bill? If this was the first time, it would be one thing, but this is ridiculous and is starting to cost me business.”
* By November, the box was so oversold that I couldn’t get into WHM/Cpanel with any reliability and even lost 4 days of CodeSnipers due to the box crashing and hosing the database.
Over Thanksgiving, I moved everything and attempted to shut down the account. They proceeded to invoice me for the following four months for an unused server and finally stopped hounding me in July ’07.
Other than that, they were great. 😉
My “account manager” wasn’t particularly useful…
· February 15th, 2008 · 3:42 pm · Permalink
Hi Keith,
Could it be the reseller? I go through them directly. They have as a stipulation that the reseller is responsible for all their own support, billing, etc.
As well if you go to servint directly they only have VPS accounts as their lowest package, which means you’re guaranteed a minimum set of resources. And that package starts around $50/month…
For me though I’ve only used dedicated servers. I’ve never had any downtime, or network loss. And on a dedicated box, you’ll never have to worry about others. For a company I generally recommend it, it’s worth the piece of mind. If your business wasn’t virtual rent would cost more anyways 😉
I’m still wondering if we’re talking about the same company, or the reseller that was going through servint. I did try to start a hosting company that way, and I found it wasn’t worth the time for me…
· February 15th, 2008 · 3:48 pm · Permalink
I was with a reseller initially and it went to crap once I moved to them directly.
· February 15th, 2008 · 3:54 pm · Permalink
All I can say is wow! I’m surprised and shocked you had such a bad experience. Hopefully I won’t have to encounter that…
· July 3rd, 2008 · 9:57 pm · Permalink
[…] I’ve said time and time again, it’s good to air out all your successes AND failures. And today I’ll be airing out another one of my failures, well more like a mistake. In any […]