A Failed Experiment – And why It's Important to Measure Everything
For some time now I’ve been considering having a weekly post where all I’d do is list interesting blog posts, articles, news, etc. that I found online over the week. In late April I decided to go for it, and so was born the Lazy Friday Reading Assignments. These posts consisted each of a list of links with information around each link explaining why that link was interesting.
In terms of results, I mostly expected that these posts would have much higher clickthroughs than normal (because it’s a list of links to check out). I also thought that the overall traffic, and the subscriptions (Google Feedburner count), to the blog would continue it’s normal growth. And if I was lucky, other sites would learn about this blog and hopefully send additional new traffic.
The results were not what I expected! And this is why it’s important to measure everything.
In the graph above, the blue line represents “Reach”, or as Google defines it, the number of people who have taken action on your content. I had 4 Lazy Friday Reading Assignments on the graph, and all 4 resulted in higher clickthrough days. There are actually about 8 posts in the timeframe of the graph, with the last post being on May 21st (I’ve been too busy over the last while to post). In any case, the end of the graph is the baseline with no posts (about 2+ weeks since my last post), the lower levels of the blue line on the left side are the baseline for normal posts, and the higher numbers are the Lazy Friday Reading Assignment post days. Exactly as I would expect them to be!
The traffic, as measured by unique visitors, did increase over this time, but not much more than my normal growth levels (it’s not displayed on the above graph, I measure unique visitors through other sources). This is more or less what I expected.
But, and this is a big but, the subscription count as measured by Google Feedburner (the green line), was not at all what I expected! Looking again at the graph above, you’ll see that while I was publishing the Lazy Friday Reading Assignments, the green line has dips and overall didn’t really increase. Although the dips don’t exactly correlate to the Lazy Friday Reading Assignment days, I can assure you it’s the first time I’ve ever seen this behavior (there’s also some delays with when Google Feedburner sends out the newsletter by email for those that subscribed by email). Normally when I write a post I’ll see an increase in subscription count on those days (the reverse). I would’ve included an example, but I wasn’t able to find a way to generate a graph from Feedburner for a specific date other than the last 30 days (the full length is too massive).
Although those dips may not look too big, they do represent a decrease of several hundred subscribers. This is significant enough! And more importantly, it’s consistent. During the experiment the subscription rate had absolutely no growth. There’s been more subscription growth with me doing nothing for 2 weeks after the experiment than during the whole month of the experiment!
In other words the experiment was a failure. Therefore the Lazy Friday Reading Assignment is no longer. Although I thought it was a good idea, this didn’t turn out to be the case. Which is why it’s a good idea to measure what you’re doing. Had I not had these metrics, I may have continued for a long time before realizing my error.
Which is why it’s important to measure, measure, and measure again.
PS: Looking at the graph also reminds me I should be posting more consistently.
· June 10th, 2010 · 10:31 am · Permalink
Steph, you are not looking at the whole picture. You are basing success vs failure based on one stat. Granted, it seems like you are using the metric that measures what you are interested in (increase/decrease of feed subscribers). But you probably need to worry more about what your readers like to see.
Now some of your readers may have thought that the friday reading was too off-topic for your blog. I however loved them. I didn’t read every link, but did read quite a few of them.
But I don’t count in your metric, as I don’t subscribe to your blog. I read it from Planet MicroISV. As they aggregate many blogs, I don’t have to check to see who is posting and who isn’t.
· June 13th, 2010 · 4:46 am · Permalink
Hi Michael,
Firstly, thank you for the positive comments about the Lazy Friday Reading Assignments. It’s great to hear from someone who directly appreciated them!!
I absolutely agree that I can’t just look at the reader count, it’s just that it’s my main and most important metric for this blog. Just as sales conversions are my most important metric to LandlordMax.
For example, with LandlordMax if I get a million unique visitors a day but none of them buy, it’s not nearly as good as if I get a hundred visitors and half of them buy.
Similarly, with blogging you want to build your community. As part of building your community, you still do get a lot of people who follow your blog, many by checking the website on a regular basis. However those that have subscribed by RSS have perform an additional action to express their interest. That’s a big step as anyone working in sales conversion will tell you! Not only that, but they are asking you to send them each and every post. That’s a big request! I can assure you it is because increasing your feedburner count is not easy!
Therefore, as much as I do appreciate people such as yourself, and I really really do – posting a comment means you enjoyed the posts enough to take the time to write a comment which in itself says a lot – I do have to take into account what the subscribers are doing.
If they are leaving, then something is amiss that needs to be resolved. Even if it’s something I don’t agree with, I will listen to them. This is the lifeblood of any blogger. Raw traffic is great, but your subscription base is even more important.
Thank you again for the comment, it’s really appreciated!