Why SEO is Important
While I was at CNET, traffic was growing steadily on our sites, but we had a rude awakening one particular day.
The “Oh Shit” moment
I remember the morning when my boss said to me:
We’re having a conference call in a few minutes with CNET’s SEO dude, we lost half of our uniques starting last Thursday, why don’t you join in.
The first thing I did is wonder what the hell I did wrong, because in the back of my mind, this problem was inevitably engineering’s fault.
The realization that Google owns you
Chowhound was getting 85% of its traffic from Google. It has around 500k topics, averaging 4.5 posts per topic, each an opportunity to become indexed by Google.
This particular week, Google decided to purge its index of stale data, and in doing so, eliminated the majority of the pages Chowhound had indexed, thus slashing our uniques in half. An interesting side-note was that our pageviews didn’t drop nearly as much, because the core users that were still visiting the site had a much higher pageview per visit ratio than the Google visitors.
What we did wrong
The solution was actually quite simple. On every board, we originally used ‘Next’ and ‘Previous’ links that would paginate you to the corresponding page of topics. Straightforward, but horrible for Google’s bots. They’d click a few of the Next links and say the hell with this and move on.
To give you an idea, the Manhattan board alone has over 800 pages of topics, and when Google is only looking at the first three pages, you have a massive problem.
How we fixed it
Simply put, I wrote the will_paginate plugin. Instead of just having ‘Next’ and ‘Previous’, it also displays a window of page numbers that allows bots the ability to navigate hundreds more pages within one click. Now we’re getting somewhere.
Eureka!
As the traffic began to come back in a serious way, well beyond where it was before the dip, the notion of SEO became quite clear to me. Given a site with tons of great content, instead of building more features, you should be trying to leverage what you already have.
This also makes a lot of sense from a financial standpoint. How long would it take your team to build a new feature for 10k more uniques a day, how long would it take them to add meta tags and a sitemap for 10k more uniques a day. There’s no comparison.
The Checklist
SEO isn’t rocket science (unless you’re trying to push the envelope), you just have to be aware of a few guidelines.
- Unique title tag
- Unique meta description (Yahoo is the only one using meta keywords anymore)
- One h1 tag (the text used in your h2-h4 tags is still very important, though)
- Allow only quality outbound links (eg.
rel="nofollow"on all user-generated links) - Block duplicate content from being indexed with robots.txt
- Structure your CSS so the content can go first
- Consistent navigation that allows bots to get to any page on your site within 2 or 3 clicks
Piece of cake. This discussion can go far deeper than these points, but if you do the stuff above, you’re 90% of the way there. The last 10% is the most time-consuming, so I leave that to you to decide if it’s worth your time.
I know a lot of people think of SEO as a dirty word, and that it conjures up negatives connotations of link-baiting and spam, but that really couldn’t be further from the truth. I’m just going to start calling it FT…Free Traffic.
Speaking of Which
Err Free does more than just Rails, we’d be happy to apply what we’ve learned about SEO to your site. Who doesn’t love Free Traffic?
On Chowhound, the Manhattan page has 832 pages of topics. They aren’t all displayed in the page widget (only 10 or so are), so doesn’t that violate your guidelines?
The window of pages changes depending on which page you’re on, so when you’re on page 400, for example, it’ll show:
1..2..398..399..400..401..822..823
So the bots have the ability to jump around in a much faster manner using the sliding window.
Having said that, the will_paginate plugin could be improved further to handle really large pagination cases like this to make the window look something like:
page 1: 1..101..201..301..401..501..601..701..801 page 2: 2..102..202..302..402..502..602..702..802 and so on
Tweaking the code to do this wouldn’t be terribly difficult, the solution on Chowhound’s larger boards could certainly be improved.
What strikes me about your points is that many of them are good usability suggestions, too.
Structuring CSS so that content comes first is great for people using slow or intermittent connections; access to any page within two or three clicks is widely cited as a way of empoweering visitors to navigate the site; unique page title and single H1 help make bookmarking easy; etc.
The idea that search engine algorithms can directly or indirectly drive the adoption of good usability principles is a fascinating one.
Good tips.
About the H1, I’ve habitually used it for the title of the website (e.g “Bloggs & Co Widgets”) which means it doesn’t vary from page to page. Am I getting that wrong? I also include the website title in the title tag (e.g. “Steel widgets < Blogg & Co Widgets”). Wrong again?
Enough of the philosophy, let me see some code!
Sorry, comments have been closed for this post.