We’re going to start with the basics of how the search engines work, and a major component of this is a robot, or spider, which is software that slurps up your site’s text and brings it back to be analyzed by a powerful central “engine.”
This activity is referred to as crawling or spidering. There are lots of different metaphors for how robots work, but we think ants make the best one.
Think of a search engine robot as an explorer ant, leaving the colony with one thought on its mind: Find food.
In this case, the “food” is HTML text, preferably lots of it, and to find it, the ant needs to travel along easy, obstacle-free paths: HTML links.
Following these paths, the ant (search engine robot), with insect-like single-mindedness, carries the food (text) back to its colony and stores it in its anthill (search engine database).
Thousands and thousands of the little guys are exploring and gathering simultaneously all over the Internet.
If a path is absent or blocked, the ant gives up and goes somewhere else. If there’s no food, the ant brings nothing back.
So basically, when you think of a search engine, you really need to think of a database that holds pieces of text that have been gathered from millions of sites all over the web.
What sets that engine in motion? A search. When a web surfer enters the term “grape bubble gum” into the search engine, all of the sites that might be relevant for that term are brought to the forefront.
The search engine sifts through its database for sites containing terms like “grape growers,” “stock market bubble,” and “gum disease.”
It uses a secret formula-a.k.a. search ranking algorithm-to sort the results, and in a fraction of a second, a list of relevant sites, many containing the exact phrase “grape bubble gum,” will be returned in the results page.
There are lots of things that factor into the way robot search engines determine the rank for their main search results.
But, just for a start, in order to be in the running for ranks, you need to provide HTML text to feed the search engines and HTML links as clear paths to the food.
Search Results Are Blended
If you’ve spent much time searching, you have probably noticed that the search engines are not displaying one set of homogeneous results.
Most search engines take the “chef’s salad” approach, displaying a mix of robot results, directory listings, and pay-per-click (PPC) ads.
Your site is probably already represented by most of the types of results we’re about to discuss.
Knowing what each type looks like and where they come from is the first step in being able to influence your own listings in a positive way.
We saw about robot results earlier; here are the other types of results that are available to searchers.
Directories
Unlike those robot search engines, directory listings are often compiled by humans. Whether these humans are editors who work for the search engines or the site owners themselves who write and submit their own listings, it is often easy to tell the difference between a directory and a robot search result.
Take a look at this robot-generated listing from Google. It’s called a snippet-text slurped directly from the web page and spat out into the search results page.
Directories aren’t likely to come out and find you the way robots will; site owners need to submit to them manually.
Sometimes you can purchase a listing, sometimes they’re free, and sometimes you pay for the “privilege” of having your submittal reviewed whether your site is included or not.
While it’s a little extra work to achieve directory listings, at least you can be relatively certain that your submittal will be reviewed by somebody and your site will have a fair chance of getting in.
This is different from the robots, which do not guarantee review or inclusion.
Pay-Per-Click
No matter how blurred the line between unpaid and paid search gets in the search engine results, you, as the SEO expert, will always know the difference.
That’s because, while it’s possible to get listed in robot search engines, meta search engines, and even directories without actually doing anything, you (or someone you delegate) will have to actively implement and carefully manage any pay-per-click (PPC) advertising for your own site.
And, of course, there’s that little matter of the checkbook too.
Here’s how PPC Works: You open an account with a PPC engine. You decide which search terms you want your site to be seen for, and you write your own listing (or often, several different listings) to correspond with your chosen terms.
Every time a searcher clicks on your listing, you pay the PPC engine a fee. You control the amount you want to spend for each click (your bid), and this is a major factor in the placement of your listing.
PPC is the SEO marketing venue over which you have the most control. It offers you a chance to micro-manage your website marketing by being able to target specific messages to specific terms, and even specific geographical locations.
It gives you the opportunity to change your message on a whim, and it provides some of the most conclusive tracking around.
Therefore, while PPC is by no means a requirement for good SEO, it’s an Eternally Attractive Option to have available to you.
Site Feeds
Site feeds have been around for years in one form or another, but their methodology is still morphing.
Available in various forms, they are Eternally Helpful for large or frequently updated sites.
Just as you may use a feed to be notified of your favorite blog or news topic, the search engines use site feeds to sit back and receive information from websites without sending spiders out to constantly gather, gather, gather.
Feeds work well for regularly edited websites such as blogs and news sites (feeding the content of their daily posts) and online sellers (feeding up-to-the-minute commercial information such as product descriptions and prices).
You may also have heard of trusted feed or paid inclusion programs where search engines allow certain “trusted”-and, usually, paying-websites to send the engines regular updates.
Generally these types of listings get thrown into the mix with robot-gathered sites and have to fend for themselves, with no special status in the ranking algorithms.
Meta Search Engines
Some people are comparison shoppers, flitting from store to store to review all the merchandise before making a decision.
For people who like to compare search results, meta search engines make it easy to review listings from different search engines in one screen-no flitting from site to site necessary.
Simply put, meta search engines compile and display results from several search engines and rank them according to their own algorithms.
You can’t use SEO to improve your presence on meta search engines directly; if a meta search engine like Mamma.com or Dogpile.com is using Google results, the way to do better on the meta search engine is to do better on Google.
Algorithms Change
Here’s something that drives people crazy about SEO: You can’t ever be 100 percent sure that what you’re doing will be rewarded with the rank and the listing you want.
This is because the search engines keep their internal ranking mechanism, even the criteria by which the ranking is determined, under wraps.
The algorithm is the formula that a search engine uses to determine its ranks. It’s a way of sifting through a multitude of factors, including keyword repetition and page titles, inbound links, and even the age of the site.
Some elements have more weight, meaning that they are considered to be more important in determining rank, and some have less.
Each search engine uses its own algorithm to determine which results to show and in which order.
And each search engine changes its algorithm from time to time, often without so much as a friendly warning. So, the truth is this: You will never really know exactly how Google works unless you work there.
Imagine if other forms of marketing worked this way! What if you couldn’t rely on alphabetical order in the Yellow Pages anymore?
What if the TV networks chose to air only the bits of your ad that they felt were most important?
What if your billboards were periodically relocated without your consent?
We’re so glad you’ve got a good head on your shoulders because, now that you’re doing SEO, you will have to find a balance between keeping up with the algorithm and keeping your sanity.
Why do the search engines guard their algorithms so closely? Because, first and foremost, they value the searcher’s experience.
If MSN published a guide called Instructions for Ranking #1 on Our Search Engine, of course you’d use it.
And so would everyone else. Then all of the results on MSN would become so manipulated by site owners that relevance would disappear-investment sites could rank high for “grape bubble gum” on purpose-and searchers would drop the engine like a big useless hot potato.
Even without a manual, the little bits of algorithm that people figure out themselves often get so abused that the search engines eventually devalue them.
How do you find the balance between seeking the Eternally Unknowable Algorithm and making sure your SEO efforts are effective?
Matt Cutts, the popular blogger and Google employee who sometimes indulges his SEO-obsessed readers with tantalizing bits of inside information on Google’s algorithm, says, “Most of the right choices in SEO come from asking, What’s the best thing for the user?”
Bringing targeted users to your site is, of course, the point of SEO, and that’s the reason we made you clarify your audience and site goals before we started talking about how the search engines work.
what do you consider to be “Eternal” about SEO. Most agreed on Good HTML titles, good body copy, great content, ensuring that your site doesn’t have roadblocks to crawling-these have worked for nearly a decade
It’s fine to keep an eye on the latest and greatest rumors about exactly how Google works, but don’t go nuts or you will lose focus on what really matters: your site visitors.
Humans Are Smart-Computers Aren’t
Let’s face it: The search engine’s job is not easy. Take a look at your filing cabinet, multiply it by about a billion, and imagine someone throwing you a couple of words and then hovering impatiently behind you, tapping a toe, expecting you to find exactly the right document in the blink of an eye.
Nobody could! We humans are wonderfully intelligent creatures, but we’re just a tad on the slow side when compared to computers.
Unfortunately, machines are still just that: machines. They struggle with ambiguity that even a kindergarten student could handle.
Not to mention misspellings, regional dialects, and punctuation. For search engines to bring back great results, they need to combine the best of both worlds: the speed of the machines and the intelligence of the human mind.
What’s a search engine developer to do? Two things: First, combine results from several sources, as discussed earlier.
This allows the search engines to intertwine the massiveness of the machine-driven system (robot results) with the finesse of the human touch (directory and PPC results).
Second, structure the ranking algorithms to integrate “votes” from human beings. Putting the human touch into a ranking algorithm can be done in a variety of ways, and search engines continue to experiment with solutions.
Counting inbound links from other websites, for example, is a way of measuring how many votes a site has from human-and presumably intelligent-webmasters.
Other ideas have included measuring how many search engine users click through to your site and how long they stay.
Social bookmarking sites and collaborative tagging, even comparing a person’s current and past searches, are forms of artificial intelligence intended to improve the search experience.
But artificial intelligence still has as long way to go. In movies you can say to a computer, “Computer, Rotate and Enhance!” and the computer will somehow manage to turn and un-blur a grainy image from a security camera just the way you need it.
In the real world, we just aren’t there yet. Search engines remain very literal creatures, unable to improvise very much beyond the exact words, even the exact syntax of words, they are given.
Report This Post
