We use search engines every day, but how much do you actually know about them? If you’re an SEO expert, it’s all the more beneficial to understand how search engines work.
Search engines can be categorized by three separate entities: the crawler or “spider” software, indexing software, and query software. First, lets go over the crawler software.
What the heck are these “spiders” and why are the crawling my site?
Spiders is just the term that’s been coined for software scripts developed for the purpose of scanning the 60 trillion+ pages that now exist on the internet. No need to kill ’em with fire, let them do their thing. Basically, these scripts periodically request to download and create a copy of all its visited pages around the world…unless you tell them not to. They obey orders using your server’s robots.txt file.
In terms of SEO, what this crawler does is collect text, links, and the URL of your web page. So remember what I’ve mentioned in the past about images and the importance of using the alt tag? Information gathered from the alt tag is the only information gathered about the image. It’s also worth mentioning that this is why Adobe Flash has fallen out of favor when creating a website: crawlers only read HTML formatted web pages.
To give you a rough estimate, over 100 million gigabytes of data has been periodically scanned to date. And best of all, this data can be accessed by the user at any time and very quickly as you may have noticed. The information collection process is the easy part, however.
So how do search engines use this information?
All of this data is made sense of by complex algorithms that analyze character combinations among many other factors. The reason search results can be returned so quickly nowadays is the results of years of the development of proprietary technology from companies like Google, Yahoo, and Bing. This is where unique algorithms get to work and attempt to get a better idea of what you are trying to search for.
It’s all about the keywords
Based on many new and innovative ways one can use a search engine (i.e. image search, video search, news, etc.) relevant documents will be displayed. Results are ranked using hundreds of factors so I won’t get into details in this post, but if you’re curious, here is an interesting slideshow of the latest testing and theories. There are also interesting ways you can use Google’s search features to expand your query and obtain more specific results.
I know the word “keyword” is overused, so allow me to dumb it down and say that they are just “words that people write in the search box.” According to Statistic Brain, over 2 billion keyword searches were performed in 2013. After all, SEO is a marketing effort and these words represent entire markets so the importance here is obvious. Algorithms are constantly changing to think more like a human, so understanding this will help enforce the idea that much of that way SEO seems to be heading is more natural and less black and white.