Frequently Asked Question about Search Engines -- Search Engine FAQ (page 2)

How To Use Web Search Engines
Tips on using internet search sites like Google, alltheweb, and Yahoo.

Page 2-- Spidap's Basic Search Engine FAQ

1. Why do I need a search engine?

For the same reason you need a card catalogue in a library. There is lots of great and useful information in a library, but it's physically impossible to examine all the books personally. Not even the most indefatigable web-surfer could hyperlink to all the documents in the aptly named World Wide Web. There are billions of pages on the Web. And every minute of the day, folks are posting more.

The search engines and directories help you sift through all those 1's and 0's to find the specific information you need.

2. If it's impossible to examine all the documents on the Web, how do the search engines do it?

They use software programs known as robots, spiders or crawlers. A robot is a piece of software that automatically follows hyperlinks from one document to the next around the Web. When a robot discovers a new site, it sends information back to its main site to be indexed. Because Web documents are one of the least static forms of publishing (i.e., they change a lot), robots also update previously catalogued sites. How quickly and comprehensively they carry out these tasks varies from one search engine to the next.

3. Which search engine is "the biggest"?

They'd all like you to believe they're either "the biggest" or, if they can't possibly claim that, "the best."

Google currently claims to index over 3.3 billion pages, while Alltheweb.com (one of Yahoo’s many search properties) claims to index over 3.1 billion. Search engines which demand that site owners pay to have their sites indexed, such as AskJeeves, index fewer pages.

4. How Do Search Engines Work, Technologically?

Click here for our detailed explanation of search engine technology, including keywords, clustering, relevancy ranking, meta tags and more. For a more highly technical discussion, try our page on Search Engine Ranking Algorithms

5. What's the difference between a Web directory like Yahoo and a Web search engine like Google?

There is less difference now than there used to be, because many search engines, including Google, have built large subject catalogues to help you search. But think of a Web directory as a subject catalogue--something like the subject catalogue in your local library. Yahoo started out as a directory but is now de-emphasizing that aspect of their broadly-based business.

However, directories such as The Open Directory aka dmoz and the Google Directory attempt to organize Web by dividing it into topics and subtopics. Some examples include: Arts, Science, Health, Business, News, Entertainment. If you're looking for information on the Web that fits neatly into an obvious subject or category, go first to a web directory.

Think of a Web search engine as an index that enables you to seek out specific words and phrases. With the search engine's help, you can locate individual appearances of such words in documents all over the Web.

This can be both a blessing and a curse--but it's more commonly the latter! You are likely to get far too many hits. Or you might discover that your keyword has meanings you didn't anticipate. Rarely, you might get no hits at all.

In brief, here's a quick run-down of some well-known general topic search engines:

Yahoo's directory, the Google Directory, and the Open Directory Project (dmoz) are web directories -- essentially subject indices. They began as attempts to catalogue important/useful pages on the Web. Search on a subject or topic. If you know exactly what subject you're searching for, and have a good sense of how to find your subject within a hierarchies of larger subjects, a directory is a good place to start.

Google was one of the newer search engines, but it rapidly become the favorite. In fact, the word is commonly used as a verb, a synonym for searching: "I'm going to google the web to find the info I need."

Google is thorough and fast. Its technology considers pages that are linked from many other sites to be more important than pages that only have a few links from other sites. In other words, if many webmasters consider a website valuable enough to create a link to it, Google considers that a good reason to justsify a high ranking for that site.

Confusingly, while Yahoo’s search functionality was for a couple of years been based on Google. In fact, Yahoo was one of Google's investors. As of March, 2004, however, that partnership ended, and Yahoo is now developing their own search technology, which will probably be based on search algorithms they obtained when they acquired Inktomi.

Meanwhile, Microsoft’s search and MSN.com have also been based on Inktomi's technology. (Inktomi provided the technology for one of the hot search engines of the 1990s, Hotbot. For more info on 1990s search engines, see our historical information page for more details). Microsoft reports that it is working to develop its own inhouse search technology to replace Inktomi, but there is no fixed date for the cutover.

Alltheweb is also one of the newer search engines. It claims to be faster and more efficient than other search engines, with the largest index and the most rapid look-up times. Alltheweb has been acquired by Yahoo, and may at some point be integrated with Inktomi.

AltaVista was the favorite of web searchers a few years ago, but Google blew it out of the water. It still provides users with excellent search refinement capabilities, though. Altavista has also been acquired by Yahoo.

Lycos was one of the original Web search engines, but has morphed into a general web portal with something of a European focus (it was bought by a Spanish company now called Terra/Lycos). Its actual search results are based on the Inktomi engine.

HotBot, as noted above, was the original Inktomi-based search engine, but has largely faded into somewhat unjustified obscurity. It was at one time rated fastest and most accurate product for business and professional purposes, and still is a good alternative to the more popular engines if you are trying to track down information about a specific person.

Excite was a concept-based search engine. We wrote quite a bit about this concept when web search was in its infancy. It was an interesting idea, but concept-oriented search is now web history. Excite.com still maintains a website and a directory, though.

6. How Can I Find Out More Details About These Search Engines?

To learn how to search engines work, click here. To go directly to our Web Search Wizard and quickly find what you want, click here. For historical ratings and detailed info about search engines of the 1990s, click here.

7. Number One Frequently Asked Question From Our Spidap Users:

How can I get a higher search engine ranking for my website?

Our reply: It used to be true that if you posted a page on the Web, sooner or later the search engines would find it and index it, as long as it's not too deep into the particular site's hierarchy (i.e, "deep" refers to how many clicks your site is from the site's main page).

To some extent, this may still be true. But as the Web has exploded in size, it's obvious that less and less of it is actually being found and indexed by search engines. Now a website developer has to be more proactive than ever before about getting listed by search engines and directories. In many cases, this means (unfortunately) that you have to pay a fee to get listed.

Most of the popular Web search engines still provide a way for you to add your own URL. Sometimes this is obvious on their initial page; other times you have to drill down into the site to find out where and how to do this.

We continue to recommend that you use meta tags in the "head" of your document. With meta tags you can provide your own keywords for the search engine to index you with. More importantly, you can also provide a description of your site that the search engines can use to show to Web surfers or researchers when your site comes up as a hit. For more on meta tags, click here.

However, meta tagging does not provide you with any guarantee that your site will be listed or highly ranked. There has been so much abuse of the keyword meta tag in recent years that meta tagging is not as helpful to web developers as it used to be in the late 1990s. Not all search engines index the meta keyword information (Google, for example, does not).

You can hire various amateur and professional "search engine optimizaton" professionals who charge a fee in order to get a higher ranking for you. Some of these fees are quite hefty, and there is rarely any guarantee. Beware, also, that you don't hire someone who uses unscrupulous tactics to try to "spam the search engines." This could result in your site getting banned from search engine indices.

The best way to get people to find your site is to provide useful, well-organized, well-written, well-designed information. The more folks who find your site and link to it, or send around the URL to their friends, the more visitors you'll have. Word of mouth (actually, word of email) is very powerful on the Web!

Next: Planning your search strategy

Spidap, Top Page

Search Engine Ranking Algorithms

Search This Website