What is a search engine?

When you use a search engine, you type words describing the information you’re interested in into an empty box and wait for the results to appear. The search engine will display a customised web page with a list of pages that contain related content – often a very long list.

Essentially, a search engine is a computer program. The biggest ones, such as Google, Microsoft's Bing, Yahoo!, and Ask, use giant clusters of computers to search the web. Others search only the owner's site (look at the upper right corner of this page, for example).

The concept behind all of these is no different from the search function available on every personal computer that searches just that machine.

Because the web is so big, for a search engine to work quickly enough to find answers before users lose patience, it must have some idea where to look.

Search engines catalogue the web by sending out a piece of software called a spider, crawler, or 'bot’ (robot) that analyses the content on each page it finds and builds an index based on that information.

It’s a complex process and the owners of search engines are often secretive about the inner workings of their software - especially how it decides which pages are the most relevant to a particular search and therefore should appear at the top of the list of results.

Each search engine's software is a little different, so the same search terms will produce different lists of results from different search engines. That doesn't necessarily mean that one is more accurate than another, but it may mean that you find one more useful than another. So, use whichever search engine you're most comfortable with.

Website owners help search engines analyse their pages correctly by using common keywords to identify the subject and by providing additional information about their pages (known as ’metadata’). These techniques are called SEO - ‘search engine optimisation’ - and are studied in detail by those wishing to attract customers to their sites.

If you do research on the web, it's important to remember that search engines do not find everything you might want. Some people use a hidden file called robots.txt to keep their pages out of search engines. This file tells 'bots’ not to scan the site.

Search engines also typically cannot reach material behind a ‘paywall’ – that is, a registration scheme that requires you to pay to access a site's content. Many free sites that require registration also cannot be searched. Finally, search engines usually can't access material held in databases, or material displayed in Flash animation software.