“Google’s mission is to organize the world’s information and make it universally accessible and useful.”
In Web 2.0, the saying “content is king” remains a prevailing theme. With seemingly endless content available online, the findability of content becomes key. Search engines are the primary tools people use to find information on the web. Today, you perform searches with keywords, but the future of web search will use natural language (see, for example, Powerset.com). Currently, when you enter a keyword or phrase, the search engine finds matching web pages and shows you a search engine results page (SERP) with recommended web pages listed and sorted by relevance. People-assisted search engines have also emerged, such as Mahalo, which pays people to develop search results.2 The popularity of vertical search engines—ones that focus on a specific topic or industry—is on the rise, though traffic to these search engines is still far behind the major (more generalized) search engines.
Traffic to the major search engines is growing rapidly—according to a recent comScore (a web analytics company) report, Americans conducted 8 billion search queries in June 2007, up 26% from the previous year. In the same report, the comScore analysis of U.S. market share across the most popular search engines reported Google at the top with 49.5% of the U.S. search market, followed byYahoo! with 25.1%, Microsoft with 13.2%, Ask with 5.0% and Time Warner Network with 4.2%.3
John Battelle’s book aaa.he Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture provides an extensive history of search engines and presents strong arguments for the importance of search in almost every aspect of our personal and business lives. John Battelle’sSearchblog discusses search and technology issues (http://battellemedia.com).
“Telecommunications bandwidth is not a problem, but human bandwidth is.”
—Thomas Davenport and John Beck, The Attention Economy4
The abundant amounts of information being produced and people’s limited free time has led to an attention economy. More content is available than users can sort through on their own, especially given the demands on their time, such as responsibilities to children, parents, friends, employers, etc. The Attention Economy, by Thomas Davenport and John Beck, begins with the familiar story of a man whose attention is constantly demanded by work and family. The authors explain that the constant flow of information in today’s world causes attention to continually be diverted.
Though it used to be difficult to obtain diverse content, there are now seemingly endless options competing for an audience’s attention. As a result, search engines have gained popularity by helping users quickly find and filter the information they want.5
Google is the leading search and online advertising company, founded by Larry Page and Sergey Brin while they were Ph.D. students at Stanford University. Google is so popular that its name has been added to the Oxford English Dictionary—the verb “Google” means to find something on the Internet using the Google search engine. (“google” with a lowercase “g” is a cricket term, whereas “googol” or 10100 is the mathematical term Google was named after.)6
Google’s success in search is largely based on itsPageRank™ algorithm (patented by Stanford University and Larry Page) and its unique infrastructure of servers that uses linked PCs to achieve faster responses and increased scalability at lower costs.7 Estimates on the number of Google servers range from hundreds of thousands to over one million.8 The PageRank algorithm considers the number of links into a web page and the quality of the linking sites (among other factors) to determine the importance of the page. Each inbound link is a vote saying that site is valuable to someone else; however, votes are given different weights depending on the “voter” site’s own value. So, two pages could have the same PageRank even if one has numerous links in from other pages and the other has fewer links in but from pages with higher PageRank. Google search also considers all of the content on the page, its fonts, its headers and the content of neighboring pages.9 Sites with the highest PageRank will appear at the top of the search results.
In addition to its regular search engine, Google offers specialty search engines for images, news, videos, blogs and more. UsingGoogle web services, you can build Google Maps and other Google services into your applications (seeSection 3.13, Web Services, Mashups, Widgets and Gadgets).
AdWords, Google’s pay-per-click (PPC) contextual advertising program (launched in 2000), is the company’s main source of revenue. AdWords ads appear next to search results on the Google site (and are related to the search query). Advertisers write their own ads, which are unobtrusive and uniform in appearance—each ad consists of a headline, limited text and a URL. Advertisers bid on search keywords related to their ads and pay based on the number of users who click on the ads.
AdSense is Google’s advertising program for publishers (sites like http://www.deitel.com that offer content), inspired by Susan Wojcicki, the vice president of product management. (In 1998, Wojcicki rented a spare room in her house to Larry Page and Sergey Brin where they founded Google.)10 AdSense is a fundamental and popular form of website monetization, particularly for Web 2.0 startup companies. Google text ads (as well as banner and rich-media ads) are placed on participating sites with related content. Click-through rates on contextual ads are often higher than on non-contextual ads because the ads reach people expressing interest in a related topic. As a result, contextual pay-per-click ads generally pay a higher eCPM (effective cost per thousand impressions).
Yahoo! was started in 1994 by Jerry Yang and David Filo (also Stanford Ph.D. students) as a web directory rather than a search engine. The original site, “Jerry and David’s Guide to the World Wide Web,” consisted of their favorite websites manually added to a categorized directory.11 As the web grew, maintaining the directory structure became increasingly difficult, and a search capability was created for better access to the data. Focusing more on search, Yahoo! also expanded into other areas, becoming a popular provider of e-mail, user groups and more. In 2003, Yahoo! acquired Overture (now Yahoo! Search Marketing), which was the first search engine to offer sponsored search results successfully.12
MSN search was created in 1998, a year after Google was launched.13 Over the past few years, Microsoft has made search engine technology development a top priority.14 Microsoft search query volume and its search market share grew rapidly in June 2007; analysis companies comScore and Compete attribute this boost largely to MSN’s Live Search club, a program introduced in May 2007 to reward users of Live Search.15, 16 MSN’s Live Search includes a new search engine, index and crawler.17 It allows you to search the web, performing specialized searches (news, images, or local listings) or MSN content searches.18 Another approach that Microsoft is taking to increase its search market share is buying vertical search sites such as MedStory, a health search engine. 19 Microsoft is also looking to gain market share in the contextual advertising market through Microsoft adCenter (similar to Google AdWords and Yahoo! Search Marketing).
Ask (formally known as AskJeeves.com) is owned by InterActiveCorp (IAC), which also owns Ticketmaster®, Match.com®, LendingTree.com®, RealEstate.com® and many other Internet properties. In June 2007, Ask launched a new search site, which includes a new design with a simple homepage default, customizable backgrounds, new video search (powered by Blinkx) and the ability to view video previews and listen to music clips. The search results are based on the searcher’s location—Ask will report relevant local businesses and events. Searching for movies, for example, will show local show times.
Vertical search engines are specialists (focusing on specific topics) in comparison to generalists (e.g., Google and Yahoo!).20 Vertical search engines enable you to search for resources in a specific area, with the goal of providing you with a smaller number of more relevant results. Popular vertical search engines include travel sites (such as Kayak or Expedia), real-estate sites (such asZillow or Trulia), job search sites (such asIndeed or Monster) and shopping search engines (such as Shopzilla and MySimon).
Location-based search (offered by most major search engines as well as some smaller specialized ones) uses geographic information about the searcher to provide more relevant search results. For example, search engines can ask the user for a ZIP code or estimate the user’s general location based on IP address. The engine can then use this information to give higher priority to search results physically located near the user. This is particularly useful when searching for businesses such as restaurants or car services. (See Section 3.14 for more information on location-based services.)
Creating Customized Search Engines
Rollyo—a build-your-own customized search engine website—allows you to explore, create and personalize search engines (“searchrolls”) created by others. This helps you narrow your search to sites you already trust.21 Other custom search sites include Gigablast and Google Custom Search Engine.
Search Engine Optimization (SEO)
Search Engine Optimization (SEO) is the process of designing and tuning your website to maximize your findability and improve your rankings in organic (non-paid) search engine results. To maximize traffic, you need to take into consideration how search engines work when you design your website. There are two ways of employing SEO. The first, white hat SEO, refers to methods that are approved by search engines, do not attempt to deceive the search engines, and produce quality, long-term results. Top white hat techniques for SEO include: offering quality content, using proper metadata and effective keywords, and having inbound links from relevant high-quality pages.22 Black hat methods are used to deceive search engines. Although they may result in temporary improvement in search engine results, these tactics could get your site banned by the search engines. A “Googlebomb” (or link bomb) is an example of a black hat method—it attempts to trick the Google algorithm into promoting a certain page (generally for humorous reasons).23
Link building is the process of increasing search engine rankings and traffic by generating inbound links to a particular website. Search engine algorithms regard each link as a vote for the destination website’s content, so sites with the greatest link popularity (or number of high-quality inbound links) appear highest on search engine result pages (SERPs). The three most practiced methods of building links include reciprocal linking, link baiting and natural linking. Reciprocal linking is an exchange in which two related websites link to each other, increasing the link popularity of both sites and adding value for site users.Link baiting involves creating attention-grabbing web content specifically for viral (exponentially increasing) exposure through social media and social bookmarking websites.Natural linking is the process of building one-way inbound links by optimizing website content and user experience without the explicit solicitation of a backlink. Search algorithms are continuously updated to prevent black hat SEOs from deceiving search engines with automated linking software and links from directories or other low-quality websites. One-way links from websites with strong, related pages are given greater weight than reciprocal links, links from sites with unrelated content or links from sites with low PageRank.
Search Engine Marketing (SEM)
Search Engine Marketing (SEM) is the method of promoting your website to increase traffic and search results by raising the site’s visibility on search engine results pages. Danny Sullivan (founder of Search Engine Watch and, more recently, Search Engine Land) introduced the term “Search Engine Marketing” in 2001 to include SEO, managing paid listings, developing online marketing strategies and submitting sites to directories.24 SEO is the most popular form of search engine marketing, which continues to take away business from other marketing channels (especially offline sources). According to the Search Engine Marketing Professional Organization’s annual State of Search Engine Marketing survey, North American advertisers spent $9.4 billion on search engine marketing in 2006, a 62% increase over 2005 spending.25
Search Engine Watch and Search Engine Land
Search Engine Watch is a search engine marketing resource site. It includes articles, tutorials, conferences and more. The site, launched in 1997 by Danny Sullivan, was inspired by his 1996 release of “A Webmaster’s Guide To Search Engines.” Search Engine Watch incorporates Web 2.0 features (blogging and forums in addition to expert columnist articles). Other Search Engine Watch departments include search engine submission tips, web searching tips, popular search engines and search engine resources (numerous topics related to search engines). Danny Sullivan served as Search Engine Watch’s editor-in-chief until November 2006, when he left the site and became the editor-in-chief for Search Engine Land. The site provides news and information on the major search engines—Google, Yahoo!, and Microsoft—as well as search engine marketing and searching issues. The site also informs users of upcoming related conferences and webcasts.
Search Engine Strategies Conferences
Search Engine Strategies is a global conference series focused on search engine advertising (including current SEO and SEM issues). Search Engine Strategies (hosted by Search Engine Watch) offers event information given by the top experts in the field as well as representatives from search engine companies.26 Because traffic and advertising are so important to most Web 2.0 businesses, understanding the search process and making sure your site is easily found is vital.
Rather than the traditional use of search engines (searching with a topic in mind), discovery refers to finding new content you would not have otherwise sought out. For example, Yahoo!’s original directory design allowed users to browse categories, and discover new interesting sites. StumbleUpon, a social bookmarking site, addresses discovery with its recommendation system that helps you discover and share websites based on your interests. Content networks also direct users to web content they would not necessarily have looked for otherwise.