Before Google: A history of search

Extract from how-search-worked-before-google (archived)
by Steven Vaughan-Nichols
Mar 24, 2017

When I was first on the Internet, we didn't have Google, or even the Web, but we still managed to search the Internet. Here are the tools we once used...and liked.
When I started using the Internet in the 1970s, it didn't look anything like it does today, and our search tools were primitive. But when all you have is stone knives and bear skins, you make do.

Before I ever turned my hand to writing for a living, I put myself through graduate school by doing research on the very first online database systems: NASA RECON Dialog, now ProQuest, and OCLC. These systems, which are still around, are part of what's called the Matrix—and, no, I don't mean the movies. The Matrix, as defined by Carl Malamud, is the superset of all interconnected networks. Today, you can get to ProQuest and OCLC over the Internet, but you'll find yourself blocked from getting very deep into them without permission.

As for the pre-Web Internet itself, at first it didn't have search tools. It wasn't until the late 1980s that the Internet became searchable. When I started using it, we had to go through FTP file directories screen by screen and hope that the file we wanted was in there somewhere.

1990 Archie
The first major search advance was Archie, which beginning in 1990 made it possible to search through a site's file directories. Archie was painful to use, but compared with what we had been dealing with, it was wonderful.

1991 Gopher
An even bigger advance was Gopher, which made it possible to search through online databases and text files. With Archie, you really had to have a clue that a file was somewhere on a given site. With Gopher, you could simply search and let the server worry about finding which site had the information you wanted, listing files as "menu items" having hyper-linkable "titles".

1992 Veronica
Archie and Gopher were quickly followed by Veronica, a service from the University of Nevada System Computing Services, a constantly updated database of the names of almost every menu item on thousands of Gopher servers. The Veronica database could be searched from most major Gopher menus.

1993 WAIS - first developed 1990 - as an Internet text searching system (within Gopher)
While Gopher was being built in 1991, the Web was also being created (though unseen before 1992). By 1993, just as Gopher reached its maturity, I thought WAIS (Wide Area Information System) and its full-text search engine for individual Gopher servers, was going to be more important than the Web! I was putting the cart before the horse. WAIS, like Archie, Veronica, and Gopher, exist now only as Internet historical trivia.

In a way, though, I was on to something. Yes, we use the Web for everything, but it's the search engines that make it useful. After all, we say, "We Google for information," not "We webbed for the answer."

The Web quickly knocked out those earlier Internet search programs as well as online services like Prodigy, CompuServe, and GEnie. Now, only bits and pieces of them remain (such as CompuServe forums), and for the most part, they're really only interesting to digital archaeologists. Fortunately for early users, the first web designers set to work creating search engines.

1994 Yahoo
Indeed the first of them, the WWW Virtual Library, was created by the Web's creator, Tim Berners-Lee. The Virtual Library, and its most well-known clone, 1994's Yahoo!, weren't really search engines. They were human-assembled catalogs of useful web links. You could (and did) submit your own webpages to Yahoo, for instance, and suggest in which category it should be included (why, yes, search engine optimization was very different back then). Editor - Yahoo in fact dominated the web.
According to the-most-visited-websites page, Yahoo was ranked second favorite website worldwide (behind AOL) until 2000, then most favorite website worldwide until 2006, when it was overtaken by Google. Geocities (1994-2009) was third.

Except for GeoCities, these directories still exist. Indeed, sites that allow users to vote on which specific webpages are interesting, such as Reddit, are still popular. Most of the early ones, such as EINet Galaxy (1994) and Open Directory Project (1998), have either died off or are little used.

In search of search

Others, such as Yahoo, switched to a search engine model.

And what is that? The very first search engines—JumpStation, the World Wide Web Worm, and the Repository-Based Software Engineering (RBSE) spider—used automatic programs, called robots or spiders or crawlers, to request webpages and then report what they found to a database. Those first spiders weren't that useful. It took Archie-Like Indexing of the Web (ALIWEB—it's completely dead now, just its domain survives) and Excite in 1993 before we had search engines that most of you would recognize as useful tools. "Useful" is being kind, though. These first search engines were useful only if you knew the exact name of the website you were searching for.

1994 Webcrawler and Lycos
Searching inside webpages would have to wait for 1994's WebCrawler. WebCrawler is now largely forgotten (though still active), but in its first year it was so popular that it became a victim of its own success; the site was so slammed that it often couldn't be used.

That gave Lycos its chance. By pouring more servers into this project, Lycos was almost certainly the first search engine to have full page search for more than a million pages. That's nothing by today's standards, but it was remarkable in mid-1994. Lycos was also the first search engine to introduce proximity searching.

The mid-90s was a hotbed of new web search engines. InfoSeek (1994-1999), whose only long-lasting claim to fame was somehow convincing Netscape to make it the Netscape browser default search engine, came, went, and unlike the others, left no real trace behind. Inktomi, which appeared in 1996, briefly enjoyed a burst of popularity. Eventually, in 2003, it was bought out by Yahoo. Inktomi provided web search results to Microsoft's MSN portal, for many years.

1995-2010 AltaVista
The most significant web search engines to arrive were AltaVista launched by DEC employing major computing resources (at the time), my personal favorite in those days, and Ask Jeeves, now Ask.com. AltaVista was the first really fast search engine that also covered much of the Web. It also gave users the first successful Boolean search options. So, for example, you could search for "New York baseball history NOT Yankees" and get results about the New York Giants and Dodgers, but not the New York Yankees. Ask's claim to fame was its attempt to support search by natural language. So you could ask it, "Tell me about New York baseball history, but I don't want to hear about the Yankees." Whether you'd get useful results is another matter, but Ask Jeeves did better with this question style than its competition.

How the 800-pound gorilla grew so big
So why haven't you heard of most of these? Well, each had its own problems. AltaVista, for example, shared in the woes of its parent companies, Digital Equipment Corp. (DEC) and then Compaq after 1998. Yahoo, which ended up with the ownership of AltaVista in 2003, finally killed off this one great web search engine in 2010.

1998-2020 Google
But what really happened to all of them was Google. Google simply did a much better job than its rivals of making search easy and comprehensive. The key to Google's overwhelming success was PageRank. With PageRank, Google rates the relevancy of webpages to queries, based not only on whether the pages contain the search terms (the technique used by all search engines), but also by how many relevant pages link to it.

Sounds simple. Well, the basic concept is simple. It was a groundbreaking advance at the time. Perfecting Google Search and PageRank continues to this day.

Google has never rested on its laurels, either; no company has managed to give Google any serious competition in more than 10 years. Today i.e. in 2017, Google is used for more than 64 percent of all searches (in fact over 90% in 2024). In comparison, Microsoft's Bing has 23 percent of the market (3% in 2024), while Yahoo continues to decline with only 12 percent of searches (just 1% in 2024). Ask is barely hanging on with 1.4 percent of the market.

Had AltaVista managed to avoid its parent companies' troubles, it might have been a different story. None of the other BG (before Google) web engines had what it took to compete with Google's genius. Today, as a result, we really do live in AG (after Google) days. I doubt that will change anytime soon. Google certainly continues to work hard to make sure that will be the case.

Click here for some of the early statistics re Alta Vista, launched December 1995 after 2 months of testing by DEC's employees. Famously seen in Captain Marvel when she goes back in time to 1995, and runs a web-search.

** End of Page

Go Top