How Deep is your Brand?

Forbes Magazine recently stirred the pot for search engine optimizers (and Google) by writing about “Supplemental Hell.” Several well-known SEOs said on their blogs that when Forbes’ reporter came calling, looking for specific types of anecdotes, they decided not to participate in the discussion.

While it’s interesting that Web marketers who have a stake in building their visibility would pass on the opportunity to do just that, I feel it’s more telling that Forbes wrote at length about a very timely subject without really hitting at the heart of the matter.

With the Bigdaddy update of December 2005 - March 2006, Google turned its search index upside down. They literally rewrote the entire search engine and, according to Google engineer Matt Cutts, they turned off the old Google on March 29, 2006.

Bigdaddy, according to Cutts, changed the way Google crawls and indexes the Web (and maybe some other things). He did not say it changes the way Google scores documents for relevance, but in a very big way the update has impacted Google’s search results. The awful truth did not become apparent to many Web marketers for at least several more months.

In September, the SE Roundtable site mentioned that Google’s cache for pages in the Supplemental Results Index does not highlight query words. The significance of this discovery was lost on the SEO world for a few more months. It marked the beginning of a trend that continues today, although the worst effects were felt by millions of Web site operators from around mid-October through February.

Google is operating a dual-index search engine. We have seen this kind of indexing before. Years ago when Inktomi was the dominant search engine, it maintained two indexes. The smaller index, often called the “primary index”, was sold to other search engines like Hotbot, Canada.com, and Infospace (more than 30 at one point). You could not use or submit to Inktomi directly — you had to go through one of its partner sites.

The primary index contained about 100 million pages. For a while that did not seem to be a problem. But as more Web sites were created and as existing Web sites increased their content, it became increasingly difficult to get pages into the primary index. The secondary index, Inktomi proudly told people, contained hundreds of millions of pages, but no one could see those pages because they were not served up in search results.

Other search engines like Lycos, Excite, Infoseek, and Altavista began refining their technologies so they could break the 100 million page boundary. When they finally did so Inktomi came under pressure to open up its content and combine the two indexes. Inktomi resisted the idea and about that time a radical new search engine called Google started capturing people’s imaginations.

We breathed a collective sigh of relief when we could finally get away from Inktomi’s dual-index model. Now, Inktomi did not completely reject the idea of improving its search results. As more and more Web site operators turned to using spam tactics (such as joining link farms and reciprocal linking programs) to improve their chances of being included in the primary index, Inktomi developed a paid inclusion program.

For a monthly fee you could pay to have Inktomi crawl your pages and thrust them into its primary index. The index, however, still held only a certain number of pages. As new pages were included, other pages were knocked out. The secret to Inktomi’s primary index success — having a lot of links pointing to your pages — was not lost on people. Paid inclusion customers learned the hard way they would have to keep paying for inclusion every month because they did not have enough links pointing at their pages.

The hardest hit sectors were merchants and large forum operators, Web sites with sufficient content and community to have well-respected brands on the Internet. At the height of Inktomi’s dominance, however, many powerful brands were almost invisible across the Inktomi partner sites. Were it not for Yahoo!, which supplemented its directory search with Inktomi results, Inktomi would have tumbled much sooner than it eventually did.

And in the end Yahoo! bought Inktomi anyway.

To compete better with Google, Yahoo! expanded its Web search index to include billions of pages. The old dual index days were gone. Webmasters breathed collective sighs of relief and large content sites were free to build brand visibility across thousands, in some cases, millions of queries.

Google has been severely challenged though through the years. Its algorithm has always been easy to manipulate and people have devised new ways to manipulate Google’s search results faster than Google could counteract them. Starting in 2004, Google began unleashing a series of updates that emphasized “trust” over “importance” or “popularity.”

Through new algorithmic filters and other protocols, Google chiseled away at the volumes of so-called “Web spam” pages that from time to time dominated its search results. By mid-2005 people began to realize that trust was the next battlefield between Google and spammers. The spammers have been relying on rapid link-building techniques for years and they were comfortable with their linking software. Google finally pulled the rug out from under many of them.

Bigdaddy introduced a radical change in Google’s Web crawling and indexing strategies. In mid-2003 Google had introduced “Supplemental Results” as a tag placed against some pages. Webmasters were told that the tag was algorithmically assigned and often applied to pages that seemed like duplicate content or which were otherwise “low-value” pages in Google’s estimation. But those pages were, for all intents and purposes, treated as if they were part of the Google Web index. They often provided the only relevant results for many queries. So while the continued referrals from Google proved to be consoling to Web site operators, many worried about the stigma of “being Supplemental.” Needlessly, we thought.

With Bigdaddy, however, it soon became clear (thanks to Matt Cutts’ explanation cited above) that the Supplemental Results pages had been moved to a separate index. And the new index had its own crawling software. The familiar Googlebot no longer visited many pages. Instead, Supplemental Googlebot began showing up — none too often — and people noticed a severe drop in search results rankings and referrals from Google.

“Not to worry,” Google assured them. “We’re still making adjustments and we’ll increase Supplemental Googlebot’s crawl rate so that its index is refreshed more often [than once or twice a year].” That promise mollified many people for several months.

By mid-November Bigdaddy (which had undergone some retooling in July and August) was behaving in a radically different manner. Many more pages dropped out of the Main Web Index and “went Supplemental”. By December Google’s Webmaster support groups were filled with people asking why they were being penalized.

Being placed in the Supplemental Index is not really a penalty, Googlers explained. It just means you need “more quality links.” But what is a “quality link”? Link building has been a major component of many SEOs’ strategies for years, but now Google was telling people their links were not helping.

Those “low quality” links that no longer help had, in fact, “gone Supplemental.” That is, as pages slipped out of the Main Web index, they were retrieved for the Supplemental Index. There they languished, uncrawled for months. Worse, their links only passed value in the Supplemental Index.

Now it’s apparent that Google is not searching Supplemental Pages for content to resolve queries. In some cases, where links to Supplemental Pages exist in the Main Web index, some Supplemental Pages can still be served for queries. Such performance, however, is increasingly rare.

Many large content sites have again lost significant brand visibility because their pages are largely showing as Supplemental Results, and Google won’t highlight query text for them. Unique expressions on these pages won’t bring them up in search results.

Many well-branded sites have now been divided into visible Main Web index content and largely invisible Supplemental Results. The lack of visibility is very real for a large number of Web site operators, many of whom have had to increase or launch pay-per-click campaigns to rebuild referrals from Google.

The reconstructive process is not only expensive, it is often fragmented. As retailers and major brands scramble for highly visible adspace, Google has also implemented new quality guidelines intended to improve paid listing results. Instead of focusing on brand, some advertisers are struggling to build competitive query visibility.

Web brand depth, where Google is concerned, must now be measured in three ways: Main Web search, Paid Listings (Adwords), and Supplemental Results. At this time, there are no metrics capable of measuring coverage in these areas with respect to the entire picture. We cannot aggregate data and measure the impact of visibility (or the lack of visibility) in the three areas.

Google fired back at the Forbes article, indicating that the examples Forbes offered are not shining examples of good Web marketing. But the real issue is trust: Google no longer trusts the Web in general, and the Bigdaddy update finally gave Google the ability to separate the pages it trusts the most from pages it trusts the least.

In the process, they have resurrected a moral dilemma for Web site operators that many of us had once believed was laid to rest years ago. Many companies that once possessed deep brand visibility in Google now find they have only shallow brand visibility. They are vulnerable to competitive challenges and other consequences without really having done anything wrong.

The true penalty that Google’s lack of trust confers upon the typical business site operator is a loss of potential reach for content that is, in many cases, more relevant than the link-rich content that still populates the Main Web index. Web branding strategies need to adapt to the new world of dual Web-indexing, at least until Google finds a better way to manage trust or is toppled from its dominant market position.