Entries Tagged 'Search Engine Optimization' ↓

John Bell, Managing Director, over at Ogilvy PR in D.C. had a interesting post today about “Search Reputation Management”. If you don’t follow John and his colleagues on their 360 Digital Influence blog you may want to. His post highlighted the growing importance of people’s online reputations in search engines like Google, Yahoo, and MSN.

I think I’ll have my colleague and fellow VisInsights co-contributor Michael Martinez ,who heads up our TruView service, provide some additional commentary on John’s post.

Blake

Visible Technologies

We have come full circle with search (not for the first time). Early search engines were more like file indexers and if you were able to submit data at all about your Web site you could only provide a few “tags.” Even directories tended to provide better results than those early services, but we have long since become accustomed to tagging posts in social media sites.

Now AOL, A9 (Amazon), and Google have brought search technology forward into integrated search. Over the past few years search indexes have become fragmented. We have Web Search, Blog Search, Local Search, News Search, Image Search, Video Search, and now even Book Search. Search engine interfaces have become increasingly complex as they’ve added more tabs and vertical search structures.

But the user navigation experience is far from ideal when you have to jump from vertical to vertical. Earlier this year America Online introduced its FullView integrated search interface. Not long ago Amazon unrolled the new A9 integrated search interface. It should probably have come as no surprise that we would see real innovation among the syndicated search partners (AOL uses Google’s database and A9 uses Microsoft’s).

Yahoo! and Microsoft both claim they have been integrating search results for some time now, but they require the user to configure their search interfaces. A9 allows extensive user customization but it doesn’t require customization in order for users to enjoy integrated search.

AOL and A9 together capture less than 5% of search query share, so the majority of searchers have not yet been exposed to integrated search functions. Now Google has unveiled what some are calling Google 3.0, or the Searchology Update.

Google says it has completely redesigned its search engine from the ground up: the internal data management functions are different, the crawling functions are different, the indexing functions are different. And now people are wondering how this affects their search marketing and visibility.

Some search marketing consultants are playing the Fear, Uncertainty, and Dismay card in the hope of attracting new business. “Your SEO is not prepared for this new world of search. We are.” “Your competitors have been caught flat-footed by Google 3.0. We’ll help you seize the day.”

The monumental task of pulling data from up to seven search indexes in response to a single query is further complicated by the search engines’ desire to maintain their standards of quality and relevance. In some cases, search engines sacrifice quality for the sake of relevance, and in some cases they sacrifice relevance for the sake of quality. But though we have seen new user interfaces, there is no indication that the rules have changed.

That is, the standards by which the search engines judge the Web are the same as they were before they had their latest face lifts. Nonetheless, the movement toward integrated search does change the playing field somewhat.

For the several years prior to integrated search the engines have divided content on the Web into vertical indexes that reflect technologies, but this division of data is somewhat artificial. News stories, for example, are still published through standard HTML pages. So are blog posts. What distinguishes news and blog content from traditional Web content, however, are the qualities of timeliness and temporal relevance. That is, their content often becomes less useful as more time passes, as more new stories and posts are created.

Image and video content convey information in a substantially different medium. The technology to extract information from audio and graphical content is only just now emerging, and in some respects has yet to be deployed. But we have learned (both Web site operators and search engineers) to surround audio-visual content with relevant identifying information. We encapsulate our audio-visual content with indexable content. There is a convergence of technologies that has made integration more feasible.

The benefit of integration is that we can begin to look at information less as “Web documents,” “graphical files,” and “audio files” and more simply as “blocks of information” or information sources. The mode of presentation is less meaningful for search. Medium has begun to assume a significant transparency.

As we learn to embed relevance-influencing content in non-traditional media we’ll capture more searchers through a broader reach. But we’ll also find ourselves competing on more fronts. The normalization of presentation media in Web search is comparable to accelerating the introduction of competitive content by new publishers. Yesterday we had to be more relevant than 100 million documents. Today we have to be more relevant than 150 million information sources.

As the indexed, visible Web has grown, we have found ourselves faced with increasing competition anyway. We are largely already tooled up, trained, and deployed to take on a greater number of competitors for searcher attention. And that is why we should not be afraid of integrated search.

As the search engines evaluate their users’ experience they will make adjustments in both their presentation strategies and their indexing algorithms. But the indexing process can still be influenced by the meta information we provide with each information source. The type of source should become increasingly irrelevant and transparent to the average searcher.

That is not to say that vertical search indexes will become useless. On the contrary, I see a growing need for archival searches across timely content: news, Blogs, and even videos and images. As the search engines improve their ability to capture and catalogue temporal data we’ll find ourselves conducting more research into trends and patterns.

The structure of queries should evolve and grow just as the structure of Web content and of the indexes we rely upon to find that content have evolved and grown. I don’t think there is any reason to be concerned. If anything, we now have an opportunity to improve our visibility and reach across integrated search results by creating more useful, informative, and innovative content.

Forbes Magazine recently stirred the pot for search engine optimizers (and Google) by writing about “Supplemental Hell.” Several well-known SEOs said on their blogs that when Forbes’ reporter came calling, looking for specific types of anecdotes, they decided not to participate in the discussion.

While it’s interesting that Web marketers who have a stake in building their visibility would pass on the opportunity to do just that, I feel it’s more telling that Forbes wrote at length about a very timely subject without really hitting at the heart of the matter.

With the Bigdaddy update of December 2005 - March 2006, Google turned its search index upside down. They literally rewrote the entire search engine and, according to Google engineer Matt Cutts, they turned off the old Google on March 29, 2006.

Bigdaddy, according to Cutts, changed the way Google crawls and indexes the Web (and maybe some other things). He did not say it changes the way Google scores documents for relevance, but in a very big way the update has impacted Google’s search results. The awful truth did not become apparent to many Web marketers for at least several more months.

In September, the SE Roundtable site mentioned that Google’s cache for pages in the Supplemental Results Index does not highlight query words. The significance of this discovery was lost on the SEO world for a few more months. It marked the beginning of a trend that continues today, although the worst effects were felt by millions of Web site operators from around mid-October through February.

Google is operating a dual-index search engine. We have seen this kind of indexing before. Years ago when Inktomi was the dominant search engine, it maintained two indexes. The smaller index, often called the “primary index”, was sold to other search engines like Hotbot, Canada.com, and Infospace (more than 30 at one point). You could not use or submit to Inktomi directly — you had to go through one of its partner sites.

The primary index contained about 100 million pages. For a while that did not seem to be a problem. But as more Web sites were created and as existing Web sites increased their content, it became increasingly difficult to get pages into the primary index. The secondary index, Inktomi proudly told people, contained hundreds of millions of pages, but no one could see those pages because they were not served up in search results.

Other search engines like Lycos, Excite, Infoseek, and Altavista began refining their technologies so they could break the 100 million page boundary. When they finally did so Inktomi came under pressure to open up its content and combine the two indexes. Inktomi resisted the idea and about that time a radical new search engine called Google started capturing people’s imaginations.

We breathed a collective sigh of relief when we could finally get away from Inktomi’s dual-index model. Now, Inktomi did not completely reject the idea of improving its search results. As more and more Web site operators turned to using spam tactics (such as joining link farms and reciprocal linking programs) to improve their chances of being included in the primary index, Inktomi developed a paid inclusion program.

For a monthly fee you could pay to have Inktomi crawl your pages and thrust them into its primary index. The index, however, still held only a certain number of pages. As new pages were included, other pages were knocked out. The secret to Inktomi’s primary index success — having a lot of links pointing to your pages — was not lost on people. Paid inclusion customers learned the hard way they would have to keep paying for inclusion every month because they did not have enough links pointing at their pages.

The hardest hit sectors were merchants and large forum operators, Web sites with sufficient content and community to have well-respected brands on the Internet. At the height of Inktomi’s dominance, however, many powerful brands were almost invisible across the Inktomi partner sites. Were it not for Yahoo!, which supplemented its directory search with Inktomi results, Inktomi would have tumbled much sooner than it eventually did.

And in the end Yahoo! bought Inktomi anyway.

To compete better with Google, Yahoo! expanded its Web search index to include billions of pages. The old dual index days were gone. Webmasters breathed collective sighs of relief and large content sites were free to build brand visibility across thousands, in some cases, millions of queries.

Google has been severely challenged though through the years. Its algorithm has always been easy to manipulate and people have devised new ways to manipulate Google’s search results faster than Google could counteract them. Starting in 2004, Google began unleashing a series of updates that emphasized “trust” over “importance” or “popularity.”

Through new algorithmic filters and other protocols, Google chiseled away at the volumes of so-called “Web spam” pages that from time to time dominated its search results. By mid-2005 people began to realize that trust was the next battlefield between Google and spammers. The spammers have been relying on rapid link-building techniques for years and they were comfortable with their linking software. Google finally pulled the rug out from under many of them.

Bigdaddy introduced a radical change in Google’s Web crawling and indexing strategies. In mid-2003 Google had introduced “Supplemental Results” as a tag placed against some pages. Webmasters were told that the tag was algorithmically assigned and often applied to pages that seemed like duplicate content or which were otherwise “low-value” pages in Google’s estimation. But those pages were, for all intents and purposes, treated as if they were part of the Google Web index. They often provided the only relevant results for many queries. So while the continued referrals from Google proved to be consoling to Web site operators, many worried about the stigma of “being Supplemental.” Needlessly, we thought.

With Bigdaddy, however, it soon became clear (thanks to Matt Cutts’ explanation cited above) that the Supplemental Results pages had been moved to a separate index. And the new index had its own crawling software. The familiar Googlebot no longer visited many pages. Instead, Supplemental Googlebot began showing up — none too often — and people noticed a severe drop in search results rankings and referrals from Google.

“Not to worry,” Google assured them. “We’re still making adjustments and we’ll increase Supplemental Googlebot’s crawl rate so that its index is refreshed more often [than once or twice a year].” That promise mollified many people for several months.

By mid-November Bigdaddy (which had undergone some retooling in July and August) was behaving in a radically different manner. Many more pages dropped out of the Main Web Index and “went Supplemental”. By December Google’s Webmaster support groups were filled with people asking why they were being penalized.

Being placed in the Supplemental Index is not really a penalty, Googlers explained. It just means you need “more quality links.” But what is a “quality link”? Link building has been a major component of many SEOs’ strategies for years, but now Google was telling people their links were not helping.

Those “low quality” links that no longer help had, in fact, “gone Supplemental.” That is, as pages slipped out of the Main Web index, they were retrieved for the Supplemental Index. There they languished, uncrawled for months. Worse, their links only passed value in the Supplemental Index.

Now it’s apparent that Google is not searching Supplemental Pages for content to resolve queries. In some cases, where links to Supplemental Pages exist in the Main Web index, some Supplemental Pages can still be served for queries. Such performance, however, is increasingly rare.

Many large content sites have again lost significant brand visibility because their pages are largely showing as Supplemental Results, and Google won’t highlight query text for them. Unique expressions on these pages won’t bring them up in search results.

Many well-branded sites have now been divided into visible Main Web index content and largely invisible Supplemental Results. The lack of visibility is very real for a large number of Web site operators, many of whom have had to increase or launch pay-per-click campaigns to rebuild referrals from Google.

The reconstructive process is not only expensive, it is often fragmented. As retailers and major brands scramble for highly visible adspace, Google has also implemented new quality guidelines intended to improve paid listing results. Instead of focusing on brand, some advertisers are struggling to build competitive query visibility.

Web brand depth, where Google is concerned, must now be measured in three ways: Main Web search, Paid Listings (Adwords), and Supplemental Results. At this time, there are no metrics capable of measuring coverage in these areas with respect to the entire picture. We cannot aggregate data and measure the impact of visibility (or the lack of visibility) in the three areas.

Google fired back at the Forbes article, indicating that the examples Forbes offered are not shining examples of good Web marketing. But the real issue is trust: Google no longer trusts the Web in general, and the Bigdaddy update finally gave Google the ability to separate the pages it trusts the most from pages it trusts the least.

In the process, they have resurrected a moral dilemma for Web site operators that many of us had once believed was laid to rest years ago. Many companies that once possessed deep brand visibility in Google now find they have only shallow brand visibility. They are vulnerable to competitive challenges and other consequences without really having done anything wrong.

The true penalty that Google’s lack of trust confers upon the typical business site operator is a loss of potential reach for content that is, in many cases, more relevant than the link-rich content that still populates the Main Web index. Web branding strategies need to adapt to the new world of dual Web-indexing, at least until Google finds a better way to manage trust or is toppled from its dominant market position.

Internet brand management tools are still in their infancy, but so are the strategies, principles, and methodologies we use despite the fact that Internet brand management has been actively evolving for more than 10 years. The dot-com meltdown of 2000 represents a break-point in the history of online brand management because many companies retrenched their investments.

Resources vanished almost overnight as online marketing companies’ shutdown and changed their focus to less visible, more strategic verticals. For a brief space of several years all that remained in the brand management space were a few search engine optimizers whose clients took advantage of the general collapse of online business to reshape Internet economics.

Search engine optimization is not robust enough for managing a brand. Other aspects of search engine marketing can help, yet even they fall short of the mark. Search marketing is not a reactive function. It can position a Web site in the search results, but it doesn’t protect against brand challenges.

So if most search optimizers are not well-suited to manage brand in the search environment, who is? The PR industry, unfortunately, is only just in the early stages of gearing up to address online brand management — and this is a process that has been unfolding for a year or longer already.

Many PR firms have scrambled to catch up on search technology, but like so many other people before them, PR firms are falling prey to the morass of misinformation, myth, and outdated methodologies that have clouded the search optimization community for years.

The search environment is much more complex now than it was a few years ago. It takes the average SEO specialist about two years to unlearn bad ideas. So given the complexity of today’s search world and the fact that most SEOs are operating on incomplete or partially incorrect information, managing a brand’s visibility and reputation in search is not nearly as simple as it should be.

If you don’t know enough about search optimization to distinguish between the bad information (which you’ll find in just about every major media article that discusses the topic) and the useful information, how do you develop a strategy to manage your brand through search? How do you choose someone to help you develop that strategy?

Search now operates across multiple sectors. The Web is so large, includes so much diverse content, that search engines have to rely upon specialized search interfaces to document the content they find. It would help if more people were sophisticated searchers, but the vast majority of searchers are not very sophisticated at all. In fact, they’ll often place their trust in whatever search results are presented to them.

So your search brand management strategy should begin with an understanding of different sectors:

Web search - Being the most traditional sector, Web search dominates everyone’s branding strategies. In fact, Web search serves as the central repository for the most successful content from other search sectors. But Web search encompasses both the past as well as the present, which makes it almost unique among the various search sectors. Search engine optimizers are most familiar with this territory, although the industry as a whole continues to struggle with ethical concerns, ineffective methodologies, and a lackluster approach to learning about and understanding the basic technologies they strive to influence.

News search - The second oldest, news search is evolving rapidly. News search indexes are more sensitive to timely content, but now we’re starting to see archives emerge to help searchers find older news content that has not crossed over into Web search. Because news content is continually created and updated, News search confers a very brief, transient lifetime upon online news copy.

Paid search - Paid search listings, sponsored results, PPC advertising, paid directory listings, etc. all comprise paid search. Paid search is driven by a cost-per-action economy that can be more volatile and transient than news search. When a successful ad-to-sales pathway appears the advertising tends to become long-term content in your search results and Web content. Paid search has now also extended its reach into non-search copy through context-sensitive advertising.

Blog search - Blogs have become such a huge part of the Web that they now have their own dedicated search interfaces and tools. Blog content is not as transient as news content (which is often removed by news providers after a few days, weeks, or months), but blog posts are still valued in part on the basis of timeliness. The more recent a blog post, the more valuable it tends to be within blog search. Once a blog post has crossed into Web search, timeliness is not as important.

Video search - More and more companies are beginning to feel the sting of online video production and distribution. It isn’t just about who is pirating movies and television shows. People are actively promoting their own videos to build their online visibility, to detract from the visibility of other people or organizations, and to promote specific ideas. Video search may ultimately become a greater influencer than Web search currently is.

Your brand management through search should begin as soon as you decide to create or restructure a brand on the Internet. Many companies wait until they have fully defined their branding campaign, often until the Web site has been fully designed, and then they think it’s time to look at search. Search engine branding takes time. You need to plan ahead. In eCommerce, a typical campaign is often laid out six months in advance.

The brand campaign should be planned 6-12 months in advance of the Web site rollout or redesign, in advance of the offline advertising campaign launch, etc. In other words, you need to plan for search brand management as soon as you start planning for non-search brand management.

You can always step into the process later on and launch search brand management after the fact, but you lose the opportunity to leverage your non-search campaigns in assisting your search campaigns. The more lead time you invest in your search brand management, the more effective your campaign will be and the less likely you’ll find yourself in a crisis management situation when something beyond your control occurs.

Every month several research services report on “search market share.” Although their numbers for queries performed and percentages of queries per search service never agree in detail, Google consistently ranks first among the major search engines. The data sources these monthly reports draw upon include user click data provided by Internet Service Providers. A reasonable third-party assumption is that each service acquires its data directly and that they may not all be overlapping sources.

Compete leverages its custom toolbar data into its reporting. At the very least, therefore, we know that Compete has two distinct data sources to work with and can (hopefully) map trends against data sets.

Hitwise reveals the least amount of information in its public interface.

Nielsen/NetRatings offers more information than Hitwise, less than Compete.

comScore offers a report about comparable to Nielsen’s.

These are the four sources most relied upon in the search engine optimization industry for comment and analysis of which search engines provide the most coverage. The problem is that very little of the data actually looks at anything beyond number of queries performed on a search service. The quality of the metrics is thus extremely low and not very useful.

At least two of the services estimate number of visitors per month. Yahoo! consistently dominates the list, although Google is now a close second. The Microsoft network provides a challenge since MSN and Live are tracked separately, but if you assume that there may be overlap between the Live.com visitors and MSN visitors, you still see Microsoft placing a solid third place.

Ask receives the fewest visitors and fewest queries per month among the major services. Out of curiosity, I occasionally look at data for Lycos, which is the last of the “old school” search services that still makes a play for traffic in any capacity.

If I wait another week to discuss this topic I can give you some numbers for March 2007, but let me talk about February. Compete estimated queries at 6.5 billion. comScore put the number at 6.9 billion. Nielsen estimated 6.4 billion queries.

comScore estimated February unique visitors for the search services as: Yahoo! 128 million, Time/Warner (AOL) 117 million, Google 114 million, Microsoft 114 million, Ask 48 million, and Lycos 22 million. (As a basis for comparison, Compete’s March 2007 data shows 112 million visitors for Yahoo!, 109 million for Google, 74 million for MSN, 72 million for AOL, 62 million for Live, 26 million for Ask, and 12 million for Lycos.)

By themselves, neither number of visitors per month nor number of queries per month tell you anything really useful. We can look at the ratios and see where people seem to spend the most time searching. This type of data is not reported, so to demonstrate the model I have to illustrate with some averaged results that are not statistically valid. This is just an illustration of how I’d like to evaluate search engine performance.

I want to look at searches performed per visitor and searches performed per visit. Since MSN is now primarily concerned with providing content and it has rebranded search under Live, let me work with 51.9 million as Live’s visitor count for February.

Since there is no agreement between the reporting services, let me use approximate averages for query share: Google 55.3%, Yahoo! 23.2%, Live 9.7%, AOL 4%, and Ask 3.3%.

Still, we show about 521 million visitors for Yahoo!, MSN, Google, AOL, and Ask in February. That gives Yahoo! 24.5% of this micro market space of “visitors”, AOL 22.45%, Google and Microsoft both 21.88%, and Ask 9.21%. In terms of estimated visitors, Yahoo! and AOL still outperform Google and Microsoft is running neck-and-neck with Google. Even Ask looks pretty healthy.

So now we can look at an average 6.6 billion queries performed and divide them according to our averaged percentages. That gives Google 3.64 billion queries, Yahoo! 1.53 billion queries, Live 640 million queries, AOL 264 million queries, and Ask 217 million queries. That works out to 32 queries per visitor for Google, 12.3 queries per visitor for Microsoft, 11.9 queries per visitor for Yahoo!, 4.5 queries performed per visitor for Ask, and 2.25 queries per visitor for AOL.

Number of queries per visitor implies that people are either very satisfied with their Google results are very unsatisfied. It also implies that the chance of people seeing advertising on a Google search results page are greater than for any other search service.

One last piece of data I’d like to see, but for which there is no reporting mechanism whatsoever, is number of click-throughs on organic search results. It would help marketers to know how many results per query people click through on.

Still, the overwhelming perception in the industry is that Google dominates search, even if that perception is based solely on the total number of queries performed. What we see here is that each search service has a core audience that repeatedly uses the service, but the overall health of a search service may be indicated by the ratio of queries-per-visit. That particular value is not reasonably available to us. Yahoo!, Time Warner, and Microsoft all offer a great deal of non-search related content. Is it possible to adjust these metrics to show that Yahoo! and Time Warner actually produce higher ratios closer to Google and Microsoft respectively?

If search industry analysts looked at more useful data like these ratios, Google’s dramatic lead would be reduced to a bold front position. People would then rightly be led to ask whether quality of search is a measurable metric and if it is indeed a metric that should be measured (I think so, although this analysis does not make the case). The fading stars of Microsoft, AOL, and Ask would not seem quite so tarnished under this kind of more detailed scrutiny.

While Ask and Lycos (for whom I had too little data to incorporate into this example) both seem to be losing traffic according to various metrics, I think it would behoove all the search services other than Google to talk about the quality of their users’ experience. They have lost the war on number of queries performed. That metric screams GOOGLE every month. Yahoo! and Microsoft, at least, look much better if they focus on what happens when people use their search tools. That is where brand wars on the Internet can change directions. Proposing a reasonable, more informative but alternative metric to the one where you’re not performing well at least offers you a fighting chance to make your case before you really do start to lose market share.

I’m yet to be convinced that Google controls search as much as many analysts imply it does. The numbers just don’t add up. Not yet.