The Algorithm Myth And Why Google Will Be Hated

Google’s search algorithm, The Algorithm, is arguably the single most important computational routine in the world right now.  As I outlined in my last post, for many companies The Algorithm literally controls their entire economic destiny and as Google’s traffic and market share grows, so too grows the power of The Algorithm.  Traditionally, concentrating so much power in hands of one company has provoked great populist opposition, and while unease with Google’s power is clearly rising, this unease is nowhere near the level of wrath directed at the dominant companies of earlier generations such as Standard Oil, IBM, or Microsoft.  Ironically, it is the heart of Google’s power, The Algorithm, which has in many ways enabled Google to greatly mitigate the build-up of popular resentment.

The Algorithm, as we have been told many times, is an objective, unprejudiced, egalitarian and ultimately democratic computational routine.  It does not play favorites or king maker nor harbor grudges or biases, it simply reflects reality.  In short, it does no evil.  This positioning of The Algorithm provides Google with a compelling defense mechanism against anyone that would question its motives or methods.  To question Google’s search results, is in effect to question objective reality.

With all this in mind it was very interesting to read a great article in Wired recently that describes in breathless detail the process by which Google makes changes to The Algorithm, changes that are likely to number more than 500 this year alone.  Wired dutifully casts Google’s engineer’s as the High Priests of Algorithmic purity working secretly and securely to enhance the end-user search experience. The Wired article is a triumph for Google’s PR machine because it maintains and enhances the Algorithm’s mythic status even though the truth is that The Algorithm myth is dying before our eyes, an unavoidable victim of it’s all encompassing success.

Dying you say?  How in the world could The Algorithm, the very engine of Google, be dying at the same time that Google appears to be on the cusp of total search domination? Look no further than the text of Wired’s article.  While many of the hundreds, if not thousands, of changes that Google will make to The Algorithm (and the Algorithm’s lesser know supporting cast of crawlers and indexers) will indeed be focused on making it easier for the end-user to find the proverbial needle in the haystack, what’s left unsaid in this article is that many, perhaps even the majority of these changes will be made as part of war; a war against those who are actively gaming The Algorithm by manipulating the very reality it attempts to reflect.

Google vs. SEO
And just who is this that is “gaming” The Algorithm?  Just about everyone on planet earth.  There is a large and rapidly growing Search Engine Optimization (SEO) industry focused exclusively on this very task.  Within this industry there are the “white hats”, those who use a stable of Google-approved or at least Google-condoned “site optimization” techniques, and then there are the black hats, those who actively seek to game or trick Google’s indexers.  Most reputable companies try to adhere to “white hat” techniques, but with the stakes so high, the economic rewards so huge and the playing field constantly evolving, there is a very slippery slope towards gray and ultimately black techniques.

While Google lightly encourages White Hat SEO (it makes Google’s crawlers more efficient and its results more accurate and complete) it abhors the Black Hats.  The Black Hats are seen are seen as public enemy #1 (perhaps more so than click fraudsters) because they corrupt the platonic integrity of Google’s results and weaken the overall user experience thus undermining the very foundations of Google’s success.  When discovered, Black Hats are often dealt swift and summary justice from Google.  Not only are page ranks reduced but in many cases pages, sites, and even entire IP address ranges receive what has become the information age’s ultimate punishment: banishment from the Google index. But such justice doesn’t solve really the problem, because the Black Hats merely decamp to another URL, another IP and start all over again.

War Over The Algorithm

The primary focus of the Black Hats is The Algorithm.  By reverse engineering how it works, they seek to create ways to trick it into elevating specific pages within the Google index with the ultimate goal of achieving the Holy Grail of SEO: becoming the #1 organic search result.  The war goes back and forth in volleys of ever increasing complexity.  The Algorithm prizes term repetition, the Black Hats repeat a term hundreds of times on a page.  The Algorithm values home page links, the Black Hats create paid link exchanges.  The Algorithm elevates back links, Black Hats create multi-layered networks of “back link farms” for hire.  It has gotten to such an extreme that it is rumored that there are thousands, some say millions, of “robo-users” on Google now, computers designed to give false feedback to Google by clicking through on specific search results just enough to raise their index rank without tipping off Google’s counter measures.  In this way,  SEO has gone from just “optimizing” content on a single site, to networks of sites, to building an entire closed loop of sites, networks, and virtual users.

Google isn’t blind to this behavior.  They know what the Black Hat’s are doing and they are fighting them, hard.  As Wired ably chronicles, their search engineers are constantly making hundreds of changes to The Algorithm, many with the explicit or implicit goal of filtering out the “noise” introduced by SEO.  While these high-tech efforts to fight Black hat SEO within the context of The Algorithm are reasonably well known, what is less well known is that increasingly Google is manually tweaking output from The Algorithm to ensure “search quality”.  Literally hundreds people around the world spend every day typing searches into Google and making sure that the correct results appear.  Some of this is to ensure search results reflect local culture (e.g. “football” results in Europe are focused on soccer while “football” results in the US are focused on the NFL), but much of it also focused on making sure that the Black Hats are not winning the war.  Google is also rumored to manually review the search results for particularly high profile or high value search results and to be manually intervening when the results diverge from the Google’s platonic ideal.

The Mother of All Negative Feedback Loops

The great irony in all of this is that Google’s efforts to defeat the Black Hats and to ensure “search quality”, arguably very laudable goals, are ultimately undermining the credibility and sustainability of The Algorithm.  The more Google “tweaks” The Algorithm in the name of “search quality” the less The Algorithm reflects the objective reality of the web.  A pristine mathematical routine becomes, by necessity, a mish-mashed labyrinth of math routines, rules based logic and arbitrary exceptions/exclusions.  Google is in essence caught in the mother of all negative feedback loops.  Its success is driving more and more SEO which makes reality a noisier and nosier mess which necessitates that the Algorithm depart further and further from objective reality.  Google’s efforts, laudable though they may be, to create the platonic ideal of Internet “reality” are thus doomed to become Google’s editorialized viewpoint on what should constitute Internet reality as opposed to what really does.

The End of the Algorithm Myth

In this way it is almost a done deal that The Algorithm will lose its mythic status as an objective and unprejudiced computational routine.  It will merely become Google’s point of view on what should constitute reality and once it does that, every single change Google makes to The Algorithm, every manual tweak, will be open to question: “Why did you make this change?”, “Who benefits from this change?”, “What ‘reality’ is this supposed to reflect?”. The Algorithm’s loss of credibility will thus expose Google to charges of bias, of playing favorites, of being, well, evil.

 

What’s more, it’s unavoidable that in Google’s war against the Black hat SEOs there will be collateral damage.  After all, if you are making hundreds of changes to The Algorithm a year and doing daily quality checks, someone is bound to either make a mistake or make a decision that while hurting the Black Hats also hurts some White Hats.  Mix this collateral damage with the editorialized Algorithm and with Google’s staggering economic influence and you get a readymade recipe for intense fear and resentment of Google on part of website owners around the world.

Today’s Hero, Tomorrow’s Villain?

In the 1980s, Microsoft was the darling of American industry.  Founded by a college drop-out, it took on the world’s most powerful technology company, IBM, and in the process made mass market computing cheap for everyone. Just a decade later, the popular perception of Microsoft had changed dramatically.  It was now an unscrupulous monopolist who abused its position of power to extort monopolistic rents and used its vast wealth to unfairly crush competition in adjacent markets by literally giving away things such as the browser and music player.

During the first decade of the 21st century, Google has become the new darling of American industry.  It has taken on Microsoft and Yahoo! and won.  The best and brightest want to work there.  With the inevitable decline of The Algorithm myth, Google will lose one of its most important assets, its reputation as a objective broker of the democratic truth.  How it responds to this loss will determine whether or not it faces a similar fate as Microsoft.  At one level, there may be nothing Google can do to change its fate.  There is bound to be a reaction to the accumulation of vast power and wealth no matter how innately good or evil its owner might be.  However Google will have choices.  Will it continue to develop its Algorithm in secrecy even as its objectivity clearly wanes, potentially fostering conspiracy theories and resentment? Or will it publicize the Algorithm hoping that such transparency builds trust and credibility?  Or will it open-source the Algorithm and seek to disavow ownership of it, and its associated liabilities, entirely?

Personally I have no idea what they will do.  The good news for Google is that they have a bunch of incredibly talented people at the company.  I know many of them and they are some of the brightest and least evil people you will ever meet.   But I also know that the Algorithm myth will die and its death, along with Google’s immense power, will pose one of the great management challenges of this decade.   The betting man in me says more people will hate Google than love it in 10 years, but 10 years is a long time and fate is what you make of it.