Google Base + Vertical Search + RSS = Death of Walled Gardens
A few weeks ago, I promised to write a follow-up on my post about Google Base that detailed how the launch of Google Base might affect the Internet’s so-called “Walled Gardens” (content sites that charge users and/or suppliers for access to their databases). One month and a long cruise later here it is...
When I think of Walled Gardens on the Internet I am reminded of one of the many priceless scenes from Monty Python’s Holy Grail in which the King of “Swamp Castle” explains that he has built three successive castles in the same place only to see the swamp subsequently swallow each castle. The king ends his speech by declaring that the 4th castle was the strongest in the land despite all historical experience to the contrary.
Much like the delusional King in the Holy Grail, many of the Internet’s biggest and most profitable “Walled Gardens”, sites such as Monster.com, Realtor.com, Match.com, and even EBay, appear to be in denial about the ultimate destiny of their sites, which is, that they are bound to be subsumed by the larger Internet.
History Repeats Itself
Not that this should come as a big surprise. Just look at the original “Walled Gardens”: Compuserv, Prodigy and AOL. In their heyday these powerful “online services” controlled every piece of content on their networks and extracted princely sums, often from both content providers and customers alike. With the advent of the Internet all three of these players tried to hold onto their Walled Garden strategy only to see the growth and diversity of the Internet overwhelm their own suddenly meager offerings in just a few short years.
The New Old Thing
Flash forward to today. A set of “new and improved” Walled Gardens have been built only now they are known as “paid listings” sites. Want to find a date? Better be prepared to pay Match.com a monthly fee. Want to sell a house? Better list it though a realtor with access to Realtor.com. Want to hire someone? Better pay Monster.com some coin. You’re a small business and need a heavily trafficked distribution channel? Be prepared to pay EBay a hefty fee.
The primary value-add that most paid listings sites offer is that they aggregate, structure, and index similar content into one coherent “site”. Early on in the Internet’s evolution these sites were literally the only place on the web that one could go for this kind of information. The sites that successfully built scale, brand, and network effects usually ended up the “winners” in their respective categories. Once they became winners, they were able to charge a healthy premium and thereby, in most cases, become highly profitable businesses.
What A Difference 10 Years Makes
The problem for these Walled Gardens, is that much like the online services before them, they have been built on a quickly shifting and deeply flawed foundation. The Internet of 2005 is a far different environment than the Internet of 1995. In 1995, the average user couldn’t spell “Internet”, let alone figure out set up and run their own Internet site. Even if they did set up their own site, the lack of any mechanisms for other users to easily find a site made operating one’s own site a pointless exercise as it was like building a billboard on deserted island in the middle of the Pacific.
In 2005, things are a bit different. Not only has the average Internet user become much more sophisticated, but several trends are rapidly coalescing to deliver what could be a “knock out” blow to the Walled Gardens. These trends include:
- Self-publishing: Thanks to dramatically lower hosting costs and greatly improved software, today most businesses and a rapidly increasing number of individuals have their own Internet sites. Businesses in particular have quickly taken to publishing all kinds of information on their sites. For example, most companies have a section of their site where they list job openings and in the real estate industry almost every agent has their own web site with detailed descriptions of their current listings. Of critical importance is that none of this content is behind a “wall”. It is typically out there for anyone who happens to stop by to see. The net effect of all this self-publishing is that there is now a ton of openly accessible “primary content” just sitting out there waiting to be indexed and manipulated by anyone who chooses to.
- Pervasive Search: Sophisticated index search has become the glue that ties the entire Internet together. With search, no site is an island (or a Walled Garden) unless it chooses to be. Search has in essence leveled the field when it comes to distribution and brand. You can spend $1M on an ad in the Super Bowl, but it won’t change your search results.
- RSS: RSS is a content syndication standard which makes it incredibly easy for people to subscribe to content from a particular site. Want to be notified every time someone updates their personal profile on their blog, or every time a company adds a new job opening, or every time a realtor gets a new listing? Subscribe to their RSS feed. What RSS does is that it provides an automated way for the web to “feed” new information directly to interested people and computers.
Collectively these trends are transforming the Internet from a disparate, sparsely populated frontier with a few major cities into a teeming highly integrated metropolis. It’s as if everyone on the Internet went from living on a deserted island to all living together on the same block in Manhattan.
A World Of Hurt
In such a close-knit world the foundation of paid listings sites is starting to look more than a little shaky. About the only thing that is still missing is the ability to relate listings within a taxonomy or consistent hierarchy. However that barrier is also rapidly eroding. The first visible signs of such erosion came when a set of “vertical” search start-ups put 2 and 2 together and realized that thanks to self-publishing and pervasive search, they could create highly focused “vertical” search engines that trolled self-published Internet sites for specific information and then republished this information as one aggregated site. (See Jeff Clavier's blog for more in-depth coverage of vertical search.) Seemingly overnight these start-ups have been able to build databases that in many ways rival those that the Walled Gardens have taken years to build. For example, Trulia and Home Pages are searching individual realtor sites to build up databases of real estate listings that in many areas are bigger and more detailed than the ones available on the venerable Realtor.com. Simply Hired and Indeed and doing something similar in the job listings space by indexing job listings from company sites and other “open" gardens.
That start-ups have been able to become so big so fast is a very bad omen for the existing Walled Gardens, but it is nothing compared to the world of hurt that is represented by Google Base (and the undoubtedly soon to follow copy cats from Yahoo and Microsoft). What Google Base represents is an attempt to marry the world’s best search infrastructure and technology with a highly structured and highly automated way of gathering, storing, and presenting listings information.
As I mentioned in my previous post, Google base is essentially the world’s largest XML database. If you take the time to read through the XML schema you will see that Google has essentially already built all of the components that it needs to enter the vertical search space in a big way; all it needs to do now is refine a few algorithms and flip a switch. If and when it does so, it will undoubtedly not only the have the largest collection of listings in all major categories overnight but will also have arguably the best distribution channel for those listings on the Internet. This is not good news if you are currently charging to either to display or to access similar listings (or if you are a vertical search start-up).
It is apparent that Google would prefer to have users flow information to Google Base via automated RSS feeds, and that is still probably the long term goal (services to help users do this have already been created). However, if they decide to “prime the pump” by doing a little indexing of their own, watch out, because then Google can start automatically feeding millions of listings/day into Google Base. Shortly thereafter, there will be population explosion of X.Google.com domains, as in Jobs.Google.com, houses.google.com, cars.google.com, etc., etc., etc. as Google launches “listing sites”, which will really be nothing more than a thin skin on top of Google Base and their indexing/crawling engine; think of it as Google News on steroids.
In fact, if you haven’t checked out Google Base in the last few weeks since the launch announcement you should probably take another look. In just a few short weeks of beta testing users have voluntarily submitted well over a million items to the database despite the lack of user friendly tools to do so. Once at the site, take each of the major Walled Garden Sites you can think of and match them up against each of the major categories that Google Base is currently listing. As a Wall Garden Executive might say: “Uh Oh!”
With Google Base fully in place (and ultimately similar services from Yahoo, Microsoft, and Amazon), why in world would anyone pay to have their listings displayed or pay to have to access to a database of listings? After all, if you publish the listing on your own site Google will automatically index it and then list it within Google Base within the next few days and if you want to make sure they get it immediately, you can just submit if directly to Google Base or register your RSS feed with them (a feature I'll bet they are likely to add). Instead of charging you (or its end users) for the privilege, Google will make money off of the advertising it sells around the listings. Perhaps you may even be able to pay a fee to have your particular listing “advertised” in a preferential position.
Whatever the case, the end result is clear: the Walled Garden sites are in for a world of hurt and unfortunately for them it will probably come much more quickly and severely than it did to Compuserv, Prodigy and AOL. That’s because the online service providers at least controlled people’s dial-up access to the Internet and thus were initially protected by very high switching costs. No such protections will be afforded to the listings site though. Since most of them sell short term services with little or no customer lock-in, the effect of services such as Google Base and its associated off-spring, will likely have a much more immediate and painful impact.
That’s not to say that all of the paid listings sites are headed for imminent Armageddon, but it is to say that they shouldn’t renew their lease on Swamp Castle because it is clearly going to get swallowed up yet more time.
I will follow up on this with a post on with a ranking of the “most vulnerable” Walled Garden business and what, if anything, they can do to avoid sinking into oblivion.
Is RFID Application Oriented Networking’s Killer App?
RFID is a widely hyped next generation technology for wirelessly “scanning” items. AON, or Application Oriented Networking, is widely hyped next generation technology for managing middleware messages. Put the two technologies together and not only do you have a whole lot of hype, but you may also have a match made in technology heaven, one which ends up making both technologies better.
AONs Second Generation
I have written about AON, or what I prefer to call Message Aware Networking, quite a bit in the past. AON is at the core of a revolution in software architectures that is moving a significant amount of functionality out of applications and pushing it down into the network. The first generation of AON devices act as simple gatekeepers/routers and are limited to pre-processing messages that they then deliver to applications. Datapower, which was recently sold to IBM, was the leading player in this space. While the space is still quite young, already a 2nd generation of AON devices is under development. These devices will not simply serve as intermediaries, but they will actually become an integral part of an application and an embedded part of a business process. By embracing the concepts of distribution and virtualization, as well as standards such as BPEL, these 2nd generation devices will turn the network into an integrated part of a given application and in doing so should enable significant new functionalities while reducing costs and improving performance.
RFID: Almost Ready For Prime Time
RFID seems worlds away from AON, but RFID’s future may be more closely tied to AON than one might at first suspect. Much of the attention around RFID has focused on the core hardware technologies required to make RFID a reality. Not only are there very difficult RF-related physics problems that must be solved, but the costs of silicon-based RFID “tags” must be reduced in order to make RFID economically viable. Now however, with companies such as Impinj shipping cost effective RFID 2.0 tags in volume and with retailers like Wal-Mart mandating RFID adoption, it appears as if RFID is finally poised for rapid growth.
Unfortunately for RFID, if anyone ever does install the hardware, the first thing they are likely to do is crash all of their software. That’s because a fully instrumented RFID installation is bound to kick off thousands, even millions of messages a day and processing all these messages is likely to bring even the most robust computer system to its knees. To solve these performance issues companies will either have invest a fortune in significantly upgrading their core logistics systems or they will have to figure out another way to get the job done.
That other way may be a new class of AON-based devices that are specifically focused on RFID. Such devices could not only locally process millions of RFID messages, thus greatly reducing the overall “load” generated by RFID systems, but they could also offer new capabilities, such as the ability to efficiently “multiplex” messages across an entire supply/demand chain. For example, AON-based RFID devices could use cached lists of expected shipments to identify in real time when specific shipments failed to arrive or arrive incomplete. These devices could also simultaneously notify all of the different supply chain members about missed shipments or out-of-stock events. While a retailer’s or distributor’s centralized computer systems could no doubt do all of this, constantly hitting those servers with updates on every single SKU from every single store, warehouse, truck, etc. would not only be wildly inefficient, it may also distract those systems from more important higher level tasks. In this way, AON-based RFID devices provide a relatively cheap way to “off load” to the network the processing of routine RFID related messages.
Startups To the Rescue
But don’t take my word for it, just look at the start-ups that are springing up to take advantage of this potential opportunity, the most interesting of which I have seen to date is a company called Omnitrol. Omnitrol has built a very intriguing RFID focused AON-based device. Unlike many 1st generation AON players which used Intel-based Linux “pizza boxes” as their hardware platform, Omnitrol has engineered a sophisticated hardware platform that uses Gigabit Ethernet to interconnect multiple RISC processors allowing for greatly increased message processing capabilities within a single box and the ability to seamlessly interconnect multiple boxes. Such an architecture not only guarantees that Omnitrol should be able to process massive message loads, but it also offers a great example of how 2nd generation of AON devices are likely to use much more sophisticated hardware platforms that look more like network devices than enterprise servers.
With Omnitrol’s AON-based RFID devices in place, retailers can keep a large portion of their RFID processing “local”, i.e. confined to the actual store or distribution center where it takes place. Only critical events and/or summaries of activities need be forwarded to the central system (which in most big retailers remains an IBM mainframe). The RFID devices have also been designed from the ground up to be compliant with the various EPCGlobal RFID related standards and have the ports needed to accommodate a variety of RFID readers. In this way, Omnitrol has customized both the hardware and software of its device so that it is RFID centric.
In addition to Omnitrol another start-up focused on this space is Reva Systems. Reva is already shipping a device that is focused on managing RFID networks and clearly has designs on providing more sophisticated “application aware” capabilities over time.
Despite their promise, the success of these RFID-focused AON start-ups heavily depends on how quickly large companies embrace and roll-out RFID. While recent signs seem to indicate that adoption should dramatically accelerate in the next year or two, this is still a major risk.
A Sea Change?
Despite this risk, both companies are great examples of how Application Oriented Networking is rapidly evolving. Perhaps the most interesting trend that these devices underscore is the ability to tailor an AON device, at both a hardware and software level, to a specific application. This tailoring raises the interesting question as to whether or not we will see an explosion of application specific AON devices, each targeting a different enterprise application. Such an explosion may not only provide a whole new class of investment opportunities, but it may also significantly challenge the dominance of general purpose computing platforms in enterprise software. After all, if you move enterprise software into the network, why won’t the devices used to process that software look a lot more like network devices than enterprise severs? Of course, we are a long way off from such a change, but it is interesting to contemplate and once again underscores that Application Oriented Networking is one of the most important and potentially disruptive trends within the software industry.