Since Google's meteoric rise, many self-proclaimed Google-killers have come along. Obviously, they were more smoke than flame. Google is bigger and badder than ever. The most recent is probably Cuil and Microsoft's reanointed Bing, but none of them has of yet made any meaningful dent in Google's size. The reason is simple. The search they offer is inferior to Google's. Google has diversified a great deal, but their bread and butter is search. If Google lost at the search front, their other applications would not sustain them. Sure, there is some advertising space in GMail as well as Google Earth, but virtually everyone who owns a PC has used and knows of Google search. A smaller percent use Google's more expansive offerings. So, to defeat Google one of two things must happen. Either Google has to be beaten on the search front or the internet itself as we know it must become irrelevant. The latter is hard to imagine, but, then again, the fall of the mainframe and the internet would have been hard to imagine a ways back.
So, as things stand, to take down Google, our hypothetical company must defeat Google in the search arena. Again, Microsoft and Cuil are, so far, thinking along the same lines. The problem is that they are not really building a better search than Google is. Google is not invincible, here. With Googlebombing and Googlespamming on the rise, the signal to noise ratio in Google searches as dropped off noticeably. Any search engine that mirrors Google's algorithm will fall to the same problem. Research needs to go into what it takes to build a next generation search engine. In its essence, Google's algorithm takes some combination of popularity (i.e. links to the page) with the number of times that the search words (with some fuzziness built in) occur. It is actually a very good little algorithm, but we are seeing its weakness. I propose that the next generation search engine (whether it is built by Google or someone else), hereafter titled NGSE, will have to be a little more intelligent. In fact, I would go so far as to say that this NGSE is little more than a massive artificial intelligence riding the back of the spiders that crawl the web now.
Work in artificial neural networks (ANN) and pattern matching would be key in something like this. Rather than looking dumbly at what the page says its about and what other people say about it (and you'll notice that the problem with links leading in is that it does not indicate that the person linking actually liked the site; they might be linking to it to run it down), it tries to see if the page matches the pattern of the person searching. This sort of engine would be based on what you mean, not just what you say. It would take the semantic web to the world, but without requiring the world to adapt to it, as every proposal has so far.
Case in point: Google Images. From firsthand experience, I can say that when I search for images there are almost always better matches for what I was looking for than what Google brought up. If I had to guess, I would say that the alt attributes on the img tags go a long way in determining the ranking. The caveat is obvious. Most web masters do not put alt attributes on all their images, even if they should. Imagine, instead, an ANN that could, with a high degree of success, scan the image and deduce what it is showing. The more precise it was, the better it would be at showing the users what they wanted to see.
Whether or not an ANN/spider based search engine is even really feasible is an open question. Especially for the image matcher discussed above, nothing close has even been created. Even if we could build such an engine, would its computational cost be prohibitive to using it across the whole internet? After all, one of the keys to Google's success was their ability to paralellize their algorithm on the massive scales required. Ultimately, something like this would have to be built to defeat Google on their home turf. The way to win when the opposition has a massive advantage is to be significantly better. Parity just doesn't cut it and Google got where they are for a reason. Their methods are nothing if not sound.