TedPage: What do we want to search?

As web crawlers like Google and Yahoo continue to index more and more of the web, we have to start wondering what will be the end of all their work. Will every page get indexed? Will that help me find what I really want on the Internet?

I'd conjecture that most people want to search and find websites that they've seen before, or ones that are closely related to websites they've seen before. It seems like the time may be upon us for a "personal webcrawler".

Basically the idea here would be a webcrawler that would start its search through items in your web browser cache. When you visit a website, it would look at the links and start to index those sites along with the data already in your cache. In the end, you'd end up with a very personal search space, that has things that you're very interested in. Tie this into something like Beagle, and you'd have the ability to find the stuff that you want -- not the rest of the Internet.