Byron:
It is maybe not a robot crawling deep trough websites to index them and find new links, but it does crawl trough the search results of google looking for matching lines. Since open directories are often so big, this is a lot easier. With the build-in player the issue of pop-ups etc. is eliminated.
Suika:
Thanks!
What happened to the pirate bay was one of the reasons why I questioned this. It's a bit hard of course, but I reckon the pirate bay wasn't really a criminal site. Like you said, it didn't host files, well torrents, but that is nothing more than a textfile containing the tracker(s).