Link parkin': Spidering Hacks may have to go on the bookshelf. Maybe it has the answer to the question I've been pursuing for the past few days: Where can I get a decent, off-the-shelf, extensible Web spider/crawler? All I've seen with potential is Harvest-NG.
Hmmm, maybe I just have to change my Google search from "crawler" to "spider".