I may not have to eat my words regarding S3, but the scent of them cooking is definitely starting to waft. Don MacAskill, the CEO of smugmug, describes another case study of commercial usage of Amazon's Simple Storage Service. Lots of business advantages therein, but I will point out that smugmug doesn't completely bet the farm and dispense with its own storage. Maybe that's the right tradeoff though. Develop an in-house storage architecture and carefully use S3 as a "redundant secondary storage for use in cases of outages, data loss, or other catastrophe."
Still waiting for the game changing, highly interactive, AJAXy, Web 2.0 application built using S3 as its main storage system.