I finally got around to setting up Google sitemaps on my DVD commentary database and review site. It was a lot easier than I was expecting. A lot of the sitemap structure is optional, so I just ignored things like the last modified date field when first setting it up.
Once I got it working, I added a new database field for each entry to keep track of when the page was modified (new review or new entry; new votes might change the rating, but not significantly enough for Googlebot to need to re-index it). Most DVDs in the database don’t have anything in that field, but new titles will get it added automatically, and any new reviews will update that field as well. So as time goes by more and more pages will wind up with last-modified dates. Right now it’s already up to 116 (out of 2774 pages) from what’s been added or modified in the last week or so.
A few things (like adding more commentary tracks to a DVD entry) aren’t automated for the site yet, so if I’m diving into phpMyAdmin to add those, I’ll just update the last modified date by hand at the same time.
I haven’t really seen any change in Googlebot traffic since adding it, though. Oddly enough, a week or two before I set up the sitemaps, Googlebot went nuts on my site, getting 11,659 pages over a 2 week period. 7,562 unique urls — I didn’t even realize there were that many unique urls on the site. I guess there were 35 different indexing sessions, based on the number of requests for robots.txt? I’d never gotten more than 500 requests a week from Google before that.
After that fell off and I submitted my sitemap, I’m still getting much less Googlebot traffic than that. I guess it is on the high end of what I’d seen before, though (close to 500 pages a week).
It definitely does seem to be getting new pages quickly. The last title that I added was grabbed by the bot 15 hours later, which is a lot quicker than it used to take.
So, go for it. Make your own. It’s probably easier than you expect. Unless you don’t have a web site, in which case, never mind.