One of the pitfalls of being an early adopter is that you get to be the guinea pig. I upgraded to iOS 4 and my battery is getting wiped out on my iPod touch. I have a 2nd generation iPod touch and I've never seen anything like this before. I really hope Apple releases a patch soon since I rely on my iPod when I drive to work and when I'm at the gym.
It sounds like others are experiencing similar problems on the iPod touch and iPhone. Make sure to carry a spare power source if you recently upgraded your iPhone or iPod touch to iOS 4.