It will slow down the loading that the OS has to do, once everything is loaded into the system memory though its not that big an issue. We are also talking about if a hard drive manages to die mid set, which is very very highly unlikely - so a thumb drive being slow is going to be the least of your worries if this happened.
Carrying around an Esata hard drive in case if a hard drive fails mid set is a little crazy, they are much more expensive and waaaay bigger. Your chances of actually having to ever use a backup drive is soooooooooooo slim.
Bookmarks