I did struggled sometimes to attach my USB device with my UBUNTU 12.04 virtual machine, found a solution and thought I would share it.
Open your Virualbox, Select the Session you want to start, Click on settings in VirtualBox and you will see a prompt select USB under the Ports tab
Click on the plus icon, which will list all the USB devices attached with your host computer.
Select the USB device you want to access in the Virtualbox, here I have selected the HP USB drive.
The most important thing is unmount the usb device from your host operating system desktop.
Open the Virtualbox Ubuntu session, now your USB device will be mounted automatically in the Virtualbox session.
If you want to reattach the USB device to the host, unselect the USB device from the menu bar (Devices -> USB Devices), now USB device will be reattached back to your host desktop.