I find an Alfred workflow to be much quicker (especially with an incoming Skype call!): http://michael.thegrebs.com/2013...

Yeah, this app came out in 2012, before Yosemite. Just hold Option and click the sound icon. You don't need this app.

I thought you could change source by Shift or Alt clicking the sound icon?

Handy method that works better than the system preferences for sound, especially when you are switching between many audio sources.