Mfcttrf

You begin by listening to individual instruments, to learn what they sound like, and the variations in their sound. Solo pieces, quartets, etc. are one way to do this. Searching Youtube for instrument tutorials is another. I got my start in instrument recognition with Piccolo, Saxo et Compagnie when I was about 3 years old.
Then try recognizing instruments in larger ensembles, working your way up to full orchestra. This is a matter of practice.

TV registrations of an orchestra can help, because they allow you to see who's playing. Or find an orchestral score and read along.

You won't be able to hear everything at all times. The difference between 12 and 14 violins is subtle. Softer instruments get drowned out by louder ones: if the difference in volume is large enough, you won't be able to hear the softer instrument at all, due to limitations of our hearing.

A good sound system (and good recordings) helps. Cheap loudspeakers sound 'muddy', obscuring detail when you play complex music. Youtube sound quality is pretty bad. Play a well-recorded CD on a decent hifi system, and things get much clearer.

All this is one aspect of a more general process for training your hearing. Sound engineers do a lot of this: learning to distinguish sounds, the effect various settings on the mixer have on sound (equalization, dynamics processing, other effects).

Musicians do this too, I suspect: when you play your instrument a lot, you learn how to produce various sounds with it (depending on the instrument, you may have more or fewer options for this), you learn what happens when you play in an ensemble etc.

It all starts with active listening: focusing entirely on what you hear, and trying to analyze it, rather than treating music as a background to other activities. It helps to practice with a more experienced person who can point out interesting bits in the music: I've had many occasions where a sound I was struggling with became clear (and recognizable to me ever after) when another engineer pointed out what was happening.

There are several compositions written in purpose to present the orchestra, the instrument sections and the single instruments to the audience and especially for children like Peter and the wolf (Sergej Prokofiev) and The young persons guide to the orchestra (Benjamin Britten).

You may listen to any youtube videos, watching or only listening.

a) camera focusing on soloists and sections

b) videos showing sheet music and the score

Starting with Solos, ensembles, chamber orchestras, symphony orchestras.

I use to practice this game like a puzzle when listening radio music:

Identify the instrument, the era, the composer, the form, the genus ...

Edit: So lets add some videos!

Peter and the wolf

Y.P.Guide

Bolero

Symphony no. 7

Moldau

Romanian folkdances

1. Get an idea, how each instrument sounds separately (as covered in
   other answers)


2. Train to read scores of ensembles with increasing
   number of instruments while listening. For orchestra pieces like symphonies start with smaller setting like Haydn. (Mahler, Berlioz, Wagner are a league above). Pieces with a singer are helpful (e.g. cantatatas and oratorios), since you can use the sung text to re-synchronize if you lost position.

I would not expect to achieve perfection soon, so you may fail to distinguish between two and three clarinets, or viola and violin, or in some ranges even between different instrument families like bassoon and French horn.

Knowledge of instrument history also helps: because the clarinet was invented comparatively late, if you detect a similar sound in Vivaldi, it is more likely to be chalumeau. Similar transitions exist with other instruments as viola da gamba
vs. violin/viola/violoncello, dulcian vs. bassoon.

I would find a good recording of a work of one of those composers and get a printed full score. In many cases you can get these from imslp.org . Listen to the recording and follow in the score. Repeat this until you know what's going to come, before you hear it. This takes time and patience, but you can really gain an insight into the orchestration and works, as well as recognizing the instruments by their sound.

The simple answer is, if you can, join an orchestra (or concert band) yourself. Or, if you can't play yourself, sit in on some rehearsals (with a score or borrow a part) and pay attention. I seem to have absorbed this ability without even trying or even being aware of it. And, it probably only actually took about a 6 months to a year of weekly rehearsal attendance (bonus - you also get to find out the shapes of the boxes for all the instruments, too). A conductor will often call for (e.g.) "flutes, oboes, clarinets only" during a rehearsal - some tricky part where those instruments play together. You get the visual feedback of seeing people pick up their instruments and a sound that is unhampered by speakers or digital compression. It doesn't have to be a professional group, in fact, it might even be better if it isn't. More rehearsal and more breakdown of the group will help you to identify different instruments.

If you can't do this, then solo instruments (unaccompanied) first, build up through quartets (say, wind quintets) to brass bands, wind bands and chamber orchestras and eventually full orchestras. Understand that sections (brass, strings, wind, percussion) can often sound very similar, so try to identify sections first rather than individual instruments.

Thought about this for a while and finally came down on this division: You gotta start by breaking the entire set of sounds into groups. Usually called "range" When considering the frequency range we have to agree on some terms..but you probably have already heard them. Bass sounds are the lowest, Tenor is mid low, Altos are mid high and Sopranos are the highest. The way to think of this visually is by looking at a piano. Humans generally can hear in a frequency range between 20hz (left end of the piano) and 20,000hz (or 20kHz, the right end of piano) with a 440h "A" note near the middle of the piano.
This classification increases along an exponential curve, which is why a piano with 88 keys represents a difference of around 19,000hz, rather than about 100hz.
You can look at a visual EQ monitor in iTunes..or on your dad's 70's era "receiver" that he still keeps plugged into a 5 disc changer, despite the fact you keep buying him the newest iPhone.. and get a quick understanding of how this works. If you turn the two left-most EQ knobs all the way up and the rest all the way down, you'll hear a lot of Bass and probably mess up your speakers pretty quickly, but you'll lose practically everything else.
... and at the risk of stating the obvious, we know the low end of the frequency range is bass sounds, but in an orchestra that might include Tuba, Bass Clarinet, Baritone saxophone and the kettle drums. The very high end frequencies will be flutes and clarinets, trumpets or even higher pitched brass horns, cymbals etc..
A dog whistle is pitched above 25khz and usually closer to 40kHz...which is why it will never bother you but it will send Spot scrambling in circles. If you DO find you can hear a dog whistle, check with your doctor to see if you're turning into a Werewolf.

This division along frequency range might sound a bit arbitrary, or at least not specific enough for learning to distinguish instruments, but you'll find that training your ear for timbre (an instrument's specific quality) is a lot harder than training your ear to notice frequency. Once you learn to recognize frequency range (and you'd be surprised how with a little bit of training you'll quickly be able to pick out notes AND their exact Hz) you'll find you can easily separate groups of instruments IN TO range .. and after THAT you can start distinguishing the instruments themselves.

Personal Anecdote feel free to skip
I started training as a violin player at age 3, and I was taught by the Suzuki method which wasn't that common when I was taught. Suzuki method worked thusly: The teacher would play a phrase. The student endeavors to play it back with no other visual cues (written music, etc) The crazy thing is I hate the violin now, but I've been trained on several instruments and that ability to repeat a phrase after hearing it once has never failed to make learning a new instrument easier. I can't really know if other ways are better, but I think it helped my music career.

Years later, studying audio production, the only ear training method that all recording students were graded on (regardless of whether you played an instrument) was called "GoldenEars" and it was based solely on frequency recognition.

So that's my story, and I'm stickin' to it.

PS. =Can Software Pull the Saxophone out of an Ensemble Recording?=
I really like the concept of software that could pick out an instrument from a recording, and I imagine it isn't as far off as we might think, the codecs that allow us to make mp3 files out of much larger "real sound" WAV files could be viewed as the opposite of this, as they work by removing bits of "redundant" information from the larger files. Most people now 20 years on from Napster can hear the difference between an MP3 and a WAV but there's an accepted level of decay that most people are willing to live with. If we can program the removal of a certain percentage of information in a sound file and still hear basically what we began with, I have to imagine that pulling a certain classification of samples and bits could give you just vocals or just instrumentals. It would also depend a great deal on the type of ensemble. Pulling just JOhn Coltrane out of the Miles Davis Quartet wouldn't be all that hard, as the Tenor sax is easily distinguished from Miles' Trumpet and Paul Chambers drums. But an orchestra situation obviously creates new hurdles. Regardless, the people who will crack this will of course be DJs. Necessity is the mother of invention and DJ's are constantly looking for instrumentals and Vox-only tracks... So, as usual, the street will break it, probably via something we already have access to, and then ProTools will quickly commodify it and we'll pay big bucks for said filters. You heard it here first!

@bobbymobetta

I am not too good in music, but, different instruments sound different due to their sound containing different amplitudes of frequencies [mainly harmonics, but also "fixed" ones e.g. due to a violin frame vibrating] in addition to the "nominal" one [ =the tone played, like the 440Hz A ]

With the available computing power, I would be surprised unless you could find a mobile app or at least some Apple/PC SW that filters out a given instrument from the orchestra sound. Maybe also gives you an "equalizer" that allows you to increase/decrease the volume of a certain instrument, and mute others if you wish.

If you can't find one, someone will hopefully make one soon, inspired by your question, as, anyway, this is an SE forum.