If you're not using iZotope or Weiss Saracon for sample rate conversion, you really should be.
All SRC processes unavoidably degrade the audio quality of the original source. SRC algorithms require very steep lowpass filters to reject unwanted high frequency energy at greater than half the sample rate, which is introduced into the signal regardless of whether the target rate is higher or lower than it started out. This level of steepness is necessary in order to retain the full signal within the audible bandwidth while falling to a sufficiently low level at the Nyquist frequency to prevent aliasing, meaning that signals greater than half the sample rate are folded back down into the audible range, creating inharmonic distortion.
Unfortunately, the steeper the filter, the greater the amount of ringing (time-smearing) that is introduced into the processed signal. This ringing can be either pre or post: that is, it can manifest either forwards or backwards in time relative to its original position. Linear phase filters are usually used in SRC in order to preserve the phase relationships of the signal, and this type of filter generates both pre- and post-ringing in equal measure. Such ringing obscures the clarity of transients (high-level, short duration peaks) and gives a general sense of blurriness in the sound.
From these inherent problems with SRC (and it should be noted that the quality of analog-to-digital and digital-to-analog converters is affected by these exact same issues), we can see that there are three parameters the algorithm designer must contend with to obtain the best possible sound after conversion: filter steepness, which determines both the amount of aliasing that is allowed through and the amount of ringing that is introduced; filter phase, which determines the time relationship of the ringing to the original signal; and cutoff point, which is the frequency at which the filter begins rolling off the audio. Higher steepness allows a greater frequency range to be represented and rejects most aliasing, at the expense of the ringing becoming far more severe. Lower steepness avoids problematic ringing artifacts, but at the expense of greater aliasing. Lowering the cutoff point can alleviate both issues, but restricts the bandwidth of the resulting audio by cutting off some of the high frequency range.
Optimal quality can only come from achieving a balance between these parameters, and there aren't many conversion algorithms out there that can be said to preserve the sound quality of the source signal while minimizing distortion. That's really the best you can hope for with SRC—it's never going to be a perfect result, but a well-designed filter can keep the distortion products small enough that they won't affect the quality of what you hear. The same is also true of ADCs and DACs, as well. This is one of the caveats of digital audio: it's only as good as the math that was programmed into it, and outside the usable range of those equations it will fall apart completely, generating extremely nasty inharmonic distortions that analog audio never had to contend with. So if you're going to do digital processing on your sound, you'd better be sure that the programmers knew what they were doing before entrusting your work to their knowledge of mathematics.
I should also point out that SRC should be performed at the highest possible bit depth your system can handle, in order to avoid quantization error (another form of digital distortion, resulting from the bit depth being too low to represent the signal with complete accuracy). Afterwards, you can dither it back down to your target format. The noise floor will be increased slightly by doing this, but the sound quality will be far better preserved.
My recommendation for sample rate conversion is to use iZotope's solution, which is the most affordable high quality converter out there. Some implementations of it allow the user to control the parameters, so you ought to understand what they mean before changing them—hopefully the above explanation is helpful for this. The settings I personally use are: Filter Steepness 32; Cutoff 0.95; Pre-Ringing 0.99. To me, this represents an ideal result, with a relatively shallow filter in order to minimize ringing, a frequency response up to about 19.8 khz before filter rolloff begins, and the filter being nearly linear phase but not quite, in order to push the ringing slightly later in time (since post-ringing is less noticeable than pre-ringing). A small amount of aliasing is allowed through, but it is at a very low level compared to the actual signal and is entirely confined to the range above 20 khz, where it is inaudible to anyone who isn't a dog or a liar. ;)
I hope this clears up some confusion about SRC. It is a complicated subject, and there's a lot of misinformation about it out there, but I'll be glad to answer any further questions. If you follow the advice I've given, your converted audio will be difficult to distinguish from the source, which is the best that it can be.