S/PDIF (digital coaxial/optical) cannot pass high resolution multichannel audio. To do so you must use an HDMI connection to the receiver, and the receiver must be able to decode the new codecs.
Decoding can also take place in the player, with the audio being sent via multichannel analogue connection, but your player supports stereo analogue output only, so this is not an option.
Obviously, you will need a second HDMI cable to go from the receiver to the tv. Such cables can be outrageously overpriced in stores, particularly the Monster brand. Try a website such as www.monoprice.com instead. I don't have HDMI myself but I ordered my S/PDIF cables and speaker wire from there and recommend them wholeheartedly.
Be aware that some devices have issues with 'HDMI handshake', the communication each component must perform with the others to ensure compliance with the copy protection scheme. Problems may arise if a manufacturer has not implemented the spec correctly on a certain model, or in a fashion incompatible with the component on the other end.
Of course, you can still obtain good results from using S/PDIF. While it will not be the full high resolution audio signal, all DTS-HDMA Bluray discs contain a 1509 kbps 'core' track for backwards compatibility. Depending on the quality of your speakers and audio equipment (not to mention room acoustics), this may well be nearly indistinguishable from the uncompressed version. Similarly, discs with PCM or Dolby TrueHD tracks also contain AC3 versions, usually at 640 kbps.
Regardless of your connection type, check your Bluray player settings to ensure that it is outputting a 5.1 signal and not a stereo downmix.