Here is a picture of my settings screen for audio: the selected one will give analogue audio out on the 3.5mm socket as well as optical, from the same (there should be a red light inside the socket if the optical is enabled)
What I have found is that sometimes after a reboot or an upgrade (but not often enough to figure it out yet!), I have to select PI:HDMI, let it be enabled (leave the screen so the option is set) then go back and select the ALSA: Default (snd_slice Analog) and the sound comes back fine.
Another thought: have you tried reducing the audio output from the slice: set too hight it can cause distortion on the analogue out. I am not sure I can see why it would do so on the optical, but worth a try.