So that's got nothing to do with the encoding then it's all about the TV mode... First check that if you go to the setup and video the 'change tv mode to match video' is selected.
If not this could be the problem...
Secondly plug in a USB keyboard and whilst you are viewing the offending video press the 'o' button. This will display a bunch of real time information (buffer sizes etc) at the beginning of the third or fourth line it should give the media frame rate, check this says 23.97.
Finally if this is all correct then you are going to need to ssh into the box and do (while the video is playing):
This will tell you the current video mode (I assume 60fps)...
tvservice -d dump.edid
ls -l dump.edid
Check that the file is the correct length... (should be 256 bytes)
And check for a 23.97 mode... Does one exist?