Even with the best computer hardware, playing 720p / 1080p mkv large video files in most of the video players might cause the video to stutter / hang / pause very frequently and skip. This is a very common issue with VLC video player. VLC’s h264 AVC support is still a little outdated so it’s definitely not recommended for playing HD movies, whether MKV or not. Below are the recommended solutions.
Some changes in VLC might also help. You could try raising the file cache. To do this go to: Settings >> Preferences… >> Input / Codecs >> Access modules >> File
Then tick “Show advanced options”. Enter a larger value for “Caching value in ms” (e.g. 1000) and press “Save”. You may need to restart VLC, then try playing DTS through S/PDIF.
If that doesn’t make any difference then you could test out one of the latest VLC nightly builds (0.9.0-svn):