I recently increased the swap on my server from 256MB to 1024MB. I seems to make GitLab run a little faster when loading the details of large commits.
Annoying that I had to throw more memory at the problem instead of actually fixing it. At least the site works, and you can now view details of random commits (like this one).