Swap being empty is not a problem. Swap can be used perfectly fine either as a normal partition, or as is here as a file on the XFS partition. I do that a lot and don’t have freezing problems. Swap being empty means you haven’t used all your RAM for it to swap processes. The freeze sounds more likely to be GUI-related, maybe 3D stuff. Compiz used to be known for doing that quite a lot in certain circumstances with specific configuration options.
Could be graphics driver-related, but would need digging through the log files under /var/log as well as dmesg for a start.