I've recently transitioned to using a PC for all of my media transcoding and hosting Plex, so I can keep my Macs unburdened and ready for real work. The only thing is -- when Handbrake is compressing something, it pegs all of my CPU cores at 100%, leaving not much left for Plex Media Server to use.
I found a great solution on this page at Windows Loop. I'll let you go there for all of the details, but I'll post an abbreviated version here, just in case that site disappears someday:
Lower Priority
Go to Handbrake > Tools > Preferences > Advanced, and set Priority Level to Below Normal or Low. I chose Low.
You can also do this from Task Manager: In the Details tab, right-click on Handbrake.exe and Set Priority.
I also set Plex Media Server to High priority here, for good measure.
CPU Affinity
From Task Manager, when you right-click, one of the choices is Set Affinity. This will allow you to set which CPU cores the app is allowed to run on. I unchecked cores 0 and 1, so on my processor, it still has access to the six other cores.
I find this lets the system stay responsive, and hope that this will make it able to serve Plex media without any trouble, even while I'm transcoding media.