I do not know how to limit RAM and GPU for some application.
But for CPU you can check this
https://mion.yosei.fi/BES/ . I was using it for application which was constantly using one core in idle state (it was bad programmed).
I would also advise instead of limit CPU percentually use lowest process priority in task manager, which will automatically free CPU for other tasks with default priority.
For better temperature undervolting CPU and GPU is best solution for desktop, notebook, ...
Limit RAM for some application is not good idea, because then operating system starts using swap file on disk, which is terrible slow and if it is on SSD, then the swapping process decreases SSD life.