I was not sure what it is, as it could've been a problem with my PC, faulty memory or else. But now I'm confident its a PS bug.
I tried to dig deeper into this problem, as it became a royal pain. As those dirty pixels are not always on the same places and jump around from build to build. I process one texture a few times and then clean it up in Photoshop
So I removed half of of my memory, and still had it. Then tried different memory configurations. As I have 8 DDR3 slots and 8x8GB memory units here. One of which could've been faulty (although memory tests say its ok). Kept getting it no matter what I did, and it was clear its not a memory problem. Although I do get very dirty textures on my Xeon workstation. On other computers, not so much, but still can see some areas are dirty.
Please download test scene here. It might be hard to find those pixels sometimes, but I'm sure some are always there.
https://www.dropbox.com/s/aviptlvu4n25eut/Dirty_pixels_test_scene.zip