There's no set formula for this, you just go through the code and looks where the developer was lazy or left a thing for you to exploit.
For example:
If you find an anti-cheat with the file grabber, then you just look up the way it detects your cheats and then you counter it. Like Blue Kirby did with the GmodZ anti cheat thing. It was just checking your sv_allowcslua cvar with a timer. So he countered the anti-cheat by destroying the timer.
So if you want to find an exploit, you look up the function you want to exploit, get how it works and maybe find an exploit.