It depends what kind of games you are recording. If it's Minecraft, you don't need any more RAM, if it's Call of Duty in full 1080P HD, you most likely want 8 gigs of RAM + a 580 or something.
That's the GPU I have. I can vouch for it. I can Fraps pretty much any game I want at 1920x1080 high settings + a good frame rate. Go for it. I never even get up to using all nine gigs of RAM i've got, but I would consider upgrading to a better GPU before more RAM. Plus your CPU is shit, consider buying a new one too.