To be honest, you cant really expect any hackers to get to you that way. Given the ease of giving out key loggers, and stealing passwords and phishing accounts, its unlikely.
But if you are really worried, use something like hamachi to run your server. And if you dont want to do that, you can always forward your ports, and then set them back when your done.