That's the problem that is happening whenever I play the game. If you really want to fix being the only person before the match starts my favorite choice is to use cheat engine and to lower the speed of the game. Because you are the host everyone else slows down so the lobby takes longer to start the game. Slow the game down when the countdown begins. As for the P2P System that kicks everyone out of the lobby, disabling VAC may solve the issue and that is all I can contribute to this issue.