I can't get this to work. When I decrease height in my game.cfg by whatever number (tried 200 and 300 first as you suggested) and start a custom game and change it to borderless mode it doesn't work (It just stays at my max resolution which is 1680x1050). It also doesn't work if I set the mode to borderless first, then exit the game and decrease the height in game.cfg. As soon as the loading screen pops up the config resets itself to 1680x1050 and full screen mode.
Any idea why it doesn't work? I can see the changes in game.cfg if I for example decrease my resolution in game from 1680x1050 to 1600x900 but it doesn't work when I manually edit it (I'm using notepad).