Both "+" and "&" can be used for concatenation in Vb.net. However vb.net understands the context of "&" better and won't end up attempting to "+" (add) the string, which will give you the error you mentioned above. Your result is a Valid String but not a Valid Double. Usually this is a "problem" you see when Option Explicit is ON because Option Explicit ON will disallow the explicit conversion of String to Double. But it's a good idea to keep it on, so "&" is the better method.
Kevin had it right, I just wanted to quickly explain why, because (IMO) if you don't understand why, you may continuously make the same "mistake".