You should always disable locations because you never know who is "listening" on the other side.
Also I encountered some strange things myself even when I turn location off. One day I was having conversation with my brother about our home adress and noticed my telephone started listening to us. Don't know what happened there because I am sure I also turned voice commands off.