You are correct that it's a good idea to remove the red wires from your wax if they have a bed built in.
I could be wrong, but in my experience I am pretty sure it's a PID issue with your quad as your flight is exactly what happens if your P term is set too low. Also what esc firmware are you running and have you set them up with cleanflight? Are you using one shot mode? Of so yet disable that to see if things improve.
Also are there any noticable vibration issues on your quad due to imbalance motors/props as excessive vibrations can confuse the sensors.