I was raised to be a southern gentleman and will remain one but I have never heard of removing your hat when speaking to a female.