Oh yes. A first "And where do you come from" is understandable. Badgering someone until it's clear that what you mean is "You're not from round here, tell me where you're really a person of" is much worse.