Just gotta give up deer coming from down wind. Most of my places I know where deer come from MOST of the time. If you scout a place well enough you should know where they should come from and should hunt that spot depending on the wind. But like you said you have some random passer-bys every now and then, but there's nothing you can do about that.