Phone has to be powered on and also location services on the phone has to be on. Otherwise you can't track it.

If you have a Find Iphone or other tracking device, and it shows the location then goes "offline" then comes back "online" a bit later it means the person turned off location services knowing they could not be tracked, then turned it back on. I busted my 16 yr old doing this.