I only use wifi if its somewhere I can't run a wire. My Xbox One on wifi only gets 20/10up but wired gets 85/10up.
If you have a router, give the xbox one a static IP address, make sure it's 'doled' out by it's MAC address.
Do that and if it's still giving you problems like that, toss that static IP on a DMZ.