This is a common issue I've had with 360 controllers, it seems to be caused by two reasons:
1) The controller is old and has been used a lot, so it's wear...
2) 360 controllers seem to have no deadzone at all.
When you combine the wear with the lack of a deadzone, the resting position will cause the camera/character to move even if you're not touching the stick at all. It's one of the most annoying damn things ever and I'm so mad about it because a simple option to modify the controller's deadzone would fix it...
I KNOW the issue is caused by the lack of a proper deadzone because on my PC I can customize it and make the resting position not move the camera/character.
So yeah, I don't know of any way to fix it other than buy a new controller that isn't so worn...