Yup just plain ol belt walk. It won't do to much but rub the hell out of the outside edge of the belt. And shorten the life of the belt like to 55-65K miles as oppose to the 100K that it can handle.
But yes. If you re-adjust the tension, that will help "walk" the belt back the other way. And it should stay there.
Adjusting the tensioner while the belt is on may or may not help. Sometimes the belt is too tight to move it incrementally. But you can try that. Just be careful not to pass the windown to much. too loose and yes, it can be catastrophic.