Probably inconsistency. Whenever I get lazy about making sure the sight housing is perfectly centered inside the peep my groups will shift. Also, try to have at least three anchor point references. Knuckle in the ear socket, string kissing the nose, vane touching the chin, etc. Slight inconsistencies in anchor point throw the whole system out of whack. You might be shooting a few shots with anchor in one spot, then shifting to another spot and the arrows are following suit.
Also, if your form is not consistent, it will cause arrow flight disturbances, such as kicking the tail up and down or left and right or both. This will cause POI shifts because the arrows are going up and down, side to side several inches back and forth as it goes down range. Fixed blade broadheads will make it even worse. You want the shooter and the bow tuned up so it is spinning arrows straight as a laser.