Won't cause damages. The factory Colt FA selectors I've seen all have the detent groove 360 degrees, even where there are no detent holes. Without the selector stops, the selector is able to rotate the selector all the way around.
When we (while I was still at BAD) made revisions to the BAD-CASS-3P (3 position), as it was called at the time, we got rid of the 360 degree channel groove. During its development in early 2010, we were wondering why the detent groove went all the way around, past the detent holes, but we made our FA selector according to milspec anyway, and only deleted the mystery groove in later revisions when we've determined it didn't do anything useful.
On a semi auto selector, the detent groove only exists between the SAFE and FIRE detents, but that's not the only thing that's different. Semi auto detent holes are also shaped differently to stop over rotation into a position where the detent groove doesn't continue. With enough force, i.e. with a mallet hitting the lever, a semi auto selector can overcome the detent hole and continue to rotate, I've never seen anyone do it without a tool though.
A semi auto selector can't use the M16 selector detent hole geometry. With or without a detent groove, the shape of a select fire selector detent groove will allow the detent to over rotate without much effort. Ask me how I know
On a semi auto receiver, selector stops are unnecessary. The shape of the detent hole is more than adequate to prevent over rotation.