Several reasons.
1: Military acquisition BS, even for SOF. It's easier to lay out a niche position that is unfilled, IE, a reliable, suppressed SBR capable of F/A fire and breaking contact drills, rather than get improvements made to an existing contract weapon. Same schtick they used to get the Glock 19; lightweight mid-sized/compact handgun.
2: HK marketing BS. Those fuckers will promise you the world to sell a gun; they'll tell you it's self-cleaning, comes with a complimentary blowjob and retirement gig, can fire with a barrel full of water, is made of unobtanium and capable of 50K rounds of F/A before barrel is shot out to 4 MOA. They had former SOF guys on board for development, for additional marketing, most like. They trash talked the M4 for decades with the intent of making our men doubt the effectiveness of their weapon, and frankly, they ought to be sued into bankruptcy for it.
3: While others can give better testimony on the development of the CQBR/MK18, it took awhile for Crane to perfect when used with a suppressor. Lots of trial and error, I'd imagine, depending on the suppressor, gas port size, exact barrel length, recoil spring, buffer weight, temperature variation, ammo selection, magazines, etc. Once they were rolled out, they began replacing 416's as they went down. Lighter, fewer parts breakages, easier to service, and much cheaper. Simpler logistics, too, since they use the same parts as the M4. The 416, on the other hand, is easier to tune to different barrel lengths and suppressor use, since the piston assembly allows excess gas to vent. This is simpler and allows for more loose tolerances than trying to get exact gas port sizes needed for a DI gun to function optimally.