I figured it out. It appears that when I installed the brace adapter in the lower that I had the rubber disk positioned to far inboard, not allowing the bolt carrier to ride rearward enough to allow the BHD to engage. When it would lock back on the USGI mag it would actually lock back on the bolt carrier, not on the bolt itself. Weird!
Not a big fan of the brace adapter mounting system. I have had trouble getting it on correctly before.