One of those things. We have spent 4 hours, 2 persons combined hunting down this problem. Google GTM code, which we for specific reason enabled on node add forms, broke all ajax forms and made crayze problem of opening new browser window instead of editing the node reference field with inline entity form. The same thing happened also on image upload widget. Remove google GTM code from page, solved the issue.