I did some extra digging on this. Lazy loading also screws up the builder loading. If you enable it, then go into the settings Performance> User Experience and then disable Process HTML Image Tags and leave Process Background Images active, the loader still works. Additionally, the suggestion from @logarank worked for me.
Currently waiting on Divi to reply to me on the suggestion from @rossanino because, from what I can see, enabling the minify options under the themes section slows down the site.
I dont know whos end this needs to be fixed on, but it would be great if it could get fixed! Im telling Divi the same thing.