Well, you have tried everything. You need to grant delete rights to the database user which is connecting your wordpress to database 🙂
Hopefully that is the issue.
I have the same problem even if I do not put the caption. It’s not inserting images at all. Firebug is clean, no JS errors.
