Just found a solution if someone finds this post later down the line.
Apparently it’s not something that changed in WordPress but rather in how I packaged the plugin.
When packaging the plugin make sure inside the zip archive you have a folder, not just the plugin files. It looks like when there are files inside the archive WordPress leaves the temporary folder as is, but if you have only one folder inside (which contains the plugin files) then WordPress will move that folder up one level to the wp-content/plugins directory and remove the temporary folder.
Apparently it was one of my old plugins. Guess it just doesn’t work with the latest changes and got broken in the recent update.
I did try to disable the recent plugins I installed, but it seems that it’s not one of them.
The other plugins are installed on my blog for a few years so it can’t be them…
Is there any way to validate the wordpress files without re-uploading all them?
