Having the same issue since I updated to 4.1. is_sidebar_active comes back as true for my first 4 sidebars and gets assigned a widget even when no widget was been assigned to those sidebars… and therefore that sidebar shows up on my page.
Please see screenshots here:
https://www.dropbox.com/sh/xr0ybrozpf31cgd/AAAAbyfuIhJRlLDwWQhecNWsa?dl=0
I have added an echo function to my sidebar.php file to show how it outputs to true even when no sidebar has been assigned.
This problem does not exist in wordpress 4.0.1.