I was able to fix the issues by deleting the files in ~Library/Accounts/ and ~/Library/Preferences/MobileMeAccounts.plist
then logging out and in again before opening the panels again.
Warning: This also removes every other account you have listed in the accounts folder.