I was just giving you a hard time actually but, if you want to remove it, there is an option to leave the group at the bottom of the page where you joined it (added the badge)