The fact that usually they don't doesn't mean they can't, and therefore it isn't actually secret. Also, it is well known that in the 1960s and 1970s the security services did ocassionally make illegal use of this to find out who voted for certain 'fringe' parties. I know that numbers are a fraud prevention device but they do have this additional feature, sorry bug.

Over in Ireland, where they use broadly similar voting methods to us, I understand they abolished the numbers because of concerns over this very issue.