Yet when I say I am a hacker, people often think I am making a naughty admission, presenting myself specifically as a security breaker. How did this confusion develop?

Playfully doing something difficult, whether useful or not, that is hacking.

You can help correct the misunderstanding simply by making a distinction between security breaking and hacking—by using the term “cracking” for security breaking.

Richard Stallman, in Richard Stallman’s personal site