I’m not “willing” to do anything, it is a fact. You may as well argue about the origins of the Swastika or the Iron Cross. Pepe is a hate symbol, and while not everyone using it might be a nazi, they are using nazi imagery. The fact that “many people” aren’t willing to drop it, despite its extensive, well documented use by extremists is a well made point, but not the one you think.
And there’s no “overestimating” of nazis in the world. We live in a culture of white supremacy. There’s no point in splitting hairs about how offensive or not a cartoon frog is. The easiest solution is to simply not use it.
Culling out hate speech isn’t censorship.