Yes. But I think a society needs a room of discussion to define what measures are accepted for defending yourself.
But without this place to discuss, no shared decisions can be accomplished. So freedom of speech is has to be the fundamant to build upon.