"how could that even be considered criticism or harmful?"
Because it takes away control over discoverability. Hashtags increase discoverability surface here on Mastodon. I for one value being able to control parts of my visibility and don't want every word I said getting picked apart through hashtags I didn't put there.