semisol on Nostr: The hidden states across layers are huge, especially for big models, for example 32KB ...

The hidden states across layers are huge, especially for big models, for example 32KB per token. Especially for prefill, that means 3.2G of data per layer to do a prefill on 100K context