ben on Nostr: what kind of gpu and how much vram in it? afaik running inference is constrained by ...
what kind of gpu and how much vram in it? afaik running inference is constrained by gpu vram.
8B models run very fast on my 8GBvram gpu
Published at
2024-06-26 18:12:49Event JSON
{
"id": "4a4cfd62232044a7af42cff9edde606a6f7feb71e8255751d33e4c3cc41fb07b",
"pubkey": "7bdef7bdebb8721f77927d0e77c66059360fa62371fdf15f3add93923a613229",
"created_at": 1719418369,
"kind": 1,
"tags": [
[
"e",
"776cbbbbd791dbe6d57a04a6c607074d4775e7c985f9a2aa8a6846b8fb700bec",
"",
"root"
],
[
"e",
"cac5c02e96850c0bb96e0cea89d35bd55987019030038f2f6aa6798dd4459bee",
"",
"reply"
],
[
"p",
"6140478c9ae12f1d0b540e7c57806649327a91b040b07f7ba3dedc357cab0da5"
],
[
"p",
"45f195cffcb8c9724efc248f0507a2fb65b579dfabe7cd35398598163cab7627"
]
],
"content": "what kind of gpu and how much vram in it? afaik running inference is constrained by gpu vram. \n\n8B models run very fast on my 8GBvram gpu",
"sig": "44354b95c866efae722ed9644f02cae34defda20e1dbdbb7ac3c3a9ad914dc126ad63f5474e2fc4a06d2e815b10f84b3ac666db333e276b60973cea04884acd9"
}
