It's fine lol it is chat the more the better
Gordon Penland PRO
Zoberzzz
·
AI & ML interests
None yet
Recent Activity
updated a Space 1 day ago
Zoberzzz/llama-3-8b-demo published a Space 1 day ago
Zoberzzz/llama-3-8b-demo updated a bucket 2 months ago
Zoberzzz/mineOrganizations
replied to their post 4 months ago
replied to their post 5 months ago
A few honest questions:
What is T3-BOOST actually doing? Distributing 1M units across 7 nodes is simple integer division — 1,000,000 ÷ 7. A local function doing that arithmetic would be extremely fast compared to a cloud LLM generating a step-by-step explanation. That's not a fair comparison — it's comparing a calculator to a professor asked to show their work.
What is Qwen 2.5 Cloud being asked to do? It's generating a detailed explanation, not just computing the answer. Of course it's slower.
What does TGLYPH3 actually do under the hood? I can see the benchmark UI but not what T3-BOOST is doing internally.
I'm not dismissing it — I'm asking because the 400,000x and 9,700,000x numbers need context to mean anything. What problem is being solved and are both sides solving the same problem?
posted an update 5 months ago
Post
187
Hackernews post · TXT
Show HN: I compressed a 160GB KV cache to 640MB at 0.9994 fidelity on a $300 GPU
Title: Show HN: DenseMem — 256x KV cache compression, 0.9994 fidelity, runs on consumer hardware
---
A 72B model at 32K context needs 160GB of KV cache. That's an H100 and $32,000 in HBM3e memory.
I built a protocol that stores the same KV cache in 640MB of DDR5 RAM — on a consumer RTX 4090 and Core i9.
256x compression. 0.9994 cosine similarity. 1.95ms average fetch latency. Verified.
**How:**
Transformer KV cache activations are highly structured and correlated. SVD at rank=64 exploits that structure. Random noise compresses to 0.12 fidelity. Real KV cache activations compress to 0.9994. The math works because the data isn't random — it has geometry.
The system manages a two-tier hierarchy: VRAM is the hot tier, DDR5 is the warm tier. An attention-weighted evictor (0.5 attn + 0.3 recency + 0.2 freq) decides what stays hot. A prefetcher using layer lookahead and token prediction pre-positions pages before they're needed. Average fetch latency: 1.95ms. Max under load: 3.96ms.
Current hit rate is 25% — bottlenecked by my i9's 2-channel DDR5 bandwidth (~38 GB/s). On an 8-channel Threadripper PRO (~224 GB/s) I'm projecting 65-75%.
**Running live:**
- Qwen2.5-7B on RTX 4090 at 32K context (was 4K)
- Every inference tick compressed INT8 via PCA → DDR5
- 2.4s cold start
**The cost math:**
- Uncompressed 72B KV cache: $32,000 in HBM3e
- FoldedMemory: $1.88 in DDR5
- 99.4% cost reduction. Verified on consumer hardware.
GitHub: https://github.com/thorshammerztp-arch/densemem-protocol
Patent Pending: US 64/045,595
Solo developer. Navy veteran. No funding. Consumer hardware.
Show HN: I compressed a 160GB KV cache to 640MB at 0.9994 fidelity on a $300 GPU
Title: Show HN: DenseMem — 256x KV cache compression, 0.9994 fidelity, runs on consumer hardware
---
A 72B model at 32K context needs 160GB of KV cache. That's an H100 and $32,000 in HBM3e memory.
I built a protocol that stores the same KV cache in 640MB of DDR5 RAM — on a consumer RTX 4090 and Core i9.
256x compression. 0.9994 cosine similarity. 1.95ms average fetch latency. Verified.
**How:**
Transformer KV cache activations are highly structured and correlated. SVD at rank=64 exploits that structure. Random noise compresses to 0.12 fidelity. Real KV cache activations compress to 0.9994. The math works because the data isn't random — it has geometry.
The system manages a two-tier hierarchy: VRAM is the hot tier, DDR5 is the warm tier. An attention-weighted evictor (0.5 attn + 0.3 recency + 0.2 freq) decides what stays hot. A prefetcher using layer lookahead and token prediction pre-positions pages before they're needed. Average fetch latency: 1.95ms. Max under load: 3.96ms.
Current hit rate is 25% — bottlenecked by my i9's 2-channel DDR5 bandwidth (~38 GB/s). On an 8-channel Threadripper PRO (~224 GB/s) I'm projecting 65-75%.
**Running live:**
- Qwen2.5-7B on RTX 4090 at 32K context (was 4K)
- Every inference tick compressed INT8 via PCA → DDR5
- 2.4s cold start
**The cost math:**
- Uncompressed 72B KV cache: $32,000 in HBM3e
- FoldedMemory: $1.88 in DDR5
- 99.4% cost reduction. Verified on consumer hardware.
GitHub: https://github.com/thorshammerztp-arch/densemem-protocol
Patent Pending: US 64/045,595
Solo developer. Navy veteran. No funding. Consumer hardware.
- 9 replies