이미지: METAL LAB 생성
Summary
- A Reddit user ran DeepSeek v4 Flash 0731 locally using an RTX 4090, a Tesla P40, and 128GB of DDR4
- Unsloth's 4-bit K_XL quantization, at about 144GB, ran at 2 tokens per second without MTP applied
- Adding MTP to IQ4_XS quantization (about 127GB) achieved 3 tokens per second, with prompt processing at 30 tokens per second
- 하드웨어 구성
- RTX 4090 + Tesla P40, DDR4 RAM 128GB (총 176GB RAM+VRAM)
- K_XL 양자화
- 약 144GB, 12k 컨텍스트에서 초당 2토큰(MTP 미적용)
- IQ4_XS 양자화
- 디스크 약 127GB, MTP 포함 약 137GB, 초당 3토큰
- 프롬프트 처리 속도
- 5k+ 컨텍스트 기준 초당 약 30토큰
- 제약 사항
- llama.cpp가 DeepSeek v4 Flash용 텐서 분할을 아직 지원하지 않음
A case of running DeepSeek v4 Flash 0731 locally on consumer-grade hardware was shared on the Reddit community r/LocalLLaMA. The poster said they upgraded their DDR4 RAM to 128GB using two 32GB sticks and combined an RTX 4090 with a Tesla P40, securing a total of 176GB of RAM plus VRAM.
Performance differences by quantization
Running Unsloth's 4-bit K_XL quantization (about 144GB, reportedly around 97% accuracy) with a 12k context yielded about 2 tokens per second, and MTP (DSpark) could not be applied at this stage due to insufficient memory. The poster then switched to the smaller IQ4_XS quantization (about 127GB on disk, about 137GB with MTP included) and enabled MTP, achieving about 3 tokens per second, with prompt processing reaching about 30 tokens per second at a 5k+ context.
GPU order and layer placement as key variables
The poster noted that specifying CUDA0 (the RTX 4090) first in llama.cpp's -dev flag pushed prompt processing speed up to 40–80 tokens per second, but placing the Tesla P40 first dropped it to 17 tokens per second. They also explained that because the output layer could not be placed on the Tesla P40 due to unsupported operations related to Gated Delta Net, the embedding and output layers were assigned to the RTX 4090, while other layers were manually split between the P40 and the CPU.
The poster said, "llama.cpp doesn't yet support tensor splitting for DeepSeek v4 Flash, so layers had to be distributed manually." They also shared an expectation that future llama.cpp updates could bring DSpark support and improved speeds.
