Large models of languages (LLMS) are increasingly sent to the EDGE devices of long settings, creating a growing demand for faster and effective mind. In these cases, the key cache (kv) is the main borkleneck depending on the GPU memory and latency, as the full KV memory should be loaded each decorative action. While the imagination is a widely accepted method of exposure to automatic failure, the methods are often struggling to achieve the main KV cachization shares due to misconceptions of unemployment. Dealing with these challenges, weigh the outline of the novel, quote, where the circular model is in the construction of the target model but the KV quantspec saves the full-speed faster ~ 2.5 ×, exceeding other ways To synchronize synchronizing using sparse kV cache of the submission of the submission. Quantspec also reduces memory requirements by ~ 1.3 × compared with these methods.
- * Equal donation
- 30 University of California, Berkeley
- ‡ International Computer Science Institute
- § Lawrence Berkeley Laboratory
