Hlangana ne-Nemotron Labs 3 Puzzle 75B A9B: I-Compressed Hybrid MoE LLM Iletha 2.03x Server throughput
Amamodeli amakhulu ayingxube ye-MoE efana ne-Nemotron-3-Super anembile kodwa ayabiza ukuwahlinzeka. Amapharamitha wabo asebenzayo, inqolobane ye-KV, kanye nekhephu yesifunda se-Mamba bangaki abasebenzisi i-node engababamba ngenani lethokheni elinikeziwe lomsebenzisi ngamunye. Ithimba le-NVIDIA AI likhululiwe I-Nemotron-Labs-3-Puzzle-75B-A9Bokuhlukile okucindezelwe kwe-Nemotron-3-Super. Imodeli yomzali inenani eliphelele elingu-120.7B kanye namapharamitha asebenzayo angu-12.8B. Imodeli ecindezelwe inengqikithi engu-75.3B kanye namapharamitha asebenzayo angu-9.3B.
Ithagethi yokuthunyelwa yalungiswa ngaphambi kokuthi kuqale ukusesha kwezakhiwo. Ithagethi yokuqala kwakuwukuphuma kweseva okungu-2x kumathokheni ayi-100 ngesekhondi ngomsebenzisi ngamunye. Okuhlosiwe kwesibili kwakuyizicelo eziyi-8 zethokheni ezihambisanayo eziyi-1 ku-H100 eyodwa. Izindawo zokuhlola ezintathu ku-Hugging Face: BF16, FP8, kanye ne-NVFP4.
I-TL;DR
- 120.7B/12.8B imiminyanisa esebenzayo iye ku-75.3B/9.3B esebenzayo, nesakhiwo esihlanganisiwe samabhulokhi angu-88 silondoloziwe.
- 8xB200 isamba sokuphuma sikhuphuka ngo-1.60x siye ku-2.14x ngaphezu kwe-Super ku-NVFP4 efanisiwe kanye nokuphuma komsebenzisi okufanayo.
- I-Single-H100 1M-token concurrency iya ku-1 kuya ku-8, iqhutshwa ukwehla kwesisindo okungu-70 GB kuya ku-44.5 GB.
- I-Iterative Puzzle ihlula i-Puzzle yesinyathelo esisodwa ngamaphoyinti ayisilinganiso angu-0.57 endaweni eqondisiwe yokuminyanisa efanayo.
- I-Arena-Hard-V2 (-4.2) kanye ne-SWE-Bench (-2.6) yizindleko zangempela; I-RULER ne-AA-LCR azinyakazi ngokulambisa.
I-Nemotron-Labs-3-Puzzle-75B-A9B
I-Nemotron-3-Super iwuhlobo oluxubile lwe-Mamba-Transformer MoE. I-Puzzle-75B-A9B igcina ukwakheka kwebhulokhi yomzali ngokuqondile. Inamabhulokhi angu-88: 40 Mamba, 40 MoE, kanye namabhulokhi okunakwa angu-8.
Okushintshile umthamo ngaphakathi kwalawo mabhulokhi:
| Ubuningi | Super | I-Puzzle-75B-A9B | Isilinganiso |
|---|---|---|---|
| Ingqikithi yamapharamitha | 120.7B | 75.3B | 62.4% |
| Amapharamitha asebenzayo | 12.8B | 9.3B | 73.1% |
| Usayizi wesimo se-Mamba SSM | 128 | 96 | 75% |
| I-MoE ihambise usayizi omaphakathi ophakathi | 2688 | 1280-2688 | Kusho u-59.9% |
| Ochwepheshe bomzila basebenza ngethokheni ngayinye | 22 | 4-18 | Kusho u-50% |
| Amandla ochwepheshe omzila osebenzayo (isihlobo) | 100% | 8.7%-62.3% | Kusho u-30.9% |
Inombolo yochwepheshe abathunyisiwe, usayizi wochwepheshe okwabelwana ngawo, kanye nosayizi ocashile we-MoE awushintshile. Izendlalelo zokunaka zishiywe zingathintwa. Isizathu esishiwo socwaningo oluhlongozwayo ukuthi i-Nemotron-3-Super isivele isebenza kahle kakhulu nge-KV-cache. Izendlalelo ze-Mamba zithenwe ngokufanayo, ngenxa yokuthi izinhlaka zokucabangela azisekeli usayizi wesimo se-SSM ohlukile isendlalelo ngasinye.
Umphumela awuyena uthisha owehliswe ngendlela efanayo. Umfanekiso ongenhla ubonisa ukwabiwa ngokujula. Indida igcine umthamo ezindlini ezikhethiwe ezimaphakathi nezakamuva, futhi isikwe kanzima kwenye indawo.
Ibhentshimakhi kanye nokusebenza
Ithebula elingezansi libika inani eliphelele le-Pareto eliphelele endaweni eyodwa engu-8xB200, ngokuqoshwa kwesinyathelo esisodwa.
| Isimo (ngaphakathi/ngaphandle) | UT phansi | Super (tok/s) | I-Puzzle-75B-A9B (tok/s) | Thuthukisa |
|---|---|---|---|---|
| 50K / 2K | >= 100 | 5,128 | 8,210 | 1.60x |
| 50K / 2K | = 125 | 3,784 | 6,412 | 1.69x |
| 50K / 2K | > = 150 | 2,532 | 4,523 | 1.79x |
| 8K / 64K | >= 100 | 20,939 | 42,601 | 2.03x |
| 8K / 64K | = 125 | 13,074 | 27,918 | 2.14x |
| 8K / 64K | > = 150 | 8,522 | 18,047 | 2.12x |
Womabili amamodeli anikezwe izisindo ezifanayo ze-NVFP4, inqolobane ye-FP8 KV, kanye nesifunda se-FP16 Mamba. Ngakho-ke igebe libonisa ukucindezelwa, hhayi ushintsho kufomethi yezinombolo. Umbuso wokugcwalisa-osindayo we-50K/2K uzuza kancane. Umbuso we-decode-heavy 8K/64K uzuza kakhulu.
Kunodi eyodwa engu-8xH100 e-UT = 100, izinzuzo zincane. Zingu-1.91x ku-50K/2K kanye no-1.82x ku-8K/64K. Zombili izinhlobo lapho zisebenzisa izisindo ze-FP8, inqolobane ye-FP8 KV, kanye nesifunda se-FP32 Mamba.
Ku-H100 eyodwa kumongo we-1M, isibophezelo esibophezelayo siyaphenduka sisuka kukhompyutha siye kumemori. Izisindo ze-Super's NVFP4 zithatha cishe u-70 GB wesabelomali esingu-80 GB HBM. Isicelo ngasinye sethokheni ye-1M sengeza cishe u-4 GB wenqolobane ye-KV. Ngakho-ke ukuvumelanisa okusebenzayo kungu-1.
Izisindo ze-Puzzle-75B-A9B's NVFP4 zithatha cishe u-44.5 GB. Isakhiwo sokunaka asishintshile, ngakho-ke izindleko ze-KV zesicelo ngasinye azishintshile. I-Concurrency eku-1M ikhuphuka iye ku-8. Ukuphuma kwekhodi okuhlanganisiwe kulokho kuvumelana cishe ku-4x isicelo esisodwa se-Super. Ukugcwalisa ngaphambili kokwaziswa kwethokheni engu-990K kucishe kusheshe okungu-1.2x.
Indlela Indida Ephindaphindwayo Esebenza Ngayo
Indida kuwuhlaka lokusesha lwe-neural architecture olubolile, olusetshenziswa lapha njenge-Puzzletron. Ichaza isikhala sosesho esihlukile sokunye ukusetshenziswa kwesendlalelo. Enye nenye ithola amaphuzu ekhwalithi. Uhlelo lwe-mix-integer bese lukhetha enye indlela eyodwa isendlalelo ngasinye ngaphansi komkhawulo wokuthunyelwa.
Izindlela ezintathu zokuthena zakha indawo yokusesha:
- Ukuthenwa kwesiteshi okumaphakathi: Iziteshi ezingaphakathi kochwepheshe ngamunye othungathwayo zilinganiswa ngokunikela kokuphumayo kochwepheshe. Bonke ochwepheshe abangaphakathi kwesendlalelo esisodwa se-MoE bathenwa babe usayizi ofanayo, ukuze kuhambisane ne-kernel.
- Ukuncishiswa okuphezulu-k: Inombolo yochwepheshe ithokheni ehanjiswayo iyahlukahluka kusendlalelo ngasinye, kufika ku-k=22 yomzali.
- Ukuthenwa kweMamba SSM: Usayizi wesifunda se-SSM wehla ukusuka kuziteshi eziyi-128 kuye kwezingu-96.
Umphumela we-SSM uyalinganiswa. Ukwehlisa iziteshi ezingu-128 kuya ku-96 kusheshisa i-SSM kernel 1.2x ukuya ku-1.3x ngesikhathi sokuqokwa. Lokhu kubamba osayizi beqoqwana phakathi kuka-8 no-512. Amashaneli alinganiswa ngokunikela okulinganiselwe ekuphumeni kongqimba lweMamba. Isilinganiso singaphezu kwamathokheni angu-67M wedatha yokuqinisekisa. Isithasiselo A sibonisa lokhu kwehlula ukukhetha kwesiteshi okungahleliwe ngaphansi kokuthenwa okunonya.
Ukwakhiwa kwangempela kuthatha ukuthi imithelela yekhwalithi yokushintsha cishe iyisengezo. Ibhulokhi ngayinye yekhandidethi inikezwa amaphuzu ngaphakathi komzali ongashintshiwe. Lokho kuziba ukusebenzisana kwezinga eliphezulu phakathi kokumiselela.
I-Iterative Puzzle ishintshanisa ukucindezelwa okuboshwe ngolwazi olufushane lokubuyiswa kwe-distillation. Yakha ukulandelana okungu-M0, M1, … MR esikhundleni sokugxumela kulokho okuhlosiwe. Izikolo zibalwe kabusha ngokuqhathaniswa nemodeli yamanje ecindezelwe, hhayi umzali wokuqala.
Kusetshenziswe izigaba ezintathu:
- Isisindo se-MoE sifika ku-75% wamandla othisha, i-Mamba SSM ifinyelela ku-75%. Kwelaphelwe amathokheni angu-24B.
- Isisindo se-MoE sifika ku-60% wamandla othisha. Kwelaphelwe amathokheni 43.2B.
- Isabelomali esenziwe sasebenza sochwepheshe safinyelela ku-50%, sabiwe ngokuhlukahlukana. Kwelaphelwe amathokheni angu-52.8B.
Ithebula elingenhla liqhathanisa lokhu nesisekelo sendida yesinyathelo esisodwa endaweni eqondisiwe efanayo. Inqubo yezinyathelo ezintathu yenza isilinganiso esingu-69.05 kuwo wonke amabhentshimakhi ayishumi, uma iqhathaniswa no-68.48. Izinzuzo zivela ku-MMLU-Pro, GPQA, HLE, AA-LCR, LiveCodeBench, SciCode, kanye ne-RULER-256K. I-IFBench-Instruction yehle ngamaphuzu angu-0.2 kwathi i-IFBench-Prompt yehla ngo-0.5.
Ukubuyisela: I-Distillation, i-RL, ne-Verbosity
Ukuchithwa kolwazi kusebenze kudatha engu-30% yokuqeqeshwa kwangaphambili kanye nedatha engu-70% ye-SFT evela ku-Nemotron-3-Nano. Phakathi nesigaba Sendida, i-KD yasebenzisa ubude bokulandelana obungu-32K. Ukubuyisela kwabe sekuqeqeshelwa ku-128K, futhi kwafinyelela ku-512K. Isabelomali sasifika kumathokheni angu-100B, ne-batch yomhlaba wonke ye-16M-token, e-Megatron-LM.
Ukuqeqeshwa kwangemuva kwe-RL kwamukele Isigaba 2 sepayipi le-Nemotron-3-Super RL, eligxile kubunjiniyela besofthiwe. Isigaba 2.1 senze ukuqhathanisa kwethuluzi lokusebenzisa isinyathelo esisodwa. Isigaba 2.2 sithuthelwe ku-end-to-end sandbox RL, lapho ama-ejenti enza amajika afika kwangu-200. Zombili izigaba zisebenzise inhlawulo ye-KL engu-0. Ithimba lishanele izilinganiso zokufunda, lase lilinganisa izisindo eziwumphumela.
Umfanekiso 4 ongenhla ukhombisa ukuthi isigaba ngasinye sibe nesandla. I-KD yokuqukethwe okufushane ibuyisela izigaba eziningi ngaphezu kuka-97% we-Nemotron-3-Super. I-KD yokuqukethwe okude bese iphakamisa amabhentshimakhi okokufaka okude nawesizukulwane eside ngokuqondile. Ithimba locwaningo lithi umthelela we-RL kulokhu kuhlolwa ubumncane.
I-Verbosity iyimininingwane ethule. Ngemva kokuphindwaphindwa kwendida kokugcina, imodeli ikhiqize u-132% wesibalo samathokheni e-Super. Lokho kwehle kwafinyelela ku-99% ngemva kwepayipi eligcwele lokutakula.
Ukuthunyelwa: I-Quantization kanye ne-Multi-Token Prediction
Amaresiphi amabili okulinganisa ngemva kokuqeqeshwa akhiqizwa: i-FP8 W8A8 eqondiswe ku-Hopper kanye ne-NVFP4 W4A4 eqondiswe ku-Blackwell.
| Isakhi | Isisekelo se-BF16 | Indawo yokuhlola ye-FP8 | Indawo yokuhlola ye-NVFP4 |
|---|---|---|---|
| Hlukanisa futhi wabelane ngama-MOE GEMM | BF16 | FP8 | I-NVFP4 |
| Mamba GEMMs | BF16 | FP8 | FP8 |
| Inqolobane ye-Mamba ye-SSM | FP32 | FP32 | I-FP16+SR |
| KV inqolobane | FP8 | FP8 | FP8 |
| Irutha | FP32 | FP32 | FP32 |
| Qaphela i-QKV/okuphumayo, ukuqagela okucashile kwe-MoE, inhloko ye-LM | BF16 | BF16 | BF16 |
Zombili izindlela zokupheka zilinganiswe kumasampula e-SFT angemuva kokuqeqeshwa angama-256. I-NVFP4 isebenzise ukulinganisa okuphezulu, hhayi ukusesha kokuzwela kwe-AutoQuantize okusetshenziselwa i-Super. Indawo yokuhlola ewumphumela ilinganiselwa kancane kancane, futhi yenziwa ngendlela efanayo.
I-NVFP4 ayisekelwe ngokomdabu ku-Hopper. Isasetshenziselwa i-1M-context H100 target, ngoba umthamo we-HBM ubophezela lapho.
I-Puzzle-75B-A9B ithola ifa lekhanda le-MTP elabelwe kwa-Super. Amapharamitha abiwa ezinyathelweni zonkana ze-MTP, ngakho-ke ikhanda elilodwa lisebenza ngokuphindaphindiwe ekuqondeni. Ukudlulisa ikhanda eliqeqeshiwe likaSuper ngokuqondile kunikeza ubude obufanayo bokwamukela.
Ithimba labacwaningi libe selihlonza ukungqubuzana kokuqeqeshwa-okuhloswe ngakho. Ukuqeqeshwa kwe-MTP ephoqelelwe ngothisha kuphakela ukulandelana okugcwele kwesimo esifihliwe esiguquliwe. Ukubhala okuzenzakalelayo esikhundleni sokuphakela ingxube yemodeli eqondiwe kanye nezimo ezifihliwe ezikhiqizwe yi-MTP. Amazinga okwamukela awela ezikhundleni ezijulile ezisalungiswa.
Ukuqeqeshwa okuqhubekayo ngekhanda elidlulisiwe likhuluma ngalokhu. Ku-SPEED-Bench ngobude obusalungiswa obungu-7, ubude bokwamukelwa bukhuphuke busuka ku-3.45 baya ku-4.34. Lokho cishe ku-25% kuya ku-30%, kugxilwe ezikhundleni ezisalungiswa kamuva. Ngokungafani ne-Super, indawo yokuhlola ye-NVFP4 alehlisi neze isithunzi: 4.31 ngokumelene ne-4.34.
Lapho Ukucindezelwa Kuyasiza Futhi Lapho Kubuhlungu Khona
| I-Benchmark (BF16) | Super | I-Puzzle-75B-A9B | I-Delta |
|---|---|---|---|
| I-MMLU-Pro | 83.8 | 82.4 | -1.4 |
| I-AIME25 (awekho amathuluzi) | 92.2 | 89.7 | -2.5 |
| I-GPQA (awekho amathuluzi) | 80.5 | 78.6 | -1.9 |
| I-LiveCodeBench | 82.1 | 81.1 | -1.0 |
| I-SciCode (umsebenzi ongaphansi) | 42.3 | 40.6 | -1.7 |
| I-SWE-Bench (OpenHands) | 59.5 | 56.9 | -2.6 |
| I-Arena-Hard-V2 | 72.8 | 68.6 | -4.2 |
| I-AA-LCR | 56.8 | 56.9 | +0.1 |
| UMbusi 1M | 93.9 | 92.2 | -1.7 |
| I-MMLU-ProX | 79.5 | 77.5 | -2.0 |
Isifinyezo sephepha locwaningo siqu ukuthi ukulandela imiyalelo kanye nokuhlolwa kwe-ejensi kulahlekelwa kakhulu. I-Arena-Hard-V2 yisimo esibi kakhulu, ngamaphoyinti angu–4.2. I-RULER ihlala phakathi kwamaphoyinti angu-1 kuya kwangu-2 kokuthi 256K, 512K, kanye no-1M.
Imiphumela emithathu ye-BF16 ayihlehli. I-AA-LCR izuza ngo-0.1, i-Scale AI Multi-Challenge ties ku-56.6, futhi i-TauBench Telecom izuza u-0.4.
I-NVFP4 ibiza kancane phezu kokucindezelwa. Ku-RULER 1M indawo yokuhlola ye-NVFP4 ithola amaphuzu angu-93.2, ngaphezu kwe-BF16's 92.2. I-HLE yizindleko ezicacile ze-NVFP4, zehla zisuka ku-16.5 ziye ku-15.7. Imiphumela ye-FP8 ihlezi ku-Appendix E, futhi ilandelela i-BF16 eduze. I-SWE-Bench ayibikwanga endaweni yokuhlola ye-FP8.
Sebenzisa Amacala
- I-RAG yokuqukethwe okude kakhulu ku-GPU eyodwa: Isevisi yokuhlaziya idokhumenti ngomongo ongu-1M isuka esicelweni esi-1 ngesikhathi esisodwa iye kweziyisi-8. Ukuhlanganisa kokukhipha ikhodi kuleyo ngqikithi yemali cishe ku-4x.
- Abasizi bekhodi abasebenzisanayo: Ku-UT >= 100 tok/s kumbuso we-8K/64K, inodi eyodwa isebenzela amathokheni angu-2.03x. Kulungiselwe i-verbosity, lokho kungu-2.16x wezicelo eziqediwe ngomzuzu.
- Gcwalisa kusengaphambili imigqa yedokhumenti esindayo: Umbuso we-50K/2K uzuza u-1.60x kuphela. Ukuminyanisa kusiza kancane uma ukucubungula ngokushesha kubusa ikhompuyutha.
- Amalophu e-Agentic SWE: Hlola igebe le-SWE-Bench elinamaphuzu angu-2.6 ngokumelene nengxube yakho yomsebenzi. Ukutholwa kwe-RL kuqondise leli khono, futhi kwalibuyisela kancane.
I-Deployment Explorer
