I-Unsloth vs Axolotl vs TRL vs LLaMA-Factory: A Fine-Tuning Framework Comparison on Speed, VRAM, and Multi-GPU

“>

I-Unsloth: izinzuzo zezinga le-kernel ku-GPU eyodwa

Amabhentshimakhi ashicilelwe we-Unsloth abonisa isivinini sokuqeqeshwa esingu-2x se-Llama 3.1 8B kanye ne-Llama 3.3 70B. Ukusetha kusebenzise idathasethi ye-Alpaca, usayizi weqoqwana 2, kanye nokunqwabelana kwegradient 4. I-QLoRA yayisezingeni elingu-32 kuzo zonke izendlalelo zomugqa.

Imiphumela ye-MoE mikhulu. I-unsloth ishuniwe kahle unsloth/gpt-oss-20b-BF16 ku-NVIDIA B200. Ibika u-712.33 ms ngesinyathelo ngasinye kumongo we-8K, uma iqhathaniswa no-5,226.86 ms we-Transformers v5. Leso yigebe elingu-7.3x. Ku-4K igebe ngu-4.82x, futhi ku-1K ingu-1.37x kuphela.

Isiqondiso sethrendi sincike kumodeli, futhi amadokhumenti e-Unsloth afinyelela lesi simangalo ku-gpt-oss. Lapho, i-speedup ikhula ngobude bokulandelana, okubizwa ngokuthi i-Flex Attention kanye nezinhlamvu ze-MoE.

I-Qwen3-30B-A3B ku-B200 isebenza ngenye indlela. Isivinini sayo esibikiwe sisuka ku-1.7x ku-1K siye ku-1.1x ku-16K. Ukonga inkumbulo kuhambisa ngakolunye uhlangothi, kukhuphuka kusuka cishe ku-2% kuya ku-15%.

I-Qwen3-30B-A3B ku-H100 ifinyelela ku-1.77x. I-GLM-4.7-Flash ku-RTX PRO 6000 ifinyelela ku-2.1x. Ukusebenzisana ne-AMD kukalwe i-Llama-3.1-8B LoRA SFT ngo-2.07 s/step. I-TRL kanye ne-FlashAttention-2 ithathe 2.87 s/step, igebe elingu-1.39x elinamajika okulahlekelwa afanayo.

I-Axolotl: izinhlamvu ezibolekiwe, ukufana komdabu

I-Axolotl yengeze izikhwebu ze-Triton ngokwezifiso nemisebenzi ye-autograd ye-LoRA ngoFebruwari 2025, icaphuna ngokucacile i-Unsloth njengogqozi. Bakhetha ukungena lora_mlp_kernel, lora_qkv_kernelfuthi lora_o_kernel.

Amanothi okukhishwa kwakamuva engeza ukwesekwa kwe-SonicMoE LoRA. Iletha isivinini esingafika ku-1.45x kanye nokunciphisa inkumbulo okungama-30% ngaphezu kwe-a grouped_mm isisekelo. Leso sibalo ese-Qwen3.5-35B-A3B 8-bit LoRA ku-H100 SXM eyodwa.

I-Axolotl iphinde ithumele i-FlashAttention 2/3/4, xFormers, Flex Attention, SageAttention, Liger Kernel, Cut Cross Entropy, kanye ne-ScatterMoE.

I-TRL: isisekelo wonke umuntu esikala ngaso

I-TRL ivamise ukuba indawo eyireferensi esikhundleni sokuba owinile ku- throughput ye-single-GPU eluhlaza. Inxephezela ngobubanzi benkumbulo namalever esivinini abhalwe kokuthi Ukunciphisa Ukusetshenziswa Kwenkumbulo kanye Nokuqeqeshwa Okusheshisa.

Lawo ma-levers afaka ukupakisha, i-padding-free batching, i-truncation, i-Liger Kernel, kanye nemodi yokulala ye-vLLM ye-GRPO. I-TRL futhi inokuhlanganiswa kwe-Unsloth yenkampani yokuqala, ngakho kokubili akuhlukani.

I-LLaMA-Factory: isivinini ngokuthumela

I-LLaMA-Factory ayizibhali izimbewu zayo. Idalula umsebenzi wabanye abantu ngamafulegi we-config.

Ukusetha use_unsloth: true yenza kusebenze ipheshi ye-Unsloth. I-changelog yephrojekthi ibika isivinini esihlobene esingu-170% ukusuka kuleyo ndlela. Ukuqeqeshwa kokulandelana okude kwe-Unsloth kubhalwe ngesivinini esingu-117% kanye nenkumbulo engu-50%. Iphinde isekele enable_liger_kernel: true kanye ne-FlashAttention-2 nge flash_attn: fa2.

Izitezi zememori ezibikiwe

I-Unsloth ishicilela ithebula lezimfuneko ze-VRAM elihlelwa ngokubalwa kwepharamitha. Ibala u-6 GB wemodeli engu-8B ku-4-bit QLoRA kanye no-41 GB we-70B. I-LoRA ku-16-bit ibiza u-22 GB no-164 GB kumamodeli afanayo.

Ithebula lehadiwe le-LLaMA-Factory le-README lihlanganisa isimiso esifanayo se-4-bit QLoRA. Ifaka ku-6 GB ku-7B, 24 GB ku-30B, kanye no-48 GB ku-70B. Ukucushwa okuphelele kwe-bf16 kwe-70B kufakwe kuhlu ku-600 GB.

Womabili amathebula achaza ubuncane. Usayizi weqoqo, ubude bokulandelana, kanye nokukhetha kwe-optimizer hambisa inombolo yangempela.

Ubude bomongo buyisihlukanisi esibukhali

I-VRAM ephezulu emongweni engaguquki ayithakazelisi kangako kunomongo omkhulu ovunyelwe isabelomali se-VRAM esinikeziwe. Amabhentshimakhi obude bomongo we-Unsloth we-Llama 3.1 8B QLoRA ezingeni 32 kanye nosayizi weqoqo 1 aqinile.

I-Unsloth ibeka lokhu ku-algorithm yokuhlola i-gradient ehlanganiswe ne-Apple's Cut Cross Entropy. Ku-Llama 3.3 70B ku-80 GB A100, ibika amathokheni angu-89,389. Isisekelo se-FA2 sifinyelela ku-6,916.

Indaba yenkumbulo ye-MoE

Ukuqeqeshwa kwe-MoE yilapho ukuziphatha kwenkumbulo kushintshe kakhulu ngo-2026. Imibiko ye-Unsloth gpt-oss-20b ukulungisa kahle ngaphakathi kwe-12.8 GB, kuyilapho i-Qwen3-30B-A3B ku-16-bit LoRA idinga u-63 GB.

I-B200 gpt-oss run yayo isebenzise i-47.43 GB kumongo we-8K lapho i-Transformers v5 isebenzisa u-73.80 GB. Ku-16K, i-Transformers v5 iphelelwe yinkumbulo futhi i-Unsloth yasebenzisa i-55.13 GB.

Indlela yokwenza iwukwakhiwa kwe-split-LoRA. I-PEFT yenza i-delta ye-LoRA ibe ngokoqobo kubo bonke ochwepheshe ngaphambi kwe-matmul ye-MoE. I-Unsloth ihlela kabusha imisebenzi esikhundleni salokho, efana nezibalo kodwa igwema ukwenziwa kwezinto.

I-Axolotl ihlasela inkinga efanayo ngokuhlukile. Ukulinganisa kwayo uchwepheshe be-MoE kulinganisa izisindo zochwepheshe ngesikhathi sokulayisha imodeli, kukhulula i-bf16 tensor yoqobo ngokushesha.

Isizathu wushintsho lwe-Transformers v5. Izendlalelo zochwepheshe zisusiwe nn.Linear ukuhlanganisa nn.Parameter I-3D tensor. ama-bitandbyte awakwazanga ukuwalinganisa ngomthwalo. Amadokhumenti e-Axolotl abika i-GLM-4.7-Flash QLoRA yehla isuka cishe ku-127 GiB iye cishe ku-23 GiB egciniwe inkumbulo nge quantize_moe_experts: true.

Yilapho izinga lishintsha khona. I-Unsloth's single-GPU lead ayidluli.

I-Axolotl: i-matrix ye-parallelism ejulile

Umhlahlandlela wama-GPU amaningi we-Axolotl unikeza amasu amathathu okuhlukanisa okukhethekile: Izigaba ze-DeepSpeed ​​ZeRO 1 ukuya ku-3, FSDP, kanye ne-DDP. I-FSDP2 iyindlela enconyiwe, futhi i-FSDP1 yehlisiwe.

Ngaphezu kwalokho, umhlahlandlela wayo we-ND Parallelism uhlanganisa idatha, i-tensor, umongo, kanye nokufana kochwepheshe ngokusebenzisa i-PyTorch's. DeviceMesh. I-matrix yokwesekwa ebhaliwe iqinisekisa i-FSDP+TP, HSDP+TP, FSDP+CP, FSDP+TP+CP, kanye ne-FSDP+EP.

Izinhlanganisela ezimbili azisekelwa ngokusobala. Ukufana kochwepheshe akukwazi ukuqamba ne-TP noma i-CP ku-v1. I-DDP emsulwa ayikwazi ukuqamba nazo futhi.

Ukuhambisana kokulandelana kwe-Axolotl kusebenzisa i- ring-flash-attention umtapo wolwazi. Ibhentshimakhi yayo eshicilelwe ye-H100 ye-Llama 3.1 8B QLoRA isekela i-tradeoff.

Izikali zomongo zisondele kumugqa. Ukusebenza kahle kokusebenza kuyawa. Ku-4090s ku-SP degree 8, ibhentshimark efanayo irekhoda ukusheshisa okungu-0.88x. Ukuqeqeshwa kwehla kancane ngenkathi umongo ufinyelela amathokheni angu-32,768.

I-Axolotl iphinde isekele ukuqeqeshwa kwe-multi-node ngokusebenzisa i-torchrun ne-Ray.

I-TRL: izingemuva ezihlukanisa ngokulandelana okubili

Umhlahlandlela wokuqeqesha osabalalisiwe we-TRL uveza umehluko ohlanzekile. I-Context Parallelism isho Ukunaka Kwendandatho ku-FSDP2. Ukulandelana kwe-Parallelism kusho i-ALST/Ulysses ku-DeepSpeed.

I-Ring Attention idinga Sheshisa i-1.11.0+, isebenzisa cp_size nge cp_backend="torch"futhi okwamanje isekela i-SDPA kuphela. I-FlashAttention ayisekelwe kuleyo ndlela. Ukulandelana kufanele kuhlukaniswe cp_size * 2.

I-ALST/Ulysses idinga i-DeepSpeed ​​0.18.1+ kanye ne-Accelerate 1.12.0+. Isebenzisa sp_size nge sp_backend="deepspeed" futhi isebenza nge-FlashAttention-2. Kuboshwe ukubala kwekhanda lokunaka, okudingayo num_heads >= sp_size.

Iziqondiso ze-TRL ziqondile. I-Ring Attention ifanela ukulandelana kwamathokheni okungu-1M+ kanye ne-topology yenethiwekhi elinganiselwe. I-Ulysses ifanela ukuxhuma kwe-NVLink noma i-InfiniBand futhi ilandelana kufika cishe kumathokheni angu-500k.

Ibhentshimakhi ye-TRL's Ring Attention ecushwe kahle i-Qwen3-8B kuwo wonke ama-1, 2, 4, kanye nama-GPU angu-8 H100. Kuma-GPU angu-8, ubude bomongo obungaphezu kwamathokheni angu-300k buye baqeqesheka.

I-LLaMA-Factory: izinjini ezijwayelekile ezineMegatron

Amadokhumenti okuqeqesha asabalalisiwe e-LLaMA-Factory amboza i-DDP, i-DeepSpeed, ne-FSDP, okuhlanganisa i-FSDP2 ne-Ray yokugijima kwe-single-node kanye nama-multi-node. Amadokhumenti aphinde achaze i-DeepSpeed ​​AutoTP, ehlanganisa i-tensor parallelism ne-ZeRO.

Ukwengezwa kwayo okulandela kakhulu ngo-2025 kwaba ukuqeqeshwa kwe-Megatron-core backend nge-mcore_adapter. Lokho kuvula umzila wangempela wokuqeqeshwa kwangaphambi kwesikhathi.

Indlela ye-FSDP+QLoRA ishuna kahle imodeli engu-70B kuma-GPU amabili angu-24 GB. Leyo yindlela eshibhile ebhalwe phansi eya ku-70B kulesi siqhathaniso.

Iphoyinti lokushayisana yi-interface. Ukucushwa okusabalalisiwe kuhlala ku-YAML ne-CLI, hhayi ku-LlamaBoard. Amaqembu e-Dev amukele i-LLaMA-Factory nge-UI yekhodi enguziro alahlekelwa yileso sakhiwo lapho edlula i-GPU eyodwa.

Unsloth: igebe elivulekile

Imibhalo ye-Unsloth ene-GPU eminingi ithi ama-GPU amaningi asebenza nge-Accelerate ne-DeepSpeed, enikeza ukufinyelela ku-FSDP ne-DDP. Iphinde ithi inqubo iyinkimbinkimbi futhi idinga ukusethwa mathupha, nokusekelwa okusemthethweni kusamenyezelwa.

Umzila ongokoqobo uwukuthi accelerate launch train.py noma torchrun --nproc_per_node N_GPUS train.py. Kumamodeli amakhulu kakhulu ku-GPU eyodwa, device_map = "balanced" ihlukanisa imodeli kuwo wonke amadivayisi.

Uhlu lwe-Unsloth's PyPI lumaka i-multi-GPU njengoba itholakala nokuthuthuka okukhulu okulindile. I-studio changelog ichaza ukwabiwa kwe-multi-GPU okuzenzakalelayo kokuqala kokuqondiswa nokuqeqeshwa kusukela ngoMashi 2026.

Fundani ndawonye, ​​isikhundla sicacile. I-Unsloth isekela i-multi-GPU. Ayinikezi okwamanje i-compostable parallelism matrix ebhalwe yi-Axolotl ne-TRL.

• Ukungaziphathi kahle ikhefu lapho udinga i-tensor, umongo, noma ukufana kochwepheshe njengokucushwa kwesigaba sokuqala. Iphinda iphule uma imodeli yakho ingekho ohlwini lwayo olusekelwe, njengoba izinzuzo zivela kuma-kernels aqondene nezakhiwo.
• I-Axolotl aphule ijika lokufunda. Ulungiselela i-FSDP2 ngokumelene ne-DeepSpeed, idigri ye-SP, kanye nezingqinamba zokuhlukaniswa. Lezo zingqinamba zithatha isibalo se-GPU, ubude bokulandelana, namakhanda okunaka.
• I-TRL iphula kokuzenzakalelayo. Ikunikeza ezakudala ezifanele, hhayi ezishuniwe. Uhlinzeka ngokulungiselelwa kwe-Accelerate, ukulungiselelwa kwememori, nohlelo lokuhambisana.
• I-LLaMA-Factory iphula emngceleni we-UI. Ukuhunyushwa kwayo kuhle kakhulu kuze kufike endaweni eyodwa futhi kuzacile ngaphezu kwayo.

• I-GPU yomthengi oyedwa, izakhiwo ezisekelwayo, i-LoRA noma i-QLoRA: Ukungaziphathi kahle. I-headroom yobude bomongo iyodwa iyakuthethelela.
• Ama-GPU amabili kuya kwayisishiyagalombili, umongo omude, ukulungisa kahle okugcwele noma amapayipi e-RLHF: I-Axolotl. I-FSDP2 kanye nokuhambisana kokulandelana kuyindlela ebhalwe kahle kakhulu.
• Izihibe zokuqeqesha ngokwezifiso, ama-algorithms anoveli angemuva kokuqeqeshwa, ukuhlanganisa ubuso obuqinile: I-TRL. Wakha phezu kongqimba abanye abagoqayo.
• Ukufakwa kwemodeli ebanzi, ama-opharetha angewona onjiniyela, ukugijima kokuqala okushesha kakhulu: I-LLaMA-Factory. Bese uya ku-CLI uma ukala.
• Lezi zinketho azikhethekile. I-LLaMA-Factory ingasebenzisa i-Unsloth njenge-backend. I-TRL ithumela ukuhlanganiswa kwe-Unsloth. I-Axolotl ishayela abaqeqeshi be-TRL ngaphakathi.

• I-Unsloth iwina isivinini se-GPU eyodwa nobude bomongo; i-multi-GPU ihlala iyiphoyinti layo elibuthakathaka elibhaliwe.
• I-Axolotl ihambisa i-parallelism matrix ejulile: i-FSDP2, i-DeepSpeed, i-TP, i-CP, ne-EP ehlanganiswe nge-DeviceMesh.
• I-TRL isendlalelo sakudala esigoqwayo, manje esinokunakwa kweRing kanye nokuhlukaniswa kokulandelana kwe-ALST/Ulysses.
• I-LLaMA-Factory ihweba ngokujula ngobubanzi: amamodeli angu-100+, i-UI yekhodi enguziro, i-Megatron backend, ukusetha okusatshalaliswe kwe-CLI kuphela.
• Ukulandelana kwezilinganiso zokulinganisa umongo eduze-ngomugqa kodwa ukusebenza kahle kwe-output kwehla kakhulu kudlule ama-GPU amane.