I-Robbyant Open-Sources yeqembu le-Ant LingBot-Vision: Imodeli Yesisekelo Sombono we-1B Boundary-Centric Yokubona Kwendawo Eminyene

I-Robbyant, inkampani ehlanganisiwe ye-AI ngaphakathi kwe-Ant Group, inemithombo evulekile I-LingBot-Visionumndeni wama-Vision Transformers azigadile akhelwe ukubona indawo eminyene. Izisindo zihamba ngaphansi kwe-Apache-2.0 ku-Hugging Face ngamasayizi amane – i-ViT-giant, i-ViT-large, i-ViT-base, ne-ViT-small – kanye nombiko wezobuchwepheshe kanye nekhodi yokukhomba.

Amamodeli amaningi esisekelo sombono aqeqeshelwe ukungaguquki kwe-semantic: afunda ukuphendula ini isesithombeni ngenkathi ilahla ngqo ukwakheka kwendawo elungiswe kahle – imingcele yezinto, amakhonsathi, ukungaqhubeki kokujula – amarobhothi namanye amasistimu ahlanganiswe ngokomzimba ancike kuwo. I-LingBot-Vision iguqula lokho okubalulekile. Iphatha imingcele njenge-a isignali yomdabu yokuqeqesha esikhundleni sokuphuma komfula ongezansi, futhi inzuzo ingumgogodla wepharamitha engu-1B efana noma edlula amamodeli afika ku-7× amakhudlwana emisebenzini yendawo eminyene, okuhlanganisa ne-7B DINOv3.

Iyini i-LingBot-Vision?

I-LingBot-Vision iyisishumeki esizigadele ngaphambili semisebenzi ehlelwe ngokwendawo eya phansi. I-flagship ViT-g/16 inamapharamitha acishe abe ngu-1.1B futhi iqeqeshelwe injongo entsha ebizwa ngokuthi. ukumodeliswa komngcele ofihliwe kukhorasi ekhethiwe yezithombe ezingaba ngu-161M – ezikhethwe kuwebhu pool ye-2B – engenawo amalebula abantu, azikho izitholi zonqenqema zangaphandle, futhi awukho umgogodla oqeqeshwe kusengaphambili ongasuswa kuwo. Ukuqeqeshwa futhi kungokonga ngokuphawulekayo: ikhophasi i-oda lobukhulu lincane kune-LVD-1689M ye-DINOv3, futhi imodeli idla ngaphansi kwengxenye eyodwa kwezintathu yamasampuli okuqeqesha e-DINOv3.

Isifaki khodi sikhipha izici zethokheni ecinene ehloselwe ukufundwa okufriziwe. Ukuze sitshalwe kumabhajethi amancane, ifulegi lichithwa libe yi-ViT-L (300M), ViT-B (86M), kanye nezitshudeni ze-ViT-S ezihola ukuqagela okuminyene ngaphakathi kwamakilasi azo osayizi.

Indlela Imodeli Yemingcele Efihliwe Esebenza Ngayo

Indlela yakhela phezu kwe-paradigm ye-DINO/iBOT yokuzithiba distillation: uthisha – ikhophi ye-EMA yomfundi – ukhiqiza okuhlosiwe okuku-inthanethi, futhi umfundi ukubuyisela ekubukeni okufihliwe.

Ukumodeliswa kwesithombe esifihlekile okujwayelekile kufihla amapeshi ngokungahleliwe, kuziba ukuthi ipheshi ngalinye libonisani. I-patch yangaphakathi eyisicaba ishibhile ukuyibuyisela komakhelwane bayo; isiqeshana esinqamula umngcele wento siphethe isakhiwo umongo uwodwa ongeke wasinikeza. Imingcele iyizifunda ezingafuneki kangako, ezifundisa kakhulu isithombe – futhi ukufihla okungahleliwe kubaphatha njengayo yonke enye into.

I-LingBot-Vision ivala lelo gebe ngemibono emibili.

Ukuphoqelela umngcele. Uthisha ubikezela inkambu yomngcele ominyene ku-inthanethi futhi akhombe amathokheni aphethe umngcele B. Lawa aphoqelelwa kusethi eyisifihlwe yomfundi phezu kwemaski engahleliwe engu-M, enikeza imaski ehlanganisiwe engu-M⁺ = M ∪ B. Amathokheni afihliwe ahanjiswa ngejometri: amathokheni emngcele athola ithagethi yejiyomethri ecacile. Ngaphezu kwe i-semantic self-distillation target, kuyilapho amathokheni ambozwe ngaphakathi agcina umgomo we-semantic ojwayelekile wodwa. Lo mzila ubalulekile ngoba okuhlosiwe kwe-semantic ngokwemvelo akucaci kahle lapho izifunda ezimbili zihlangana khona – okuqondiwe kwejiyomethri kubekwe kahle lapho ukumodela okufihlakele okujwayelekile kubuthaka kakhulu, okuyikhona okuvumela ukumelwa kwe-semantic nejometri ukuthi kuqhamuke kunokuthi kuqhudelane.

Inkambu yomngcele wesigaba. Imingcele imodela njengamasegimenti omugqa aphakanyiselwe endaweni eminyene: iphikseli ngalinye eliseduze ligcina i-vector yesichasiso a(p) = (d, θ, φ¹, φ²) eqopha ibanga layo ukuya engxenyeni eseduze nama-engeli amathathu ayitholayo. Ukuhlehlisa ngokuqondile lo mkhakha ku-loop kathisha nomfundi kuyawa. Ukulungiswa kuwukuhlukanisa isiteshi ngasinye sibe yimigqomo engu-K = 32, ukusakaza kabusha ukubikezela komngcele njengokuhlukaniswa kwephikiseli ngayinye – okuvumela igatsha eliwumngcele lizuze ifa lomshini ofanayo wokugxilisa nokulola ozinzisa ukuzihluza kwesimanjemanje.

Ifomu lesigaba linomphumela omuhle ohlangothini. Ngaphansi kwe-classical a-contrario null hypothesis yokuthi “akukho sakhiwo,” umumo womngcele usatshalaliswa ngokufanayo – futhi lokho okuyi-null manje kuwukusabalaliswa okufanayo phezu kwemigqomo. Ukuchezuka ekufaneni kuwubufakazi bomngcele wangempela, ngakho ukuhlolwa kwe-Number-of-False-Alarms (NFA) okungenayo ipharamitha kuqinisekisa yonke ingxenye ekhokhiwe ngaphandle kwezindleko ezengeziwe. Uthisha usebenzisa lokhu ekuphindaphindeni ngakunye: ukhipha amasegimenti ekhandidethi ekubikezeleni kwenkambu yakhe, agcine kuphela abasindile abaqinisekiswe yi-NFA, futhi iphinde ibanikeze emkhakheni okuhlosiwe – ukuze isakhiwo esingasekelwe singalokothi sibe isignali yokufundisa.

Inhloso ephelele ihlanganisa amagama amane:

L = L_DINO + λᵢ · L_iBOT + λᵦ · L_bnd + λₖ · L_KoLeo

Amabhentshimakhi kanye nokusebenza

Yonke imiphumela eminyene ngezansi isebenzisa izici ezifriziwe ezinesendlalelo esisodwa somugqa, ngakho ukusebenza kubangelwa ukumelwa esikhundleni sedekhoda.

Ku-NYU-Ukujula v2, i-LingBot-Vision ithumela i-RMSE ehamba phambili kukho konke ukuqhathanisa (0.296), ngaphambi kwe-7B DINOv3 (0.309) cishe ngamapharamitha angu-7× ambalwa, nangaphambi kwe-2B V-JEPA 2.1 (0.307). Ku-KITTI iyimodeli engcono kakhulu engaphansi kwamapharamitha angu-2B. Ekuhlukanisweni kwe-semantic ilingana ne-Distilled DINOv3 ViT-H+ – 1.3 mIoU ngemuva ku-ADE20K, imesha ku-Cityscapes, phambili ku-VOC12 – kuyilapho ithuthuka ngaphezu kwe-DINOv2 yosayizi ofanayo ngo-4+ mIoU kuwo womathathu amabhentshimakhi; okuwukuphela kwegebe elisele elomndeni wakwa-DINOv3 ngokwawo (2.4 mIoU ku-ADE20K kuya kumodeli ye-7B), amandla awo aminyene avela ku-distillation nezinjongo zesici esiminyene.

Ukuhlukaniswa kwento yevidiyo kusebenzisa ukusakazeka kwelebula ngaphandle kokuqeqeshwa phezu kwezici ezifriziwe. I-LingBot-Vision ifinyelela ku-70.0 J&F ku-DAVIS-2017 kanye no-73.5 ku-YouTube-VOS — ngokuhambisana ne-DINOv3 ViT-H+ (71.1 / 74.0) kanye ne-7B DINOv3 (71.1 / 74.1), futhi ehamba phambili kuwo wonke amamodeli asele kunoma yisiphi isikali. Amathokheni omngcele ngokwawo azinzile ngokwanele ukuthi angalandelelwa ngevidiyo ngokufana okusobala kwe-cosine kwezici ezifriziwe, ngaphandle kokugadwa kwesikhashana.

Ukuhwebelana ukuqashelwa kweleveli yesithombe: Ukuhlola ngomugqa we-ImageNet-1K kufinyelela ku-86.32 kanye ne-k-NN 83.39, ilandela i-DINOv3-7B, echitha umthamo wayo ekuguquguqukeni kweleveli yesithombe. Izinzuzo nazo zisinda ekufakweni kwe-distillation – umfundi we-0.3B ViT-L ufana ne-7B DINOv3 ekujuleni kwe-NYUv2 (0.310 vs. 0.309) cishe namapharamitha angu-23× ambalwa.

Sebenzisa Amacala nokuthi Ungawalayisha Kanjani

Amathokheni epeshi afriziwe anikeza imithwalo yemisebenzi embalwa eminyene ngokuqondile: ukulinganisa ukujula kufunda i-geometry ngokuqondile kusukela ezicini, izinzuzo zesegimenti ye-semantic ezivela ezinguqukweni zesici ezihlala ncamashi kumakhonsana wento, futhi ukuhlukaniswa kwento yevidiyo kusebenza ngokufanisa ithokheni yokufana kwe-cosine. Isishumeki siphinde sisebenze njengokuqaliswa kokuqeqeshwa kokujula komfula.

Ukulayisha umgogodla kulandela inqolobane esemthethweni:

git clone 
cd lingbot-vision
conda create -n lingbot-vision python=3.10 -y
conda activate lingbot-vision
python -m pip install -r requirements.txt
python -m pip install -e .

import torch
from lingbot_vision import load_pretrained_backbone, extract_patch_tokens, load_image


device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32


# Downloads model.pt from Hugging Face on first use.
backbone, embed_dim = load_pretrained_backbone(
    variant="small",   # giant | large | base | small; defaults to large
    device=device,
    dtype=dtype,
)


img_norm, _, _ = load_image(
    "examples/example.png",
    size=512,
    patch_size=backbone.patch_size,
    mode="square",
)
patch_tokens, patch_grid = extract_patch_tokens(backbone, img_norm, device, dtype)


print(patch_tokens.shape, patch_grid, embed_dim)
# torch.Size([1, 1024, 384]) (32, 32) 384

I-agumenti ehlukile ikhetha usayizi bese ishintsha ibe nkulu. Amathokheni okukhiphayo anomumo [B, H*W, C]. Izimfuneko i-Python ≥ 3.10 ne-PyTorch ≥ 2.0, ene-GPU enconyelwe iqolo elikhudlwana.

I-LingBot-Depth 2.0: Inkokhelo Ephansi

Ukuze ubonise ukuthi isifaki khodi somdabu wendawo-sokubona sithenga ini ezansi nomfula, ithimba lithuthukise isistimu yalo yokuqedela ukujula ukuze I-LingBot-Depth 2.0. Iresiphi ye-masked-depth-modeling ayishintshiwe kusuka kunguqulo 1.0; izingxenye ezimbili eziqondile zihanjisiwe: ukuqaliswa komshini wokufaka ikhodi kushintshile ukusuka ku-DINOv2 kuya ku-LingBot-Vision (ezinhlobonhlobo ze-ViT-L ne-ViT-g), kanye nedatha yokuqeqeshwa ekhethiwe ikhule isuka kumasampuli ayi-3M akhishwe esidlangalaleni yaya ku-150M.

Lezo zinguquko ezimbili zibeka imiphumela eholayo kumabhentshimakhi okuqedela ukujula angu-14 ahlanganisa imaski yebhulokhi, i-sparse, kanye nezimiso zenzwa yangempela. Ku-DIODE-Indoor, i-RMSE ihlukaniswe ngohhafu isuka ku-0.132 iye ku-0.062. Isistimu iqine kakhulu ekuthwebuleni kwe-transparent-object ClearGrasp (0.010 / 0.012 RMSE) — isimo sokuhluleka sakudala sokuzwa ukujula okusebenzayo.

Ngokuphawulekayo, lezi zinhlanganisela ezimbili zishintsha esikhundleni sokukhansela: njengoba idatha yokuqeqeshwa ikhula isuka ku-3M iye ku-150M, ijika eliqaliswe i-DINOv2 ligcwala ngaphezu kwamasampuli angu-20M kuyilapho ijika le-LingBot-Vision liqhubeka lithuthuka. Idatha eyengeziwe ikhulisa, kunokuba igeze, inzuzo yendawo yokuqala engcono.

Okuthathwayo Okubalulekile

• I-LingBot-Vision yenza imingcele ibe isignali yomdabu yokuqeqeshelwa kusengaphambili, exhunywe ezithombeni ezingavuthiwe ezingenawo amalebula, izitholi ezisemaphethelweni, noma iqolo eliqeqeshwe kusengaphambili.
• Ukuphoqelela umngcele kanye nenkambu yomngcele wesigaba kuvumela i-geometry ne-semantics ukuthi ziqhamuke – futhi kuveza ukuhlolwa kokuqinisekisa kwe-NFA okunganapharamitha mahhala.
• I-backbone ye-1B ithumela i-NYU-Depth v2 RMSE ehamba phambili ekuqhathanisweni kwayo, ngaphambi kwe-7B DINOv3, kuyilapho iziqeqeshela ikhorasi encane yohlelo lobukhulu.
• Izinzuzo ziyasinda ekufakweni kwe-distillation: i-0.3B ViT-L ifana ne-7B DINOv3 ku-NYUv2 namapharamitha angu-23× ambalwa.
• Ukushintshanisa kuphela isifaki khodi nedatha yokukala kuthathe i-LingBot-Depth 2.0 emiphumeleni eholayo kuma-benchmarks wokuqedela ukujula okungu-14, futhi unqenqema lwesishumeki lunwebeka ngedatha eyengeziwe.
• Izisindo zithunyelwa ngaphansi kwe-Apache-2.0 ngamasayizi we-ViT-g/L/B/S kuso sonke isabelomali sokusebenza.

I-Interactive Dynamic Explainer