Ukuhlola i-TabPFN: Imodeli Yesisekelo Eyakhelwe Idatha Yethebula

I-TabPFN ngephepha le-ICLR 2023 – I-TabPFN: I-Transformer Exazulula Izinkinga Zokuhlukaniswa Kwethebula Encane Ngomzuzwana. Iphepha lethule i-TabPFN, imodeli ye-open-source transformer eyakhelwe ngqo amasethi edatha ethebula, isikhala esingazuzanga ngempela ekufundeni okujulile nalapho amamodeli ezihlahla ezinqumayo akhulisa izinqumo asabusa khona.

Ngaleso sikhathi, i-TabPFN yayisekela kuphela amasampula okuqeqesha afika kwayi-1,000 nezici zezinombolo eziyi-100, ngakho ukusetshenziswa kwayo kuzilungiselelo zomhlaba wangempela kwakunomkhawulo. Ngokuhamba kwesikhathi, nokho, kube nokuthuthuka okuningana okungeziwe okubandakanya i-TabPFN-2, eyethulwa ngo-2025 ngephepha – Izibikezelo Ezinembile Ngedatha Encane ene-Tabular Foundation Model (TabPFN-2).

Muva nje, i-TabPFN-2.5 ikhishwe futhi le nguqulo ingakwazi ukuphatha amaphuzu edatha asondele ku-100,000 kanye nezici ezingaba ngu-2,000, okuyenza isebenze kahle emisebenzini yokubikezela umhlaba wangempela. Ngichithe iminyaka yami eminingi yobungcweti ngisebenza ngamasethi edatha ethebula, ngakho-ke lokhu kubambe intshisekelo yami ngokwemvelo futhi kwangiphusha ukuthi ngibheke ngijule. Kulesi sihloko, nginikeza umbono wezinga eliphezulu le-TabPFN futhi ngiphinde ngidlule ekusetshenzisweni okusheshayo ngisebenzisa umncintiswano we-Kaggle ukukusiza ukuthi uqalise.

Yini i-TabPFN

I-TabPFN imele Inethiwekhi Efakwe I-Tabular-data Yangaphambili, imodeli yesisekelo ukuthi isekelwe embonweni wokufaka imodeli a ngaphambi kwedathasethi yethebula, esikhundleni sedathasethi eyodwa, yingakho igama.

Njengoba ngifunda imibiko yobuchwepheshe, bekukhona izingcezu nezingcezu eziningi ezithokozisayo kulawa mamodeli. Isibonelo, i-TabPFN ingaletha izibikezelo eziqinile zethebula nge-latency ephansi kakhulu, evame ukuqhathaniswa nezindlela zokuhlanganisa ezishuniwe, kodwa ngaphandle kwama-loops okuqeqesha aphindaphindiwe.

Ngokombono wokuhamba komsebenzi futhi alikho ijika lokufunda njengoba lingena ngokwemvelo ekusetheni okukhona ngokusebenzisa scikit-learn isitayela esibonakalayo. Ingakwazi ukuphatha amanani angekho, okungaphandle kanye nezinhlobo zezici ezixutshiwe ezinokucubungula kusengaphambili okuncane esizozihlanganisa phakathi nokuqaliswa, kamuva kulesi sihloko.

Isidingo semodeli eyisisekelo yedatha yethebula

Ngaphambi kokungena ekutheni i-TabPFN isebenza kanjani, ake siqale sizame ukuqonda inkinga ebanzi ezama ukuyilungisa.

Ngokufunda komshini okuvamile kumasethi edatha ethebula, ngokuvamile uqeqesha imodeli entsha yayo yonke idathasethi entsha. Lokhu kuvame ukubandakanya imijikelezo yokuqeqesha emide, futhi kusho nokuthi imodeli eqeqeshwe ngaphambilini ayikwazi ukuphinda isetshenziswe.

Kodwa-ke, uma sibheka amamodeli ayisisekelo ombhalo nezithombe, umbono wabo uhluke kakhulu. Esikhundleni sokuziqeqesha kabusha kusukela ekuqaleni, inani elikhulu lokuqeqeshwa kwangaphambili lenziwa ngaphambili kuwo wonke amasethi edatha amaningi futhi imodeli ewumphumela ingase isetshenziswe kumasethi edatha amasha ngaphandle kokuqeqeshwa kabusha ezimweni eziningi.

Lokhu ngokubona kwami ​​igebe imodeli elizama ukulivala kudatha yethebula okungukuthi ukunciphisa isidingo sokuqeqesha imodeli entsha kusukela ekuqaleni kwayo yonke idathasethi futhi lokhu kubukeka njengendawo ethembisayo yocwaningo.

Ukuqeqeshwa kwe-TabPFN kanye nepayipi le-Inference ezingeni eliphezulu

I-TabPFN iyasebenza ukufunda ngaphakathi kokuqukethwe ukuze kulingane inethiwekhi ye-neural ngaphambi kwedathasethi yethebula. Okushiwo lokhu ukuthi esikhundleni sokufunda umsebenzi owodwa ngesikhathi, imodeli ifunda ukuthi izinkinga zethebula zivame ukubukeka kanjani ngokujwayelekile bese isebenzisa lolo lwazi ukwenza izibikezelo kumasethi wedatha amasha ngokudlula okukodwa kokuya phambili. Nansi isiqephu sephepha le-TabPFN's Nature:

I-TabPFN ithuthukisa ukufunda kokuqukethwe (ICL), indlela efanayo eholele ekusebenzeni okumangalisayo kwamamodeli amakhulu olimi, ukukhiqiza i-algorithm enamandla yokuqagela ithebula efundwa ngokugcwele. Nakuba i-ICL yaqala ukubonwa kumamodeli ezilimi ezinkulu, umsebenzi wakamuva ubonise ukuthi ama-transformer angafunda ama-algorithms alula njengokuhlehla kwezinto nge-ICL.

Ipayipi lingahlukaniswa ngezinyathelo ezintathu ezinkulu:

1. Ukukhiqiza Idathasethi Yokwenziwa

I-TabPFN iphatha yonke idathasethi njengephoyinti ledatha elilodwa (noma ithokheni) elifakwe kunethiwekhi. Lokhu kusho ukuthi kudinga ukuchayeka enanini elikhulu kakhulu lamadathasethi ngesikhathi sokuqeqeshwa. Ngalesi sizathu, ukuqeqesha i-TabPFN kuqala amasethi edatha ethebula yokwenziwa. Kungani zokwenziwa? Ngokungafani nombhalo noma izithombe, awekho amasethi edatha ethebula amaningi amakhulu futhi ahlukahlukene emhlabeni wangempela atholakalayo, okwenza idatha yokwenziwa ibe yingxenye ebalulekile yokusetha. Ukuze sikubeke kucace, i-TabPFN 2 yaqeqeshwa kumadathasethi ayizigidi eziyi-130.

Inqubo yokukhiqiza idathasethi yokwenziwa iyathakazelisa ngokwayo. I-TabPFN isebenzisa i-parametric ephezulu imodeli yembangela yesakhiwo ukuze udale amasethi edatha ethebula anezakhiwo ezihlukahlukene, ubudlelwano besici, amazinga omsindo nemisebenzi eqondiwe. Ngokuthatha isampula kule modeli, isethi enkulu nehlukahlukene yamadathasethi ingakhiqizwa, ngalinye lisebenza njengesignali yokuqeqeshwa yenethiwekhi. Lokhu kukhuthaza imodeli ukuthi ifunde amaphethini ajwayelekile kuzo zonke izinhlobo zezinkinga zethebula, kunokuthi ifake ngokweqile kunoma iyiphi idathasethi eyodwa.

2. Ukuqeqeshwa

Isibalo esingezansi sithathwe ephepheni leMvelo, okukhulunywe ngalo ngenhla sikhombisa ngokusobala ukuqeqeshwa kanye nenqubo yokucabanga.

Ngesikhathi sokuqeqeshwa, isethi yedatha yethebula yokwenziwa iyathathwa bese ihlukaniswa ibe isitimela esingu-X,Y isitimela, Ukuhlolwa kwe-Xfuthi Ukuhlolwa kuka-Y. I Ukuhlolwa kuka-Y amanani abanjiwe, futhi izingxenye ezisele zidluliselwa kunethiwekhi ye-neural ekhipha ukusabalalisa kwamathuba ngayinye Ukuhlolwa kuka-Y iphuzu ledatha, njengoba kukhonjisiwe emfanekisweni ongakwesokunxele.

Ibambe ngaphandle Ukuhlolwa kuka-Y amanani abe esehlolwa ngaphansi kwalokhu kusatshalaliswa okubikezelwe. A cross entropy ukulahlekelwa bese kubalwa futhi inethiwekhi ibuyekezelwa ukuze nciphisa lokhu kulahlekelwa. Lokhu kuqeda isinyathelo esisodwa sokubuyisela emuva kudathasethi eyodwa futhi le nqubo ibe isiphindaphinda izigidi zamasethi edatha okwenziwa.

3. Incazelo

Ngesikhathi sokuhlolwa, imodeli ye-TabPFN eqeqeshiwe isetshenziswa kudathasethi yangempela. Lokhu kuhambisana nesibalo esingakwesokudla, lapho imodeli isetshenziselwa ukucabangela. Njengoba ubona, isikhombimsebenzisi sihlala sifana nesikhathi sokuqeqeshwa. Uyahlinzeka X isitimela, Y isitimelafuthi Ukuhlolwa kwe-Xkanye nezibikezelo zemodeli ye Ukuhlolwa kuka-Y ngokusebenzisa iphasi eyodwa eya phambili.

Okubaluleke kakhulu, akukho ukuqeqeshwa kabusha ngesikhathi sokuhlolwa futhi i-TabPFN yenza lokho okusebenza ngempumelelo i-zero-shot inferenceekhiqiza izibikezelo ngokushesha ngaphandle kokubuyekeza izisindo zayo.

Izakhiwo

Masiphinde sithinte ukwakheka kwemodeli njengoba kushiwo ephepheni. Ezingeni eliphezulu, i-TabPFN ivumelanisa i-architecture yesiguquli ukuze ivumelane kangcono nedatha yethebula. Esikhundleni sokucaba itafula ngokulandelana okude, imodeli iphatha inani ngalinye etafuleni njengeyunithi yalo. Isebenzisa indlela yokunaka enezigaba ezimbili lapho iqala khona ukufunda ukuthi izici zihlobana kanjani phakathi komugqa owodwa bese ifunda ukuthi isici esifanayo siziphatha kanjani emigqeni ehlukene.

Le ndlela yokuhlela ukunaka ibalulekile njengoba ihambisana nendlela idatha yethebula ehlelwe ngayo. Lokhu kusho nokuthi imodeli ayinandaba nokuhleleka kwemigqa noma amakholomu okusho ukuthi ingakwazi ukuphatha amathebula amakhulu kunalawo eqeqeshwe kuwo.

Ukuqaliswa

Manje ake sidlule ekusetshenzisweni kwe I-TabPFN-2.5 bese uyiqhathanisa ne-vanilla XGBoost uhlukanisa ukuze unikeze iphuzu elijwayelekile lereferensi. Ngenkathi izisindo zemodeli zingalandwa ku-Hugging Face, kusetshenziswa Kaggle Notebooks iqonde kakhudlwana njengoba imodeli itholakala kalula lapho futhi nokusekelwa kwe-GPU kuphuma ebhokisini ukuze kucatshangelwe ngokushesha. Kunoma ikuphi, udinga ukwamukela imigomo yemodeli ngaphambi kokuyisebenzisa. Ngemva kokwengeza imodeli ye-TabPFN endaweni ye-notebook ye-Kaggle, sebenzisa iseli elandelayo ukuze uyingenise.

# importing the model
import os
os.environ["TABPFN_MODEL_CACHE_DIR"] = "/kaggle/input/tabpfn-2-5/pytorch/default/2"

Ungathola ikhodi ephelele encwadini yokubhalela ye-Kaggle ehambisana naso lapha.

Ukufakwa

Ungafinyelela i-TabPFN ngezindlela ezimbili noma njenge- Iphakheji ye-Python futhi uyiqhube endaweni noma njenge Iklayenti le-API ukusebenzisa imodeli efwini:

# Python package
pip install tabpfn


# As an API client
pip install tabpfn-client

Idathasethi: Isethi yedatha yomncintiswano we-Kaggle Playground

Ukuthola umqondo ongcono wokuthi i-TabPFN isebenza kanjani esimweni somhlaba wangempela, ngiyihlolile emqhudelwaneni we-Kaggle Playground ophele ezinyangeni ezimbalwa ezedlule. Umsebenzi,I-Binary Prediction ene-Rainfall Dataset (ilayisensi ye-MIT), idinga ukubikezela amathuba okuna kwemvula ngayinye id kusethi yokuhlola. Ukuhlola kwenziwa kusetshenziswa i-ROC–AUC, okwenza lokhu kulingane kahle kumamodeli asekelwe emathubeni afana ne-TabPFN. Idatha yokuqeqeshwa ibukeka kanje:

Ukuqeqesha I-TabPFN Classifier 

Ukuqeqesha I-TabPFN Classifier iqondile futhi ilandela ejwayelekile scikit-funda isitayela esibonakalayo. Nakuba kungekho ukuqeqeshwa okuqondene nomsebenzi othile ngokomqondo wendabuko, kusabalulekile ukunika amandla Usekelo lwe-GPUngaphandle kwalokho ukucatshangwa kungahamba kancane ngokuphawulekayo. Amazwibela ekhodi alandelayo ahamba ekulungiseni idatha, aqeqeshe isihlukanisi se-TabPFN nokuhlola ukusebenza kwaso kusetshenziswa isikolo se-ROC–AUC.

# Importing necessary libraries
from tabpfn import TabPFNClassifier
import pandas as pd, numpy as np
from sklearn.model_selection import train_test_split

# Select feature columns
FEATURES = [c for c in train.columns if c not in ["rainfall",'id']]
X = train[FEATURES].copy()
y = train["rainfall"].copy()

# Split data into train and validation sets
train_index, valid_index = train_test_split(
    train.index,
    test_size=0.2,
    random_state=42
)

x_train = X.loc[train_index].copy()
y_train = y.loc[train_index].copy()

x_valid = X.loc[valid_index].copy()
y_valid = y.loc[valid_index].copy()

# Initialize and train TabPFN
model_pfn = TabPFNClassifier(device=["cuda:0", "cuda:1"])
model_pfn.fit(x_train, y_train)

# Predict class probabilities
probs_pfn = model_pfn.predict_proba(x_valid)

# # Use probability of the positive class
pos_probs = probs_pfn[:, 1]

# # Evaluate using ROC AUC
print(f"ROC AUC: {roc_auc_score(y_valid, pos_probs):.4f}")

-------------------------------------------------
ROC AUC: 0.8722

Okulandelayo masiqeqeshe isihlukanisi se-XGBoost esiyisisekelo.

Ukuqeqesha I-XGBoost Classifier

from xgboost import XGBClassifier

# Initialize XGBoost classifier
model_xgb = XGBClassifier(
    objective="binary:logistic",
    tree_method="hist",
    device="cuda",
    enable_categorical=True,
    random_state=42,
    n_jobs=1
)

# Train the model
model_xgb.fit(x_train, y_train)

# Predict class probabilities
probs_xgb = model_xgb.predict_proba(x_valid)

# Use probability of the positive class
pos_probs_xgb = probs_xgb[:, 1]

# Evaluate using ROC AUC
print(f"ROC AUC: {roc_auc_score(y_valid, pos_probs_xgb):.4f}")

------------------------------------------------------------
ROC AUC: 0.8515

Njengoba ubona, i-TabPFN isebenza kahle ngaphandle kwebhokisi. Ngenkathi i-XGBoost ingacutshungulwa ngokuqhubekayo, inhloso yami lapha ukuqhathanisa eziyisisekelo, vanilla ukuqaliswa esikhundleni samamodeli athuthukisiwe. Kungibeke esikhundleni sama-22 ebhodini labaphambili lomphakathi. Ngezansi kukhona amaphuzu aphezulu angu-3 okubhekisela kuwo.

Kuthiwani ngokuchazwa kwemodeli?

Amamodeli e-Transformer awachazeki ngokwemvelo ngakho-ke ukuze kuqondwe izibikezelo, amasu okutolika angemuva kwe-hoc afana ne-SHAP (I-SHApley Additive Explanations) avame ukusetshenziselwa ukuhlaziya ukuqagela ngakunye kanye neminikelo yesici. I-TabPFN ihlinzeka ngesandiso esizinikele sokutolika esihlanganisa ne-SHAP, okwenza kube lula ukuhlola nokubonisana mayelana nezibikezelo zemodeli. Ukuze ufinyelele lokho uzodinga ukufaka isandiso kuqala:

# Install the interpretability extension:
pip install "tabpfn-extensions[interpretability]"

from tabpfn_extensions import interpretability

# Calculate SHAP values
shap_values = interpretability.shap.get_shap_values(
    estimator=model_pfn,
    test_x=x_test[:50],
    attribute_names=FEATURES,
    algorithm="permutation",
)

# Create visualization
fig = interpretability.shap.plot_shap(shap_values)

Isakhiwo esingakwesokunxele sibonisa isilinganiso sokubaluleka kwesici se-SHAP kuyo yonke idathasethi, okunikeza umbono womhlaba wonke wokuthi yiziphi izici ezibaluleke kakhulu kumodeli. Isakhiwo esingakwesokudla ngu-a SHAP summary (beesswarm) isakhiwookunikeza ukubuka okuyimbudumbudu kakhudlwana ngokubonisa amanani e-SHAP esici ngasinye kuzo zonke izibikezelo ngazinye.

Kulezi ziqephu ezingenhla, kusobala ukuthi ikhava yefu, ukukhanya kwelanga, umswakamafuthi iphuzu lamazolo abe nomthelela omkhulu kunayo yonke ekuqaguleni kwemodeli, kuyilapho izici ezifana nendlela yomoya, ingcindezi, neziguquko ezihlobene nezinga lokushisa zidlala indima encane ngokuqhathaniswa.

Kubalulekile ukuqaphela ukuthi i-SHAP ichaza ubudlelwano obufundiwe bemodeli, hhayi imbangela yomzimba.

Isiphetho

Kuningi okuningi ku-TabPFN kunalokho engikumbozile kulesi sihloko. Engikuthandile mathupha yiwo womabili umqondo oyisisekelo nokuthi kulula kangakanani ukuqalisa. Kunezici eziningi engingazithintanga lapha, njengokusetshenziswa kwe-TabPFN ekubikezelweni kochungechunge lwesikhathi, ukutholwa okungaqondakali, ukukhiqiza idatha yethebula yokwenziwa, kanye nokukhipha okushumekiwe kumamodeli we-TabPFN.

Enye indawo engithakasela kakhulu ukuyihlola wukuhlela kahle, lapho lawa mamodeli angashintshwa ukuze abe idatha evela esizindeni esithile. Sesikushilo lokho, lesi sihloko besihloselwe ukuba isingeniso esilula esisekelwe kokuhlangenwe nakho kwami ​​​​kokuqala. Ngihlela ukuhlola lawa makhono engeziwe ngokujula okwengeziwe kokuthunyelwe okuzayo. Okwamanje, imibhalo esemthethweni iyindawo enhle yokungena ujule.

Qaphela: Zonke izithombe, ngaphandle uma kushiwo ngenye indlela, zidalwe umbhali.