Ngale Komugqa Oqondile: Ukukhetha Phakathi Kwe-OLS, Imibandela Yokusebenzisana, kanye Nokwehliswa Kwe-Tweedie

0 0 10 minutes read

Ngale Komugqa Oqondile: Ukukhetha Phakathi Kwe-OLS, Imibandela Yokusebenzisana, kanye Nokwehliswa Kwe-Tweedie

imodeli yokufunda yomshini yokubikezela ukuthi ingakanani imali amakhasimende azoyisebenzisa kunkundla ye-e-commerce onyakeni olandelayo. Umsebenzi ibhizinisi lesayensi yedatha njengenjwayelo: layisha idatha, ihlanze, iqonde, futhi wenze imodeli yomphumela.

Umbuzo omkhulu ofika ezingqondweni zethu: Iyiphi i-algorithm oyikhipha ku-toolkit yakho?

Kumamodeli okubuyisela emuva, ngicabanga ukuthi sivame ukuzenzakalelayo kuma-classics afana ne-OLS, noma ngezinye izikhathi, vele sigxume siqonde endleleni ehlanganisiwe eyinkimbinkimbi efana ne-XGBoost. Kepha ngokuhlangenwe nakho, bengilokhu ngibona ukuthi imvamisa uhlaka olujwayelekile lomugqa yilo kanye oludingayo ukuze uthole incazelo, isivinini, kanye nokutolika.

Inselele yangempela isekukhetheni ukunambitheka okulungile.

Ingabe kufanele sibambelele ekuhlehleni okuvamile kwe-Ordinary Least Squares (OLS)?
Ingabe sidinga ukwethula imigomo yokusebenzisana?
Noma ingabe idatha yethu iyinqaba ngokwanele ukugunyaza a Tweedie ukuhlehla?

Ukukhetha okungalungile kungaholela kumamodeli akhipha izibikezelo ezingenakwenzeka, njengokusebenzisa ikhasimende inani elibi lemali.

Ake sihlukanise lezi zindlela ezintathu ukuze ukhethe ngokuzethemba ithuluzi elilungile lendawo yedatha yakho ethile.

Isethi yedatha

Idathasethi esetshenziselwa lo msebenzi kuzoba yidathasethi Yezimangalo Zesikweletu Senkampani Yase-French, kusukela kuphakheji engu-R CASDatasets, ngaphansi kwelayisensi GPL >2.

UChristophe Dutang no-Arthur Charpentier (2026). Amasethi wedatha we-CAS: Amasethi wedatha womshwalenseinguqulo yephakheji ye-R 1.2-1, DOI 10.57745/P0KHAG.

# Basics
import pandas as pd
import numpy as np

# Dataviz
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')
import plotly.express as px

# Stats
import statsmodels.api as sm
import statsmodels.formula.api as smf
import scipy.stats as stats

# Preprocessing & Modeling
from sklearn.linear_model import TweedieRegressor
from feature_engine.encoding import OneHotEncoder
import warnings
from sklearn.metrics import mean_absolute_error

import warnings
warnings.filterwarnings('ignore')

Ukuze uthole idathasethi, kukhona amazwibela ekhodi ongawasebenzisa kulesi sifundo se-sklearn [4].

# Load Data
df = load_mtpl2(n_samples=250_000)

Ukubuka kwedathasethi. Isithombe sombhali.

Ngemuva kokuhlola idatha kancane, kukhona izinguquko ezimbalwa ezithokozisayo esingayenza, njengokusika idatha ku-$500,000 esiphezulu futhi sigcine inani lezicelo lingaphansi koku-4, njengoba izinombolo ezingaphezulu kwalokho zihlanya kakhulu (ama-outliers) futhi zizohlanekezela ngokuqinisekile izibikezelo zethu.

# Clipped
df = df.query('ClaimAmount < 500000')
df = df.query('ClaimNb < 4')

1. I-OLS Evamile: Umgwaqo Omkhulu Othembekile

Ake siqale ngemodeli yesisekelo. Ukuhlehla okulula kakhulu kwe-Ordinary Least Squares (OLS) kuwumgogodla wokumodela kwezibalo. Uma isayensi yedatha ibinenkinobho yezilungiselelo “ezenzakalelayo”, bekungaba i-OLS.

Kodwa “okubi” ukuthi kudinga ukucabanga ukuthi idatha izoziphatha ngendlela ethile.

Isibonelo, ingacabanga ukuthi:

Ithagethi ishintsha ngezinga elingaguquki, elibikezelwayo njengoba okuguquguqukayo kwakho okuzimele kushintsha.
Uma unyusa inani lezimangalo ngo-1, Inani lakho Lesimangalo likhuphuka ngenani elinqunyiwe, kungakhathaliseki ukuthi lesi isicelo sakho sokuqala noma cha.

Ngokwezibalo, i-OLS ibheka umugqa owehlisa isamba somehluko oyisikwele phakathi kwamaphoyinti akho edatha wangempela nalowo mugqa.

Isebenza nini?

I-OLS inamandla amangalisayo uma idatha yakho ihlangabezana nokuqagela okumbalwa okubalulekile:

Linearity: Ubudlelwano phakathi kwezici zakho kanye nenhloso yakho buqondile ngendlela efanele.
I-Homoscedasticity: Ukwehluka kwezinsalela zakho (amaphutha) kuhlala njalo. NgesiNgisi esilula, imodeli yakho ayilungile neze ngamavelu amakhulu kunamanani amancane.
Ukusabalalisa Okuvamile: Amaphutha avamise ukusatshalaliswa.

Uma ubikezela okuthile okuzinzile nokuqhubekayo (isb, ukuphakama kwesihlahla ngokusekelwe eminyakeni yaso, noma izinga lokushisa elisekelwe ekuphakameni), i-OLS evamile ngokuvamile iwukubheja kwakho okuhle kakhulu. Isobala, iyashesha, futhi iqondakala kalula.

Kule nkinga, nokho, sizobona ukuthi akukhona ukubheja okungcono kakhulu, njengoba idatha yethu igcwele amanani esimangalo angamaziro.

# Define the OLS model formula
formula = "ClaimAmount ~ Exposure + VehPower + VehAge + DrivAge + BonusMalus + Density + C(Area) + C(VehBrand) + C(VehGas) + C(Region)"

# Fit the OLS model
ols_model = smf.ols(formula=formula, data=df).fit()

# Display the model summary
print(ols_model.summary())

Ngemva kokusebenzisa imodeli yethu yokuqala, ake sibheke imiphumela.

Uma sibheka izinsalela zemodeli, izinsalela zemodeli ye-OLS, njengoba kuboniswa i-histogram necebo le-QQ, ngokusobala azisatshalaliswa ngokujwayelekile, okuyinkinga evamile lapho kubhekwana namasethi edatha anamanani amaningi anguziro njengamanani esimangalo somshuwalense.

I-Histogram kanye ne-QQ-Plot yezinsalela: Imodeli idonsa kanzima ngamavelu aphezulu. Isithombe sombhali.

2. I-OLS Nemibandela Yokusebenzelana: Isici “Sincike”

Kodwa kwenzekani lapho umhlaba wangempela uba yinkimbinkimbi? Ake sithi ubikezela amanani ezindlu ngokusekelwe ezicini ezimbili: inombolo yamagumbi okulala nokuthi indlu inalo yini idamu lokubhukuda.

Imodeli ye-OLS evamile ibheka lokhu ngokuzimela. Ithi, “Igumbi lokulala elengeziwe liyengeza $50,000nedamu liyanezela $30,000.” Kodwa ingabe lokho kuyiqiniso ngaso sonke isikhathi? Ingabe ichibi lengeza inani elifanayo ku-condo enekamelo elilodwa njengoba lenza endaweni ewubukhazikhazi enamagumbi ama-5?

Cishe cha. Inani le-pool kuncike ngobukhulu bendlu. Kulapho-ke amatemu okusebenzisana aqala khona ukusebenza.

Itemu lokusebenzisana litshela imodeli yakho ukuthi umthelela wokuguquguquka okukodwa kuya ngezinga lokunye okuguquguqukayo. Kudatha yakho, umele lokhu ngokuphindaphinda izici ezimbili ndawonye:

Enye indlela yokuyibuka iwukusebenzisa lesi sifaniso nokupheka. Imigomo yokusebenzisana ifana nokubhaka.

Ufulawa wodwa womile.
Amanzi ngokwawo amanzi.
Kodwa uma uzihlanganisa ndawonye, awuvele ube “manzi omile”; uthola inhlama, indawo entsha ngokuphelele.

Kufanele uyisebenzise nini?

Kufanele uthuthukise imodeli yakho ye-OLS ngemibandela yokusebenzisana uma usola ukusebenzisana noma ukuphikisa phakathi kwezici.

I-Synergy: Izici ezimbili zihlangene zinamandla kakhulu kunesamba sezingxenye zazo (isb., I-Marketing Spend $times$ Holiday Season).
Ukuphikisa: Isici esisodwa sidambisa umphumela wesinye (isb, Umthamo Womuthi A $times$ Umthamo Womuthi B).

Uma imodeli yakho ye-OLS evamile ingasebenzi kahle futhi unolwazi oluqinile lwesizinda oluphakamisa ukuthi okuguquguqukayo kwakho kukhulume zodwa, ungamane uphonse idatha eyengeziwe kuyo. Zama ukungeza itemu lokusebenzisana.

Nokho, kulesi simo, ake sibone ukuthi isebenza kanjani.

# Define the OLS model formula with another interaction term and without 'Region'
formula_no_region = "ClaimAmount ~ Exposure * VehBrand + Exposure * BonusMalus + DrivAge + BonusMalus + Density + C(Area) + VehPower + C(VehGas)"

# Fit the OLS model with two interaction terms and without 'Region'
ols_no_region_model = smf.ols(formula=formula_no_region, data=df).fit()

# Display the model summary
print(ols_no_region_model.summary())

I-OLS enemiphumela yemibandela yokusebenzisana. Isithombe sombhali.

Ngaphandle kokwethula imigomo yokusebenzisana, ukusebenza kwemodeli ye-OLS ku- ClaimAmount idatha ayishintshile. Lokhu kungenxa yokugxila okuphezulu kwezimangalo ezinguziro kanye nokusabalalisa okugwegwile kakhulu kwezimangalo ezingezona uziro.

Amamodeli e-Ordinary Least Squares athatha amaphutha avame ukusatshalaliswa, ukucabanga ukuthi kwephulwa ngokusobala kule dathasethi egcwele iqanda. Ngenxa yalokho, ukwengeza amatemu okusebenzelana kungase kuthwebule ubudlelwano obuyinkimbinkimbi, kodwa akukwazi ukumodela ukusatshalaliswa kwangempela kwale datha, okuholela ekuthuthukisweni okuncane kuphela.

3. Ukuhlehla kwe-Tweedie: Ukwamukela ama-Messy Zeroes

Kuthiwani uma idatha yakho ingeyona ijika lensimbi elihlanzekile, eliqhubekayo? Kuthiwani uma idatha yakho ibukeka njengodonga olukhulu loziro olulandelwa umsila omude, olandelanayo wezinombolo eziphozithivu?

Njengoba singabona esibonelweni sethu, kunoma yimuphi unyaka othile, iningi labanikazi bomshwalense bafaka i-$0 ncamashi ezimangalweni. Abangeni ezingozini. Kodwa kumaphesenti amancane abenzayo, izimangalo zabo aziwona nje ama-$5 noma ama-$10. Zingaba izinkulungwane noma amashumi ezinkulungwane zamaRandi.

Uma uzama ukufaka ulayini ojwayelekile we-OLS kule datha, imodeli iyaphuka. Kungani? Ngoba i-OLS ithatha ukusatshalaliswa okuvamile. Ukuze kuhlangatshezwane nawo wonke lawo ma-zero kanye nezitolo ezimbalwa ezinkulu, i-OLS ingase iqale ukubikezela ukuthi amanye amakhasimende asengozini encane negative izimangalo.

Lesi yisizathu sokuthi sizozama Ukusatshalaliswa kwe-tweedie.

Ukusatshalaliswa kwe-Tweedie kuwumndeni okhethekile wokusabalalisa okungenzeka ukwazi ukuphatha ingxube yenqwaba yeqanda kuqanda kanye nokusabalalisa okuqhubekayo, okutsheke kwesokudla ngamavelu amahle. Isebenza njengengxube phakathi kokusatshalaliswa kwe-Poisson (okubala imicimbi) nokusabalalisa kwe-Gamma (okulinganisa amanani aqhubekayo avumayo).

Cabanga ngokuhlehla kwe-Tweedie njengokulinganisa imvula ogwadule. Ezinsukwini eziningi, imvula isuke inguziro. Kodwa lapho lina, liyathulula, futhi uthola inani eliqhubekayo, elishintshashintsha kakhulu lamanzi. Awukwazi ukumodela imvula yasogwadule usebenzisa ijika lensimbi elibushelelezi, elilinganayo.

Kufanele uyisebenzise nini?

Ukwehla kwe-Tweedie kuyizinga eligolide ledatha elibonakala ngokuthi:

Amanani angewona anegethivu ngokuqinile (awukwazi ukuba nemali engaphansi kuka-$0 ekuthengisweni noma ezimangalweni).
I-spike enkulu ku-zero (zero-inflation).
Ukusabalalisa okugwegwile kakhulu ngamavelu angewona aziro.

Uma usebenza kubuchwepheshe bomshwalense, ubungozi besikweletu, noma ukumodela inani lokuphila kwekhasimende lapho abasebenzisi abaningi bengaguquki khona, u-Tweedie ungaba umngane wakho.

# OHE categorical variables
ohe = OneHotEncoder(drop_last=True)
df_ohe = ohe.fit_transform(df.drop(columns=['OLS_pred', 'OLS_Int_Term_pred'], axis=1))

# X and Y
X = df_ohe.drop(columns=['ClaimAmount'], axis=1)
y = df_ohe['ClaimAmount']

# More granular search for tweedie_powers around 1.75-1.8
tweedie_powers = np.arange(1.7, 1.8, 0.01) # Example: From 1.7 to 1.8 with step 0.01
alpha_values = [0.1, 0.5, 1.0] # Test different regularization strengths

results = []

print("Searching for best Tweedie power and alpha...")
for power in tweedie_powers:
    for alpha in alpha_values:
        sklearn_tweedie_model = TweedieRegressor(
            power=power, 
            link='auto', 
            solver='newton-cholesky', 
            max_iter=500, 
            alpha=alpha # Add regularization
        )
        try:
            # Fit the model, using Exposure as an offset
            sklearn_tweedie_model.fit(X, y, sample_weight=df['Exposure'])
            mae = mean_absolute_error(y, sklearn_tweedie_model.predict(X))
            results.append((power, alpha, mae))
        except Exception as e:
            # Catch potential convergence issues for some parameter combinations
            results.append((power, alpha, float('inf'))) # Assign high MAE for failed fits
            

# Sort results to find the best model
best_result = min(results, key=lambda x: x[2])
best_power, best_alpha, best_mae = best_result

print(f"Best Tweedie Power: {best_power:.2f}, Best Alpha: {best_alpha:.1f}, Best MAE: {best_mae:.2f}")

# Now, fit the best model
sklearn_tweedie_model = TweedieRegressor(
    power=best_power, 
    link='auto', 
    solver='newton-cholesky', 
    max_iter=500, 
    alpha=best_alpha
)
sklearn_tweedie_model.fit(X, y, sample_weight=df['Exposure'])

print("Sklearn Tweedie Regressor fitted successfully with optimized parameters!")
print(results)

Searching for best Tweedie power and alpha...
Best Tweedie Power: 1.76, Best Alpha: 1.0, Best MAE: 110.31
Sklearn Tweedie Regressor fitted successfully with optimized parameters!

Ake sibikezele ngale modeli, futhi ngokulandelayo sihlole ukuthi icebo le-QQ libukeka kanjani.

# Make predictions using the optimized model
df['Tweedie_pred'] = sklearn_tweedie_model.predict(X)

Lokhu ukuqhathanisa okwangempela nokubikezela kwale modeli yokugcina.

Izibikezelo ze-Tweedie Model vs zangempela. Isithombe sombhali.

Qaphela ukuthi amaphutha asekhona kakhulu, ikakhulukazi ClaimNb == 1. Nakuba izinsalela zemodeli ye-Tweedie zisabonisa ukuphambuka kokujwayelekile, le modeli ilandela okunye ukusatshalaliswa, okuyi-Tweedie. Ngakho-ke, ukuthuthukiswa kwangempela kungabonwa kumethrikhi ye-MAE, eletha inani elicishe libe ngama-35%.

Ukuqhathanisa Amamodeli

cols=['ClaimNb','ClaimAmount', 'OLS_pred', 'OLS_Int_Term_pred', 'Tweedie_pred']
df[cols].sample(10).round()

Siyabona ukuthi izimangalo ezinguziro ziseduze noziro ku-Tweedie Regression, imiphumela engcono kakhulu kunezilinganiso ze-OLS, njengoba lezo zizoletha amanani eduze kwesilinganiso sawo wonke ClaimNb amanani. I-Tweedie regressor ingaqonda ukwehla kwamandla emali okuyiziro, nokho ngeke ibikezele uziro, kodwa inikeza inani elincane.

from sklearn.metrics import mean_absolute_error

# Calculate MAE for OLS
mae_ols = mean_absolute_error(df['ClaimAmount'], df['OLS_pred'])
print(f"MAE for OLS Model: {mae_ols:.2f}")

# Calculate MAE for OLS Interaction Terms
mae_ols = mean_absolute_error(df['ClaimAmount'], df['OLS_Int_Term_pred'])
print(f"MAE for OLS Model w/ Interaction Terms: {mae_ols:.2f}")

# Calculate MAE for Statsmodels Tweedie
mae_sm_tweedie = mean_absolute_error(df['ClaimAmount'], df['Tweedie_pred'])
print(f"MAE for Tweedie Model: {mae_sm_tweedie:.2f}")

# Calculate MAE for Zero Inflated
mae_zero_inflated = mean_absolute_error(df['ClaimAmount'], df['ZeroInflated_pred'])
print(f"MAE for Zero-Inflated Model: {mae_zero_inflated:.2f}")

MAE for OLS Model: 174.17
MAE for OLS Model w/ Interaction Terms: 172.24
MAE for Tweedie Model: 111.97

I-Decision Matrix: Ukukhetha Indlela Yakho

Uma ugqolozele kudathasethi yakho, uzama ukunquma ukuthi iyiphi indlela ozoyithatha, sebenzisa leli thebula ukuze uqondise isinqumo sakho:

Imodeli	Uhlobo Oluguquguqukayo oluqondiwe	Ukuqagela Okubalulekile / Amandla	Isetshenziswa Kakhulu E…
I-OLS evamile	Okuqhubekayo, okulinganayo, okumise okwensimbi	Ubudlelwano buqondile futhi buzimele.	Izilinganiso zomzimba ezinzile, ukubikezela okuyisisekelo.
OLS nge Ukusebenzisana	Iyaqhubeka, i-symmetric	Umphumela wesici A uyashintsha ngokusekelwe kusici B.	Isici sokumaketha, amanani ezindlu, ezomnotho.
I-Tweedie Regression	Itsheke kakhulu, alinayo inegethivu, ikakhulukazi iziro	Iphatha i-spike enkulu ku-zero elandelwa umsila omude.	Izimangalo zomshwalense, inani lokuphila kwekhasimende, izindleko zemicimbi eyivelakancane.

Imodeli Yebhonasi

Kodwa namanje angijabule ngalo modeli. Impela, imiphumela ayimihle kangako ekusebenzeni. Siyabona ukuthi imodeli ye-Tweedie, empeleni, inganciphisa kakhulu iphutha, kodwa isakwenza lokho ngokulawula izinombolo eziphakeme kakhulu neziphansi kakhulu (0) nokushayela yonke into eduze kwesilinganiso.

Okuwukuphela kwesixazululo esisele ukudala imodeli enezinyathelo ezimbili lapho sizomodela khona ukusatshalaliswa kweZero:

Qalisa isihlukanisi esizositshela ukuthi belikhona yini inani lesimangalo noma cha. Ibuyisela amathuba okuba nenani lesimangalo.
Ngomphumela wemodeli yokuqala, singasebenzisa ukusatshalaliswa kwe-Tweedie ukuze senze isibonelo kuphela lokho okuphawuliwe ngenani lesimangalo. Uma kungenjalo, imodeli izovele ibuyise uziro.

Masibhale lokho.

Siqala ngokulungiselela idathasethi yethu nge-One Hot Encoding ukuze sisuse izigaba.

import lightgbm as lgb
from sklearn.metrics import classification_report

# Remove previous predictions from df
cols = ['ClaimNb', 'Exposure', 'Area', 'VehPower', 'VehAge', 'DrivAge',
       'BonusMalus', 'VehBrand', 'VehGas', 'Density', 'Region', 'ClaimAmount']
df2 = df[cols].copy()

# 1. Create the binary outcome variable for having a claim
df2['has_claim'] = (df2['ClaimAmount'] > 0).astype(int)

# Prepare the data for "zero claim" classification
df_ohe = ohe.fit_transform(df2)
X = df_ohe.drop(columns=['ClaimAmount', 'has_claim'])
y = df_ohe['has_claim']

Okulandelayo, kunenqwaba yamakhodi.

Sizohlukana phakathi isitimela futhi ukuqinisekiswa isethi ye-LightGBM.
Ibala izisindo ukuze isingathe ukungalingani kwekilasi.
Isetha amapharamitha ashuniwe wemodeli ye-LightGBM.
Iqeqesha imodeli ye-LightGBM ngokuma kusenesikhathi.

from sklearn.model_selection import train_test_split

print("Tuning LightGBM for P(Claim > 0) with updated parameters...")

# Split the training data into training and validation sets for early stopping
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

train_data_split = lgb.Dataset(X_train, label=y_train)
val_data_split = lgb.Dataset(X_val, label=y_val)

# Calculate scale_pos_weight for better handling of class imbalance
scale_pos_weight_value = (y_train == 0).sum() / (y_train == 1).sum()

# Define updated parameters for better control and to mitigate the warning
params_tuned = {
    'objective': 'binary',
    'metric': 'auc',
    'learning_rate': 0.05,  # Reduced learning rate
    'num_leaves': 31,       # Increased num_leaves (default is 31), allowing for more complexity
    'max_depth': 7,         # Increased tree depth, allowing for more complexity
    'min_child_samples': 15, # Slightly decreased minimum data in leaf
    'min_gain_to_split': 0.01, # Decreased minimum gain for a split
    'force_col_wise': True, # Keep for dataset compatibility
    'scale_pos_weight': scale_pos_weight_value, # Explicitly handle imbalance
    'verbose': -1           # Suppress verbose output during training
}

# Re-train LightGBM model with tuned parameters
# Increase num_boost_round given the reduced learning rate and increased complexity
model_tuned = lgb.train(
    params_tuned,
    train_data_split,
    num_boost_round=300, # Increased number of boosting rounds
    valid_sets=[val_data_split], # Provide validation set for early stopping
    callbacks=[lgb.early_stopping(20, verbose=False)] # Add early stopping and suppress verbose, increased patience
)

Ibikezela amathuba okwenzeka kwesimangalo. Iguqula ibe kanambambili 0 noma 1 ukuze kubalwe umbiko wesigaba.
Ihlola imodeli isebenzisa umbiko wokuhlukanisa.
Izitolo zibikezele amathuba ku-dataframe.
Imisa amathuba angaphansi kuka-0.5 ukuya kuziro.

# Predictions and evaluation with the tuned model
y_pred = (model_tuned.predict(X) > 0.5).astype(int)
print("nClassification Report for Tuned LightGBM Model:")
print(classification_report(y, y_pred))

# Store the probabilities for the next step of the Zero-Inflated model
df2['prob_has_claim'] = model_tuned.predict(X)

# If prob_has_claim is under 0.5, make it 0
df2.loc[df2['prob_has_claim'] < 0.5, 'prob_has_claim'] = 0

Manje, ingxenye yesi-2 yemodeli ihlunga kuphela lezo zimangalo ngenani ledola elihlotshaniswayo futhi sebenzisa i-Tweedie Regression ukuze ilinganisele.

# Part 2: Model the severity (ClaimAmount > 0) using Tweedie Regression
# Filter data for positive claims only
df_positive = df_ohe[df_ohe['has_claim'] == 1].copy()
X_positive = df_ohe.drop(columns=['ClaimAmount', 'has_claim'], axis=1)
y_positive = df_ohe['ClaimAmount']

# Use Exposure as offset for the severity model
print("nFitting Tweedie Model E[ClaimAmount | Claim > 0]...")
sklearn_tweedie_model = TweedieRegressor(
    power=1.85,
    link='auto',
    solver='newton-cholesky',
    max_iter=500,
    alpha=1
)
sklearn_tweedie_model.fit(X_positive, y_positive, sample_weight=df['Exposure'])

Ekugcineni, ukubikezela kwale modeli kuzoba yi Amathuba alinganiswa imodeli yokuqala iphindaphindwe ngu Tweedie Regressor inani elilinganiselwe.

# Predict conditional severity for all observations (even those with 0 claims)
# This assumes the severity model learned from positive claims can generalize.
df['pred_severity_if_claim'] = sklearn_tweedie_model.predict(df_ohe.drop(columns=['ClaimAmount', 'has_claim'], axis=1))

# Calculate the final Zero-Inflated Model prediction
df['ZeroInflated_pred'] = df2['prob_has_claim'] * df['pred_severity_if_claim']

# Display head of predictions and MAE
print("nZero-Inflated Model Predictions (first 5 rows):")
print(df[['ClaimAmount', 'ZeroInflated_pred']].sample(5))

mae_zero_inflated = mean_absolute_error(df['ClaimAmount'], df['ZeroInflated_pred'])
print(f"nMAE for Zero-Inflated Model: {mae_zero_inflated:.2f}")

Lona umphumela.

Fitting Tweedie Model E[ClaimAmount | Claim > 0]...

Zero-Inflated Model Predictions (first 5 rows):
         ClaimAmount  ZeroInflated_pred
IDpol                                  
1179635          0.0           0.000000
108738           0.0           0.000000
2066923          0.0           0.000000
1007186          0.0           0.000000
1117109       1128.0         729.428288

MAE for Zero-Inflated Model: 87.79

Kuhle, sithole omunye u-21% wokuthuthuka ku-MAE. Lokhu kuyamangalisa!

Imiphumela yemodeli Eshisiwe Ayiziro Okungempela x Izibikezelo. Isithombe sombhali.

Isiphetho

Isayensi yedatha ayivamisile ukuthola i-algorithm eyodwa “ephelele” exazulula zonke izinkinga. Kunalokho, imayelana nokufanisa ukuqagela kwezibalo kwemodeli yakho nokuziphatha komhlaba wangempela kwedatha yakho.

Qala elula. Bheka ukusatshalaliswa kokuguquguquka okuhlosiwe kwakho. Uma kubukeka njengejika lensimbi, qala ngalo I-OLS evamile. Uma wazi ukuthi izici zakho ziyathonya enye kwenye, yazisa Imigomo yokusebenzisana. Kepha uma ugqolozele intaba yoziro nomsila omude wezinombolo ezinkulu, yeqa amakhanda e-OLS ngokuphelele futhi usebenzise amandla I-Tweedie Regression.

Ukuthatha amaminithi ambalwa ukuhlaziya ukusatshalaliswa kwalokho okuqondiwe ngaphambi kokumodela kungakongela izinsuku zokulungisa izakhiwo ezingalungile kamuva.

I-GitHub Repository

Izithenjwa

[1. Dataset] (

[2. Data License and citation] (

UChristophe Dutang no-Arthur Charpentier (2026). Amasethi wedatha we-CAS: Amasethi wedatha womshwalenseinguqulo yephakheji ye-R 1.2-1, DOI 10.57745/P0KHAG.

[3. Tweedie Regression Documentation] (

[4. Tweedie Regression Tutorial] (

Source link

nimda 2 hours ago

0 0 10 minutes read