Izingibe Zokuhlola I-A/B: Yini Esebenzayo Nengasebenzi Ngedatha Yangempela

# Isingeniso

Uthumele lokho okubukeka njengokuhlola okuwinile: ukuguqulwa kwenyuke ngo-8%, amamethrikhi okuzibandakanya acwebezela ngokuluhlaza. Bese iphahlazeka ekukhiqizweni noma ihluleke buthule ngemva kwenyanga.

Uma lokho kuzwakala kujwayelekile, awuwedwa. Ukwehluleka okuningi kokuhlolwa kwe-A/B akuveli emibonweni emibi yomkhiqizo; bavela emikhubeni emibi yokuhlola.

Idatha ikudukisile, umthetho wokumisa awuzange uzitshwe, noma akekho ohlolile ukuthi “win” bekuwumsindo nje ogqokiswe njengesignali. Nali iqiniso elingathandeki: ingqalasizinda ezungeze ukuhlolwa kwakho ibaluleke kakhulu kunokwehluka ngokwako, futhi amaqembu amaningi akuthola kungalungile.

Ake sihlukanise ababulali abane abangathuli bokuhlolwa kwe-A/B – kusukela kudatha edukisayo kuye ekucabangeni okunephutha – futhi siveze imikhuba eqondiswa izigwegwe ehlukanisa okuhle kakhulu kokunye.

Isithombe nguMbhali

# Lapho Idatha Iqamba Amanga: I-SRM kanye Nokwehluleka Kwekhwalithi Yedatha

I-Pitfall: Iningi lemiphumela yokuhlolwa “emangazayo” akuyona imininingwane; zingamaphutha ekhwalithi yedatha egqoke isembozo.

Ukungafani Kwesampula Yesilinganiso (I-SRM) i-canary emayini yamalahle. Ulindele ukuhlukaniswa okungu-50/50, uthole u-52/48. Kuzwakala kungenangozi. Akunjalo. Izimpawu ze-SRM eziphukile, umzila wethrafikhi ochemile, noma ukwehluleka kokungena okonakalisa imiphumela yakho buthule.

Indaba yomhlaba wangempela: Microsoft ithole ukuthi i-SRM ikhombisa izinkinga zekhwalithi yedatha eqinile eyenza imiphumela yokuhlolwa ingavumelekile, okusho ukuthi ukuhlola nge-SRM kuvame ukuholela ezinqumweni zomkhumbi ezingalungile.
I-DoorDash ithole i-SRM ngemuva kokuthi abasebenzisi bezinhloso eziphansi baphume ngokungafani eqenjini elilodwa kulandela ukulungiswa kweziphazamisi, imiphumela yokuskena nokudala ukuwina kwe-phantom.

Yini okufanele uyihlole uma une-SRM:

Isithombe nguMbhali

• Ukuhlolwa kwe-Chi-squared yokuhlukaniswa kwethrafikhi: zenzele lokhu ngaphambi kwanoma yikuphi ukuhlaziya.
• Izinga lomsebenzisi uma liqhathaniswa nokuloga kweleveli yeseshini: ubumbudumbudu obungafani budala imiphumela ye-phantom.
• Izimbungulu zamabhakede ezisekelwe esikhathini: Abasebenzisi bangoMsombuluko bayalawula, abasebenzisi bangoLwesihlanu ekwelashweni = imiphumela edidekile.

Isixazululo: Ukulungisa akukona ubuhlakani bezibalo. Ukuhlanzeka kwedatha. Yenza ukuhlola kwe-SRM ngaphambi kokubheka amamethrikhi. Uma ukuhlolwa kwehluleka ukuhlola isilinganiso, yima. Phenya. Lungisa i-randomization. Akukho okuhlukile.

Ingabe ufuna ukuzijwayeza ukubona izinkinga zekhwalithi yedatha njenge-SRM noma ukugawula okungafani? Zama izinselele ezimbalwa zangempela zokuhlanza idatha ye-SQL kanye nokutholwa okungaqondakali I-StrataScratch. Uzothola amasethi edatha ezinkampanini zangempela ukuze uhlole ukulungisa iphutha lakho kanye namakhono okuqinisekisa idatha.

Amaqembu amaningi aseqa lesi sinyathelo. Kungakho izivivinyo eziningi “eziyimpumelelo” zehluleka ekukhiqizeni.

# Misa Ukulunguza: Ukuthi Kubukeka Kusengakanani Konakalisa Ukufaneleka

I-Pitfall: Ukuhlola imiphumela yakho yokuhlolwa njalo ekuseni kuzwakala kuphumelela. Akunjalo. Kwenyusa ngokuhlelekile izinga lakho lokuphozithivu okungelona iqiniso.

Nasi isizathu: ngaso sonke isikhathi uma ubheka amanani we-p futhi unquma ukuthi uyeke, unikeza ukungahleleki elinye ithuba lokukukhohlisa. Qalisa ukubuka okungu-20 kumphumela oyize, futhi ekugcineni uzobona u-p <0.05 ngenhlanhla emsulwa. NgokufanelekileUcwaningo luthole ukuthi ukulunguza okungalungisiwe kungakhuphula amaphuzu angamanga ukusuka ku-5% kuye ngaphezulu kuka-25%, okusho ukuthi “ukuwina” okukodwa kokune kuwumsindo.

Indlela yokubona indlela engenangqondo:

• Yenza ukuhlolwa amasonto amabili.
• Hlola nsuku zonke.
• Yima lapho p <0.05.
• Umphumela: Usebenzise iziqhathaniso eziningi eziyi-14 ngaphandle kokulungiswa.

Isixazululo: Sebenzisa ukuhlola okulandelanayo noma izindlela zokukhomba ezisebenza njalo ezilungisela ukubukeka okuningi.

Indaba yomhlaba wangempela:

• SpotifyIndlela yeqembu: Ukuhlola okulandelanayo kweqembu (GST) okunomsebenzi wokusebenzisa imali we-alpha ngokufanelekile kulandisa ukubukeka okuningi ngokusebenzisa ukwakheka kokuhlobana phakathi kokuhlolwa kwesikhashana.
• Isixazululo se-Optimizely: Amanani we-p ahlala evumelekile abangela ukuqapha okuqhubekayo, okuvumela ukubuka okuphephile ngaphandle kokwenyusa amazinga amaphutha.
• I-NetflixIndlela 's: Ukuhlola okulandelanayo okunokulandelana kokuzethemba okuvumelekile noma nini kuyashintsha kusuka kumkhathizwe kuya ekuhloleni okuqhubekayo kuyilapho kugcinwa iziqinisekiso zephutha lohlobo I.

Uma kufanele ulunguze, sebenzisa amathuluzi akhelwe wona. Ungayihlanganisi nokuhlolwa kwe-t.

Okubalulekile: Chaza kusengaphambili umthetho wakho wokumisa ngaphambi kokuthi uqale. “Yeka lapho kubukeka kukuhle” akuwona umthetho; liyiresiphi yegolide lesiwula.

# Amandla Asebenzayo: CUPED kanye Nokwehliswa Kwezinhlobonhlobo Zanamuhla

Ugibe: Ukwenza izivivinyo ezinde akuyona impendulo. Ukwenza izivivinyo ezihlakaniphile.

Isixazululo: CUPED (Ukuhlola okulawulwayo Ngokusebenzisa Idatha Yokuhlola Ngaphambilini) kuyisixazululo se-Microsoft kumamethrikhi anomsindo. Umqondo uhilela ukusebenzisa ukuziphatha kokuhlolwa kwangaphambili ukuze ubikezele imiphumela yangemva kokuhlolwa, bese ukala umehluko oyinsalela kuphela. Ngokususa ukuhlukahluka okungabikezelwa, unciphisa izikhathi zokuzethemba ngaphandle kokuqoqa idatha eyengeziwe.

Isibonelo somhlaba wangempela: I-Microsoft ibike ukuthi ethimbeni elilodwa lomkhiqizo, i-CUPED ifana nokungeza u-20% wethrafikhi ngaphezulu ekuhlolweni. I-Netflix ithole ukuncishiswa kokuhluka okucishe kube ngu-40% kumamethrikhi abalulekile wokuzibandakanya. Statsig ibone ukuthi i-CUPED yehlise ukuhluka ngo-50% noma ngaphezulu kumamethrikhi amaningi avamile, okusho ukuthi ukuhlola kufinyelele ukubaluleka engxenyeni yesikhathi, noma ngohhafu wethrafikhi.

Isebenza kanjani:

Adjusted_metric = Raw_metric - θ × (Pre_period_metric - Mean_pre_period)

Ukuhumusha: Uma umsebenzisi echitha u-$100/iviki ngaphambi kokuhlolwa, futhi iqoqo lakho lokuhlola liba nesilinganiso esingu-$90/ngeviki sokuhlola kwangaphambili, i-CUPED ilungisela phansi abasebenzisi asebevele besebenzisa imali enkulu. Ukala umphumela wokwelapha, hhayi ukwehluka okukhona kakade.

Isetshenziswa nini i-CUPED?

Isithombe nguMbhali

Nini akufanele usebenzise i-CUPED?

Isithombe nguMbhali

Izindlela ezintsha ezifana I-CUPAC (ukuhlanganisa ama-covariates kuwo wonke ama-metrics) kanye namasampula ane-stratified kuqhuba lokhu ngokuqhubekayo, kodwa umgomo uhlala unjalo: nciphisa umsindo ngaphambi kokuhlaziya, hhayi ngemva kwalokho.

Inothi lokusetshenziswa: Izinkundla eziningi zokuhlola zesimanje (Ngokufanelekile, Eppo, GrowthBook) isekela i-CUPED ngaphandle kwebhokisi. Uma wenza okwakho, engeza ama-covariate enkathi yangaphambili epayipini lakho lokuhlaziya; ukuphakama kwezibalo kuwufanele umzamo wobunjiniyela.

# Ukulinganisa Okubalulekile: Ama-Guardrails kanye Nokuhlola Kwangempela Kwesikhathi Eside

Ugibe: Ukulungiselela imethrikhi engalungile kubi kunokungahloli nhlobo.

I-trap yakudala: Uhlola isici esithuthukisa ukuchofoza ngo-12%. Ithumele. Ezinyangeni ezintathu kamuva, ukugcinwa kwehle ngo-8%. Kwenzenjani? Uthuthukise i-metric eyize ngaphandle kokuvikela ekulimaleni okwehla nomfula.

Isixazululo: Amamethrikhi e-Guardrail ayinethi yakho yokuphepha. Amamethrikhi ongawalungiseleli, kodwa uyagada ukuze uthole imiphumela engahlosiwe:

Isithombe nguMbhali

Isibonelo somhlaba wangempela: I-Airbnb bathole ukuthi ukuhlolwa okwandisa ukubhuka kuphinde kwehlise izilinganiso zokubuyekeza; ushintsho luhehe ukubhuka okwengeziwe kodwa lwalimaza ukwaneliseka kwesikhathi eside. I-Guardrail metrics ibambe inkinga ngaphambi kokukhishwa okugcwele. Ezinkulungwaneni zokuhlolwa kwanyanga zonke, abaqaphi be-Airbnb bahlaba umkhosi ngokuhlolwa okungaba ngu-25 ukuze kubuyekezwe ababambiqhaza, kuvinjwe imithelela engaba mikhulu engaba khona emikhulu emihlanu inyanga ngayinye.

Indlela yokuhlela ama-guardrails:

Isithombe nguMbhali

Inkinga entsha: Ukuhlolwa kwesikhashana kuthwebula imiphumela emisha, hhayi umthelela oqhubekayo. Abasebenzisi bachofoza izinkinobho ezintsha ngoba zisha, hhayi ngoba zingcono. Izinkampani zisebenzisa amaqembu e-holdout ukukala ukuthi ingabe imiphumela iyaqhubeka emavikini noma izinyanga ngemva kokwethulwa, ngokuvamile kugcina u-5–10% wabasebenzisi benolwazi loshintsho lwangaphambili kuyilapho kuqashwe amamethrikhi esikhathi eside.

Ukwenza okuhle kakhulu: Konke ukuhlolwa kudinga ukuqinisekiswa ngale kokuhlolwa kokuqala:

• Isigaba 1: Ukuhlolwa okujwayelekile kwe-A/B (amaviki angu-1–4) ukuze kukale umthelela osheshayo.
• Isigaba 2: Ukuqapha isikhathi eside namaqembu e-holdout noma ukulandelela okunwetshiwe ukuze uqinisekise ukuphikelela.

Uma umphumela unyamalala esigabeni sesi-2, bekungekona ukuwina kwangempela: bekuwukufuna ukwazi.

# Okwenziwa Abahloli Abaphezulu Ngokuhlukile

Igebe phakathi kwamaqembu amahle namakhulu okuhlola alikona ukuba yinkimbinkimbi kwezibalo; yisiyalo sokusebenza.

Nakhu okwenziwa yizinkampani ezifana ne-Booking.com, Netflix, ne-Microsoft ezingakwenzi abanye:

Isithombe nguMbhali

// Ukuhlola okuzenzakalelayo kwe-SRM

Ukuzijwayeza komkhakha: Izinkundla zokuhlola zesimanje njengo-Optimizely futhi Statsig sebenzisa ngokuzenzakalelayo ukuhlola kwe-SRM kukho konke ukuhlola. Uma isheke lehluleka, ideshibhodi ikhombisa isexwayiso. Ayikho inketho yokukhipha. Cha “sizophenya ngokuhamba kwesikhathi.” Yilungise noma ungathumeli.

Booking.comIsiko lokuhlola lidinga ukuthi izindaba zekhwalithi yedatha zibanjwe ngaphambi kokuthi imiphumela ihlaziywe, iphatha amasheke e-SRM njengemigqa yokuqapha okungaxoxiswana ngayo, hhayi ukuxilonga ongakukhetha.

// Amamethrikhi okubhalisa ngaphambilini

Ukwenza okuhle kakhulu: Chaza amamethrikhi ayisisekelo, esibili, nawe- guardrail ngaphambi kokuthi kuqale ukuhlolwa. Azikho izimayini ze-post-hoc metric. Cha “ake sihlole ukuthi nayo ihambise imali engenayo.” Uma ungazange uhlele ukukukala, awutholi ukukumangalela njengokuwinile.

Indlela ye-Netflix: Ukuhlola kufaka phakathi amamethrikhi ayisisekelo achazwe ngaphambilini kanye namamethrikhi e-Guardrail (njengamazinga okuxhumana amakhasimende) ukuze abambe imiphumela engemihle engahlosiwe.

// Ukwenza ama-Postmortems kukho konke ukwethulwa

I-Microsoft's ExP platform practice: Wina noma ulahlekelwe, konke ukuhlolwa okuthunyelwayo kuthola i-postmortem:

• Ingabe umphumela ufana nokubikezela?
• Ingabe i-Guardrails ibambe?
• Yini esingayenza ngokuhlukile?

Lokhu akuwona umsebenzi we-bureaucracy; yingqalasizinda yokufunda.

// Ukuhlola Esikalini

Imiphumela ye-Booking.com: Isebenzisa izivivinyo ezingu-1,000+ ngesikhathi esisodwa, bafunde ukuthi ukuhlola okuningi (90%) kuyafeyila, kodwa iphuzu yilokho. Ivolumu yokuhlola ayikona mayelana nokuwina; imayelana nokufunda ngokushesha ukwedlula izimbangi.

Amaqembu akalinganiswa ngenani lokuwina, kodwa nge:

• Isivinini sokuhlola (izivivinyo ngekota ngayinye).
• Ikhwalithi yedatha (ukugcina amanani e-SRM ephansi).
• Landela (i-% yokuwina okuvumelekile okuthunyelwayo).

Lokhu kudikibalisa uhlelo lokudlala futhi kuvuza ukubulawa kanzima.

// Ukwakha I-Centralised Experimentation Platform

Amaqembu amahle awavumeli onjiniyela bazenzele ezabo izivivinyo ze-A/B. Bakha (noma bathenge) inkundla ethi:

• Iphoqelela ukulunga kwe-randomization.
• Ibala ngokuzenzakalelayo osayizi besampula.
• Isebenzisa i-SRM futhi amandla ahlola ngokuzenzakalelayo.
• Ifaka zonke izinqumo ukuze zihlolwe.

Kungani lokhu kubalulekile: Impumelelo ekuhloleni ayikho mayelana nokwenza izivivinyo ezengeziwe. Imayelana nokwenza izivivinyo ezithembekile. Amaqembu awinile yiwo azenzela ukuqina.

# Isiphetho

Iqiniso elinzima kakhulu ekuhlolweni kwe-A/B alinazo izibalo; kungokwesiko. Ungaba umpetha wokuhlola okulandelanayo, usebenzise i-CUPED, futhi uchaze ama-guardrails aphelele, kodwa akukho lutho olubalulekile uma ithimba lakho lihlola imiphumela kusenesikhathi kakhulu, lingaziba izixwayiso ze-SRM, noma umkhumbi uwina ngaphandle kokuqinisekiswa.

Umehluko phakathi kwamaqembu alinganisa ukuhlolwa kanye namaqembu acwila ezintweni ezingamanga akubona ososayensi bedatha abahlakaniphe kakhulu; kuwukuqina okuzenzakalelayo, isiyalo esiphoqelelwe, kanye nesivumelwano esabiwe sokuthi “sibukeke sibalulekile” asilungile ngokwanele.

Ngokuzayo lapho ulingeka ukuthi uhlole uhlolo noma weqe ukuhlola kwe-SRM, khumbula: iphutha elibiza kakhulu ekuhloleni ukuziqinisekisa ukuthi idatha ihlanzekile uma injalo.

Nate Rosidi ungusosayensi wedatha nakusu lomkhiqizo. Uphinde abe nguprofesa osizayo ofundisa izibalo, futhi ungumsunguli we-StrataScratch, inkundla esiza ososayensi bedatha ukulungiselela izingxoxo zabo ngemibuzo yenhlolokhono yangempela evela ezinkampanini eziphezulu. U-Nate ubhala ngamathrendi akamuva emakethe yemisebenzi, unikeza izeluleko zenhlolokhono, wabelane ngamaphrojekthi wesayensi yedatha, futhi uhlanganisa yonke into ye-SQL.