Ucwaningo lwe-Google DeepMind Luvumela i-LLM Ibhale Kabusha I-Theory Yayo Yomdlalo Ama-algorithms – Futhi Yenze Ngomsebenzi Kahle Ochwepheshe

Ukudizayina ama-algorithms we-Multi-Agent Reinforcement Learning (MARL) kumageyimu wolwazi olungaphelele — izimo lapho abadlali benza khona ngokulandelana futhi bengakwazi ukubona imininingwane eyimfihlo yomunye nomunye, njenge-poker — ngokomlando bethembele ekuphindaphindweni mathupha. Abacwaningi bahlonza izikimu zesisindo, imithetho yokwehlisa, nezixazululi zokulingana ngokusebenzisa intuition kanye nesivivinyo-namaphutha. Abacwaningi be-Google DeepMind bahlongoza i-AlphaEvolve, i-ejenti ye-LLM ye-evolutionary coding ethatha indawo yaleyo nqubo yezandla ngokusesha okuzenzakalelayo.

Ithimba labacwaningi lisebenzisa lolu hlaka kuma-paradigms amabili amisiwe: I-Counterfactual Regret Minimization (CFR) kanye ne-Policy Space Response Oracles (PSRO). Kuzo zombili izimo, isistimu ithola okuhlukile okusha kwe-algorithm esebenza ngokuncintisana ngokumelene noma okungcono kunezisekelo ezikhona eziklanywe ngesandla zesimanjemanje. Konke ukuhlola kwenziwe kusetshenziswa uhlaka lwe-OpenSpiel.

Isendlalelo: CFR NE-PSRO

I-CFR iyi-algorithm ephindaphindayo ebolisa ukunciphisa ukuzisola kuwo wonke amasethi olwazi. Ekuphindaphindweni ngakunye kuqongelela 'ukuzisola okungekhona iqiniso' – ukuthi umdlali ubezozuza malini ngokudlala ngendlela ehlukile – futhi athole inqubomgomo entsha ngokulingana nokuzisola okunqwabelene okuhle. Ngokuphindaphinda okuningi, isu lesilinganiso sesikhathi lishintshela ku-Nash Equilibrium (NE). Izinhlobonhlobo ezifana ne-DCFR (i-CFR Eyehlisiwe) ne-PCFR+ (Predictive CFR+) zithuthukisa ukuhlangana ngokusebenzisa isaphulelo esithile noma imithetho yokubuyekeza ebikezelwayo, konke okuthuthukiswe ngokuklama okwenziwa mathupha.

I-PSRO isebenza ngezinga eliphezulu lokuzikhipha. Igcina isibalo sezinqubomgomo zomdlali ngamunye, yakha i-payoff tensor (i-meta-game) ngokwenza ikhompuyutha izinsiza ezilindelekile kuyo yonke inhlanganisela yezinqubomgomo zabantu, bese isebenzisa isixazululi se-meta-su ukukhiqiza ukusabalalisa kwamathuba phezu kwesibalo sabantu. Izimpendulo ezingcono kakhulu ziyaqeqeshwa ngokumelene nalokho kusatshalaliswa futhi zengezwe kubantu ngokuphindaphindiwe. Isixazululi se-meta-strategy – indlela ukusatshalaliswa kwabantu okubalwa ngayo – ukukhetha kwedizayini emaphakathi iphepha eliqondiswe ekutholweni okuzenzakalelayo. Konke ukuhlolwa kusebenzisa i-oracle yokusabela engcono kakhulu (ebalwa ngenani eliphindaphindwayo) kanye namanani enkokhelo aqondile akho konke okufakiwe kwe-meta-game, kususa umsindo wesampula ye-Monte Carlo emiphumeleni.

UHLAKA lwe-AlphaEvolve

I-AlphaEvolve iwuhlelo lokuziphendukela kwemvelo olusabalalisiwe olusebenzisa ama-LLM ukuze iguqule ikhodi yomthombo kunemingcele yezinombolo. Inqubo: inani labantu liqaliswa ngokusetshenziswa okujwayelekile (i-CFR+ njengembewu yokuhlolwa kwe-CFR; Okufanayo njengembewu yazo zombili izigaba zesixazululi se-PSRO). Esizukulwaneni ngasinye, i-algorithm yomzali ikhethwa ngokusekelwe ekuqineni; ikhodi yayo yomthombo idluliselwa ku-LLM (Gemini 2.5 Pro) ngokushesha ukuze ilungiswe; umuntu ozongenela ukhetho uyahlolwa kumidlalo ye-proxy; amakhandidethi avumelekile anezelwa kubantu. I-AlphaEvolve isekela ukuthuthukiswa kwezinhloso eziningi – uma kuchazwa amamethrikhi amaningi okufaneleka, eyodwa ikhethwa ngokungahleliwe ngesizukulwane ngasinye ukuze iqondise amasampula omzali.

Isignali yokufaneleka iwukuxhashazwa okungekuhle ngemva kokuphindaphinda kuka-K, okulinganiselwe kusethi egxilile yemidlalo yokuqeqesha: abadlali abangu-3 u-Kuhn Poker, abadlali abangu-2 i-Leduc Poker, i-Goofspiel yamakhadi angu-4, kanye neDayisi Lamanga elinamacala angu-5. Ukuhlola kokugcina kwenziwa kusethi yokuhlola ehlukile yemidlalo emikhulu, engabonakali.

Ku-CFR, indawo yokusesha eguqukayo iqukethe amakilasi amathathu ePython: RegretAccumulator, PolicyFromRegretAccumulator, kanye nePolicyAccumulator. Lokhu kubusa ukuqoqwa kokuzisola, ukutholwa kwenqubomgomo yamanje, kanye nokuqoqwa kwenqubomgomo okumaphakathi ngokulandelanayo. I-interface icacile ngokwanele ukumela zonke izinhlobo ze-CFR ezaziwayo njengamakesi akhethekile. Ku-PSRO, izingxenye eziguqulekayo yi-TrainMetaStrategySolverand EvalMetaStrategySolver— izixazululi ze-meta-strategy ezisetshenziswa phakathi nokuqeqeshwa kwezwi kanye naphakathi nokuhlola ukusebenziseka.

Kutholwe i-algorithm 1: I-VAD-CFR

Okuhlukile kwe-CFR okuguquliwe yi-Volatility-Adaptive Discounted CFR (VAD-CFR). Esikhundleni sesephulelo esilinganiselwe nesimile esisetshenziswa emndenini we-CFR, ukusesha kukhiqize izindlela ezintathu ezihlukene:

1. Isaphulelo esishintshashintshayo. Esikhundleni sezinto zesaphulelo ezigxilile u-α no-β ezisetshenziswa ekuzisoleni okwandayo (njengaku-DCFR), i-VAD-CFR ilandelela ukuntengantenga kwenqubo yokufunda isebenzisa i-Exponential Weighted Moving Average (EWMA) yobukhulu bokuzisola obusheshayo. Uma ukuguquguquka kuphezulu, isaphulelo siyakhuphuka ukuze i-algorithm ikhohlwe umlando ongazinzile ngokushesha; lapho ukuntengantenga kwehla kugcina umlando owengeziwe. I-EWMA decay factor ingu-0.1, enesisekelo esingu-α = 1.5 nesisekelo esingu-β = −0.1.
2. I-asymmetric ithuthukisa ngokushesha. Ukuzisola okuphazima kweso kuphindaphindwa into engu-1.1 ngaphambi kokuthi kwengezwe ekuzisoleni okwandayo. Le asymmetry isetshenziswa ekubuyekezeni okusheshayo, hhayi umlando oqoqiwe, okwenza i-algorithm isebenze kakhulu ezenzweni ezinhle zamanje.
3. Isiqalo esifudumele esinesisindo sokuzisola-ubukhulu. Ukulinganiswa kwenqubomgomo kuhlehliswa ngokuphelele kuze kuphinde kuphindwe 500. Inqubo yokuqoqwa kokuzisola iqhubeka ngokujwayelekile phakathi nalesi sigaba. Uma ukunqwabelana sekuqalile, izinqubomgomo zilinganiswa ngenhlanganisela yesisindo sesikhashana kanye nobukhulu bokuzisola obusheshayo — ukubeka phambili ukuphindwaphindwa kolwazi oluphezulu lapho kwakhiwa isu elimaphakathi. I-threshold ye-500-iteration yenziwe yi-LLM ngaphandle kolwazi lomkhathizwe wokuhlola we-1000-iteration.

I-VAD-CFR imakwe uma iqhathaniswa ne-CFR evamile, i-CFR+, i-Linear CFR (LCFR), i-DCFR, i-PCFR+, i-DPCFR+, ne-HS-PCFR+(30) phakathi kokuphindwayo okungu-1000 nge-K = 1000. Ukusebenziseka kubalwa ngokunembile. Ekuhlolweni okugcwele kwemidlalo engu-11, i-VAD-CFR ifana noma idlula ukusebenza kwesimanje Imidlalo engu-10 kweyi-11ngabadlali abangu-4 u-Kuhn Poker njengoyedwa ohlukile.

Kutholwe i-Algorithm 2: SHOR-PSRO

Okuhlukile kwe-PSRO okuthuthukisiwe yi-Smoothed Hybrid Optimistic Regret PSRO (SHOR-PSRO). Ukusesha kukhiqize i-hybrid meta-solver eyakha isu lemeta ngokuthi ukuhlanganisa ngokulinganayo izingxenye ezimbili kukho konke ukuphindaphinda kwangaphakathi kwesixazululi:

• σ_ORM (Ukufanisa Ukuzisola Okunethemba): Inikeza ukuzinza kokunciphisa ukuzisola. Izinzuzo zibalwa ngekhompyutha, zikhethwa ngokujwayelekile futhi zilungiswe ngokwehlukahlukana, bese zisetshenziselwa ukubuyekeza ukuzisola okunqwabelene ngokufanisa ukuzisola. Igama lomfutho lisetshenziswa ezinzuzweni eziyinkokhelo.
• σ_Softmax (Isu Elihlanzekile Elingcono Kakhulu): Ukusatshalaliswa kwe-Boltzmann ngamasu amsulwa achemile ezindleleni zokukhokha okuphezulu. Ipharamitha yezinga lokushisa ilawula ukugxiliswa kwezinga lokushisa – izinga lokushisa eliphansi lisho ukuthi ukusabalalisa kugxile kakhulu esu elihlanzekile elingcono kakhulu.

Isixazululi sesikhathi sokuqeqesha sisebenzisa ishejuli yokuhlanganisa eguquguqukayo ekuphindaphindweni kwangaphandle kwe-PSRO. Isici esihlanganisayo esingu-λ sisuka ku-0.3 → 0.05 (ukusuka ekuxhashazweni okuhahayo kuye ekutholeni ukulingana), ukubola kwebhonasi kokuhlukahluka kusuka ku-0.05 → 0.001 (ivumela ukuhlolwa kwesibalo sangaphambi kwesikhathi bese kuthuthukiswa kwesigaba sekwephuzile), kanye nezinga lokushisa le-softmax lehla lisuka ku-0.5 → 0.5 Inani lokuphindaphinda kwesixazululi sangaphakathi liphinda likala ngosayizi wabantu. Isixazululi sokuqeqeshwa sibuyisela isu lesilinganiso sesikhathi kukho konke ukuphindaphinda kwangaphakathi ukuze kube nokuzinza.

Isixazululi sesikhathi sokuhlola sisebenzisa amapharamitha angashintshi: λ = 0.01, ibhonasi yokuhlukahluka = ​​0.0, izinga lokushisa = 0.001. Isebenzisa ukuphindaphinda kwangaphakathi okwengeziwe (isisekelo esingu-8000, ukukala ngosayizi wesibalo sabantu) futhi ibuyisela isu lokugcina eliphindaphindwayo kunesilinganiso, ukuze kube nesilinganiso esisebenzayo, somsindo ophansi. Le-asymmetry yokuqeqesha/yokuhlola yona ngokwayo iwumkhiqizo wokusesha, hhayi umklamo womuntu.

I-SHOR-PSRO imakwe iqhathaniswa ne-Uniform, i-Nash (ngohlelo oluwumugqa lwemidlalo yabadlali ababili), i-AlphaRank, i-Projected Replicator Dynamics (PRD), kanye ne-Regret Matching (RM), kusetshenziswa iziphindaphindo ze-K = 100 ze-PSRO. Ekuhlolweni okugcwele kwemidlalo engu-11, i-SHOR-PSRO ifana noma idlula ukusebenza kwesimanje 8 kweyi-11 imidlalo.

Ukusethwa Kokulinga

Iphrothokholi yokuhlola ihlukanisa ukuqeqeshwa nemidlalo yokuhlola ukuze kuhlolwe ukwenziwa okuvamile. Ukuqeqeshwa okusethiwe kukho kokubili ukuhlolwa kwe-CFR kanye ne-PSRO kuhlanganisa abadlali abangu-3 u-Kuhn Poker, abadlali ababili i-Leduc Poker, i-Goofspiel yamakhadi angu-4, kanye ne-Liars Dice enamacala angu-5. Isethi yokuhlola esetshenziswa engxenyeni eyinhloko yephepha iqukethe abadlali abangu-4 u-Kuhn Poker, i-Leduc Poker yabadlali abangu-3, ​​i-Goofspiel yamakhadi angu-5, kanye neDayisi Lamanga elinazinhlangothi ezingu-6 – okuhlukile okukhulu nokuyinkimbinkimbi kakhulu okungabonakali ngesikhathi sokuvela. Ukushanela okuphelele kuyo yonke imidlalo engu-11 kufakiwe kusithasiselo. Ama-algorithms ayalungiswa ngemuva kokutholwa kwesigaba sokuqeqesha ngaphambi kokuthi kuqale ukuhlolwa kokuhlolwa.

Okuthathwayo Okubalulekile

1. I-AlphaEvolve yenza ngokuzenzakalelayo ukwakheka kwe-algorithm – esikhundleni sokushuna ama-hyperparameter, ishintsha ikhodi yomthombo wangempela we-Python yama-algorithms e-MAR kusetshenziswa i-Gemini 2.5 Pro njengo-opharetha wokuguqula, ithola imithetho emisha ngokuphelele yokuvuselela kunokwehluka kwaleyo ekhona.
2. I-VAD-CFR ingena esikhundleni sesaphulelo esimile ngokuqwashisa ngokushintshashintsha – ilandelela ubukhulu bokuzisola ngaleso sikhathi nge-EWMA futhi ilungisa izici zayo zesaphulelo ngamandla, kanye nenqubomgomo yokubambezeleka ibe yisilinganiso ngokuphelele kuze kube i-iteration 500, umkhawulo i-LLM ewutholile ngaphandle kokutshelwa ukuthi umkhathizwe wokuhlaziya wawuphindaphinda izikhathi ezingu-1000.
3. I-SHOR-PSRO yenza ngokuzenzakalelayo uguquko lokuhlola-kuya-ukuxhashazwa – ngokuhlanganisa isici esihlanganisayo phakathi kwe-Optimistic Regret Matching kanye nengxenye ye-Softmax best-pure-strategy phezu kokuqeqeshwa, kususa isidingo sokushuna mathupha lapho isixazululi se-meta se-PSRO kufanele sisuke ezinhlobonhlobo zabantu siye ekuthuthukisweni kokulingana.
4. Ukujwayelekile kuyahlolwa, akucatshangwa – womabili ama-algorithms athuthukiswa kusethi eyodwa yemidlalo emine futhi ahlolwa kusethi ehlukile yemidlalo emikhulu, engabonakali. I-VAD-CFR ibambe imidlalo engu-10 kwengu-11; I-SHOR-PSRO kokungu-8 kwangu-11, ngaphandle kokucushwa kabusha phakathi kokuqeqeshwa nokuhlolwa.
5. Izinqubo ezitholiwe aziwona ngokuklama – izinto ezinjengokuqala okufudumele kokuphindaphinda okungu-500, ukukhushulwa kokuzisola okuhle ngo-1.1, kanye nokuhlelwa okuhlukene kwesixazululi sokuqeqeshwa/ukuhlola akulona uhlobo lwezinketho ezivame ukufika kuzo abacwaningi abangabantu, okuyimpikiswano eyinhloko yalolu cwaningo yokusesha okuzenzakalelayo phezu kwalesi sikhala sokuklama.

Hlola Iphepha. Futhi, zizwe ukhululekile ukusilandela Twitter futhi ungakhohlwa ukujoyina wethu 120k+ ML SubReddit futhi Bhalisela ku Iphephandaba lethu. Linda! ukutelegram? manje ungasijoyina kuthelegramu futhi.