The paper is accepted in the ongoing continuous continuous learning of LifeDolong Founds (SCRllFM) workshop at Neurips 2024.
Large models of language (llms) are trained in expired website information. We investigate test strategies and methods of renewal of llms as new data is available. We launched a web-scale dataset for the 114th time of the Crawl Crawl (CC) Our findings indicate that in the General CC Data, Autorgreate Meta-combined schedules with the Replay-Ratio Replay Information can reach the restricted losses from the original writing, while requiring a minimum integration (2.6x). However, the appropriate balance between entering new data and multiplying data data is unique as replay is important to avoid forgetting regular web data but as much to specific domains.
