Ukulungiselela idatha yevidiyo yokufunda okujulile: Kwethulwa i-Vid Prepper
Ukulungiselela amavidiyo wokufunda komshini / ukufunda okujulile. Ngenxa yosayizi kanye nezindleko ze-computational zedatha yevidiyo, kubalulekile ukuthi icutshungulwa ngendlela esebenza kahle ngecala lakho lokusebenzisa. Lokhu kufaka izinto ezinjengokuhlaziywa kwe-metadata, okujwayelekile, ukwengeza, ukuvunjwa nokutholwa kwento, kanye nokulayishwa kwe-tensor. Lo mbhalo uhlola izindlela ezithile zokuthi zingakwenziwa kanjani nokuthi kungani sasizokwenza. Ngiphinde ngakha nephakheji evulekile yePython ebizwa ngeVid-Prepper. Ngakha iphakethe ngenhloso yokuhlinzeka ngendlela esheshayo nephumelelayo yokusebenzisa amasu ahlukile wokufaka idatha yevidiyo yakho. Iphakheji eyakha iziqhwaga ezifundwayo zomshini kanye nomhlaba ojulile wokufunda, ngakho-ke ngenkathi leli phakethe liwusizo ekubangenise ndawonye ngendlela evamile futhi kulula ukuyisebenzisa, umsebenzi wangempela ubunjalo kubo!
Ividiyo ibiyingxenye ebalulekile yomsebenzi wami. Ngiqale umsebenzi wami wedatha enkampanini eyakha iplatifomu ye-SAAS ye-Video Analytics yezinkampani eziphambili zevidiyo ezihola phambili (ezibizwa nge-NPAW) futhi njengamanje zisebenzela i-BBC. Ividiyo njengamanje ibusa indawo yeWebhu, kepha nge-AI isalinganiselwe kahle, yize ikhula i-SuperFast. Bengifuna ukudala okuthile okusiza ukusheshisa amandla abantu okuzama izinto futhi kufaka isandla kule ndawo ethakazelisa ngempela. Lo mbhalo uzoxoxa ngalokho okwenziwa amamojula wephakeji ahlukile nokuthi ungazisebenzisa kanjani, uqala ngokuhlaziywa kweMetadata.
Ukuhlaziywa kweMetadata
from vid_prepper import metadataE-BBC, nginenhlanhla enkulu yokusebenza enhlanganweni yobungcweti ngabantu abanethalente elikhulu kakhulu bedala amavidiyo asezingeni elifanele. Kodwa-ke, ngiyazi ukuthi idatha eminingi yevidiyo akuyona lokhu. Imvamisa amafayela azoba amafomethi ahlanganisiwe, imibala, osayizi, noma angonakaliswe noma anezingxenye ezingekho, angahle abe nama-quirks aqhamuka kumavidiyo amadala, njengokuhlangana. Kubalulekile ukuqaphela noma yikuphi kwalokhu ngaphambi kokucubungula amavidiyo wokufunda komshini.
Sizobe siqeqesha amamodeli ethu ku-GPUs, futhi lokhu kumnandi kakhulu ekubaleni kweziqu ezilinganisweni kodwa kubiza kakhulu ukugijima. Lapho siqeqesha amamodeli amakhulu ku-GPUS, sifuna ukuba senziwe kahle ngangokunokwenzeka ukugwema izindleko eziphakeme. Uma sinamavidiyo asonakele noma amavidiyo ezifomethi ezingalindelekile noma ezingasekelwa kuzochitha isikhathi nezinsizakusebenza, angenza amamodeli akho anembe kancanyana noma aze abangele ukuba ipayipi lokuqeqeshwa liphule. Ngakho-ke, ukubheka nokuhlunga amafayela akho kusengaphambili kuyisidingo.
Ngizakhele imodyuli yokuhlaziywa kweMetadata kuLabhulali ye-FFProbe, ingxenye yomtapo wezincwadi we-FFMPEG owakhiwe ku-C nokubuthana. Le umtapo wezincwadi enamandla kakhulu futhi esebenza kahle esetshenziswe kakhulu emsebenzini kanye nemodyuli ingasetshenziselwa ukuhlaziya ifayela elilodwa levidiyo noma iqembu lazo njengoba kukhonjisiwe kwikhodi engezansi.
# Extract metadata
video_path = [“sample.mp4”]
video_info = metadata.Metadata.validate_videos(video_path)
# Extract metadata batch
video_paths = [“sample1.mp4”, “sample2.mp4”, “sample3.mp4”]
video_info = metadata.Metadata.validate_videos(video_paths)Lokhu kunikeza umphumela wesichazamazwi we-metadata yevidiyo kufaka ama-codecs, osayizi, amazinga ozimele, isikhathi, amafomethi we-pixel, i-audio metadata nokuningi. Lokhu kuwusizo ngempela ekutholeni idatha yevidiyo ngezinkinga noma izingqinamba eziyinqaba, noma nokukhetha idatha ethile yevidiyo noma ukukhetha amafomethi kanye ne-codec ukuze ulinganise kakhulu ku-aidges evame kakhulu.
Ukuhlunga kususelwa ezindabeni zeMetadata
Njengoba kunikezwe lokhu kwakubonakala sengathi kuyicala elijwayelekile lokusebenzisa, ngakhekela amandla okuhlunga uhlu lwamavidiyo asuselwa kusethi yamasheke. Isibonelo, uma kukhona ividiyo noma i-audio elahlekile, ama-codec noma amafomethi hhayi njengamazinga acacisiwe, noma amazinga noma isikhathi esihlukile kulawo acacisiwe, khona-ke lawa mavidiyo angabonakala ngokubeka isihlungi kanye kuphela_pholors.
# Run tests on videos
videos = ["video1.mp4", "video2.mkv", "video3.mov"]
all_filters_with_params = {
"filter_missing_video": {},
"filter_missing_audio": {},
"filter_variable_framerate": {},
"filter_resolution": {"min_width": 1280, "min_height": 720},
"filter_duration": {"min_seconds": 5.0},
"filter_pixel_format": {"allowed": ["yuv420p", "yuv422p"]},
"filter_codecs": {"allowed": ["h264", "hevc", "vp9", "prores"]}
}
errors = Metadata.validate_videos(
videos,
filters=all_filters_with_params,
only_errors=True
)Ngokususa noma ukukhomba izingqinamba ngedatha ngaphambi kokuthi sifinyelele emsebenzini wangempela wokuqeqeshwa omodeli kusho ukuthi sigwema ukuchitha isikhathi nemali, okwenza kube yisinyathelo sokuqala esibalulekile.
Ukwenza phakathi
from vid_prepper import standardizeUkulinganisa okujwayelekile kuvame ukuba kubaluleke kakhulu ku-PreProcessing for Video Machine Learning. Kungasiza ekwenzeni izinto zisebenze kahle kakhulu futhi zingaguquki, futhi kaningi amamodeli wokufunda ajulile adinga osayizi abathile (isib. 224 x 224). Uma unedatha eningi yevidiyo lapho isikhathi esichithwa kulesi sigaba sivame ukubuyiselwa kaningi esigabeni sokuqeqeshwa kamuva.
Ama-codecs
Amavidiyo avame ukuhlelwa ukuze abekelwe isitoreji esisebenzayo nokusatshalaliswa kwe-Intanethi ukuze asakazeke eshibhile futhi ngokushesha. Lokhu kuvame ukubandakanya ukucindezelwa okunamandla ukwenza amavidiyo ancanyana ngangokunokwenzeka. Ngeshwa, lokhu kufanekiswa kakhulu kubumbiso kulokho okuhle kokufunda okujulile.
I-bottleneck yokufunda okujulile cishe ihlala ifaka amavidiyo futhi ilayishe izifiso, ngakho-ke ifayela levidiyo licindezelwe kakhulu, isikhathi eside esithathayo. Lokhu ngokujwayelekile kusho ukugwema ama-codecs acindezelwe we-Ultra afana ne-H265 ne-VVC futhi akwazi ukusheshiswa kwezinye izindlela ezicindezelwe nge-H264 noma i-VP9, noma inqobo into efana ne-MJPEEG ethambisa ukukhiqizwa njengoba kuyindlela esheshayo yokulayisha ngezifiso.
Ukukalwa kwe fulemu
Amanani ajwayelekile wohlaka (FPS) wevidiyo angama-24 nge-cinema, ama-30 we-TV nokuqukethwe okuku-inthanethi kanye nokuqukethwe okusheshayo okuqukethwe. Lezi zilinganiso zozimele zinqunywa yinombolo yezithombe ezidingekayo ukuze zikhonjiswe ngomzuzwana ukuze amehlo ethu abone ukunyakaza okubushelelezi okukodwa. Kodwa-ke, amamodeli wokufunda ajulile awadingeki njengezinga eliphakeme ngohlaka kumavidiyo wokuqeqesha ukuze adale izethulo zezinombolo zokunyakaza futhi akhiqize amavidiyo abukeka kahle. Njengoba lonke uhlaka luyisengezo esingeziwe sokuhlanganisa, sifuna ukunciphisa inani lohlaka lube luncane kakhulu esingaluthola.
Izinhlobo ezahlukene zamavidiyo kanye necala lokusebenzisa amamodeli ethu zizonquma ukuthi singahamba kangakanani. Ukunyakaza okuncane kwividiyo, ephansi singasetha inani lokufaka uhlaka ngaphandle kokuyekethisa imiphumela. Isibonelo, idatha yokufaka yezithombe zezindaba zesitudiyo noma imibukiso yenkulumo izodinga isilinganiso sohlaka oluphansi kunedatha eyenziwe ngemidlalo ye-ice hockey. Futhi, uma sisebenza ekuqondeni kwevidiyo noma imodeli yevidiyo-to-umbhalo, kunokuba akhiqize ividiyo yokusetshenziswa kwabantu, kungenzeka ukuthi usethe isilinganiso sefreyimu ephansi.
Ukubala isilinganiso sefreyimu ephansi
Empeleni kungenzeka ngokubona izilinganiso ezinhle zohlaka oluhle lwedatha yakho yevidiyo esekelwe kwizibalo ezinyakazayo. Usebenzisa i-algorithm ye-raft noma ye-farneback kwisampula yedatha yakho, ungabala ukugeleza kwe-optical nge-pixel ngayinye yokushintsha kohlaka ngalunye. Lokhu kunikeza ukufuduka okuvundlile nokumi mpo kwe-pixel ngayinye ukubala ubukhulu boshintsho (impande yesikwele yokwengeza amanani akwenziwe).
Ukuvimbela leli nani phezu kohlaka kunikeza ifreyimu yesikhashana futhi kuthathe i-median ne-95th percentile yawo wonke amafreyimu kunikeza ama-values ongawafaka ku-equation engezansi kwedatha yakho yokuqeqesha efanelekile.
Optimal FPS (Lower) = Current FPS x Max model interpolation rate / Median momentum
Optimal FPS (Higher) = Current FPS x Max model interpolation rate / 95th percentile momentumLapho ukuhlukaniswa kwemodeli yeMax kunguhlaka oluphezulu ngefreyimu ngayinye imodeli ingaphatha, imvamisa inikezwe ekhadini lemodeli.
Ungase ugijime izivivinyo ezincane zepayipi lakho lokuqeqesha ukuthola isilinganiso sohlaka oluphansi kunazo zonke ongaluthola ukuze usebenze kahle.
I-VID Prepper
Imodyuli evamile ku-Vid-Prepper ingamisa usayizi, i-codec, ifomethi yombala kanye nesilinganiso sohlaka sevidiyo eyodwa noma i-batch yamavidiyo.
Futhi, yakhelwe ku-FFMPEG futhi inamandla okusheshisa izinto ku-GPU uma lokho kutholakala kuwe. Ukuze uzenze amavidiyo ajwayelekile, ungamane usebenzise ikhodi engezansi.
# Standardize batch of videos
video_file_paths = [“sample1.mp4”, “sample2.mp4”, “sample3.mp4”]
standardizer = standardize.VideoStandardizer(
size="224x224",
fps=16,
codec="h264",
color="rgb",
use_gpu=False # Set to True if you have CUDA
)
standardizer.batch_standardize(videos=video_file_paths, output_dir="videos/")Ukuze wenze izinto zisebenze kahle, ikakhulukazi uma usebenzisa i-GPU ebizayo futhi ungafuni i-Io Bottleneck kusuka kumavidiyo alayisha, imodyuli nayo yamukela ama-webdataset. Lokhu kungalayishwa ngokufanayo nekhodi elandelayo:
# Standardize webdataset
standardizer = standardize.VideoStandardizer(
size="224x224",
fps=16,
codec="h264",
color="rgb",
use_gpu=False # Set to True if you have CUDA
)
standardizer.standardize_wds("dataset.tar", key="mp4", label="cls")Umthwalo we-TESONOR
from vid_prepper import loaderI-tensor yevidiyo imvamisa ubukhulu obu-4 noma obu-5, obuqukethe umbala we-pixel (imvamisa i-RGB), ukuphakama nobubanzi bohlaka, isikhathi kanye nezakhi ze-batch (ngokuzithandela). Njengoba kushiwo ngenhla, ukudicilela amavidiyo abe yizinsika kuvame ukuba yibhodlela elikhulu kunazo zonke epayipini elingaphambi kwesikhathi, ngakho-ke izinyathelo ezithathwe kuze kube manje zenza umehluko omkhulu wokuthi singalayisha kahle kangakanani.
Le module iguqula amavidiyo abe yizingqinamba ze-Pytorch ezisebenzisa i-FFmpeg ngesampula yohlaka ne-NVDEC ukuvumela ukusheshiswa kwe-GPU. Ungaguqula usayizi wezinsika ukuze uvumelane nemodeli yakho kanye nokukhetha inani lamafreyimu ukuze kuboniswe isampula ngesiqeshana ngasinye nesigameko (ukuhlukaniswa phakathi kwamafreyimu). Njengokwenziwa okujwayelekile, inketho yokusebenzisa i-WebDatasets iyatholakala. IKhodi engezansi inikeza isibonelo sokuthi lokhu kwenziwa kanjani.
# Load clips into tensors
loader = VideoLoader(num_frames=16, frame_stride=2, size=(224,224), device="cuda")
video_paths = ["video1.mp4", "video2.mp4", "video3.mp4"]
batch_tensor = loader.load_files(video_paths)
# Load webdataset into tensors
wds_path = "data/shards/{00000..00009}.tar"
dataset = loader.load_wds(wds_path, key="mp4", label="cls")Umtshingo
from vid_prepper import detectorNgokuvamile kuyingxenye edingekayo yevidiyo eyandulelayo ukuthola izinto ngaphakathi kokuqukethwe kwevidiyo. Lokhu kungenzeka kube yizinto ezithile, izibhamu noma izinguquko. Le module iletha ndawonye izinqubo ezinamandla namamodeli avela ku-PyScenetenet, i-huggafaface, i-Idea Research ne-Pytorch ukuhlinzeka ngokutholwa okusebenzayo.
Ukutholwa kwe-Shot
Emaqenjini amaningi okufunda kwevidiyo asebenzisa amacala okusetshenziswa kwevidiyo (isib. Ukusesha kwe-semantic, isizukulwane se-seq2seq trailer nokunye okuningi), ukuhlukanisa amavidiyo kusibhamu ngasinye isinyathelo esibalulekile. Kunezindlela ezimbalwa zokwenza lokhu, kepha i-PYSCEPETECT ingenye yezindlela ezinembile nezinokwethenjelwa zokwenza lokhu. Lo mtapo wezincwadi uhlinzeka nge-wrapper yendlela yokutholwa kokuqukethwe kokuqukethwe kwePyscesettectet ngokubiza indlela elandelayo. Ikhipha amafreyimu wokuqalisa nokugcina ngokudutshulwa ngakunye.
# Detect shots in videos
video_path = "video.mp4"
detector = VideoDetector(device="cuda")
shot_frames = detector.detect_shots(video_path)Ukutholwa kokushintsha
Ngenkathi i-PySceseteTEct iyithuluzi eliqinile lokuhlukanisa amavidiyo abe yizigcawu ngazinye, akuhlali kunembile njalo ngo-100%. Kunezikhathi lapho ungakwazi khona ukuze ukwazi ukusebenzisa ngokunenzuzo okuqukethwe okuphindaphindwayo (isib. Izinguquko) ukuphula isibhamu. Isibonelo, i-BBC News inenguquko ebomvu ebomvu futhi emhlophe evulekile phakathi kwezingxenye ezingatholwa kalula kusetshenziswa okuthile okufana ne-PyTorch.
Ukutholwa kwenguquko kusebenza ngqo kwizifiso ngokuthola izinguquko zePixel emabhulokini amaphikiseli adlula ushintsho oluthile lweveni ongasenza. Ikhodi eyisibonelo engezansi ikhombisa ukuthi isebenza kanjani.
# Detect gradual transitions/wipes
video_path = "video.mp4"
video_loader = loader.VideoLoader(num_frames=16,
frame_stride=2,
size=(224, 224),
device="cpu",
use_nvdec=False # Use "cuda" if available)
video_tensor = loader.load_file(video_path)
detector = VideoDetector(device="cpu" # or cuda)
wipe_frames = detector.detect_wipes(video_tensor,
block_grid=(8,8),
threshold=0.3)Ukutholwa Kwento
Ukutholwa kwento kuvame ukuba imfuneko yokuthola iziqeshana ozidingayo kwidatha yakho yevidiyo. Isibonelo, ungadinga iziqeshana nabantu kuzo noma ezilwaneni. Le ndlela isebenzisa imodeli ye-Dino evulekile ngokumelene nesethi encane yezinto ezivela kumalebula we-Coco Dataset ajwayelekile wokuthola izinto. Kokubili ukukhetha okuyimodeli nohlu lwezinto zingenziwa ngezifiso ngokuphelele futhi zingabekwa nguwe. I-Model Loader iyiphakethe le-Transform Transformers ukuze imodeli oyisebenzisayo izodinga ukutholakala lapho. Okwamalebula ngokwezifiso, imodeli ezenzakalelayo ithatha intambo ngesakhiwo esilandelayo ku-tex_queries parameter – “inja. Ikati. I-ambulensi.”
# Detect objects in videos
video_path = "video.mp4"
video_loader = loader.VideoLoader(num_frames=16,
frame_stride=2,
size=(224, 224),
device="cpu",
use_nvdec=False # Use "cuda" if available)
video_tensor = loader.load_file(video_path)
detector = VideoDetector(device="cpu" # or cuda)
results = detector.detect_objects(video,
text_queries=text_queries # if None will default to COCO list,
text_threshold=0.3,
model_id=”IDEA-Research/grounding-dino-tiny”)Ukwengezelela kwedatha
Izinto ezinjengeziguquli zevidiyo zinamandla amakhulu futhi zingasetshenziswa ukudala amamodeli amasha amasha. Kodwa-ke, zivame ukudinga inani elikhulu ledatha elingatholakali kalula ngezinto ezifana nevidiyo. Kulezo zimo, sidinga indlela yokukhiqiza idatha ehlukahlukene eyeka amamodeli ethu ngokweqile. Ukwengezwa kwe-Data kungenye yezixazululo ezinjalo ukusiza ukukhulisa ukutholakala kwemininingwane okunomkhawulo.
Ividiyo, kunenqwaba yezindlela ezijwayelekile zokuthiza imininingwane futhi iningi lalezo zisekelwa izinhlaka ezinkulu. I-Vid-Prepper ihlanganisa ndawonye ezimbili ezinhle kakhulu – i-Kornia ne-TORTVision. Nge-Vid-Prepper, ungenza ukwengezelwa ukwengezelwa okufana nokutshala, ukufuya, izibuko, ukufiphala, ukugqama, ukugqama, ukugqama, lapho izingxenye zohlaka lwevidiyo zimiswe khona). Ungazicaphuna ndawonye ukuze usebenze kahle.
Ukulala wonke umsebenzi ku-tessors yevidiyo kunokuba ngqo kumavidiyo nokusekelwa kokusheshisa kwe-GPU uma unayo. Ikhodi eyisibonelo engezansi ikhombisa ukuthi ungazibiza kanjani izindlela ngawodwana nokuthi ungazicaphuna kanjani.
# Individual Augmentation Example
video_path = "video.mp4"
video_loader = loader.VideoLoader(num_frames=16,
frame_stride=2,
size=(224, 224),
device="cpu",use_nvdec=False # Use "cuda" if available)
video_tensor = loader.load_file(video_path)
video_augmentor = augmentor.VideoAugmentor(device="cpu", use_gpu=False)
cropped = augmentor.crop(video_tensor, type="center", size=(200, 200))
flipped = augmentor.flip(video_tensor, type="horizontal")
brightened = augmentor.brightness(video_tensor, amount=0.2)
# Chained Augmentations
augmentations = [
('crop', {'type': 'random', 'size': (180, 180)}),
('flip', {'type': 'horizontal'}),
('brightness', {'amount': 0.1}),
('contrast', {'amount': 0.1})
]
chained_result = augmentor.chain(video_tensor, augmentations)Ukufingqa
I-Video PreProcessing ibaluleke kakhulu ngokufunda okujulile ngenxa yobukhulu obukhulu bemininingwane ngokuqhathaniswa nombhalo. Izidingo zemodeli yeTransformer yezilwandlekazi zedatha zihlanganisa lokhu nangaphezulu. Izinto ezintathu ezibalulekile zenza inqubo yokufunda ejulile – isikhathi, imali kanye nokusebenza. Ngokwenza kahle idatha yethu yevidiyo yokufaka, singanciphisa inani lezinto ezimbili zokuqala esikudingayo ukuze siphumelele kokugcina.
Kunamanye amathuluzi womthombo avulekile amangalisayo atholakalayo wokufunda komshini wevidiyo, ngokuza okuningi kanye nosuku nsuku zonke. I-Vid-Prepper imele emahlombe athile ahamba phambili futhi asetshenziswe kabanzi emzameni wokuzama futhi ahlanganise ndawonye kwiphakheji elula yokusebenzisa. Ngiyethemba uthola inani elithile kulo futhi kukusiza ukudala isizukulwane esilandelayo samamodeli wevidiyo, okujabulisa kakhulu!
