Umhlahlandlela osebenzayo kuma-multimodal data analytics

Okuqukethwe okuxhasiwe

I-Google Cloud

Ukuqalisa

Amabhizinisi aphatha ukuhlanganiswa kwemininingwane ehlelekile kumatafula ahlelekile kanye nevolumu ekhulayo yedatha engahleliwe efana nezithombe, umsindo, kanye nemibhalo. Ukuhlaziya lezi zinhlobo zemininingwane ezahlukahlukene ndawonye kuyinkimbinkimbi ngokwesiko, njengoba zivame ukudinga amathuluzi ahlukile. Imidiya engahleliwe ngokuvamile idinga ukuthunyelwa kwizinsizakalo ezikhethekile zokucutshungulwa (isib. I-Computer Vision Service for Ukuhlaziywa Kwezithombe, noma Injini Yezinkulumo Yombhalo Wokulalelwayo), okwakha ama-silos wedatha futhi avimbela umbono wokuhlaziya wedatha futhi avimbela umbono ophelele wokuhlaziya.

Cabanga ngohlelo lokusekelwa lwe-e-commerce oluqanjiwe: imininingwane yamathikithi ahlelekile ihlala etafuleni elibuyabusuku, kuyilapho kubhalwa okuqoshwa kwamakholi noma izithombe zemikhiqizo eyonakele ihlala ezitolo ze-Cloud Object. Ngaphandle kwesixhumanisi esiqondile, ukuphendula umbuzo ocebile oqondile onjengokuthi “Khomba wonke amathikithi okusekela imodeli ethile ye-laptop lapho umsindo wokulahla ukhombisa ukukhungatheka kwamakhasimende okuphezulu futhi isithombe sibonisa isikrini esiqhekekile” yinqubo enezinyathelo eziningi.

Le ndatshana iyinkomba esebenzayo, yezobuchwepheshe ye-OnyRREF eBigQuery, isici esenzelwe ukuhlanganisa lokhu kuhlaziywa. Sizohlola ukuthi singawakha kanjani, sibuze, futhi silawule ama-multimodal datasets, enika amandla ukuqonda okuphelele kusetshenziswa izindawo zokuxhumana ezijwayelekile ze-SQL nePython.

Ingxenye 1: I-OnybleRef – Isihluthulelo Sokuhlanganisa idatha ye-multimodal

Ukwakheka kwento kanye nomsebenzi

Ukubhekana nenselelo yedatha ethulisiwe, iBigQuery ingenisa i-OnyPref, uhlobo olukhethekile lwedatha yedatha. I-Onyblef isebenza njengereferensi eqondile yento engahleliwe egcinwe kwisitoreji se-Google Cloud (GCS). Akuqukethe idatha engahleliwe ngokwayo (isib. I-BASE64 Encode Izithombe ku-database, noma olalelwayo); Esikhundleni salokho, ikhomba indawo yaleyo datha, okuvumela ukufinyeleleka okune-biobquery ukuyifinyelela futhi ifake imibuzo ukuze ihlaziywe.

I-Onyblerref ehlelwe yakhiwa amasimu aphezulu asemqoka:

• U-URI (Intambo): Indlela ye-GCS entweni
• ukugunyaza (I-String): Ivumela iBigQuery ifinyelele ezintweni eziphephile ze-GCS
• ukuhumushela (I-String): Igcina i-ID ethile yesizukulwane sento ye-GCS, ukukhiya ireferensi enguqulweni eqondile yokuhlaziywa okuvela kabusha
• bonisana (Json): Into ye-json evame ukuqukethe i-GCS metadata efana contentType noma size

Nasi isethulo se-JSson senani lento:

JSON

{
  "uri": "gs://cymbal-support/calls/ticket-83729.mp3",
  "version": 1742790939895861,
  "authorizer": "my-project.us-central1.conn",
  "details": {
    "gcs_metadata": {
      "content_type": "audio/mp3",
      "md5_hash": "a1b2c3d5g5f67890a1b2c3d4e5e47890",
      "size": 5120000,
      "updated": 1742790939903000
    }
  }
}

Ngokuhlanganisa lolu lwazi, i-OnyFREF ihlinzeka ngobukhulu nayo yonke imininingwane edingekayo yokuthola, ukufinyelela ngokuphephile, futhi uqonde izakhiwo eziyisisekelo zefayela elingahleliwe kuma-GCS. Lokhu kwakha isisekelo sokwakha amatafula we-multimodal kanye ne-dafaframes, okuvumela idatha ehlelekile ukuze ibuphile eceleni nezinkomba ezinokuqukethwe okungahleliwe.

Dala amatafula amaningi

A Ithebula le-multimodal iyitafula elijwayelekile elisezingeni eliphakeme elifaka ikholomu eyodwa noma ngaphezulu ye-abref. Lesi sigaba simboza ukuthi ungawakha kanjani la matafula bese uwahlukanisa nge-SQL.

Ungachaza amakholamu we-onessrref lapho udala ithebula elisha noma ungeze kumatafula akhona. Lokhu kuvumelana nezimo kukuvumela ukuthi uvumelanise amamodeli wakho wedatha wamanje ukuze usebenzise amakhono amaningi we-multimodal.

Ukwakha ikholomu ye-OnyFREF enamatafula ento

Uma unamafayela amaningi agcinwe kubhakede le-GCS, ithebula lento liyindlela ephumelelayo yokukhiqiza izinto ezingenzi kahle. Ithebula lento yitafula elifundwayo kuphela elibonisa okuqukethwe yi-GCS Directory futhi ngokuzenzakalelayo kufaka phakathi ikholomu ebizwa ngegama refzohlobo lwento.

SQL

CREATE EXTERNAL TABLE `project_id.dataset_id.my_table`
WITH CONNECTION `project_id.region.connection_id`
OPTIONS(
  object_metadata="SIMPLE",
  uris = ['gs://bucket-name/path/*.jpg']
);

Okuphumayo ithebula elisha eliqukethe a ref ikholomu. Ungasebenzisa ref ikholomu ngemisebenzi efana AI.GENERATE noma ujoyine kwamanye amatafula.

Ukukhahlela okuhle kwe-Ontrefs

Ukuze uthole ukugeleza okunamandla okwengeziwe, ungakha ama-pragrams wecebo ngokusenzela OBJ.MAKE_REF() sebenza. Kuvamile ukusonga lo msebenzi ku OBJ.FETCH_METADATA() ukuveza details into ene-GCS metadata. Ikhodi elandelayo nayo isebenza uma ungena esikhundleni gs:// indlela enenkambu ye-URI etafuleni elikhona.

SQL

SELECT 
OBJ.FETCH_METADATA(OBJ.MAKE_REF('gs://my-bucket/path/image.jpg', 'us-central1.conn')) AS customer_image_ref,
OBJ.FETCH_METADATA(OBJ.MAKE_REF('gs://my-bucket/path/call.mp3', 'us-central1.conn')) AS support_call_ref

Ngokusebenzisa amatafula ento noma OBJ.MAKE_REFungakha futhi ulondoloze amatafula we-multimodal, ubeka isigaba sama-analytics ahlanganisiwe.

Ingxenye 2: Amathebula E-Multimodal nge-SQL

Ukufinyelela okuphephile nokulawulwa

I-OnyPref ihlangana nezici zokuphepha zendabuko ze-Greequery, okuvumela ukuphathwa kwemininingwane yakho ye-multimodal. Ukutholakala kwezinto ezingaphansi kwe-GCS azinikezwe umsebenzisi wokugcina ngqo. Esikhundleni salokho, idluliselwe kwimithombo yokuxhumeka ebabazekayo echazwe enkundleni ye-Accerref's Authorizer field. Le modeli ivumela izingqimba eziningi zokuphepha.

Cabanga ngetafula le-multimodal elilandelayo, eligcina imininingwane ngezithombe zomkhiqizo esitolo sethu se-e-commerce. Ithebula lifaka ikholomu ye-Onyblef image.

Ezokuphepha Eleveli Yekholomu: khawula ukufinyelela kukholomu yonke. Ukuze kubekwe ngabasebenzisi okufanele bahlaziye kuphela amagama omkhiqizo nezilinganiso, umlawuli angafaka ukuphepha okusezingeni lekholomu ku- image ikholomu. Lokhu kungavumeli labo abahlaziyi ekukhetheni image Ikholomu ngenkathi kuvumela ukuhlaziywa kwezinye izinkambu ezihlelekile.

Ukuphepha okudala: I-Bhroquery ivumela ukuhlunga ukuthi yimiphi imigqa umsebenzisi angabona ngokususelwa kwimithetho echaziwe. Inqubomgomo yezinga le-Row-Level ingakhawulela ukufinyelela ngokuya ngeqhaza lomsebenzisi. Isibonelo, inqubomgomo ingasho ukuthi “ungavumeli abasebenzisi ukuthi babuze imikhiqizo ehlobene nezinja”, okuhlunga le migqa emiphumeleni yemibuzo kube sengathi ayikho.

Abagunyazayo abaningi: Leli thebula lisebenzisa ukuxhumeka okubili okuhlukile ku image.authorizer element (conn1 na- conn2).

Lokhu kuvumela umlawuli ukuthi aphathe izimvume ze-GCS maphakathi nokuxhunyaniswa. Ngokwesibonelo, conn1 kungafinyelela ibhakede lesithombe somphakathi, ngenkathi conn2 Ifinyelela ibhakede elikhawulelwe ngemiklamo emisha yomkhiqizo. Noma umsebenzisi angabona yonke imigqa, amandla abo okubuza ifayili elingaphansi kwe- “bird yembewu” kuncike ngokuphelele ekutheni banemvume yokusebenzisa ilungelo lokusebenzisa ilungelo elikhulu conn2 ukuxhumana.

Ukutholwa kwe-Ai-eqhutshwa nge-SQL

Le khasi AI.GENERATE_TABLE Umsebenzi udala itafula elisha elihlelekile ngokusebenzisa imodeli ye-AI ekhiqizayo kwimininingwane yakho ye-multimodal. Lokhu kulungele imisebenzi yokucebisa idatha ngezinga. Masisebenzise isibonelo sethu se-e-commerce ukudala amagama angukhiye we-SEO kanye nencazelo emfushane yokumaketha yomkhiqizo ngamunye, usebenzisa igama lawo nesithombe njengemithombo yomthombo.

Umbuzo olandelayo uyasebenza products ithebula, ukuthatha product_name na- image I-OnyPref njengokufakwa. Kwakha itafula elisha eliqukethe okwangempela product_iduhlu lwamagama angukhiye we-SEO, nencazelo yomkhiqizo.

SQL 

SELECT
  product_id,
  seo_keywords,
  product_description
FROM AI.GENERATE_TABLE(
  MODEL `dataset_id.gemini`, (
    SELECT (
		'For the image of a pet product, generate:'
            '1) 5 SEO search keywords and' 
            '2) A one sentence product description', 
            product_name, image_ref) AS prompt,
            product_id
    FROM `dataset_id.products_multimodal_table`
  ),
  STRUCT(
     "seo_keywords ARRAY, product_description STRING" AS output_schema
  )
);

Umphumela uba itafula elisha elihlelekile elinamakholomu product_id, seo_keywordsfuthi product_description. Lokhu kusebenza umsebenzi wokuthengisa odla isikhathi futhi kukhiqize idatha elungele ukusetshenziswa engalayishwa ngqo ohlelweni lokuphathwa kokuqukethwe noma olusetshenziselwa ukuhlaziya okwengeziwe.

Ingxenye 3: I-Multimodal DataFrames ngePython

Bridging python kanye birquery ukubonwa kwe-multimodal

I-Python lulimi lokuzikhethela ososayensi abaningi bedatha kanye nabahlaziyi bedatha. Kepha abasebenza ngokuvamile bagijimela izingqinamba lapho idatha yabo inkulu kakhulu ukuba ingenele inkumbulo yomshini wendawo.

I-BoodQuery DataFrames inikeza isisombululo. Inikeza i-API ye-pandas-like uxhumana nedatha egcinwe eBigQuery Ngaphandle kwalokhu ukudonsa kwimemori yasendaweni. Umtapo wezincwadi uhumusha ikhodi yePython kwi-SQL edonswa phansi futhi yenziwa ngenjini enobukhulu obukhulu kakhulu. Lokhu kuhlinzeka nge-syntax ejwayelekile yomtapo wezincwadi odumile we-Python ehlanganiswe namandla e-biobquery.

Lokhu ngokwemvelo kufinyelela kuma-multimodal analytics. I-dataFeery DataFrame ingamelela zombili imininingwane yakho ehlelekile nezinkomba kumafayela angahleliwe, ndawonye I-Multimodal DataFrame. Lokhu kukuvumela ukuthi ulayishe, uguqule, uhlaziye ama-dafaframes aqukethe ama-metadata akho aqukethe zombili ama-metadata nama-pointres kumafayili angahleliwe, ngaphakathi kwendawo eyodwa yePython.

Dala i-multimodal dafazi

Uma usune-Library Bigframes efakiwe, ungaqala ukusebenza nedatha ye-multimodal. Umqondo osemqoka yi- ikholomu ye-blob: Ikholomu ekhethekile ephethe izinkomba kumafayela angahleliwe kuma-GCS. Cabanga ngekholomu ye-Blob njengokwembulwa kwe-Python ye-OnybleTef – ayiphathi ifayela uqobo, kepha likhomba futhi linikeze izindlela zokusebenzisana nazo.

Kunezindlela ezintathu ezijwayelekile zokwenza noma ukuklama ikholomu ye-Blob:

PYTHON

import bigframes
import bigframes.pandas as bpd

# 1. Create blob columns from a GCS location
df = bpd.from_glob_path(  "gs://cloud-samples-data/bigquery/tutorials/cymbal-pets/images/*", name="image")

# 2. From an existing object table
df = bpd.read_gbq_object_table("", name="blob_col")

# 3. From a dataframe with a URI field
df["blob_col"] = df["uri"].str.to_blob()

Ukuchaza izindlela ezingenhla:

1. Indawo ye-GCS: Sebenzisa from_glob_path ukuskena ibhakede le-GCS. Ngemuva kwezigcawu, lokhu kusebenza kudala ithebula lento yesikhashana, futhi likuveze njenge-dataframe ngekholomu ye-blob elungele ukusetshenziswa.
2. Ithebula lento ekhona: Uma usuvele unetafula lento eBogquery, sebenzisa read_gbq_object_table Umsebenzi wokulayisha. Lokhu kufunda itafula elikhona ngaphandle kokudinga ukuskena kabusha i-GCS.
3. I-DataFrame ekhona: Uma une-dataFAMRAme eyiBrowquery equkethe ikholomu yentambo ye-GCS Uris, mane nje usebenzise .str.to_blob() Indlela kuleyo kholamu ukuze ithuthukise “kuyo kukholamu ye-blob.

Ukutholwa kwe-AI kuqhutshwa ngePython

Inzuzo eyinhloko yokwakha i-multimodal dataframe ukwenza ukuhlaziya okuqhutshwa yi-AI ngqo kwidatha yakho engahleliwe esikalini. I-BookQuery DataFrames ikuvumela ukuthi usebenzise amamodeli amakhulu wezilimi (LLMS) kwidatha yakho, kufaka phakathi noma yimaphi amakholomu we-BLOB.

Ukuhamba komsebenzi okujwayelekile kufaka izinyathelo ezintathu:

1. Dala i-multimodal dataframe ngekholomu ye-blob ekhomba amafayela angahleliwe
2. Layisha imodeli ye-BoodQuery ekhona ngaphambi kokuya entweni yama-Bigframes Model
3. Shayela indlela ye-.prict () entweni yemodeli, edlula idatha yakho ye-multimodal njengokufaka.

Masiqhubeke nesibonelo se-e-commerce. Sizosebenzisa i- gemini-2.5-flash Imodeli yokukhiqiza incazelo emfushane yesithombe ngasinye somkhiqizo wesilwane.

PYTHON

import bigframes.pandas as bpd

# 1. Create the multimodal dataframe from a GCS location
df = bpd.from_glob_path(
"gs://cloud-samples-data/bigquery/tutorials/cymbal-pets/images/*", name="image_blob")


# Limit to 2 images for simplicity
df = df.head(2)

# 2. Specify a large language model
from bigframes.ml import llm


model = llm.GeminiTextGenerator(model_name="gemini-2.5-flash-preview-05-20")

# 3. Ask the LLM to describe what's in the picture

answer = model.predict(df_image, prompt=["Write a 1 sentence product description for the image.", df_image["image"]])

answer[["ml_generate_text_llm_result", "image"]]

Lapho ufona model.predict(df_image)Ukwakha i-dataframes eBreequery futhi kwakhiphe umbuzo we-SQL usebenzisa ML.GENERATE_TEXT sebenza, kudluliswa ngokuzenzakalelayo izinkomba zefayela kusuka ku- blob ikholomu nombhalo prompt njengokufakwayo. Injini eBogQuery icubungula lesi sicelo, ithumela imininingwane kwimodeli ye-Gemini, futhi ibuyisa izincazelo zemibhalo ekhiqizwayo kwikholamu entsha kwi-DataFrame ephumela.

Lokhu okuhlanganisiwe okunamandla kukuvumela ukuthi wenze ukuhlaziya okuningi kwe-multimoral kuwo wonke amakhulu noma izigidi zamafayela usebenzisa imigqa embalwa yekhodi yePython.

Ukuya ngokujulile nge-dataframes ye-multimodal

Ngaphezu kokusebenzisa i-LLMS yesizukulwane, The bigframes ILabhulali inikeza isethi ekhulayo yamathuluzi aklanyelwe ukucubungula futhi ahlaziye idatha engahleliwe. Amakhono asemqoka atholakala kwikholomu yeBlob kanye nezindlela zawo ezihlobene zifaka:

• Ukuguqulwa okwakhelwe ngaphakathi: Lungiselela izithombe zokumodela ngokuguqulwa kwendabuko ngemisebenzi ejwayelekile efana nokufiphala, okujwayelekile, kanye nokukhulisa kabusha ngezinga.
• Ukushumeka isizukulwane: Nika amandla ukusesha kwe-semantic ngokukhiqiza ukushumeka kusuka kudatha ye-multimodal, kusetshenziswa amamodeli aphethwe yi-vertex ai ukuguqula idatha ibe yimininingwane ekunikezelweni kwemali eyodwa.
• I-PDF Chunking: Hlukanisa ukugeleza kokusebenza kwe-rag ngokuhlukanisa ngokuhlelwa ngokuhlelwa ngokuhlelwa kwezingxenye ezincanyana, ezinenjongo – isinyathelo esijwayelekile sokusebenza.

Lezi zici zisayina ukuthi i-dataFRAMRAMR yama-BookQuery yakhelwe njengethuluzi lokugcina lokugcina lokuhlaziywa kwe-multimodal ne-AI ngePython. Njengoba intuthuko iyaqhubeka, ungalindela ukubona amathuluzi amaningi ngokwesiko etholakala emitatsheni ehlukene, ekhethekile ehlanganiswe ngqo kuyo bigframes.

Isiphetho:

Amatafula we-multimodal kanye ne-dafaframes amele ukuguquka kokuthi izinhlangano zezinhlangano zingasondela kanjani kwedatha. Ngokwakha isixhumanisi esiqondile, esivikelekile phakathi kwemininingwane ye-tabular namafayela angahleliwe kuma-GCS, ama-biobquery akhiphe ama-silos wedatha anenkimbinkimbi yokuhlaziywa kwe-multimodal ende.

Lo mhlahlandlela ukhombisa ukuthi noma ngabe ungumuntu obhala ngedatha ubhala i-SQL, noma usosayensi wedatha esebenzisa i-Python, manje unekhono lokuhlaziya ngokucophelela amafayela we-multimodal ngokulandelana ngokulandelana okulula.

Ukuze uqale ukwakha ezakho izixazululo ze-multimodal analytics, hlola izinsiza ezilandelayo:

1. Imibhalo esemthethweni: Funda ukubuka konke okusendleleni yokuhlaziya idatha ye-multimoral eBigQuery
2. Paython Notebook: Thola izandla nge-dataFeery DataFRAMS Isibonelo Sokubhala
3. Tutorials step-by-step:

Umbhali: UJeff Nelson, Unjiniyela Wokuhola Konjiniyela