Creating a Modern Data Location in Google Cloud by Apache Iceberg and Apache Spark

Supported content

The state of a large analysis area is constantly evolving, with tangible materials. In the heart of this revolution, the table formats are open as Apache Iceberg and powerful processing engines as Apache sparks, all are powered with a strong for Google's cloud infrastructure.

Apache Iceberg Resurrection: Data Lakes Stock Potter

Years, data lakes, usually constructed in a cloud of property such as Google Cloud storage (GCS), provided uniformly lack of cost. However, they are often lacking important aspects that are available in the final database, such as the harmony of exchanging, the appearance of Schema, and performance performance questions. This is where Apache Iceberg shines.

Apache Iceberg is an open table format designed to deal with these restrictions. It remains on top of your data files (such as parquet, orc, or avido) to the Metadata layer that converts a very functional set, SQL-Like Table. Here's what makes the iceberg have a great power:

• Following Acid: The Iceberg brings an amomolicity, consistency, division, and stability (acid) buildings in your data pond. This means that data writing is a transaction, to ensure data integrity and even function. No more part or relevant writing.
• Schema evolution: One of the big pain points in traditional data pools control schema changes. The iceberg move a schema outside the seams, allowing you to add, slow, reboot, or re-organize the column without re-writing basic data. This is important for the development of old data.
• Hidden Division: The iceberg is smarter with claying partition, to remove the formation of your data. Users no longer need to know how to classify effective questions, and you can change your separating strategy later without data immigration.
• Time and Recovery: The iceberg keeps the complete history of table abbreviations. This enables “the time to go”, allowing you to ask the data as they were at any time in the past. It also provides expanding skills, allowing you to return the table to a good priority, which is very important for redemption and data recovery.
• Effective performance: The rich Metadata of the Iceberg allows questions to focus on unsuitable data files and proper classification, accelerate the killing of the question. It avoids the installation of expensive files list, skip directly to the relevant data based on its metadata.

By providing these data features such as data on the data pool, the iceberg Apache gives the construction of frronity “the cost of the cost and operation of the cost of cloudy.

Biglake Cloud tables in Apache Iceberg in the Iceberg in Bigberg offered a fully managed tablet similar to the common tables. Support features include:

• Table Dections with GoogleSQL Deletitive Data Data (DML)
• A batch of united batch and highest distribution using storage Write API through the plastermy connectors such as Sparks
• Iceberg V2 snapshot Export and auto update to each table changes
• SCHEMA evolution renewing Colomum Metadata
• Automatic storage usage
• Sound Time With Soundical Data Access
• Cololm – Level Elevent and Data Masking

Here is an example of how to build a Biglake Iceberg Table using GoogleSSSQL:

SQL

CREATE TABLE PROJECT_ID.DATASET_ID.my_iceberg_table (
  name STRING,
  id INT64
)
WITH CONNECTION PROJECT_ID.REGION.CONNECTION_ID
OPTIONS (
file_format="PARQUET"
table_format="ICEBERG"
storage_uri = 'gs://BUCKET/PATH');

You can import data to data using LOAD INTO importing data from a file or INSERT INTO from one table.

SQL

# Load from file
LOAD DATA INTO PROJECT_ID.DATASET_ID.my_iceberg_table
FROM FILES (
uris=['gs://bucket/path/to/data'],
format="PARQUET");

# Load from table
INSERT INTO PROJECT_ID.DATASET_ID.my_iceberg_table
SELECT name, id
FROM PROJECT_ID.DATASET_ID.source_table

In addition to a full-owned contribution, Apache icberg is also supported as a study table in Benery. Use this to identify the existing method with data files.

SQL

CREATE OR REPLACE EXTERNAL TABLE PROJECT_ID.DATASET_ID.my_external_iceberg_table
WITH CONNECTION PROJECT_ID.REGION.CONNECTION_ID
OPTIONS (
  format="ICEBERG",
  uris =
    ['gs://BUCKET/PATH/TO/DATA'],
  require_partition_filter = FALSE);

Apache Spark: The Lathouse Analytics Data Engine

While Apache The Iceberg provides a structure and management of your Lathouse data, Apache spark engine is a life-giving engine. Spark is a powerful source of open, a distributed processing system that is famous for their speed, international fluctuations, and the ability to handle large amounts of large data. Sparks In-Memory In-Memory, Robst Ecosystem Includes ML and SQL-based operations, the deepest support of Iceberg makes it a very good decision.

Apache sparks are mostly integrated into Google Cloud Ecosystem. Benefits of Apache Spark on Google Cloud:

• To Access the Real feature spark experience without group management using Google Cloud Server for Apache Spark.
• The full-owned spark experience with configuration of changing groups and the DataProc management.
• Prompt Spark jobs use a new feature of the Apache Spark testing lightning.
• Prepare your running time with GPU and drivers Preinsaled.
• Run AI / ML activities using a strong set of libraries available in spark times, including XGBOost, Pytorch and converts.
• Write a Pyspark code directly within the Breequery studio with Colab Enterprise Boothen and Gemini-Powered Pyspark Code Generation.
• Connect easily on your personal information in the indigenous tables, BigLake Iceberg tables, external tables and GCS
• Compilations with Vertex AI of Mlops-to-End

Iceberg + spark: it's better together

Together, the Iceberg and spark creates a powerful combination of building a place to combat reliable data. Spark can earn the ICEberg's Metadata profits to perform effective data systems, make effective data trees, and ensure the agreement of transactions in your data pool.

Your tables in the Iceberg and biggest native tables are available in BigLake metalore. This discloses your tables to open the source engines in the highest matching, including spark.

Python

from pyspark.sql import SparkSession

# Create a spark session
spark = SparkSession.builder 
.appName("BigLake Metastore Iceberg") 
.config("spark.sql.catalog.CATALOG_NAME", "org.apache.iceberg.spark.SparkCatalog") 
.config("spark.sql.catalog.CATALOG_NAME.catalog-impl", "org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog") 
.config("spark.sql.catalog.CATALOG_NAME.gcp_project", "PROJECT_ID") 
.config("spark.sql.catalog.CATALOG_NAME.gcp_location", "LOCATION") 
.config("spark.sql.catalog.CATALOG_NAME.warehouse", "WAREHOUSE_DIRECTORY") 
.getOrCreate()
spark.conf.set("viewsEnabled","true")

# Use the blms_catalog
spark.sql("USE `CATALOG_NAME`;")
spark.sql("USE NAMESPACE DATASET_NAME;")

# Configure spark for temp results
spark.sql("CREATE namespace if not exists MATERIALIZATION_NAMESPACE");
spark.conf.set("materializationDataset","MATERIALIZATION_NAMESPACE")

# List the tables in the dataset
df = spark.sql("SHOW TABLES;")
df.show();

# Query the tables
sql = """SELECT * FROM DATASET_NAME.TABLE_NAME"""
df = spark.read.format("bigquery").load(sql)
df.show()
sql = """SELECT * FROM DATASET_NAME.ICEBERG_TABLE_NAME"""
df = spark.read.format("bigquery").load(sql)
df.show()

sql = """SELECT * FROM DATASET_NAME.READONLY_ICEBERG_TABLE_NAME"""
df = spark.read.format("bigquery").load(sql)
df.show()

Expanding BigLake Metastore performance is the Iceberg resting-up-to-date access to the ICEberg data in any data processing engine. Here's how you can connect to it using Spark:

Python

import google.auth
from google.auth.transport.requests import Request
from google.oauth2 import service_account
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog = ""
spark = SparkSession.builder.appName("") 
    .config("spark.sql.defaultCatalog", catalog) 
    .config(f"spark.sql.catalog.{catalog}", "org.apache.iceberg.spark.SparkCatalog") 
    .config(f"spark.sql.catalog.{catalog}.type", "rest") 
    .config(f"spark.sql.catalog.{catalog}.uri",
" 
    .config(f"spark.sql.catalog.{catalog}.warehouse", "gs://") 
    .config(f"spark.sql.catalog.{catalog}.token", "") 
    .config(f"spark.sql.catalog.{catalog}.oauth2-server-uri", "                    .config(f"spark.sql.catalog.{catalog}.header.x-goog-user-project", "")      .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") 
.config(f"spark.sql.catalog.{catalog}.io-impl","org.apache.iceberg.hadoop.HadoopFileIO")     .config(f"spark.sql.catalog.{catalog}.rest-metrics-reporting-enabled", "false") 
.getOrCreate()

Laking Lakehouse

Google Cloud provides suite full Suite for Apache Iceberg and Apache Sparks, to make it possible, manage, and measure your open data while you put a lot of open-open technology while putting multiple open technology while you put a lot of open-open technology while putting multiple open technology while putting multiple open technology while putting multiple open technology while you set many open technology while you put a lot of open technology while you set a lot of open technology while you set many open technology while you set a lot of open technology while you set many open technology while you set many open technology while you set many open technology.

• The All Data Catalog: Data data catalog provides integrated data management, monitoring, and displaying data on all data ponds, data warehouse, and data day. It meets Biplontarore, to ensure that administrative policies apply regularly in your Iceberg tables, as well as enabling skills such as semantic search, data list, and data quality checks.
• The Service held by Google Cloud of Apache Kafka: Run Kafka clusters held full in Google Cloud, including Kofka Connect. Data streams can be read directly into bobey, including iceberg tables controlled with lower latency readings.
• Former composer: Full-owned-held service is a full-owned operating area of ​​Apache Airflow.
• VERTEX AI: Use VerTex AI to handle the full ML SPS experience. You can also use the vertex Ai workbench controlling Jussyterlab experience to connect to your non-defective spark and data data conditions.

Store

The combination of Apache Iceberg and Apache Sparks in Google Cloud provides a compelling solution to create today's residence, working well. The Iceberg provides an exchange of exchange, the appearance of Schema, and efficient performance lost in data ponds, while the spark provides a variable engine and consideration of these large scriptures.

To learn more, check our free Webinar on July 8th at 11am PST where we will get into the use of Apache sparks with Google's class.

The author: Brad Miro, Top Developer's lawyer – Google