In this lesson, we show that the Pipeline for fully functional and full Modar is using fully using Thaweas Library, without leaning from the signal. It includes Lilac Dataset management skills with a valid Pardigm Pythigm to create a pure job movement, more. From the project searches and generate a logical sample data for issuing the deleted information and export, the lesson emphasizes the reuse, the data structures. Working resources, such as a pipeline, map_of, and filtering_by, are used to form an announcement, while the panda helps the model modification and quality analysis.
!pip install lilac[all] pandas numpyTo get started, we install the required libraries using the command! Pip to add lilac[all] Pandas Numpy. This ensures that we have a full liite of lilac beside pandas and numpy to manage smooth and evaluation data. We should work this in our writing book before continuing.
import json
import uuid
import pandas as pd
from pathlib import Path
from typing import List, Dict, Any, Tuple, Optional
from functools import reduce, partial
import lilac as llWe import all the important libraries. This includes JSON and UUID management and generating unique project names, operating pandas and data in the form of tabar, and how from Pathlib for managing directions. We also make plans for the type of advanced clarification of clarity and functools of nuts cleaning patterns. Finally, we take a core lilac library as a ll to handle our datasets.
def pipe(*functions):
"""Compose functions left to right (pipe operator)"""
return lambda x: reduce(lambda acc, f: f(acc), functions, x)
def map_over(func, iterable):
"""Functional map wrapper"""
return list(map(func, iterable))
def filter_by(predicate, iterable):
"""Functional filter wrapper"""
return list(filter(predicate, iterable))
def create_sample_data() -> List[Dict[str, Any]]:
"""Generate realistic sample data for analysis"""
return [
{"id": 1, "text": "What is machine learning?", "category": "tech", "score": 0.9, "tokens": 5},
{"id": 2, "text": "Machine learning is AI subset", "category": "tech", "score": 0.8, "tokens": 6},
{"id": 3, "text": "Contact support for help", "category": "support", "score": 0.7, "tokens": 4},
{"id": 4, "text": "What is machine learning?", "category": "tech", "score": 0.9, "tokens": 5},
{"id": 5, "text": "Deep learning neural networks", "category": "tech", "score": 0.85, "tokens": 4},
{"id": 6, "text": "How to optimize models?", "category": "tech", "score": 0.75, "tokens": 5},
{"id": 7, "text": "Performance tuning guide", "category": "guide", "score": 0.6, "tokens": 3},
{"id": 8, "text": "Advanced optimization techniques", "category": "tech", "score": 0.95, "tokens": 3},
{"id": 9, "text": "Gradient descent algorithm", "category": "tech", "score": 0.88, "tokens": 3},
{"id": 10, "text": "Model evaluation metrics", "category": "tech", "score": 0.82, "tokens": 3},
]In this section, it describes practical resources. Pipeline work helps chains conversion, while Map_over and filter_by allows us to change or filter unused data by working. After that, we build a sample data that adjusts Real-World records, containing fields such as text, category, and tokens, will automatically refine Lilac data skills.
def setup_lilac_project(project_name: str) -> str:
"""Initialize Lilac project directory"""
project_dir = f"./{project_name}-{uuid.uuid4().hex[:6]}"
Path(project_dir).mkdir(exist_ok=True)
ll.set_project_dir(project_dir)
return project_dir
def create_dataset_from_data(name: str, data: List[Dict]) -> ll.Dataset:
"""Create Lilac dataset from data"""
data_file = f"{name}.jsonl"
with open(data_file, 'w') as f:
for item in data:
f.write(json.dumps(item) + 'n')
config = ll.DatasetConfig(
namespace="tutorial",
name=name,
source=ll.sources.JSONSource(filepaths=[data_file])
)
return ll.create_dataset(config)For the Setworking_Lilac_projectter, we start a unique directory of operation of our Lilac project and registered using the API of Lilac. Using Card_Dataset_from_Data, we turn our green dictionary list to .jsonl file and build a lilac dataset by describing its configuration. This adjusts clean and formal analysis data.
def extract_dataframe(dataset: ll.Dataset, fields: List[str]) -> pd.DataFrame:
"""Extract data as pandas DataFrame"""
return dataset.to_pandas(fields)
def apply_functional_filters(df: pd.DataFrame) -> Dict[str, pd.DataFrame]:
"""Apply various filters and return multiple filtered versions"""
filters = {
'high_score': lambda df: df[df['score'] >= 0.8],
'tech_category': lambda df: df[df['category'] == 'tech'],
'min_tokens': lambda df: df[df['tokens'] >= 4],
'no_duplicates': lambda df: df.drop_duplicates(subset=['text'], keep='first'),
'combined_quality': lambda df: df[(df['score'] >= 0.8) & (df['tokens'] >= 3) & (df['category'] == 'tech')]
}
return {name: filter_func(df.copy()) for name, filter_func in filters.items()}We remove the dataset in Pandas data data using extract_datframe, which allows us to work with the formats selected in a normal way. After that, using Apper_Filtal_filters, it describes and applying a logical set, such as the selection of the highest points, a double-release transfusions, duplicate, to produce many poor data.
def analyze_data_quality(df: pd.DataFrame) -> Dict[str, Any]:
"""Analyze data quality metrics"""
return {
'total_records': len(df),
'unique_texts': df['text'].nunique(),
'duplicate_rate': 1 - (df['text'].nunique() / len(df)),
'avg_score': df['score'].mean(),
'category_distribution': df['category'].value_counts().to_dict(),
'score_distribution': {
'high': len(df[df['score'] >= 0.8]),
'medium': len(df[(df['score'] >= 0.6) & (df['score'] < 0.8)]),
'low': len(df[df['score'] < 0.6])
},
'token_stats': {
'mean': df['tokens'].mean(),
'min': df['tokens'].min(),
'max': df['tokens'].max()
}
}
def create_data_transformations() -> Dict[str, callable]:
"""Create various data transformation functions"""
return {
'normalize_scores': lambda df: df.assign(norm_score=df['score'] / df['score'].max()),
'add_length_category': lambda df: df.assign(
length_cat=pd.cut(df['tokens'], bins=[0, 3, 5, float('inf')], labels=['short', 'medium', 'long'])
),
'add_quality_tier': lambda df: df.assign(
quality_tier=pd.cut(df['score'], bins=[0, 0.6, 0.8, 1.0], labels=['low', 'medium', 'high'])
),
'add_category_rank': lambda df: df.assign(
category_rank=df.groupby('category')['score'].rank(ascending=False)
)
}Testing data data, using analye_Data_quality, which helps measure key records as complete records and unique, classification, and distribution. This gives us a clear picture of the data and trust. We also explain the activities of the Reform Using Crades_Data_Traptions, enabling enhances such as Scound Oounization, class length.
def apply_transformations(df: pd.DataFrame, transform_names: List[str]) -> pd.DataFrame:
"""Apply selected transformations"""
transformations = create_data_transformations()
selected_transforms = [transformations[name] for name in transform_names if name in transformations]
return pipe(*selected_transforms)(df.copy()) if selected_transforms else df
def export_filtered_data(filtered_datasets: Dict[str, pd.DataFrame], output_dir: str) -> None:
"""Export filtered datasets to files"""
Path(output_dir).mkdir(exist_ok=True)
for name, df in filtered_datasets.items():
output_file = Path(output_dir) / f"{name}_filtered.jsonl"
with open(output_file, 'w') as f:
for _, row in df.iterrows():
f.write(json.dumps(row.to_dict()) + 'n')
print(f"Exported {len(df)} records to {output_file}")Then, through the Apperson_Tranchhorvations, we use the required selection of valid transformation, to ensure that our data is advisable and organized. On the filtering, we use Export_Filterled_Data to write each separate data from a separate file .JSsonl. This enables us to store subsets, such as high quality installation or recordings that may be repeated, in a planned manner of low use.
def main_analysis_pipeline():
"""Main analysis pipeline demonstrating functional approach"""
print("🚀 Setting up Lilac project...")
project_dir = setup_lilac_project("advanced_tutorial")
print("📊 Creating sample dataset...")
sample_data = create_sample_data()
dataset = create_dataset_from_data("sample_data", sample_data)
print("📋 Extracting data...")
df = extract_dataframe(dataset, ['id', 'text', 'category', 'score', 'tokens'])
print("🔍 Analyzing data quality...")
quality_report = analyze_data_quality(df)
print(f"Original data: {quality_report['total_records']} records")
print(f"Duplicates: {quality_report['duplicate_rate']:.1%}")
print(f"Average score: {quality_report['avg_score']:.2f}")
print("🔄 Applying transformations...")
transformed_df = apply_transformations(df, ['normalize_scores', 'add_length_category', 'add_quality_tier'])
print("🎯 Applying filters...")
filtered_datasets = apply_functional_filters(transformed_df)
print("n📈 Filter Results:")
for name, filtered_df in filtered_datasets.items():
print(f" {name}: {len(filtered_df)} records")
print("💾 Exporting filtered datasets...")
export_filtered_data(filtered_datasets, f"{project_dir}/exports")
print("n🏆 Top Quality Records:")
best_quality = filtered_datasets['combined_quality'].head(3)
for _, row in best_quality.iterrows():
print(f" • {row['text']} (score: {row['score']}, category: {row['category']})")
return {
'original_data': df,
'transformed_data': transformed_df,
'filtered_data': filtered_datasets,
'quality_report': quality_report
}
if __name__ == "__main__":
results = main_analysis_pipeline()
print("n✅ Analysis complete! Check the exports folder for filtered datasets.")Finally, in Main_NaySis_Pipeline, we remove full operating flow, from the data export flow, showing how the Lilac, how integrated with the active program, allow us to make effective, synonyms and synthesical. We also print high quality entries as a quick summary. This function represents our full loop of data, which is powered by lilac.
In conclusion, users will be receiving understanding of the data pipe that will promote Dataset's Dataset's Dataset and applicable analysis patterns, clean analysis. The pipe includes all sensitive categories, including creating data data, modification, filters, quality analysis, and export, provides assessment and distribution. It also shows how to embed a purpose medadata such as normal scores, level tiers, lengths, which can have money in good jobs or personal reviews.
Look Codes. All credit for this study goes to research for this project. Also, feel free to follow it Sane and don't forget to join ours 100K + ml subreddit Then sign up for Our newspaper.
Nikhil is a student of students in MarktechPost. Pursuing integrated graduates combined in the Indian Institute of Technology, Kharagpur. Nikhl is a UI / ML enthusiasm that searches for applications such as biomoutomostoments and biomedical science. After a solid in the Material Science, he examines new development and developing opportunities to contribute.
