Photo by Editor
# Introduction
Feature engineering it's a critical process in data science and machine learning workflows, as well as in any AI program in general. It involves the construction of meaningful variable definitions from raw – and often dirty – data. The processes behind feature engineering can be very simple or overly complex, depending on the volume, structure, and diversity of the dataset and the objectives of the machine learning model. While the most popular Python libraries for data manipulation and modeling, such as Pandas again scikit-learnenable basic and balanced feature engineering to some extent, there are specialized libraries that go a long way in dealing with large data sets and complex automated transformations, yet they are generally unknown to many.
This article lists 7 under-the-radar Python libraries that push the boundaries of feature engineering processes at scale.
# 1. Acceleration with NVTabular
First, we have NVIDIA-Merlin's NVTabular: a library designed to apply pre-processing and feature engineering to existing datasets — yes, you guessed it! – table. Its unique feature is its GPU-accelerated approach built to easily handle the very large-scale datasets required to train deep learning models. The library is specifically designed to help scale pipelines for recommender system engines based on deep neural networks (DNNs).
# 2. Automation with FeatureTools
FeatureToolsdesigned by Alteryx, focuses on automating the process of feature engineering. This library uses deep feature synthesis (DFS), an algorithm that creates new, “deep” features when analyzing relationships statistically. The library can be used for both relational and time series data, making it possible for both to generate complex features with minimal coding burden.
This code snippet shows an example of how DFS is used with featuretools the library looks like, in the customer dataset:
customers_df = pd.DataFrame({'customer_id': [101, 102]})
es = es.add_dataframe(
dataframe_name="customers",
dataframe=customers_df,
index="customer_id"
)
es = es.add_relationship(
parent_dataframe_name="customers",
parent_column_name="customer_id",
child_dataframe_name="transactions",
child_column_name="customer_id"
)# 3. Compatibility with Dask
Dask increasing its popularity as a library to make parallel Python computations faster and easier. The main recipe behind Dask is to scale traditional Pandas and scikit-learn feature changes using cluster-based calculations, thus enabling fast and affordable feature engineering pipelines on large data sets that would otherwise consume memory.
This article demonstrates a practical Dask workflow for performing data preprocessing.
# 4. Developing with Polars
Competing with Dask in terms of growing popularity, and Pandas vying for a place in the Python data science platform, Timber: A Rust-based data framework library that uses a lazy expression API and lazy calculations to perform efficient, unmanaged feature engineering and transformation on very large datasets. Considered by many to be Panda's most efficient counterpart, Polars is very easy to learn and get used to if you are familiar with Pandas.
Interested in knowing more about Polars? This article shows Polars a few one-liners for common data science tasks, including feature engineering.
# 5. Ending with a Deal
A feast is an open source library considered as a feature store, which helps to bring structured data sources to the production level or production-ready AI applications at scale, especially those based on large-scale linguistic models (LLMs), both for model training and index operations. One of its attractions includes ensuring consistency between the two sections: training and the description of the product. Its use as a feature store is closely tied to engineering practices, i.e. by using it in conjunction with other open source frameworks, for example, denormalized.
# 6. Extracting with tsfresh
Shifting focus to large time series datasets, we have tsfresh library, with a package that specializes in scalable feature extraction. From statistical to visual features, this library is able to compute up to hundreds of logical features in large time series, as well as apply adaptive filtering, which includes, as its name suggests, filtering features in accordance with the machine learning process.
This example code snippet takes a DataFrame which contains a time series dataset that has been previously windowed, and is running tsfresh attribute domain to:
features_rolled = extract_features(
rolled_df,
column_id='id',
column_sort="time",
default_fc_parameters=settings,
n_jobs=0
)# 7. Moving with the river
Let's finish dipping our toes in the river basin (pun intended), with The river library, designed to simplify machine learning workflows on the Internet. As part of its functionality, it has the ability to allow the modification of the online or streaming feature and the included learning methods. This can help effectively deal with problems such as unlimited data and a sense of productivity drift. River is designed to robustly address problems that rarely occur in batch machine learning applications, such as the appearance and disappearance of data features over time.
# Wrapping up
This article has listed 7 notable Python libraries that can help make feature engineering processes more scalable. Some of them are specifically focused on providing engineering methods for a different feature, while others can be used to further support feature engineering activities in certain situations, in conjunction with other agencies.
Iván Palomares Carrascosa is a leader, author, speaker, and consultant in AI, machine learning, deep learning and LLMs. He trains and guides others in using AI in the real world.
