In [ ]:
Copied!
import marimo as mo
import marimo as mo
Intro to the core interfaces of mloda¶
mloda is a robust and flexible data framework tailored for professionals to efficiently manage data and feature engineering. It enables users to abstract processes away from data, in contrast to the current industry setup where processes are usually bound to specific data sets.
This introductory notebook provides a practical demonstration of how MLoda helps machine learning data workflows by emphasizing data processes over raw data manipulation.
- It begins by loading data from various sources, such as order, payment, location, and categorical datasets.
- Next, we showcase mloda's versatility in handling diverse compute frameworks, including PyArrow tables and Pandas DataFrames.
- Then we leverage mloda's advanced capabilities to integrate data from various sources into cohesive and unified feature sets (details on feature sets are covered in chapter 3).
Finally, we will conclude by discussing the broader implications of what was done.
In [ ]:
Copied!
# Load all available plugins into the python environment
from mloda.user import PluginLoader
plugin_loader = PluginLoader.all()
# Since there are potentially many plugins loaded, we'll focus on specific categories for clarity.
# Here, we demonstrate by listing the available 'read' and 'sql' plugins.
print(plugin_loader.list_loaded_modules("read"))
print(plugin_loader.list_loaded_modules("sql"))
# Load all available plugins into the python environment
from mloda.user import PluginLoader
plugin_loader = PluginLoader.all()
# Since there are potentially many plugins loaded, we'll focus on specific categories for clarity.
# Here, we demonstrate by listing the available 'read' and 'sql' plugins.
print(plugin_loader.list_loaded_modules("read"))
print(plugin_loader.list_loaded_modules("sql"))
['mloda_plugins.feature_group.experimental.llm.tools.available.read_file_tool', 'mloda_plugins.feature_group.input_data.read_context_files', 'mloda_plugins.feature_group.input_data.read_file', 'mloda_plugins.feature_group.input_data.read_db_feature', 'mloda_plugins.feature_group.input_data.read_document', 'mloda_plugins.feature_group.input_data.read_db', 'mloda_plugins.feature_group.input_data.read_file_feature', 'mloda_plugins.feature_group.input_data.read_document_feature', 'mloda_plugins.feature_group.input_data.read_files.markdown_document_reader', 'mloda_plugins.feature_group.input_data.read_files.json', 'mloda_plugins.feature_group.input_data.read_files.csv', 'mloda_plugins.feature_group.input_data.read_files.json_document_reader', 'mloda_plugins.feature_group.input_data.read_files.parquet', 'mloda_plugins.feature_group.input_data.read_files.feather', 'mloda_plugins.feature_group.input_data.read_files.text_file_reader', 'mloda_plugins.feature_group.input_data.read_files.orc', 'mloda_plugins.feature_group.input_data.read_files.yaml_document_reader', 'mloda_plugins.feature_group.input_data.read_dbs.sqlite'] ['mloda_plugins.feature_group.input_data.read_dbs.sqlite', 'mloda_plugins.compute_framework.base_implementations.sqlite.sqlite_filter_engine', 'mloda_plugins.compute_framework.base_implementations.sqlite.sqlite_framework', 'mloda_plugins.compute_framework.base_implementations.sqlite.sqlite_pyarrow_transformer', 'mloda_plugins.compute_framework.base_implementations.sqlite.sqlite_relation', 'mloda_plugins.compute_framework.base_implementations.sqlite.sqlite_merge_engine', 'mloda_plugins.compute_framework.base_implementations.sqlite.sqlite_mask_engine', 'mloda_plugins.compute_framework.base_implementations.sql.sql_window', 'mloda_plugins.compute_framework.base_implementations.sql.sql_base_pyarrow_transformer', 'mloda_plugins.compute_framework.base_implementations.sql.sql_base_merge_engine', 'mloda_plugins.compute_framework.base_implementations.sql.sql_base_mask_engine', 'mloda_plugins.compute_framework.base_implementations.sql.sql_utils', 'mloda_plugins.compute_framework.base_implementations.sql.sql_base_filter_engine']
In [ ]:
Copied!
# Optional!
# We use synthetic dummy data to demonstrate the basic usage.
# You can run this cell in your own jupyter notebook.
# They are however not relevant for further understanding.
#
# from examples.mloda_basics import create_synthetic_data
# create_synthetic_data.create_ml_lifecylce_data()
# Optional!
# We use synthetic dummy data to demonstrate the basic usage.
# You can run this cell in your own jupyter notebook.
# They are however not relevant for further understanding.
#
# from examples.mloda_basics import create_synthetic_data
# create_synthetic_data.create_ml_lifecylce_data()
We should see 4 files in the base_data folder. One sqlite example for a db and 3 different file formats.
Now we want to load the data to look at the content, so we can look at the data.
In [ ]:
Copied!
# Step 1: We want to load typical order information like order_id, product_id, quantity, and item_price.
from mloda.user import Feature
order_features: list[str | Feature] = ["order_id", "product_id", "quantity", "item_price"]
payment_features: list[str | Feature] = ["payment_id", "payment_type", "payment_status", "valid_datetime"]
location_features: list[str | Feature] = ["user_location", "merchant_location", "update_date"]
categorical_features: list[str | Feature] = ["user_age_group", "product_category", "transaction_type"]
# Step 1: We want to load typical order information like order_id, product_id, quantity, and item_price.
from mloda.user import Feature
order_features: list[str | Feature] = ["order_id", "product_id", "quantity", "item_price"]
payment_features: list[str | Feature] = ["payment_id", "payment_type", "payment_status", "valid_datetime"]
location_features: list[str | Feature] = ["user_location", "merchant_location", "update_date"]
categorical_features: list[str | Feature] = ["user_age_group", "product_category", "transaction_type"]
In [ ]:
Copied!
# Step 2: We specify the data sources to load
import os
from mloda.user import DataAccessCollection
from mloda_plugins.feature_group.input_data.read_dbs.sqlite import SQLITEReader
# Initialize a DataAccessCollection object
data_access_collection = DataAccessCollection()
# Define the folders containing the data
# Note: We use two paths to accommodate different possible root locations as it depends where the code is executed.
base_data_path = os.path.join(os.getcwd(), "docs", "docs", "examples", "mloda_basics", "base_data")
if not os.path.exists(base_data_path):
base_data_path = os.path.join(os.getcwd(), "base_data")
# Add the folder to the DataAccessCollection
data_access_collection.add_folder(base_data_path)
# As a db cannot work with a folder, we need to add a connection for the db.
data_access_collection.add_credentials({SQLITEReader.db_path(): os.path.join(base_data_path, "example.sqlite")})
# Step 2: We specify the data sources to load
import os
from mloda.user import DataAccessCollection
from mloda_plugins.feature_group.input_data.read_dbs.sqlite import SQLITEReader
# Initialize a DataAccessCollection object
data_access_collection = DataAccessCollection()
# Define the folders containing the data
# Note: We use two paths to accommodate different possible root locations as it depends where the code is executed.
base_data_path = os.path.join(os.getcwd(), "docs", "docs", "examples", "mloda_basics", "base_data")
if not os.path.exists(base_data_path):
base_data_path = os.path.join(os.getcwd(), "base_data")
# Add the folder to the DataAccessCollection
data_access_collection.add_folder(base_data_path)
# As a db cannot work with a folder, we need to add a connection for the db.
data_access_collection.add_credentials({SQLITEReader.db_path(): os.path.join(base_data_path, "example.sqlite")})
In [ ]:
Copied!
# Step 3: Request Data Using the Defined Access Collection and Desired Features
from mloda.user import mloda
from mloda_plugins.compute_framework.base_implementations.pyarrow.table import PyArrowTable
all_features = order_features + payment_features + location_features + categorical_features
_result = mloda.run_all(
all_features, data_access_collection=data_access_collection, compute_frameworks={PyArrowTable}
)
for _data in _result:
# Retrieve data based on the specified feature list and access collection
# Display the first five entries of each result table and its type
print(_data[:2], type(_data))
# Step 3: Request Data Using the Defined Access Collection and Desired Features
from mloda.user import mloda
from mloda_plugins.compute_framework.base_implementations.pyarrow.table import PyArrowTable
all_features = order_features + payment_features + location_features + categorical_features
_result = mloda.run_all(
all_features, data_access_collection=data_access_collection, compute_frameworks={PyArrowTable}
)
for _data in _result:
# Retrieve data based on the specified feature list and access collection
# Display the first five entries of each result table and its type
print(_data[:2], type(_data))
pyarrow.Table user_location: string update_date: int64 merchant_location: string ---- user_location: [["East","West"]] update_date: [[1640995200000,1641632290909]] merchant_location: [["North","East"]] <class 'pyarrow.lib.Table'> pyarrow.Table order_id: int64 product_id: int64 item_price: double quantity: int64 ---- order_id: [[1,2]] product_id: [[282,355]] item_price: [[74.86,154.56]] quantity: [[6,2]] <class 'pyarrow.lib.Table'> pyarrow.Table valid_datetime: timestamp[ns, tz=UTC] payment_status: string payment_type: string payment_id: int64 ---- valid_datetime: [[2024-01-11 23:01:49.090909090Z,2024-01-15 09:41:49.090909090Z]] payment_status: [["failed","pending"]] payment_type: [["debit card","debit card"]] payment_id: [[1,2]] <class 'pyarrow.lib.Table'> pyarrow.Table transaction_type: string product_category: string user_age_group: string ---- transaction_type: [["online","online"]] product_category: [["clothing","home"]] user_age_group: [["26-35","26-35"]] <class 'pyarrow.lib.Table'>
In [ ]:
Copied!
# The data is initially loaded as a Pyarrow table. However, we can easily load it also as a PandasDataFrame.
from mloda_plugins.compute_framework.base_implementations.pandas.dataframe import PandasDataFrame
_result = mloda.run_all(
all_features, data_access_collection=data_access_collection, compute_frameworks={PandasDataFrame}
)
# Request data using the Pandas compute framework
for _data in _result:
# Display the first five entries of each result table and its type
print(_data[:2], type(_data))
# The data is initially loaded as a Pyarrow table. However, we can easily load it also as a PandasDataFrame.
from mloda_plugins.compute_framework.base_implementations.pandas.dataframe import PandasDataFrame
_result = mloda.run_all(
all_features, data_access_collection=data_access_collection, compute_frameworks={PandasDataFrame}
)
# Request data using the Pandas compute framework
for _data in _result:
# Display the first five entries of each result table and its type
print(_data[:2], type(_data))
user_location update_date merchant_location
0 East 1640995200000 North
1 West 1641632290909 East <class 'pandas.core.frame.DataFrame'>
order_id product_id item_price quantity
0 1 282 74.86 6
1 2 355 154.56 2 <class 'pandas.core.frame.DataFrame'>
valid_datetime payment_status payment_type payment_id
0 2024-01-11 23:01:49.090909090+00:00 failed debit card 1
1 2024-01-15 09:41:49.090909090+00:00 pending debit card 2 <class 'pandas.core.frame.DataFrame'>
transaction_type product_category user_age_group
0 online clothing 26-35
1 online home 26-35 <class 'pandas.core.frame.DataFrame'>
In [ ]:
Copied!
# Define features with specific compute frameworks
order_id = Feature(name="order_id", compute_framework="PandasDataFrame")
product_id = Feature(name="product_id", compute_framework="PyArrowTable")
specific_framework_feature_list: list[Feature | str] = [order_id, product_id]
_result = mloda.run_all(specific_framework_feature_list, data_access_collection=data_access_collection)
# Request data for the defined features
for res in _result:
print("The resulting data structure differs based on the compute framework:")
# Display the first few rows and data types of the results
print("\n", res[:3], type(res))
# Define features with specific compute frameworks
order_id = Feature(name="order_id", compute_framework="PandasDataFrame")
product_id = Feature(name="product_id", compute_framework="PyArrowTable")
specific_framework_feature_list: list[Feature | str] = [order_id, product_id]
_result = mloda.run_all(specific_framework_feature_list, data_access_collection=data_access_collection)
# Request data for the defined features
for res in _result:
print("The resulting data structure differs based on the compute framework:")
# Display the first few rows and data types of the results
print("\n", res[:3], type(res))
The resulting data structure differs based on the compute framework:
order_id
0 1
1 2
2 3 <class 'pandas.core.frame.DataFrame'>
The resulting data structure differs based on the compute framework:
pyarrow.Table
product_id: int64
----
product_id: [[282,355,395]] <class 'pyarrow.lib.Table'>
In [ ]:
Copied!
from typing import Any, Optional
from mloda.provider import FeatureGroup, FeatureSet
from mloda.user import FeatureName, Options, Index, Link, JoinSpec
from mloda_plugins.feature_group.input_data.read_file_feature import ReadFileFeature
index = Index(("order_id",))
class ReadFileFeatureJoin(ReadFileFeature):
@classmethod
def index_columns(cls) -> Optional[list[Index]]:
return [index]
link = Link.inner(JoinSpec(ReadFileFeatureJoin, index), JoinSpec(ReadFileFeatureJoin, index))
class ExampleMlLifeCycleJoin(FeatureGroup):
def input_features(self, options: Options, feature_name: FeatureName) -> Optional[set[Feature]]:
quantity = Feature(name="quantity", compute_framework="PandasDataFrame")
product_id = Feature(name="product_id", compute_framework="PyArrowTable")
return {product_id, quantity}
@classmethod
def calculate_feature(cls, data: Any, features: FeatureSet) -> Any:
print(
"Data from two different sources is now combined into one feature within one data technology: \n",
data,
type(data),
"\n",
)
return {"ExampleMlLifeCycleJoin": [1, 2, 3]}
_result = mloda.run_all(["ExampleMlLifeCycleJoin"], data_access_collection=data_access_collection, links={link})
print(
"Final result: ",
_result[0],
"\nNote: As no specific compute framework was defined for the result, the output could be in either format.",
)
from typing import Any, Optional
from mloda.provider import FeatureGroup, FeatureSet
from mloda.user import FeatureName, Options, Index, Link, JoinSpec
from mloda_plugins.feature_group.input_data.read_file_feature import ReadFileFeature
index = Index(("order_id",))
class ReadFileFeatureJoin(ReadFileFeature):
@classmethod
def index_columns(cls) -> Optional[list[Index]]:
return [index]
link = Link.inner(JoinSpec(ReadFileFeatureJoin, index), JoinSpec(ReadFileFeatureJoin, index))
class ExampleMlLifeCycleJoin(FeatureGroup):
def input_features(self, options: Options, feature_name: FeatureName) -> Optional[set[Feature]]:
quantity = Feature(name="quantity", compute_framework="PandasDataFrame")
product_id = Feature(name="product_id", compute_framework="PyArrowTable")
return {product_id, quantity}
@classmethod
def calculate_feature(cls, data: Any, features: FeatureSet) -> Any:
print(
"Data from two different sources is now combined into one feature within one data technology: \n",
data,
type(data),
"\n",
)
return {"ExampleMlLifeCycleJoin": [1, 2, 3]}
_result = mloda.run_all(["ExampleMlLifeCycleJoin"], data_access_collection=data_access_collection, links={link})
print(
"Final result: ",
_result[0],
"\nNote: As no specific compute framework was defined for the result, the output could be in either format.",
)
Data from two different sources is now combined into one feature within one data technology:
order_id quantity product_id
0 1 6 282
1 2 2 355
2 3 4 395
3 4 9 319
4 5 5 275
.. ... ... ...
95 96 4 170
96 97 3 328
97 98 5 361
98 99 5 192
99 100 6 271
[100 rows x 3 columns] <class 'pandas.core.frame.DataFrame'>
Final result: ExampleMlLifeCycleJoin
0 1
1 2
2 3
Note: As no specific compute framework was defined for the result, the output could be in either format.
What Have We Observed So Far?¶
mloda unifies the interfaces for data for various sources, formats and technologies for the definition of the processes and applying the processes on the data. We used the FeatureGroup, the ComputeFramework and mlodaAPI as interfaces.
It integrates with any techologies, e.g. PyArrow and Pandas, enabling flexible tool choices for data processing.
mloda combines data access and computation, reducing complexity and providing a reusable approach to ML workflows. Data Access can be controlled centrally for different sources of data. Here, we showed folders and a database access.
We will further deepen the advantages of the used approach in the next notebook.