Data Access Patterns: BaseInputData vs MatchData

Overview

mloda provides two distinct but complementary patterns for data access: BaseInputData and MatchData. While they may appear similar at first glance, they serve different purposes and are used in different contexts within the framework.

This document clarifies the differences between these concepts and their respective use cases. For practical examples of data access methods, see Data Access Overview.

BaseInputData Pattern

Purpose

BaseInputData is an abstract base class that defines how feature groups load and access data. It's the foundation for data loading mechanisms in mloda.

Key Characteristics

• Data Loading Focus: Primarily concerned with how to load data from various sources
• Feature Group Integration: Used by feature groups through the input_data() method
• Inheritance-Based: Concrete implementations inherit from BaseInputData
• Scope Management: Supports both global and feature-specific data access scopes
• Universal Usage: Used by all feature groups that need to load data

Use Cases

• Reading files (CSV, JSON, Parquet, etc.)
• Connecting to databases
• Creating synthetic/test data
• Loading data from APIs
• Managing data dependencies between features

For detailed examples of these use cases, see the data access documentation.

Example Implementation

from mloda.provider import BaseInputData, FeatureGroup, FeatureSet

class ReadFileFeature(FeatureGroup):
    @classmethod
    def input_data(cls) -> Optional[BaseInputData]:
        return ReadFile()  # BaseInputData implementation

    @classmethod
    def calculate_feature(cls, data: Any, features: FeatureSet) -> Any:
        reader = cls.input_data()
        if reader is not None:
            data = reader.load(features)
            return data
        raise ValueError("Reading file failed.")

Common BaseInputData Implementations

• ReadFile: For structured file-based data loading (see access-feature-data)
• ReadDocument: For unstructured document loading (Markdown, YAML, text). Skips file types owned by ReadFile by default.
• DataCreator: For generating synthetic data (see access-feature-data)
• ApiInputData: For runtime data injection (see access-feature-data)
• ReadDB: For database-backed loading

Writing an input-data reader

Each reader family exposes a recommended hook seam. Overriding load_data wholesale remains supported in every family.

• ReadDB: implement produce_rows, connect, and is_valid_credentials; optionally prepare_credentials and build_query.
• ReadDocument: implement produce_document and suffix; optionally document_file_type.
• ReadFile: override load_data wholesale to return the table. CsvReader resolves to a FileSource descriptor that the target compute framework materializes into its native type.

CSV inference semantics are defined by pyarrow's default CSV reader; the stdlib reader behind PythonDict follows it. Types are inferred per column: null tokens (pyarrow's default set, e.g. NA, NaN, null) become None in an int/float/bool column but stay literal text in a string column, a column of only empty cells and/or null tokens is all-None, and an int column with a value outside signed int64 range degrades entirely to float.

The stdlib reader does not yet cover pyarrow's full surface. Where they differ, a column pyarrow types stays a string column in PythonDict: dates, timestamps and times, whitespace-padded numbers (" 1 "), inf/infinity, uppercase NAN (the float value, not the NaN null token), hex literals (0x1f), and a true/1 mix (pyarrow reads 1/0 as bools too).

Readers are classified structurally; no reader code is executed for classification. is_final_reader() is True when a class overrides load_data wholesale, or when it overrides all hooks named by its family's _final_reader_requires() (for example ("produce_rows", "connect") for ReadDB). Family bases (ReadDB, ReadDocument, ReadFile) are never discovered as final readers themselves.

Warning: classification is structural (declared is overridden), so an intermediate base that re-declares a hook or load_data with a bare raise NotImplementedError body is classified as a final reader and enters discovery. Intermediate bases must not re-declare bare hooks; re-anchor the family by declaring _final_reader_requires instead.

_final_reader_requires is underscore-named but is a stable, documented extension point for third-party reader families.

Selecting among sibling readers

A feature selects a specific reader with an Option whose key equals the reader's BaseInputData.data_access_name(), which defaults to cls.__name__ (unique per class, so sibling readers cannot collide) and which a reader that overrides it keeps unique within its family itself:

Feature("value", options={UbaAirReader.__name__: url})

The reader class itself is also accepted as the key, e.g. Feature("value", options={UbaAirReader: url}); it is normalized to the class-name string when the Options object is constructed, so both forms are one identity.

The matched (ReaderClass, data_access) pair is stored under the reserved "BaseInputData" options key and consumed by init_reader at load time.

For non-file sources such as HTTP endpoints, subclassing ReadFile and overriding match_subclass_data_access plus load_data is a supported pattern; on that path suffix() is never consulted (it is inert). ApiInputData injects in-memory data passed through the API request and is not an HTTP client.

Reader selection vs feature-group resolution

Reader selection answers "which plugin handles this input" the way feature-group resolution answers "which feature group owns this name". It is not a second resolver: it runs nested inside the criteria gate of feature-group resolution, where match_feature_group_criteria calls the reader family's matches(). The two deliberately share no request, environment, or outcome abstractions; they share only the low-level rejection channel described below.

Aspect	Feature-group resolution	Reader selection
Candidate discovery	Registered accessible plugins	Structural walk over the family's final readers (is_final_reader()); no reader code executed
Auto-loading	Up-front plugin loading	Lazy per-family _auto_load_group, triggered only when no final readers are found
Accessibility policy	Strict mode, collector policy, enabled compute frameworks	None: every final reader of the family is a candidate
Matching	Criteria, domain, scope, capability, framework-pin, and links gates	Per-reader file, suffix, column-validation, and pinning rules
Ambiguity	Multiple winners resolved by subclass preference, then reported	First match wins; a second conflicting reader for the same feature raises
Outcome and diagnostics	Structured evaluation result rendered into failure messages	A matched (ReaderClass, data_access) pair written into options; declines surface through the shared rejection channel

Declining with an attributable reason

A reader that owns an input but cannot serve the requested feature can record why it declined; the reason then appears in the near-miss block of the "No feature groups found" error message, labeled (input data). record_match_rejection is exported via mloda.provider. A custom reader owns its own suffix and overrides load_data wholesale; its own decline points sit beyond the automatic column validation, for example a required schema marker in the header:

from typing import Any

from mloda.provider import INPUT_DATA_STAGE, FeatureSet, record_match_rejection
from mloda_plugins.feature_group.input_data.read_file import ReadFile

class SensorCsvReader(ReadFile):
    @classmethod
    def suffix(cls) -> tuple[str, ...]:
        return (".sensorcsv",)

    @classmethod
    def load_data(cls, data_access: Any, features: FeatureSet) -> Any: ...

    @classmethod
    def validate_columns(cls, file_name: str, feature_names: list[str]) -> bool:
        if super().validate_columns(file_name, feature_names) is False:
            return False
        with open(file_name, encoding="utf-8") as handle:
            header = handle.readline()
        if "#sensor-schema" not in header:
            record_match_rejection(
                cls.get_class_name(),
                f"{cls.get_class_name()} matched the suffix of {file_name} "
                f"but its header lacks the #sensor-schema marker",
                stage=INPUT_DATA_STAGE,
            )
            return False
        return True

The recorded decline renders as a near-miss line of the resolution failure:

  - SensorFeatureGroup (input data): SensorCsvReader matched the suffix of /data/run1.sensorcsv but its header lacks the #sensor-schema marker

Rules for reader authors:

• Record only when ownership is established but the content fails: right suffix but a missing column, valid credentials but a declined feature.
• Plain non-matches (wrong suffix, invalid credentials) stay silent. NotImplementedError from is_valid_credentials is a silent non-match; from check_feature_in_data_access it is an accept (the reader matches on credentials alone).
• Never raise to decline: anything but NotImplementedError in the DB match hooks aborts matching for every reader sharing the DataAccessCollection. Record, then return a falsy value.
• Recording outside an engine-opened window is a no-op, so readers stay usable standalone.
• Recorded reasons are discarded at the enclosing candidate level: when the reader ultimately matches, when a sibling reader matches, or, for unowned recordings, when the feature group matches by another rule. An owned veto instead gates the name-based rules (see the paragraph below). Only a decline surfaces them.
• Name the reader and the concrete input in the reason, as the example does. Any label works as the owner name, an overridden data_access_name() included, but it must be distinct among the reader's own decline points: the first recording per owner wins, so a later reason under a name already used in the same window is dropped and never reaches the owned stage.

ReadFile column validation and the ReadDB feature check (check_feature_in_data_access) already record automatically; a custom reader only needs this for its own decline points.

A veto recorded while the user explicitly addressed the reader family (an option key equal to the reader's data_access_name()) gates the candidate's name-based match rules: the feature group fails at resolution with that reason instead of resolving by name and crashing at load time in init_reader. A content decline on that path gates the same way: if the addressed reader records a decline and its probe still matches nothing, the recording counts as owned. A decline followed by a match on another input of the same probe stays discarded as usual. An unowned decline on the global probe stays near-miss material only, and the MatchData rule is not gated.

MatchData Pattern

Purpose

MatchData is a specialized matching mechanism specifically designed for feature groups that require framework connection objects. It determines which data access method should be used when stateful connections are needed.

Key Characteristics

• Connection-Specific: Only used for feature groups that need framework connection objects
• Matching Logic: Determines which data source matches when connections are involved
• Scope Resolution: Resolves conflicts between feature-scope and global-scope data access for stateful frameworks
• Limited Usage: Only applies to specific compute frameworks (like DuckDB) that require persistent connections

When MatchData is Used

MatchData is only used in these specific scenarios: - Feature groups that work with stateful compute frameworks (e.g., DuckDB) - When framework connection objects are required - For data sources that need persistent connections (databases, connection pools)

Use Cases

• Matching DuckDB features to appropriate DuckDB connections
• Routing features to specific database connections based on credentials
• Resolving data access when multiple connection objects are available
• Enabling flexible connection configuration for stateful frameworks

Example Implementation

from mloda.provider import MatchData, FeatureGroup

class DuckDBFeatureGroup(FeatureGroup, MatchData):
    @classmethod
    def match_data_access(
        cls,
        feature_name: str,
        options: Options,
        data_access_collection: Optional[DataAccessCollection] = None,
        framework_connection_object: Optional[Any] = None,
    ) -> Any:
        # Logic to determine if this matcher handles DuckDB connections
        if framework_connection_object and isinstance(framework_connection_object, duckdb.DuckDBPyConnection):
            return framework_connection_object
        if data_access_collection is not None:
            return data_access_collection.resolve(
                "connection",
                predicate=lambda c: isinstance(c, duckdb.DuckDBPyConnection),
                hint=options.get("data_access_handle"),
            )
        return None

Key Differences

Aspect	BaseInputData	MatchData
Primary Purpose	Data loading and access	Connection object matching for stateful frameworks
When Used	All feature groups that load data	Only feature groups requiring framework connection objects
Scope	Universal data access pattern	Specialized for stateful compute frameworks
Usage Pattern	input_data() method in feature groups	Multiple inheritance: FeatureGroup, MatchData
Connection Dependency	Works with or without connections	Specifically designed for connection objects
Framework Support	All compute frameworks	Only stateful frameworks (DuckDB, database connections)

How They Work Together

BaseInputData and MatchData serve different purposes and are used in different scenarios:

BaseInputData Workflow

1. Feature groups define their data loading strategy via input_data() method
2. BaseInputData implementations handle the actual data loading
3. Works with all compute frameworks (stateful and stateless)

MatchData Workflow (Connection-Specific)

1. Only used when feature groups need framework connection objects
2. MatchData determines which connection object to use for stateful frameworks
3. Only applies to specific compute frameworks like DuckDB that require persistent connections

Combined Usage Example

class DuckDBAnalyticsFeature(FeatureGroup, MatchData):
    @classmethod
    def input_data(cls) -> Optional[BaseInputData]:
        # BaseInputData for general data loading
        return ReadFile()

    @classmethod
    def match_data_access(cls, feature_name: str, options: Options,
                         data_access_collection: Optional[DataAccessCollection] = None,
                         framework_connection_object: Optional[Any] = None) -> Any:
        # MatchData for connection object matching
        if framework_connection_object and isinstance(framework_connection_object, duckdb.DuckDBPyConnection):
            return framework_connection_object
        return None

Practical Examples

Scenario 1: Standard File Processing (BaseInputData Only)

Use Case: Reading CSV files with Pandas Pattern: Only BaseInputData is needed

class CsvProcessingFeature(FeatureGroup):
    @classmethod
    def input_data(cls) -> Optional[BaseInputData]:
        return ReadFile()  # BaseInputData handles file reading

Scenario 2: DuckDB Analytics (BaseInputData + MatchData)

Use Case: Analytics with DuckDB requiring connection objects Pattern: Both BaseInputData and MatchData are needed

class DuckDBAnalyticsFeature(FeatureGroup, MatchData):
    @classmethod
    def input_data(cls) -> Optional[BaseInputData]:
        return ReadFile()  # BaseInputData for data loading

    @classmethod
    def match_data_access(cls, ...):
        # MatchData for connection matching
        return appropriate_duckdb_connection

For database connection patterns, see Framework Connection Object.

Scenario 3: In-Memory Processing (BaseInputData Only)

Use Case: Creating synthetic data with Pandas Pattern: Only BaseInputData is needed

class SyntheticDataFeature(FeatureGroup):
    @classmethod
    def input_data(cls) -> Optional[BaseInputData]:
        return DataCreator({"synthetic_data"})  # BaseInputData for data creation

Integration with Other Concepts

Compute Frameworks

• BaseInputData: Works with all compute frameworks
• MatchData: Only works with stateful frameworks requiring connection objects

For more information, see: - Compute Frameworks - Framework Connection Object - Compute Framework Integration

Feature Groups

These patterns are fundamental to how feature groups access data. For more details, see: - Feature Groups - Feature Group Matching

Best Practices

When to Use BaseInputData Only

• Working with stateless compute frameworks (Pandas, PyArrow, Polars)
• File-based data loading
• mloda data injection
• Synthetic data generation
• Most standard data processing scenarios

When to Use BaseInputData + MatchData

• Working with stateful compute frameworks (DuckDB)
• Database connections requiring persistent state
• Connection pooling scenarios
• When framework connection objects are required

Design Considerations

• BaseInputData: Focus on robust data loading, error handling, and performance
• MatchData: Focus on accurate connection matching and state management
• Integration: Use MatchData only when framework connection objects are actually needed

Related Documentation

• (Feature) data - Comprehensive guide to data access in mloda
• Framework Connection Object - Managing stateful connections (essential for understanding MatchData)
• Feature Groups - Introduction to feature groups
• Compute Frameworks - Overview of compute framework system
• Feature Group Matching - How features are matched to implementations

Summary

BaseInputData and MatchData serve different and specialized roles in mloda's data access architecture:

• BaseInputData is the universal pattern for data loading - used by all feature groups that need to load data
• MatchData is a specialized pattern for connection object matching - only used by feature groups that require framework connection objects

Key Understanding: - Most feature groups only use BaseInputData - MatchData is only needed when working with stateful compute frameworks like DuckDB - They are not alternatives - they solve different problems in different contexts

Understanding this distinction is crucial for: - Choosing the right pattern for your use case - Implementing feature groups correctly - Working with stateful vs stateless compute frameworks - Leveraging mloda's connection management capabilities

This separation allows mloda to provide both universal data access (BaseInputData) and specialized connection management (MatchData) while keeping the complexity contained to only those scenarios that actually need it.