Changelog

All notable changes to Moltres will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

1.1.0 - 2026-07-07

Added

  • Public API surface – register_performance_hook / unregister_performance_hook exported from top-level moltres; contract tests in tests/api/test_public_imports.py.

  • Optional extras – moltres[parquet], moltres[fastapi], and moltres[duckdb] install groups.

  • Path sandboxing – MoltresConfig.allowed_paths and validation for file readers to restrict reads to approved directories.

  • Deprecation helpers – moltres.utils._compat utilities for staged API removals.

  • Tiered integration CI – tier2_integration / tier3_integration markers with dedicated CI jobs and xdist load-group scheduling.

  • Onboarding docs – docs/PUBLIC_API.md, contributor install fixes, GitHub issue/PR templates, and warning-free Sphinx builds (-W).

Changed

  • Expressions package – Removed shadowed expressions/functions.py module file; moltres.expressions.functions package is the single implementation.

  • Release workflow – Tag builds verify __init__.py versions match the release tag for both moltres and moltres-core.

Security

  • File read paths – Readers and FileScan builders validate paths against allowed_paths when configured.

Fixed

  • Flaky sampling test – Probabilistic SQL sample() test tolerates engine variance on Windows.

  • Interpreter shutdown cleanup – Database atexit cleanup regression hardened.

  • File reader errors – resolve_read_path(..., must_exist=False) preserves format-specific FileNotFoundError messages for lazy scans.

1.0.1 - 2026-06-07

Breaking

  • Minimum Python is now 3.10 – Required by the pydantable-protocol dependency (e.g. dataclass(slots=True)). Python 3.9 is no longer supported.

  • Django template tag – The query= parameter on {% moltres_query %} is removed. Templates must use table_name only (full-table SELECT). Use a view for filtered or joined queries.

  • Django management command – Arbitrary Python query strings are no longer executed. Use --table with optional --where-column, --where-op, --where-value, and --limit, or the legacy form db.table('name').select() only.

  • Async database inspection – Sync get_table_names() and get_view_names() raise when called on an AsyncDatabase. Use await db.get_table_names() / await db.get_view_names() instead.

Changed

  • moltres-core – Replaced the vendored pydantable_protocol copy with a runtime dependency on pydantable-protocol (≥1.14.0). The moltres_core.embedded_protocol module remains as a thin re-export for backward compatibility.

  • Engine lifecycle – EngineConfig.owns_engine tracks whether Moltres created the SQLAlchemy engine. Database.close() only disposes owned engines and rolls back active transactions first.

  • Streaming inserts – Chunked Records / AsyncRecords inserts run inside a transaction when no transaction is already active, so partial chunk failures roll back the whole insert.

  • dropDuplicates(subset=...) – Subset deduplication now keeps one row per key group (matching PySpark semantics) instead of treating non-key columns incorrectly.

Security

  • Django RCE – Removed eval() from the moltres_query template tag and management command; queries use a safe declarative API (safe_query.py).

  • SQL injection in expressions – Interval literals, strftime/date_format format strings, and join column names are validated or parameterized before compilation.

  • SQL injection in writer – _table_exists() uses quote_identifier() for table names.

Fixed

  • insert_rows() with streaming Records – Detects streaming record sources and delegates to insert_into() instead of materializing incorrectly.

  • Session / transaction connection sharing – connect(session=...) and db.transaction() use the same underlying connection; transaction state is tracked per context via ContextVar.

  • Async cleanup – Async database atexit handlers run real connection cleanup instead of being no-ops.

  • Database._close_resources() – Uses connection_manager.close() for consistent teardown.

  • NULL comparisons – col("x") == None and col("x") != None compile to IS NULL / IS NOT NULL instead of = NULL.

  • isnan() – Compiles to a proper NaN check instead of IS NULL.

  • Polars slice() – Offset/limit slicing uses operators.limit(..., offset=...) correctly.

1.0.0 - 2026-04-03

Breaking

  • SQL execution lives in moltres-core – Connection management, query execution, and dialect helpers are packaged as moltres-core on PyPI. pip install moltres pulls it in automatically. From a git checkout, install core first: pip install -e ./moltres-core then pip install -e ..

  • EngineConfig is unified with core – moltres.config.EngineConfig is now moltres_core.config.EngineConfig (single dataclass type for the public API and for ConnectionManager / QueryExecutor).

Added

  • moltres-core distribution – Same major version line as moltres; runtime dependency moltres-core>=1.0.0,<2.

  • PyPI release automation – GitHub Actions workflow publishes both moltres-core and moltres when you push tag v*, using the shared version in each pyproject.toml.

  • Pydantable SQL engine surface – MoltresPydantableEngine, SqlPlan, and SqlRootData for plan-driven SQL execution; see docs/PYDANTABLE_ENGINE.md.

  • Comprehensive Test Coverage – Added extensive test suite for complex plan scenarios:

    • Tests for select_for_update() with joins, aggregates, sorts, limits, distinct, and multiple filters

    • Tests for select_for_share() with complex plans

    • Async versions of all complex plan tests

    • Tests verifying improved error messages

    • 12 new tests in tests/dataframe/test_select_for_update_fixes.py

    • Enhanced existing tests in tests/table/test_enhanced_transactions.py

  • Error Handling Tests – Added comprehensive test suite for error handling paths:

    • Tests for SQLModel .exec() fallback behavior with different exception types

    • Tests for selectExpr() error handling scenarios

    • Tests verifying error context logging

    • 7 new tests in tests/engine/test_error_handling.py

Changed

  • Type Hints – Improved type annotations for better IDE support and type checking:

    • Changed transaction: Optional[Any] to transaction: Optional["Connection"] in mutation functions (insert_rows, update_rows, delete_rows, merge_rows)

    • Updated QueryExecutor.execute() and execute_many() methods with proper Connection type hints

  • Documentation – Enhanced docstrings for select_for_update() and select_for_share():

    • Added examples showing usage with joins and complex plans

    • Clarified that methods work with any plan structure

    • Documented plan type handling behavior

  • Exception Handling – Standardized exception handling patterns across codebase:

    • More specific exception catching where possible

    • Improved logging for debugging fallback scenarios

    • Better error context in all exception handlers

Fixed

  • select_for_update() and select_for_share() Logic Bug – Fixed critical bug where these methods only handled simple plan types (Project, Filter, TableScan) and failed with “This should not happen” errors on complex plans:

    • Now properly handles all plan types including Join, Aggregate, Sort, Limit, Distinct, and combinations thereof

    • Added plan tree traversal logic to find or create Project nodes for row-level locking

    • Works correctly with nested joins, filters, aggregates, and other complex query structures

    • Both sync (DataFrame) and async (AsyncDataFrame) versions fixed

  • Improved Error Messages – Enhanced defensive error messages to include helpful debugging information:

    • select_for_update() and select_for_share() now include plan type in error messages

    • GroupedPivot compilation errors now include pivot column, value column, and aggregation function details

    • Error messages provide actionable guidance for debugging

  • Code Quality Improvements – Fixed style and exception handling issues:

    • Removed extra blank line in CompilationState dataclass (PEP 8 compliance)

    • Improved exception handling in QueryExecutor.fetch() methods (both sync and async):

      • Now catches specific exceptions (AttributeError, TypeError, ValueError) before broad Exception

      • Added debug logging when SQLModel .exec() fallback occurs

      • Added explanatory comments for broad exception handling

    • Enhanced selectExpr() exception handling in both sync and async DataFrames:

      • Now catches specific exceptions (AttributeError, TypeError, KeyError) before broad Exception

      • Added debug logging with exception details when column extraction fails

      • Added comments explaining why fallback behavior is acceptable

[0.23.0] - 2025-12-04

Added

  • Transaction Utilities – Comprehensive transaction utility features for enhanced transaction management:

    • Transaction Decorator (@transaction) – Automatically wrap functions in transactions:

      • Works with both sync and async functions

      • Supports all transaction parameters (readonly, isolation_level, savepoint, timeout)

      • Automatic rollback on exceptions

      • Can accept database instance or find it from function parameters

    • Transaction Hooks – Register callbacks for transaction lifecycle events:

      • register_transaction_hook() – Register hooks for begin, commit, or rollback events

      • register_async_transaction_hook() – Async version for async transactions

      • Supports multiple hooks per event type

      • Use cases include audit logging, cache invalidation, event publishing

    • Transaction Metrics – Track transaction performance and statistics:

      • Global metrics collector tracks transaction count, duration, success/failure rates

      • Records savepoint usage, isolation levels, read-only transactions

      • Error tracking by exception type

      • get_transaction_metrics() and reset_transaction_metrics() functions

    • Transaction Retry – Automatic retry on transient failures:

      • retry_transaction() – Retry function calls within transactions

      • retry_transaction_async() – Async version

      • Detects deadlocks, lock timeouts, connection errors

      • Configurable retry parameters with exponential backoff

      • Database-specific error detection (PostgreSQL, MySQL, SQLite)

    • Transaction Testing Utilities – Helpers for testing transaction behavior:

      • ConcurrentTransactionTester – Test concurrent transaction scenarios

      • DeadlockSimulator – Simulate and test deadlock scenarios

      • test_isolation_level() – Test isolation level behavior

      • Supports both sync and async testing

    • All utilities integrated with existing transaction infrastructure

    • Comprehensive test coverage (16 new tests)

    • New guide: guides/20-transaction-utilities.md

    • Example file: docs/examples/21_transaction_utilities.py

[0.22.0] - 2025-12-02

Added

  • Context Manager Support – Added context manager protocol support to Database and AsyncDatabase classes:

    • Database now supports with statements for automatic connection cleanup

    • AsyncDatabase now supports async with statements for automatic connection cleanup

    • Connections are automatically closed when exiting the context, even if exceptions occur

    • Eliminates the need for manual db.close() or await db.close() calls

    • Updated documentation and examples to show context manager usage as the recommended approach

[0.19.6] - 2025-12-02

Changed

  • Integrations Directory Reorganization – Reorganized integrations directory for better structure and maintainability:

    • Consolidated integration/ directory into integrations/sqlalchemy/ package

    • Moved single-file integrations into organized package structure:

      • django.py and django_module.py → django/core.py

      • pytest.py and pytest_plugin.py → pytest/{fixtures,plugin}.py

      • fastapi.py → fastapi/core.py

      • streamlit.py → streamlit/core.py

      • airflow.py → airflow/core.py

      • prefect.py → prefect/core.py

    • Moved SQLAlchemy integration code from __init__.py to sync_integration.py for cleaner API exposure

    • All integration packages now follow consistent structure with __init__.py for clean exports and core.py for implementation

    • Updated all imports across codebase, tests, and documentation examples

    • Fixed exception handling order in Django middleware (check subclasses before base classes)

    • Exported FASTAPI_AVAILABLE and STREAMLIT_AVAILABLE from package __init__.py files for graceful degradation testing

Fixed

  • Type Checking – Fixed mypy type checking errors:

    • Added proper type ignore comments for optional integration imports

    • Fixed untyped decorator warnings in Streamlit integration

    • Added import-untyped ignores for Django imports (missing stubs)

  • Exception Handling – Fixed Django middleware exception handling to check more specific exceptions (subclasses) before base classes

0.19.5 - 2025-12-01

Changed

  • Single Responsibility Principle (SRP) Refactoring – Comprehensive refactoring to improve code organization and maintainability:

    • ExpressionCompiler Refactoring – Split ExpressionCompiler into specialized compiler modules:

      • expression_compilers/aggregation.py – Aggregation function compilation

      • expression_compilers/string.py – String operation compilation

      • expression_compilers/datetime.py – Datetime operation compilation

      • expression_compilers/window.py – Window function compilation

      • expression_compilers/json.py – JSON function compilation

      • expression_compilers/logical.py – Logical operation compilation (and, or, not, between, in, case_when)

      • expression_compilers/type_casting.py – Type casting compilation

      • expression_compilers/math.py – Math operation compilation

      • Significantly reduced complexity of main ExpressionCompiler class

    • DataFrame Refactoring – Split DataFrame into specialized manager classes:

      • dataframe/execution.py – DataFrameExecutor for execution methods (collect, show, take, first, head, tail)

      • dataframe/schema.py – SchemaInspector for schema methods (columns, schema, dtypes, printSchema)

      • dataframe/statistics.py – StatisticsCalculator for statistics methods (count, nunique, describe, summary)

      • dataframe/materialization.py – MaterializationHandler for materialization logic

      • dataframe/model_integration.py – ModelIntegrator for model integration logic

    • Database Refactoring – Split Database into specialized manager classes:

      • table/ddl_manager.py – DDLManager for DDL operations (create_table, drop_table, create_index, drop_index)

      • table/table_manager.py – TableManager for table operations (table, insert)

      • table/query_executor.py – DatabaseQueryExecutor for query execution (execute_plan, execute_plan_stream, execute_sql, explain)

      • table/ephemeral_manager.py – EphemeralTableManager for ephemeral table management (createDataFrame, cleanup)

    • Records Refactoring – Split Records into specialized manager classes:

      • io/records_accessor.py – RecordsAccessor for data access methods (iter, len, getitem, rows, iter, head, tail, first, last)

      • io/records_schema.py – RecordsSchema for schema operations (schema, select, rename)

      • io/records_writer.py – RecordsWriter for database write operations (insert_into)

    • All refactoring maintains full backward compatibility with existing APIs

    • Improved code maintainability and testability through better separation of concerns

Fixed

  • Type Safety – Enhanced type annotations throughout refactored modules:

    • Added proper type guards and runtime type narrowing to satisfy mypy without using type: ignore

    • Converted all quoted type annotations to real types

    • Fixed mypy errors related to ExpressionArg type narrowing

    • Added field(init=False) for dataclass attributes initialized in __post_init__

  • Error Messages – Fixed error message in ephemeral_manager.py to match expected test pattern

0.19.4 - 2025-12-01

Changed

  • Documentation Examples – Made all code examples in docs/examples/ and guides/ directories fully runnable:

    • All examples now use in-memory SQLite databases to avoid file conflicts

    • Added missing imports and variable definitions to make blocks self-contained

    • Fixed SQL errors by ensuring tables and columns are properly defined

    • Added dependency checks with helpful installation instructions for optional dependencies

    • Integration examples now include execution guards with clear usage instructions

Fixed

  • Example Code – Fixed all standalone code blocks in guides to execute successfully (175/175 blocks passing)

  • Missing Imports – Added missing imports (col, Records, pd, etc.) throughout guide examples

  • SQL Errors – Fixed “no such table/column” errors by adding proper table creation and data insertion

  • Variable Definitions – Made all code blocks self-contained by adding missing df and db definitions

0.19.3 - 2025-11-30

Changed

  • Type System Improvements – Replaced Any types with specific types across the codebase:

    • Added FillValue type alias for fill_null/fillna operations

    • Replaced database: Any with Union[Database, AsyncDatabase] in file I/O helpers

    • Added proper return type annotations for file I/O functions with overloads for type narrowing

    • Updated fill_null/fillna methods to use FillValue instead of Any

  • Pre-commit CI Checks – Fixed script to remove non-existent examples directory from ruff checks

Fixed

  • Type Checking – Fixed all mypy type errors (135 source files now pass type checking)

  • Type Annotations – Improved type safety in file I/O helpers, DataFrame operations, and fill operations

0.19.2 - 2025-11-29

Changed

  • Documentation and ReadTheDocs improvements.

  • Excluded archived planning docs from example validation to keep CI green.

0.19.1 - 2025-11-29

Fixed

  • Windows Path Handling in Streamlit Tests – Fixed Windows CI failures by replacing .replace("\\", "/") with .as_posix() for proper path conversion in Streamlit integration tests

0.19.0 - 2025-11-29

Changed

  • Documentation Organization – Moved example directories to docs/ for better organization:

    • example_data/ → docs/example_data/

    • example_output/ → docs/example_output/

    • examples/ → docs/examples/

    • Updated all references throughout codebase, guides, and documentation

    • Updated test files to use new docs/examples/ path

    • Example files now use relative paths from __file__ for data/output directories

  • Docstring Optimization for Read the Docs – Enhanced all docstrings for optimal Read the Docs deployment:

    • Added Sphinx cross-references (:class:, :func:, :meth:) throughout codebase

    • Ensured Google-style format consistency across all modules

    • Added proper type annotations in Returns sections with Sphinx references

    • Completed Args/Returns/Raises sections for all public API functions and classes

    • Enhanced module-level docstrings with comprehensive descriptions

    • Updated 948+ docstrings across 133 source files

    • Created helper scripts for docstring management (scripts/update_docstrings_for_rtd.py, scripts/find_missing_docstrings.py)

Fixed

  • CI Workflow – Updated CI workflow to check docs/examples/ instead of examples/ directory

  • README Quick Start Example – Fixed join example in README:

    • Added .select() to db.table("customers") before joining (TableHandle must be converted to DataFrame)

    • Fixed join condition to use proper column references: on=[col("orders.customer_id") == col("customers.id")]

0.19.0 - 2025-11-27

Added

  • Airflow/Prefect Workflow Orchestration Integration – Comprehensive integrations with Apache Airflow and Prefect for workflow orchestration:

    • Airflow Operators – Custom operators for executing Moltres operations in Airflow DAGs:

      • MoltresQueryOperator – Execute DataFrame queries and push results to XCom

      • MoltresToTableOperator – Write data from XCom to database tables

      • MoltresDataQualityOperator – Execute data quality checks on query results

      • Support for sync and async queries with proper error handling

      • Full integration with Airflow’s XCom system for task communication

      • Comprehensive error handling with Airflow task failure conversion

    • Prefect Tasks – Custom Prefect tasks for workflow integration:

      • moltres_query – Execute DataFrame queries as Prefect tasks

      • moltres_to_table – Write data to tables from task results

      • moltres_data_quality – Execute data quality checks with quality reports

      • Async task support with Prefect result storage integration

      • Automatic retry configuration and task logging

    • Data Quality Framework – Reusable data quality checking framework:

      • DataQualityCheck class with factory methods for common checks (not_null, range, unique, column_type, row_count, completeness, custom)

      • QualityChecker class for executing multiple checks on DataFrames

      • QualityReport class for comprehensive quality check reporting

      • Support for fail-fast and configurable error handling

    • ETL Pipeline Helpers – Generic ETLPipeline class for common ETL patterns:

      • Extract, Transform, Load pattern with validation hooks

      • Error handling and logging support

      • Available for both Airflow and Prefect workflows

    • Graceful Degradation – Optional dependency handling with clear error messages when frameworks are not installed

    • Comprehensive test coverage (32 integration tests) with mock and real framework tests

    • Example files (docs/examples/27_airflow_integration.py, docs/examples/28_prefect_integration.py)

    • Comprehensive guide (guides/16-workflow-integration.md) with detailed usage examples

    • Added apache-airflow>=2.5.0 and prefect>=2.0.0 to optional dependencies

  • Pytest Integration – Comprehensive testing utilities for Moltres DataFrames:

    • Database Fixtures – Isolated test databases with automatic cleanup:

      • moltres_db fixture for sync tests (SQLite by default, configurable to PostgreSQL/MySQL)

      • moltres_async_db fixture for async tests

      • Support for multiple database backends via pytest markers

      • Transaction rollback for test isolation

    • Test Data Fixtures – Load test data from CSV/JSON files:

      • test_data fixture automatically loads files from test_data/ directory

      • create_test_df() helper for creating DataFrames from test data

    • Custom Assertions – DataFrame comparison utilities:

      • assert_dataframe_equal() for comparing DataFrames (schema + data)

      • assert_schema_equal() for schema-only comparison

      • assert_query_results() for query result validation

      • Detailed diff reporting for test failures

    • Query Logging Plugin – Track SQL queries during tests:

      • query_logger fixture for query tracking and debugging

      • Query count assertions and performance monitoring

      • Query history inspection

    • Pytest Markers – Database-specific and performance test markers:

      • @pytest.mark.moltres_db("postgresql") for database-specific tests

      • @pytest.mark.moltres_performance for performance tests

    • Comprehensive test suite with full feature coverage

    • Example file (docs/examples/26_pytest_integration.py) and guide (guides/15-pytest-integration.md)

  • dbt Integration – Use Moltres DataFrames in dbt Python models:

    • Adapter Functions – Connect to databases from dbt configuration:

      • get_moltres_connection() to get Database instance from dbt config

      • moltres_dbt_adapter() convenience function for dbt models

      • Automatic connection string extraction from dbt profiles

    • Helper Functions – Reference dbt models and sources:

      • moltres_ref() to reference other dbt models as Moltres DataFrames

      • moltres_source() to reference dbt sources

      • moltres_var() to access dbt variables

    • dbt Macros – SQL macros for common Moltres patterns

    • Example file (docs/examples/29_dbt_integration.py) and guide (guides/17-dbt-integration.md)

    • Added dbt-core>=1.5.0 to optional dependencies

  • Streamlit Integration – Comprehensive integration with Streamlit for building data applications:

    • DataFrame Display Component (moltres_dataframe) – Display Moltres DataFrames in Streamlit with automatic conversion and query information display

    • Query Builder Widget (query_builder) – Interactive UI for building queries using Streamlit widgets (table selector, column selector, filter builder)

    • Caching Integration – Streamlit caching utilities for Moltres queries:

      • cached_query decorator with TTL and max_entries support

      • clear_moltres_cache() and invalidate_query_cache() for cache management

    • Session State Helpers – Database connection management within Streamlit sessions:

      • get_db_from_session() – Retrieve or create Database instance from session state

      • init_db_connection() and close_db_connection() for connection lifecycle management

      • Support for multiple databases with different keys

      • Connection string configuration via Streamlit secrets/config

    • Query Visualization (visualize_query) – Display SQL queries, execution plans, and performance metrics in organized Streamlit expanders

    • Error Handling (display_moltres_error) – Convert Moltres exceptions to user-friendly Streamlit error messages

    • Graceful Degradation – Optional dependency handling with clear error messages when Streamlit is not installed

    • Comprehensive test coverage using Streamlit’s AppTest framework

    • Example file (docs/examples/25_streamlit_integration.py) and comprehensive guide (guides/14-streamlit-integration.md)

    • Added streamlit>=1.28.0 to optional dependencies

Changed

  • Documentation Organization – Moved example directories to docs/ for better organization:

    • example_data/ → docs/example_data/

    • example_output/ → docs/example_output/

    • examples/ → docs/examples/

    • Updated all references throughout codebase, guides, and documentation

    • Updated test files to use new docs/examples/ path

    • Example files now use relative paths from __file__ for data/output directories

  • Docstring Optimization for Read the Docs – Enhanced all docstrings for optimal Read the Docs deployment:

    • Added Sphinx cross-references (:class:, :func:, :meth:) throughout codebase

    • Ensured Google-style format consistency across all modules

    • Added proper type annotations in Returns sections with Sphinx references

    • Completed Args/Returns/Raises sections for all public API functions and classes

    • Enhanced module-level docstrings with comprehensive descriptions

    • Updated 948+ docstrings across 133 source files

    • Created helper scripts for docstring management (scripts/update_docstrings_for_rtd.py, scripts/find_missing_docstrings.py)

Fixed

  • MySQL Test Port Conflicts – Fixed port conflicts in parallel test execution by implementing worker-specific port assignment with retry logic and port verification

  • Cleanup Regression Test – Fixed parallel execution issues in cleanup regression test by using unique database paths and working directories per test execution

  • Documentation Example Validation – Fixed syntax errors in documentation examples by properly formatting async/await code blocks and shell command examples

  • CI Configuration – Configured mypy to only check src directory (excluding examples) to prevent CI failures while maintaining strict type checking for source code

  • Pre-Commit Script – Simplified scripts/pre_commit_ci_checks.py to run the exact same commands as CI (using Python 3.11), removing custom error parsing logic and ensuring local checks match CI behavior

  • dbt Integration Type Hints – Fixed mypy type checking errors in dbt integration by correcting type ignore comments for optional dependency handling

  • README Quick Start Example – Fixed join example in README:

    • Added .select() to db.table("customers") before joining (TableHandle must be converted to DataFrame)

    • Fixed join condition to use proper column references: on=[col("orders.customer_id") == col("customers.id")]

0.19.0 - 2025-11-27

Added

  • SQLAlchemy, SQLModel, and Pydantic Integration – Comprehensive integration with SQLAlchemy ORM models, SQLModel, and Pydantic models:

    • Model-based table creation and references

    • Bidirectional type mapping between SQLAlchemy types and Moltres types

    • Automatic constraint extraction from models

    • Full async support for model operations

    • Backward compatibility with string-based API

Fixed

  • Fixed mypy type errors in examples and async DataFrame implementations

0.18.0 - 2025-11-26

Added

  • Added link to User Guides in README documentation section

  • Added meaningful outputs to guide code blocks

0.17.0 - 2025-11-26

Added

  • Polars-Style Interface – Comprehensive Polars LazyFrame-style API (PolarsDataFrame):

    • Lazy Evaluation – All operations build logical plans that execute only on collect() or fetch(), matching Polars’ lazy evaluation model

    • Core Operations – Full Polars-style API: select(), filter(), with_columns(), with_column(), drop(), rename(), sort(), limit(), head(), tail(), sample()

    • GroupBy Operations – group_by() returns PolarsGroupBy with aggregation methods: agg(), mean(), sum(), min(), max(), count(), std(), var(), first(), last(), n_unique()

    • Join Operations – Full join support with join() method: inner, left, right, outer, anti, semi joins with on parameter

    • Column Access – df['col'] returns PolarsColumn with .str and .dt accessors for pandas-like column operations

    • String Accessor – .str accessor with methods: upper(), lower(), strip(), lstrip(), rstrip(), contains(), startswith(), endswith(), replace(), split(), len()

    • DateTime Accessor – .dt accessor with methods: year(), month(), day(), hour(), minute(), second(), day_of_week(), day_of_year(), quarter(), week()

    • Window Functions – Window function support with over() clause: row_number(), rank(), dense_rank(), percent_rank(), ntile(), lead(), lag(), first_value(), last_value()

    • Conditional Expressions – when().then().otherwise() for SQL CASE statements, matching Polars’ conditional expression API

    • Data Reshaping – explode(), unnest(), pivot(), slice() for data transformation

    • Set Operations – concat(), vstack(), hstack(), union(), intersect(), difference(), cross_join() for combining DataFrames

    • SQL Expressions – select_expr() for raw SQL expression selection (e.g., select_expr("id", "name", "age * 2 as double_age"))

    • CTEs – cte(), with_recursive() for Common Table Expressions and recursive queries

    • Utility Methods – gather_every(), quantile(), describe(), explain(), with_row_count(), with_context(), with_columns_renamed()

    • File I/O – Polars-style read/write operations:

      • Read: db.scan_csv(), db.scan_json(), db.scan_jsonl(), db.scan_parquet(), db.scan_text()

      • Write: df.write_csv(), df.write_json(), df.write_jsonl(), df.write_parquet()

    • Schema Properties – columns, width, height, schema properties with lazy evaluation

    • Entry Points – db.table("name").polars() and df.polars() for easy conversion

    • Comprehensive test coverage (all tests passing)

    • Example file (docs/examples/19_polars_interface.py) and comprehensive guide (guides/10-polars-interface.md)

  • Async Polars DataFrame – Async version of Polars-style interface (AsyncPolarsDataFrame):

    • Wraps AsyncDataFrame with Polars-style API

    • All database-interactive methods are async (collect(), fetch(), height, schema, describe(), explain(), write_*)

    • Integrated via .polars() method on AsyncDataFrame and AsyncTableHandle

    • scan_* methods on AsyncDatabase return AsyncPolarsDataFrame

    • Comprehensive test coverage

  • Async Pandas DataFrame – Async version of Pandas-style interface (AsyncPandasDataFrame):

    • Wraps AsyncDataFrame with Pandas-style API

    • All database-interactive methods are async (collect(), shape, dtypes, empty, describe(), info(), nunique(), value_counts())

    • Integrated via .pandas() method on AsyncDataFrame and AsyncTableHandle

    • _AsyncLocIndexer and _AsyncILocIndexer for async pandas-style indexing

    • Comprehensive test coverage

  • Pandas Interface Enhancements – Additional pandas-style methods:

    • explode() – Expand array/JSON columns into multiple rows

    • pivot(), pivot_table() – Data reshaping operations

    • melt() – Unpivot operations (noted as NotImplementedError for future implementation)

    • sample(), limit() – Sampling and limiting operations

    • append(), concat() – Concatenation operations

    • isin(), between() – Advanced filtering methods

    • select_expr(), cte() – SQL expression selection and CTEs

    • Updated guide (guides/09-pandas-interface.md) with new features

  • Enhanced Pandas-Style Interface – Comprehensive improvements to the pandas-style interface (PandasDataFrame):

    • String Accessor – Added .str accessor for pandas-style string operations:

      • Methods: upper(), lower(), strip(), lstrip(), rstrip(), contains(), startswith(), endswith(), replace(), split(), len()

      • Full SQL pushdown execution for all string operations

      • Access via df['col'].str.upper() syntax

    • Improved Query Syntax – Enhanced query() method:

      • Supports both = and == for equality comparisons (pandas-style)

      • Supports AND/OR keywords in addition to &/| operators

      • Better error messages for syntax errors

    • Data Type Information – Implemented proper dtypes property:

      • Real schema inspection using SQL type mapping

      • Returns pandas-compatible dtype strings ('int64', 'float64', 'object', etc.)

      • Cached after first access to avoid redundant queries

    • Data Inspection Methods – Added comprehensive pandas-style inspection methods:

      • head(n=5) – Returns first n rows as list of dicts

      • tail(n=5) – Returns last n rows with stable sorting

      • describe() – Statistical summary (requires pandas, returns pandas DataFrame)

      • info() – Column info and memory usage (requires pandas)

      • nunique(column) – Count unique values in a column

      • value_counts(column, normalize=False) – Count frequency of values

    • Fixed drop_duplicates – Corrected implementation to properly handle subset parameter:

      • Uses GROUP BY with MIN/MAX aggregation for subset-based deduplication

      • Supports keep='first' and keep='last' parameters

    • Early Column Validation – Added column existence validation:

      • Validates columns before building logical plans

      • Provides helpful error messages with typo suggestions

      • Integrated into __getitem__, query(), merge(), sort_values(), groupby(), etc.

    • Enhanced GroupBy – Added pandas-style aggregation methods:

      • sum(), mean(), min(), max() – Aggregate all numeric columns

      • count() – Count rows per group

      • nunique() – Count distinct values for each column

      • first(), last() – Get first/last value per group

    • Column Access Improvements – df['col'] now returns PandasColumn:

      • Wrapper around Column that adds .str accessor

      • Forwards all Column methods and operators

      • Enables pandas-like syntax: df['name'].str.upper()

    • Shape Caching – Added caching for shape property:

      • Results cached after first computation to avoid redundant queries

      • Warnings for expensive operations

    • Comprehensive test coverage (all tests passing)

    • Updated examples (docs/examples/18_pandas_interface.py) and documentation

  • Runnable Guide Documentation – All guide code blocks are now fully runnable:

    • Created automated script (scripts/make_guides_runnable.py) to update code blocks

    • Updated 131 code blocks across 8 guides to use sqlite:///:memory:

    • All examples include complete setup (imports, database creation, data insertion)

    • All 567 examples validated and passing

  • New Pandas Interface Guide – Created comprehensive guide (guides/09-pandas-interface.md):

    • Getting started with PandasDataFrame

    • Column access and string operations

    • Query filtering with improved syntax

    • Data inspection methods

    • GroupBy operations

    • Merging, sorting, and data manipulation

    • All code examples are self-contained and runnable

  • SQLAlchemy ORM Model Integration – Comprehensive bidirectional integration between SQLAlchemy ORM models and Moltres:

    • Model-based table creation – Create tables directly from SQLAlchemy model classes:

      • db.create_table(User) – Automatically extracts schema, constraints, and types from model

      • Supports all SQLAlchemy column types with automatic type mapping

      • Extracts primary keys, foreign keys, unique constraints, and check constraints

    • Model-based table references – Query using SQLAlchemy model classes instead of table names:

      • db.table(User).select() – Get table handle from model class

      • db.table(User).select().where(col("age") > 25) – Query using model references

      • Model class stored in TableHandle for later reference

    • Bidirectional type mapping – Automatic conversion between SQLAlchemy types and Moltres types:

      • SQLAlchemy → Moltres: Integer → "INTEGER", String(100) → "VARCHAR(100)", etc.

      • Moltres → SQLAlchemy: "DECIMAL(10,2)" → Numeric(10, 2), etc.

      • Supports dialect-specific types (PostgreSQL JSONB, UUID, etc.)

    • Constraint extraction – Automatic extraction of all constraints from SQLAlchemy models:

      • Primary keys from primary_key=True columns

      • Foreign keys from ForeignKey column definitions

      • Unique constraints from UniqueConstraint in __table_args__

      • Check constraints from CheckConstraint in __table_args__

    • Async support – Full async support for SQLAlchemy model operations:

      • await async_db.create_table(User).collect()

      • await async_db.table(User).select().collect()

    • Backward compatibility – Traditional string-based API still works:

      • db.create_table("users", [column(...)]) – Still supported

      • db.table("users") – Still supported

      • All existing code continues to work unchanged

    • Comprehensive test coverage – 19 tests covering all integration features

    • Example file – docs/examples/17_sqlalchemy_models.py demonstrating usage

    • Documentation – README.md updated with SQLAlchemy integration section

  • Explode compilation – explode() now emits working SQL for SQLite (via json_each) and PostgreSQL (jsonb_array_elements), unlocking table-valued expansions for array/JSON columns on those dialects.

Fixed

  • FILTER fallback stability – the CASE-expression fallback used when a dialect lacks native FILTER support now compiles with SQLAlchemy’s sa_case, avoiding UnboundLocalError crashes on SQLite.

  • Async health checks – the dev extra now installs asyncpg, and the async PostgreSQL health test runs successfully by default.

  • Pandas interface column validation – Fixed drop_duplicates() to properly handle subset parameter using GROUP BY operations

  • SQL parser improvements – Fixed AND/OR keyword parsing in query parser by adjusting regex patterns to work correctly after whitespace skipping

  • LIKE pattern compilation – Fixed like and ilike operations to correctly handle string patterns in SQL compiler

  • Type checking – Fixed mypy type errors in async DataFrame implementations (async_polars_dataframe.py, async_pandas_dataframe.py, async_table.py)

  • Type checking – Fixed redundant cast errors in mutations.py by removing unnecessary type casts

  • Type checking – Fixed mypy errors in example files by removing unused type ignore comments and adding proper type annotations

  • CI/CD – All pre-commit CI checks now pass with Python 3.11 (ruff, mypy, tests, documentation validation)

Changed

  • Type-checking polish – records/dataframe helpers and examples were tightened so mypy passes across src/ and docs/examples/, including forward-declared pandas/polars types and stricter Records typing.

  • Documentation improvements – All guide code blocks updated to be fully runnable with SQLite in-memory databases for easy setup and testing.

  • Error handling – Enhanced error messages in pandas-style interface with column validation and typo suggestions for better user experience.

  • Type safety – Improved type annotations throughout async DataFrame implementations, removing unused type ignore comments and fixing all mypy errors

  • CI/CD – Pre-commit CI checks script now uses Python 3.11 for consistent type checking across all environments

0.16.0 - 2025-11-26

Fixed

  • Fixed mypy type errors and improved CI checks

  • Fixed type checking issues across the codebase

Changed

  • Improved type safety with better type annotations

  • Enhanced CI/CD pipeline with pre-commit checks

0.15.0 - 2025-11-25

Added

  • Added DuckDB dialect support

  • Added pandas-style interface for Moltres DataFrames

  • Added pre-commit CI checks script

Fixed

  • Fixed mypy type errors and type narrowing issues

  • Fixed redundant casts and type ignore comments

  • Updated pre-commit script to use same Python interpreter for mypy

0.14.0 - 2025-11-24

Added

  • DataFrame Attributes - PySpark-compatible introspection properties:

    • .columns property - Returns list of column names from logical plans

    • .schema property - Returns List[ColumnInfo] with column names and types

    • .dtypes property - Returns List[Tuple[str, str]] of (column_name, type_name) pairs

    • .printSchema() method - Prints formatted schema tree similar to PySpark

    • Works with both DataFrame and AsyncDataFrame

    • Supports all logical plan types: TableScan, FileScan, Project, Aggregate, Join, Filter, Limit, Sort, etc.

    • Handles edge cases: aliases, star columns, nested projects, Explode operations

    • Lazy evaluation - extracts schema information without executing queries

Changed

  • Expanded moltres.utils.inspector module with async database support

  • Improved schema introspection utilities for both sync and async databases

0.13.0 - 2025-11-24

Added

  • Schema Management - Constraints & Indexes - Comprehensive support for database constraints and indexes:

    • Unique Constraints - Single and multi-column unique constraints via unique() helper:

      • db.create_table("users", [...], constraints=[unique("email")])

      • db.create_table("sessions", [...], constraints=[unique(["user_id", "session_id"], name="uq_user_session")])

    • Check Constraints - SQL expression-based validation via check() helper:

      • db.create_table("products", [...], constraints=[check("price >= 0", name="ck_positive_price")])

    • Foreign Key Constraints - Referential integrity with cascade options via foreign_key() helper:

      • Single column: foreign_key("user_id", "users", "id", on_delete="CASCADE")

      • Multi-column: foreign_key(["order_id", "item_id"], "order_items", ["id", "id"])

      • Supports on_delete and on_update actions (CASCADE, SET NULL, RESTRICT, etc.)

    • Index Management - Create and drop indexes for performance optimization:

      • db.create_index("idx_email", "users", "email") - Single column index

      • db.create_index("idx_user_status", "orders", ["user_id", "status"]) - Multi-column index

      • db.create_index("idx_unique_email", "users", "email", unique=True) - Unique index

      • db.drop_index("idx_email", "users") - Drop index

    • SQLAlchemy DDL Integration - All DDL operations now use SQLAlchemy’s declarative API:

      • Replaced raw SQL string generation with SQLAlchemy Table, Column, Index, CreateTable, DropTable, CreateIndex, DropIndex objects

      • Better dialect compatibility and abstraction

      • Automatic handling of dialect-specific syntax differences

    • Async Support - Full async support for all constraint and index operations:

      • await async_db.create_table(..., constraints=[...])

      • await async_db.create_index(...)

      • await async_db.drop_index(...)

    • Comprehensive Test Coverage - 41 tests covering all constraint types, indexes, edge cases, and async operations

    • Example Updates - Updated docs/examples/09_table_operations.py with constraint and index examples

Changed

  • DDL Compilation - Refactored all DDL compilation to use SQLAlchemy objects instead of raw SQL strings:

    • compile_create_table() now uses SQLAlchemy’s CreateTable with Table and Column objects

    • compile_drop_table() uses SQLAlchemy’s DropTable

    • compile_create_index() uses SQLAlchemy’s CreateIndex with Index objects

    • compile_drop_index() uses SQLAlchemy’s DropIndex

    • compile_insert_select() uses SQLAlchemy’s insert().from_select()

    • Improved dialect compatibility and maintainability

  • Type Safety - Enhanced type hints with proper TYPE_CHECKING imports for constraint and index operation types

Fixed

  • Fixed foreign key constraint compilation when referenced tables aren’t in the same MetaData (fallback to string-based FK handling)

  • Fixed index compilation to properly handle column references in SQLAlchemy Index objects

  • Fixed async index operations to use correct import paths

0.11.0 - 2025-11-24

Fixed

  • Improved async PostgreSQL handling

  • Stabilized staging tables for test harness

  • Fixed harness doc formatting for validator

  • Fixed CI: format async CSV reader for ruff

  • Skip unsupported math tests on SQLite

  • Skip Postgres/MySQL tests when DB binaries missing

0.10.0 - 2025-11-23

Added

  • Chunked File Reading for Large Files - Files are now read in chunks by default to safely handle files larger than available memory:

    • Default streaming mode for all file reads

    • Opt-out mechanism with stream=False

    • Memory safety prevents out-of-memory errors

    • Schema inference from first chunk

    • Error recovery with automatic cleanup

    • Empty file handling

    • Both sync and async support

0.9.0 - 2025-11-23

Added

  • 98% PySpark API Compatibility - Major improvements to match PySpark’s DataFrame API:

    • Raw SQL query support via db.sql() method

    • SQL expression selection with selectExpr() method

    • Select all columns with select("*")

    • SQL string predicates in filter() and where()

    • String column names in aggregations

    • Dictionary syntax in aggregations

    • Pivot on GroupBy

    • Explode function

    • PySpark-style aliases (camelCase methods)

    • Improved withColumn() to correctly handle adding and replacing columns

  • PySpark-style dot notation column selection

  • LazyRecords for db.read.records. API*

  • createDataFrame function

Changed

  • API Compatibility - Moltres now achieves ~98% API compatibility with PySpark for core DataFrame operations

  • All major DataFrame transformation methods now match PySpark’s API

  • Both camelCase (PySpark-style) and snake_case (Python-style) naming conventions supported throughout the API

Fixed

  • Fixed withColumn() to correctly replace existing columns instead of duplicating them

  • Fixed pivot value inference to work automatically when values are not provided

  • Fixed column replacement logic in withColumn() to match PySpark’s behavior

  • Fixed select("*") to work correctly when combined with other columns

  • Fixed async PostgreSQL connections that forwarded DSN ?options=-csearch_path=... parameters to asyncpg

  • Fixed async PostgreSQL staging tables so createDataFrame() and file readers now create regular tables instead of connection-scoped temp tables

0.12.0 - 2025-11-24

Added

  • Comprehensive Examples Directory - Added 13 example files demonstrating all Moltres features:

    • 01_connecting.py - Database connections (sync and async)

    • 02_dataframe_basics.py - Basic DataFrame operations

    • 03_async_dataframe.py - Asynchronous DataFrame operations

    • 04_joins.py - Join operations

    • 05_groupby.py - GroupBy and aggregation

    • 06_expressions.py - Column expressions and functions

    • 07_file_reading.py - Reading files (CSV, JSON, JSONL, Parquet, Text)

    • 08_file_writing.py - Writing DataFrames to files

    • 09_table_operations.py - Table operations and mutations

    • 10_create_dataframe.py - Creating DataFrames from Python data

    • 11_window_functions.py - Window functions

    • 12_sql_operations.py - Raw SQL and SQL operations

    • 13_transactions.py - Transaction management

    • All examples use PySpark-style function imports (from moltres.expressions import functions as F)

    • All examples verified to run with real outputs documented as comments

Changed

  • README Streamlined - Significantly streamlined README for better readability:

    • Reduced from 714 lines to 446 lines (37% reduction)

    • Removed verbose release notes and repetitive marketing claims

    • Focused on essential quick start examples with links to comprehensive examples directory

    • All example links use GitHub URLs for PyPI compatibility

    • All code examples verified to run with actual outputs documented

Fixed

  • Fixed ruff F823: remove case from local imports

  • Fixed UnboundLocalError: remove literal from local imports

  • Fixed concat() function for SQLite dialect

  • Fixed Windows test failures: SQLite function support and path handling

  • Fixed linting error and SQLite path handling on Windows

  • Fixed indentation in OPS_RUNBOOKS.md code blocks

  • Fixed CI: Remove deprecated license classifier from pyproject.toml

Added (continued)

  • Chunked File Reading for Large Files - Files are now read in chunks by default to safely handle files larger than available memory:

    • Default Streaming Mode - All file reads (db.read.csv(), db.read.json(), etc.) now use chunked reading by default, similar to PySpark’s partition-based approach

    • Opt-Out Mechanism - Users can disable chunked reading for small files by setting stream=False: db.read.option("stream", False).csv("small_file.csv")

    • Memory Safety - Prevents out-of-memory errors when processing large datasets by reading and inserting data incrementally in chunks

    • Schema Inference from First Chunk - Schema is inferred from the first chunk of data, then applied consistently to all subsequent chunks

    • Error Recovery - Temporary tables are automatically cleaned up if chunk insertion fails

    • Empty File Handling - Gracefully handles empty files with or without explicit schemas

    • Both Sync and Async - Full support for both synchronous (DataFrame) and asynchronous (AsyncDataFrame) operations

    • This matches PySpark’s behavior where files are read in partitions across the cluster, adapted for single-machine processing

  • PySpark Read API Parity - Enhanced read API to match PySpark’s DataFrameReader with comprehensive option support:

    • Builder Methods - Added options() method to set multiple read options at once (PySpark-compatible):

      • db.read.options(header=True, delimiter=",").csv("data.csv")

      • Works with all read methods: csv(), json(), parquet(), text(), etc.

      • Available on both sync (DataLoader, ReadAccessor) and async (AsyncDataLoader, AsyncReadAccessor) APIs

    • Text File Method - Added textFile() method as PySpark-compatible alias for text():

      • db.read.textFile("log.txt") - Same as db.read.text("log.txt")

      • Available in both sync and async APIs

    • CSV Options - Comprehensive CSV reading options matching PySpark:

      • mode - Read mode: “PERMISSIVE” (default), “DROPMALFORMED”, or “FAILFAST”

      • encoding - File encoding (default: “UTF-8”)

      • quote - Quote character (default: ‘”’)

      • escape - Escape character (default: “")

      • nullValue - String representation of null (default: “”)

      • nanValue - String representation of NaN (default: “NaN”)

      • dateFormat - Date format string for parsing dates

      • timestampFormat - Timestamp format string for parsing timestamps

      • samplingRatio - Fraction of rows used for schema inference (default: 1.0)

      • columnNameOfCorruptRecord - Column name for corrupt records

      • sep - Alias for delimiter

      • quoteAll - Quote all fields (default: False)

      • ignoreLeadingWhiteSpace - Ignore leading whitespace (default: False)

      • ignoreTrailingWhiteSpace - Ignore trailing whitespace (default: False)

      • comment - Comment character to skip lines

      • enforceSchema - Enforce schema even if it doesn’t match data (default: True)

      • All options work with both sync and async CSV readers

    • JSON Options - Enhanced JSON reading options matching PySpark:

      • mode - Read mode: “PERMISSIVE” (default), “DROPMALFORMED”, or “FAILFAST”

      • encoding - File encoding (default: “UTF-8”)

      • multiLine - Alias for multiline (PySpark-compatible)

      • dateFormat - Date format string for parsing dates

      • timestampFormat - Timestamp format string for parsing timestamps

      • samplingRatio - Fraction of rows used for schema inference (default: 1.0)

      • columnNameOfCorruptRecord - Column name for corrupt records

      • lineSep - Line separator for multiline JSON

      • dropFieldIfAllNull - Drop fields if all values are null (default: False)

      • All options work with both sync and async JSON readers

      • Note: Some JSON parsing options (e.g., allowComments, allowUnquotedFieldNames) are not supported by Python’s json module and are ignored

    • Parquet Options - Parquet reading options matching PySpark:

      • mergeSchema - Merge schemas from multiple files (default: False)

      • rebaseDatetimeInRead - Rebase datetime values during read (default: True)

      • datetimeRebaseMode - Datetime rebase mode (default: “EXCEPTION”)

      • int96RebaseMode - INT96 rebase mode (default: “EXCEPTION”)

      • All options work with both sync and async Parquet readers

    • Text Options - Enhanced text file reading options:

      • encoding - File encoding (default: “UTF-8”)

      • wholetext - If True, read entire file as single value (default: False)

      • lineSep - Line separator (default: newline)

      • All options work with both sync and async text readers

    • Read Modes - Comprehensive error handling modes for CSV and JSON:

      • PERMISSIVE (default) - Sets other fields to null when encountering corrupted records and puts malformed strings into a field configured by columnNameOfCorruptRecord

      • DROPMALFORMED - Ignores the whole corrupted records

      • FAILFAST - Throws an exception when it meets corrupted records

    • Schema Inference Enhancements - Enhanced schema inference with date/timestamp format support:

      • dateFormat and timestampFormat options now properly influence schema inference

      • Date and timestamp columns are correctly inferred when formats are provided

      • Works with CSV, JSON, and JSONL readers

  • PySpark Write API Parity & Chunked Output:

    • Builder Enhancements - Added .format(), .options(), .bucketBy(), .sortBy() and camel-case mode aliases to DataFrameWriter and AsyncDataFrameWriter

    • Save Modes - mode("ignore") now skips both table and file targets when they already exist; file targets also honor error_if_exists/overwrite

    • File Writers Stream by Default - Any sink that requires materialization (CSV/JSON/JSONL/Text/Parquet) now streams chunks automatically unless .stream(False) is specified

    • New Sinks & Options - Added .text() helper, format("csv").options(...) parity, JSON streaming (when no indent), Parquet streaming via pyarrow.ParquetWriter, and explicit mode handling for partitioned outputs

    • Async Parity - All improvements apply to the async writer as well, using aiofiles for file sinks

    • Safety Checks - Unsupported combinations (e.g., bucketing into files, partitioned async text/parquet writes) now raise NotImplementedError instead of silently misbehaving

  • Extended Function Library - Added 38+ new PySpark-compatible functions across multiple categories:

    • Mathematical Functions - pow(), power(), asin(), acos(), atan(), atan2(), signum(), sign(), log2(), hypot() for advanced mathematical operations

    • String Functions - initcap(), instr(), locate(), translate() for enhanced string manipulation

    • Date/Time Functions - to_timestamp(), unix_timestamp(), from_unixtime(), date_trunc(), quarter(), weekofyear(), week(), dayofyear(), last_day(), months_between() for comprehensive date/time operations

    • Window Functions - first_value(), last_value() for window-based analytics

    • Array Functions - array_append(), array_prepend(), array_remove(), array_distinct(), array_sort(), array_max(), array_min(), array_sum() for array manipulation

    • JSON Functions - json_tuple(), from_json(), to_json(), json_array_length() for JSON data processing

    • Utility Functions - rand(), randn(), hash(), md5(), sha1(), sha2(), base64() for random number generation and hashing

    • Additional Functions - monotonically_increasing_id(), crc32(), soundex() for ID generation and data processing

    • All functions include dialect-specific SQL compilation for PostgreSQL, MySQL, and SQLite

    • Functions are available from moltres.expressions.functions and can be imported directly

    • Example: from moltres.expressions.functions import pow, asin, to_timestamp; df.select(pow(col("x"), 2), asin(col("y")))

  • 98% PySpark API Compatibility - Major improvements to match PySpark’s DataFrame API:

    • Raw SQL Query Support - New db.sql() method for executing raw SQL queries, similar to PySpark’s spark.sql():

      • Accepts raw SQL strings with optional named parameters (:param_name syntax)

      • Returns lazy DataFrame objects that can be chained with other operations

      • Supports parameterized queries for security and flexibility

      • Raw SQL is automatically wrapped in subqueries when chained, enabling full DataFrame API compatibility

      • Works with both synchronous (db.sql()) and asynchronous (await db.sql()) databases

      • SQL dialect is determined by the database connection

      • Example: db.sql("SELECT * FROM users WHERE id = :id", id=1).where(col("age") > 18).collect()

    • SQL Expression Selection - New selectExpr() method on DataFrame for writing SQL expressions directly, matching PySpark’s selectExpr() API:

      • Accepts SQL expression strings (e.g., "amount * 1.1 as with_tax")

      • Parses SQL expressions into Column expressions automatically

      • Supports arithmetic operations, functions, comparisons, literals, and aliases

      • Full SQL expression parser with operator precedence handling

      • Works with both synchronous (df.selectExpr()) and asynchronous (await df.selectExpr()) DataFrames

      • Returns lazy DataFrame objects that can be chained with other operations

      • Example: df.selectExpr("id", "amount * 1.1 as with_tax", "UPPER(name) as name_upper")

    • Select All Columns with select("*") - Support for df.select("*") to explicitly select all columns, matching PySpark’s API:

      • select("*") alone is equivalent to select() (selects all columns)

      • Can combine "*" with other columns: select("*", col("new_col")) or select("*", "column_name")

      • Works with both synchronous and asynchronous DataFrames

      • Example: df.select("*", (col("amount") * 1.1).alias("with_tax"))

    • SQL String Predicates in filter() and where() - Support for SQL string predicates in filtering methods, matching PySpark’s API:

      • filter() and where() now accept both Column expressions and SQL strings

      • Supports basic comparison operators (>, <, >=, <=, =, !=)

      • Works with both synchronous and asynchronous DataFrames

      • Complex predicates can be achieved by chaining multiple filters or using Column API

      • Example: df.filter("age > 18") or df.where("amount >= 100 AND status = 'active'")

    • String Column Names in Aggregations - Support for string column names in agg(), matching PySpark’s convenience:

      • String column names default to sum() aggregation

      • More convenient than PySpark’s requirement for explicit aggregation functions

      • Works with both synchronous and asynchronous DataFrames

      • Example: df.group_by("category").agg("amount") (equivalent to sum(col("amount")))

    • Dictionary Syntax in Aggregations - Support for dictionary syntax in agg(), matching PySpark’s API:

      • Dictionary maps column names to aggregation function names

      • Supports multiple aggregations in a single call

      • Can be mixed with Column expressions and string column names

      • Example: df.group_by("category").agg({"amount": "sum", "price": "avg"})

    • Pivot on GroupBy - PySpark-style pivot() chaining on groupBy(), matching PySpark’s API:

      • Supports df.group_by("category").pivot("status").agg("amount") syntax

      • Automatic pivot value inference from data (like PySpark)

      • Explicit pivot values when needed: pivot("status", values=["active", "inactive"])

      • Works with both synchronous and asynchronous DataFrames

      • Example: df.group_by("category").pivot("status").agg("amount")

    • Explode Function - PySpark-style explode() function for array/JSON column expansion:

      • Function-based API: df.select(explode(col("array_col")).alias("value"))

      • Matches PySpark’s from pyspark.sql.functions import explode pattern

      • Works with both synchronous and asynchronous DataFrames

      • Note: SQL compilation for explode() is in progress (requires table-valued function support)

      • Example: from moltres.expressions.functions import explode; df.select(explode(col("tags")).alias("tag"))

    • PySpark-style Aliases - Additional camelCase aliases for better PySpark compatibility:

      • orderBy() and sort() - PySpark-style aliases for order_by()

      • saveAsTable() - PySpark-style alias for save_as_table()

      • Both camelCase and snake_case methods available throughout the API

      • Example: df.orderBy(col("name")) or df.write.saveAsTable("results")

    • Improved withColumn() - Enhanced withColumn() to correctly handle both adding and replacing columns:

      • Adding new columns: df.withColumn("new_col", col("old_col") * 2)

      • Replacing existing columns: df.withColumn("existing_col", col("existing_col") + 1)

      • Matches PySpark’s behavior exactly

      • Works with both synchronous and asynchronous DataFrames

Changed

  • API Compatibility - Moltres now achieves ~98% API compatibility with PySpark for core DataFrame operations

  • All major DataFrame transformation methods now match PySpark’s API

  • Both camelCase (PySpark-style) and snake_case (Python-style) naming conventions supported throughout the API

Fixed

  • Fixed withColumn() to correctly replace existing columns instead of duplicating them

  • Fixed pivot value inference to work automatically when values are not provided

  • Fixed column replacement logic in withColumn() to match PySpark’s behavior

  • Fixed select("*") to work correctly when combined with other columns

  • Fixed async PostgreSQL connections that forwarded DSN ?options=-csearch_path=... parameters to asyncpg (which rejects unknown keywords) by translating them into asyncpg server_settings.

  • Fixed async PostgreSQL staging tables so createDataFrame() and file readers now create regular tables instead of connection-scoped temp tables, preventing UndefinedTableError when inserts execute on different pooled connections.

0.8.0 - 2025-11-21

Added

  • Lazy CRUD and DDL Operations - All DataFrame CRUD and DDL operations are now lazy, requiring an explicit .collect() call for execution:

    • insert(), update(), delete(), merge() now return lazy Mutation objects

    • create_table(), drop_table() now return lazy DDLOperation objects

    • Operations build a logical plan that only executes when .collect() is called

    • DataFrame write operations remain eager (similar to PySpark’s behavior)

    • New to_sql() method on lazy operations for SQL inspection without execution

  • Transaction Management - All operations within a single .collect() call are part of a single session that rolls back all changes if any failure occurs:

    • Automatic transaction management for lazy operations

    • Rollback on any exception during execution

    • Explicit transaction support via db.transaction() context manager

  • Batch Operation API - New db.batch() context manager to queue multiple lazy operations and execute them together within a single transaction:

    • Queue multiple insert, update, delete, and DDL operations

    • Execute all queued operations atomically in a single transaction

    • Automatic rollback if any operation fails

    • Supports both synchronous and asynchronous batch operations

  • Type Checking Improvements - Enhanced type safety and CI compatibility:

    • Added pandas-stubs>=2.1 to dev dependencies for proper mypy type checking

    • Fixed pandas DataFrame constructor type compatibility issues

    • Improved type annotations for lazy operation classes

Changed

  • Breaking Change: CRUD and DDL operations now require .collect() to execute:

    • table.insert([...]) → table.insert([...]).collect()

    • table.update(...) → table.update(...).collect()

    • table.delete(...) → table.delete(...).collect()

    • db.create_table(...) → db.create_table(...).collect()

    • db.drop_table(...) → db.drop_table(...).collect()

  • Improved composability of operations by making them lazy

  • Enhanced transaction safety with automatic rollback on failures

  • Better alignment with PySpark’s lazy evaluation model

Fixed

  • Fixed mypy type checking errors related to pandas DataFrame constructor

  • Fixed unused type ignore comments after adding pandas-stubs

  • Fixed transaction management to ensure atomicity of operations

  • Fixed async operation handling in batch context

Internal

  • Added OperationBatch and async_OperationBatch classes for batch operation management

  • Created Mutation and DDLOperation base classes for lazy operations

  • Enhanced test coverage for lazy operations and batch API

  • Improved code quality with proper type annotations and mypy strict checking

0.7.0 - 2025-11-21

Added

  • PostgreSQL and MySQL Testing Infrastructure - Comprehensive test support for multiple database backends:

    • Added testing.postgresql and testing.mysqld dependencies for ephemeral database instances in tests

    • New pytest markers: @pytest.mark.postgres, @pytest.mark.mysql, @pytest.mark.multidb

    • Database fixtures: postgresql_db, postgresql_connection, mysql_db, mysql_connection

    • Async database fixtures: postgresql_async_connection, mysql_async_connection

    • Test helpers: seed_customers_orders() for consistent test data across databases

    • New test suites: test_postgresql_features.py, test_mysql_features.py, test_multidb.py

    • Async test suites: test_async_postgresql_features.py, test_async_mysql_features.py, test_async_integration.py

  • Type Overloads for collect() Methods - Enhanced type safety with @overload decorators:

    • collect(stream=False) returns List[Dict[str, object]]

    • collect(stream=True) returns Iterator[List[Dict[str, object]]] or AsyncIterator[...]

    • Improved type inference for better IDE support and type checking

Changed

  • Enhanced type annotations throughout the codebase with proper @overload decorators

  • Improved type safety in SQL compiler with explicit type casting using typing_cast

  • Better type inference for DataFrame operations with overloaded method signatures

Fixed

  • Fixed mypy type checking errors related to ColumnElement[Any] return types in expression compiler

  • Fixed ruff linting errors for name conflicts between typing.cast and sqlalchemy.cast

  • Fixed async DSN parsing to correctly convert mysql+pymysql:// to mysql+aiomysql:// for async connections

  • Fixed database connection cleanup - added close() methods to Database and AsyncDatabase classes

  • Fixed test hanging issues by ensuring proper engine disposal in pytest fixtures

  • Fixed column qualification after joins to handle unqualified column names correctly

  • Fixed PostgreSQL JSON extraction to use ->> operator for direct JSONB path extraction

  • Fixed PostgreSQL array literal syntax to use ARRAY[...] format

  • Fixed MySQL JSON array functions to handle literal values correctly

  • Fixed MySQL JSON_CONTAINS to properly quote values with json_quote()

Internal

  • Added comprehensive type stubs and type annotations for better IDE support

  • Improved code quality with ruff formatting and mypy strict type checking

  • Enhanced test coverage with 301 passing tests across multiple database backends

0.6.0 - 2025-11-21

Added

  • Null Handling Convenience Methods - New na property on DataFrame for convenient null handling:

    • df.na.drop() - Drop rows with null values (wrapper around dropna())

    • df.na.fill(value) - Fill null values with a specified value (wrapper around fillna())

    • Available on both synchronous and asynchronous DataFrames

  • Random Sampling - New sample(fraction, seed=None) method for random row sampling:

    • Uses TABLESAMPLE clause for PostgreSQL and SQL Server

    • Falls back to ORDER BY RANDOM() LIMIT for SQLite and other dialects

    • Supports optional seed for reproducible sampling

  • Enhanced Type System - New data types with full SQL support:

    • Decimal/Numeric with Precision - decimal(name, precision, scale) helper for creating columns with specific precision and scale

    • UUID Type - uuid(name) helper with dialect-specific compilation (PostgreSQL UUID, MySQL CHAR(36), SQLite TEXT)

    • JSON/JSONB Type - json(name) helper with dialect-specific compilation (PostgreSQL JSONB, MySQL JSON, SQLite TEXT)

    • Date/Time Interval Types - interval(name) helper with interval arithmetic support

    • All types support precision/scale where applicable and proper casting

  • Interval Arithmetic Functions - New functions for date/time interval operations:

    • date_add(column, interval) - Add interval to date/time column (e.g., date_add(col("date"), "1 DAY"))

    • date_sub(column, interval) - Subtract interval from date/time column

    • Dialect-specific compilation with proper interval handling

  • Join Hints - New hints parameter for join() method to provide query optimization hints:

    • Supports dialect-specific join hints (e.g., USE_INDEX, FORCE_INDEX, IGNORE_INDEX)

    • Hints are passed through to the SQL compiler for database-specific optimizations

  • Complex Join Conditions - Enhanced join() method to support arbitrary Column expressions in join conditions:

    • Beyond simple column pairs, now supports complex predicates and expressions

    • Enables more sophisticated join logic while maintaining SQL pushdown

  • Query Plan Analysis - New explain(analyze=False) method on DataFrame:

    • Returns query execution plan as SQL EXPLAIN output

    • Supports analyze=True for execution plan with statistics (PostgreSQL EXPLAIN ANALYZE)

    • Helps with query optimization and debugging

  • Pivot/Unpivot Operations - New pivot() method for data reshaping:

    • df.pivot(pivot_column, value_column, agg_func="sum", pivot_values=None) - Reshape data by pivoting columns

    • Compiles to CASE WHEN with aggregation for cross-dialect compatibility

    • Supports custom aggregation functions (sum, avg, count, min, max)

    • Automatically detects pivot values if not specified

Changed

  • Enhanced cast() function to support more SQL types with precision/scale (DECIMAL, TIMESTAMP, DATE, TIME, INTERVAL)

  • Improved type annotations throughout the codebase for better IDE support and type safety

Fixed

  • Fixed mypy type checking errors related to type annotations in compiler and DDL modules

  • Fixed ruff linting errors for unused imports and code formatting

0.5.0 - 2025-11-21

Added

  • Compressed File Reading - Automatic detection and support for gzip, bz2, and xz compression formats

    • Support for compressed CSV, JSON, JSONL, and text files

    • Works with both synchronous and asynchronous file readers

    • Compression detection from file extension (.gz, .bz2, .xz) or explicit compression option

    • New compression.py module with open_compressed() and read_compressed_async() utilities

  • Array/JSON Functions - New functions for working with JSON and array data:

    • json_extract(column, path) - Extract values from JSON columns (SQLite JSON1, PostgreSQL JSONB, MySQL JSON)

    • array(elements) - Create array literals

    • array_length(column) - Get array length

    • array_contains(column, value) - Check if array contains value

    • array_position(column, value) - Find position of value in array

    • All functions include dialect-specific SQL compilation for optimal database support

  • Collect Aggregations - New aggregation functions for collecting values into arrays:

    • collect_list(column) - Collect values into a list/array (uses ARRAY_AGG in PostgreSQL, json_group_array in SQLite, GROUP_CONCAT in MySQL)

    • collect_set(column) - Collect distinct values into a set/array

  • Semi-Join and Anti-Join Operations - New DataFrame methods for efficient filtering:

    • semi_join(other, on=[...]) - Filter rows that have matches in another DataFrame (compiles to INNER JOIN with DISTINCT)

    • anti_join(other, on=[...]) - Filter rows that don’t have matches in another DataFrame (compiles to LEFT JOIN with IS NULL)

    • Both methods support column-based joins and custom conditions

  • MERGE/UPSERT Operations - New table method for upsert operations:

    • table.merge(source_df, on=[...], when_matched={...}, when_not_matched={...}) - Merge/upsert rows with conflict resolution

    • Dialect-specific SQL compilation:

      • SQLite: INSERT ... ON CONFLICT DO UPDATE

      • PostgreSQL: Full MERGE statement

      • MySQL: INSERT ... ON DUPLICATE KEY UPDATE (planned)

    • Supports both update-on-match and insert-on-no-match scenarios

  • Comprehensive Test Coverage - Added execution tests for all new features:

    • Tests for compressed file reading (gzip, bz2, xz) in sync and async modes

    • Tests for array/JSON functions with SQLite limitations handled

    • Tests for collect aggregations

    • Tests for semi-join and anti-join operations

    • Tests for MERGE/UPSERT operations

Changed

  • Improved join compilation to handle column qualification properly in semi-join and anti-join operations

  • Enhanced type safety with proper type annotations for new functions and methods

Fixed

  • Fixed type checking issues in compression utilities

  • Fixed column qualification in semi-join and anti-join to avoid ambiguous column errors

0.4.0 - 2025-11-20

Added

  • Strict Type Checking - Enabled mypy strict mode with comprehensive type annotations across the entire codebase

  • Type Stubs for PyArrow - Custom type stubs (stubs/pyarrow/) to provide type information for pyarrow library

  • PEP 561 Compliance - Added py.typed marker file to signal that the package is fully typed

  • Mypy Configuration - Comprehensive mypy configuration in pyproject.toml with strict checking enabled

Changed

  • Type Safety: All functions and methods now have complete type annotations

  • Type Safety: Removed all unused type ignore comments and fixed type inference issues

  • Type Safety: Improved type hints for async operations and Records classes

Fixed

  • Fixed AsyncRecords import issue in async_mutations.py for proper runtime type checking

  • Fixed missing pytest fixtures for example tests by creating conftest.py

  • Fixed all mypy type errors to achieve strict mode compliance

  • Fixed duplicate class and function definitions in logical/plan.py and logical/operators.py

  • Fixed missing function imports in expressions/__init__.py (removed non-existent date_add, date_sub, len, substr, pow, power, trunc)

0.3.0 - 2025-11-20

Added

  • Full async/await support for all database operations, file I/O, and DataFrame operations

  • Async API with async_connect() function returning AsyncDatabase instance

  • Async DataFrame operations - All DataFrame methods now support async execution (collect(), select(), where(), join(), etc.)

  • Async file readers - Async support for CSV, JSON, JSONL, Parquet, and text file reading

  • Async file writers - Async support for writing DataFrames to files and tables

  • Async table mutations - Async insert(), update(), and delete() operations

  • Optional async dependencies - Grouped optional dependencies for async support:

    • moltres[async] - Core async support (aiofiles)

    • moltres[async-postgresql] - PostgreSQL async driver (includes async + asyncpg)

    • moltres[async-mysql] - MySQL async driver (includes async + aiomysql)

    • moltres[async-sqlite] - SQLite async driver (includes async + aiosqlite)

  • Async streaming support - Async iterators for processing large datasets in chunks

  • SQLAlchemy async engine integration - Automatic async driver detection and configuration

  • Comprehensive async test suite (8 new async tests)

Changed

  • Improved type safety with proper async type hints

  • Enhanced error messages for async operations

  • Better separation between sync and async APIs

0.2.0 - 2025-11-20

Added

  • Comprehensive exception hierarchy with specific exception types (ExecutionError, ValidationError, SchemaError, DatabaseConnectionError, UnsupportedOperationError)

  • Input validation for SQL identifiers to prevent injection attacks

  • Batch insert support for improved performance (execute_many() method)

  • Connection pooling configuration options (max_overflow, pool_timeout, pool_recycle, pool_pre_ping)

  • Structured logging throughout the execution layer

  • Enhanced error messages with context (table names, operations, etc.)

  • Comprehensive docstrings for public APIs

  • Pre-commit hooks configuration

  • EditorConfig for consistent code formatting

  • Contributing guide (CONTRIBUTING.md)

  • CI/CD workflow with multi-OS and multi-Python version testing

  • Environment variable support for all configuration options (MOLTRES_DSN, MOLTRES_POOL_SIZE, etc.)

  • Performance monitoring hooks for query execution tracking (register_performance_hook(), unregister_performance_hook())

  • Security best practices documentation (docs/SECURITY.md)

  • Troubleshooting guide (docs/TROUBLESHOOTING.md)

  • Common patterns and examples (docs/EXAMPLES.md)

  • Comprehensive test suite with 113 test cases covering edge cases, security, and error handling

  • Modular file reader architecture - Refactored large reader.py (699 lines) into organized readers/ subdirectory with format-specific modules

Changed

  • Improved type hints throughout the codebase

  • Enhanced error messages with more context

  • Better documentation for limit() behavior

  • Code organization: Split dataframe/reader.py into modular format-specific readers (csv_reader.py, json_reader.py, parquet_reader.py, text_reader.py, schema_inference.py)

Fixed

  • Typo in pyproject.toml (deV → dev)

  • Incomplete insert_rows implementation in io/write.py

  • Missing CI/CD workflow file

  • Type checking issues with optional dependencies (pandas, pyarrow)

Security

  • Added SQL identifier validation to prevent injection attacks

  • Comprehensive security testing and documentation

0.1.0 - Initial Release

Added

  • PySpark-like DataFrame API

  • SQL compilation from logical plans

  • Support for SQLite, PostgreSQL, MySQL

  • File format readers (CSV, JSON, JSONL, Parquet, Text)

  • Streaming support for large datasets

  • Table mutations (insert, update, delete)

  • Column expressions and functions

  • Joins, aggregations, filtering, sorting

  • Type hints and mypy support