|
1 |
| -# Query Farm SQL Scan Planning |
| 1 | +# [Query.Farm](https://query.farm) SQL Scan Planning |
2 | 2 |
|
3 | 3 | A Python library for intelligent file filtering using SQL expressions and metadata-based scan planning. This library enables efficient data lake query optimization by determining which files need to be scanned based on their statistical metadata.
|
4 | 4 |
|
5 | 5 | ## Overview
|
6 | 6 |
|
7 |
| -Query Farm SQL Scan Planning provides predicate pushdown capabilities for file-based data storage systems. By maintaining metadata about file contents (min/max values, value sets, null presence), the library can quickly determine which files contain data that could satisfy a given `SQL WHERE` clause, significantly reducing query execution time. |
| 7 | +This module provides predicate pushdown capabilities for file-based data storage systems. By maintaining metadata about file contents (min/max values, value sets, null presence), the library can quickly determine which files contain data that could satisfy a given `SQL WHERE` clause, significantly reducing query execution time. |
8 | 8 |
|
9 | 9 | ## Features
|
10 | 10 |
|
11 | 11 | - **SQL Expression Parsing**: Parse and evaluate complex `SQL WHERE` clauses using [SQLGlot](https://github.com/tobymao/sqlglot)
|
12 | 12 | - **Metadata-Based Filtering**: Support for both range-based (min/max) and set-based field metadata
|
13 |
| -- **Null Handling**: Comprehensive support for NULL value semantics in SQL expressions |
14 |
| -- **Complex Predicates**: Handle AND, OR, XOR, NOT, IN, BETWEEN, CASE statements, and more |
15 |
| -- **Multiple Data Types**: Support for integers, floats, strings, decimals, and NULL values |
| 13 | +- **Null Handling**: Comprehensive support for `NULL` value semantics in SQL expressions |
| 14 | +- **Complex Predicates**: Handle `AND`, `OR`, `XOR`, `NOT`, `IN`, `BETWEEN`, `CASE` statements, and more |
| 15 | +- **Multiple Data Types**: Support for integers, floats, strings, decimals, and `NULL` values |
16 | 16 | - **Dialect Support**: Configurable SQL dialect support (default: DuckDB)
|
17 | 17 |
|
18 | 18 | ## Installation
|
@@ -216,14 +216,6 @@ rye run pytest --mypy
|
216 | 216 | 4. Ensure all tests pass
|
217 | 217 | 5. Submit a pull request
|
218 | 218 |
|
219 |
| -## License |
220 |
| - |
221 |
| -This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
222 |
| - |
223 |
| -## Related Projects |
224 |
| - |
225 |
| -- [SQLGlot](https://github.com/tobymao/sqlglot) - SQL parser and transpiler |
226 |
| - |
227 | 219 | ## Author
|
228 | 220 |
|
229 | 221 | This Python module was created by [Query.Farm](https://query.farm).
|
|
0 commit comments