Feature/programmatic api docs #27
Conversation
Add comprehensive documentation for PyneCore's programmatic API including: - README with overview and quick start - Core components reference (OHLCVReader, SymInfo, ScriptRunner) - Basic usage examples - Best practices guide for performance and error handling The documentation provides developers with everything needed to integrate PyneCore into Python applications programmatically
- Simplify README content and remove redundant examples - Streamline code examples in basic and advanced sections - Remove verbose logging and progress tracking from examples - Consolidate best practices with more concise examples
- Replace core-components.md with quick-start.md as the primary entry point - Update basic-examples.md with more practical usage patterns - Remove deprecated best-practices.md and advanced-examples.md - Update README.md to reflect new documentation structure
| ### Memory Management | ||
|
|
||
| #### Use Context Managers | ||
| Always use context managers to ensure proper resource cleanup: |
There was a problem hiding this comment.
I think this 2 is the same and both of them is good. So should be deleted
| ``` | ||
|
|
||
| #### Monitor Memory Usage | ||
| For large datasets, monitor memory consumption: |
There was a problem hiding this comment.
There's no memory consumptions! OHLCV Reader uses MMAP, it loads data row by row and all the memory is handled by the OS. So it is simply not true. Please check and try to understand it.
|
|
||
| ### Parallel Processing | ||
|
|
||
| #### Use ProcessPoolExecutor for CPU-Bound Tasks |
There was a problem hiding this comment.
I don't think it is needed here. It is not a best practice. Because PyneCore is a sync engine, you don't need to use parallel processing, only if you want to make better performance. But that is much more advanced stuff and not needed here.
| script_path=task.script_path, | ||
| ohlcv_iter=reader, | ||
| syminfo=task.syminfo, | ||
| update_syminfo_every_run=True # Required for parallel execution |
There was a problem hiding this comment.
This is not true. It is required for serial execution, when you want to run multiple scripts in one timesteps. If you have multiprocessing, then all the process have its own memory, so it is not needed, I think.
|
|
||
| ### Data Access Optimization | ||
|
|
||
| #### Minimize File I/O |
There was a problem hiding this comment.
It is not true again. Because of MMAP you can simply open it any time you want, because the OS take care of caching and preloading. And you won't use more memory and file I/O. This is the trick, and this is why the OHLCVReader is so good.
| runner.run() | ||
| # File automatically closed after use | ||
|
|
||
| # ✅ Process data in chunks for very very large datasets |
| ``` | ||
|
|
||
| ### Performance Optimization | ||
| - Use `skip_gaps=True` in `read_from()` to avoid processing invalid data |
There was a problem hiding this comment.
What is this?! It is not true!
|
|
||
| ### Performance Optimization | ||
| - Use `skip_gaps=True` in `read_from()` to avoid processing invalid data | ||
| - Set appropriate `last_bar_index` for better progress tracking |
| ### Performance Optimization | ||
| - Use `skip_gaps=True` in `read_from()` to avoid processing invalid data | ||
| - Set appropriate `last_bar_index` for better progress tracking | ||
| - Consider `update_syminfo_every_run=False` for single-threaded usage |
There was a problem hiding this comment.
It is the default and nothing about threads! PyneCore supports only single thread!
| - Use `skip_gaps=True` in `read_from()` to avoid processing invalid data | ||
| - Set appropriate `last_bar_index` for better progress tracking | ||
| - Consider `update_syminfo_every_run=False` for single-threaded usage | ||
| - Process data in time chunks for memory efficiency with large datasets |
There was a problem hiding this comment.
It is a huge lie! And repeaded!
…atures and examples - Update README with clearer title and description - Add new API reference and practical examples documentation - Enhance quick-start guide with automatic data conversion examples - Expand basic examples with more real-world use cases - Improve structure and clarity across all programmatic docs
No description provided.