diff --git a/docs/freqai.md b/docs/freqai.md index 29a45d042..730f353c8 100644 --- a/docs/freqai.md +++ b/docs/freqai.md @@ -4,8 +4,9 @@ Freqai is still experimental, and should be used at the user's own discretion. Freqai is a module designed to automate a variety of tasks associated with -training a regressor to predict signals based on input features. Among the -the features includes: +training a regressor to predict signals based on input features. + +Among the the features included: * Easy large feature set construction based on simple user input * Sweep model training and backtesting to simulate consistent model retraining through time @@ -16,6 +17,7 @@ the features includes: * Cleaning of NaNs from the data set before training and prediction. TODO: + * live is not automated, still some architectural work to be done ## Background and vocabulary @@ -43,7 +45,7 @@ directly influence nodal weights within the model. ## Install prerequisites -Use `pip` to install the prerequisities with: +Use `pip` to install the prerequisites with: `pip install -r requirements-freqai.txt` @@ -62,7 +64,9 @@ FreqaiExampleStrategy --freqaimodel CatboostPredictionModel --strategy-path freq ``` ## Configuring the bot + ### Example config file + The user interface is isolated to the typical config file. A typical Freqai config setup includes: @@ -152,8 +156,8 @@ data set timerange months. Users can think of this as a "sliding window" which emulates Freqai retraining itself once per week in live using the previous month of data. - ## Running Freqai + ### Training and backtesting The freqai training/backtesting module can be executed with the following command: @@ -196,6 +200,7 @@ The Freqai strategy requires the user to include the following lines of code in return dataframe ``` + The user should also include `populate_any_indicators()` from `templates/FreqaiExampleStrategy.py` which builds the feature set with a proper naming convention for the IFreqaiModel to use later. @@ -216,7 +221,7 @@ freqtrade trade --strategy FreqaiExampleStrategy --config config_freqai.example. By default, Freqai will not find find any existing models and will start by training a new one given the user configuration settings. Following training, it will use that model to predict for the duration of `backtest_period`. After a full `backtest_period` has elapsed, Freqai will auto retrain -a new model, and begin making predictions with the updated model. +a new model, and begin making predictions with the updated model. If the user wishes to start dry/live from a saved model, the following configuration parameters need to be set: @@ -232,13 +237,14 @@ parameters need to be set: Where the `identifier` is the same identifier which was set during the backtesting/training. Meanwhile, the `live_trained_timerange` is the sub-trained timerange (the training window) which was set during backtesting/training. These are available to the user inside `user_data/models/*/sub-train-*`. -`live_full_backtestrange` was the full data range assocaited with the backtest/training (the full time +`live_full_backtestrange` was the full data range associated with the backtest/training (the full time window that the training window and backtesting windows slide through). These values can be located inside the `user_data/models/` directory. In this case, although Freqai will initiate with a -pretrained model, if a full `backtest_period` has elapsed since the end of the user set -`live_trained_timerange`, it will self retrain. +pre-trained model, if a full `backtest_period` has elapsed since the end of the user set +`live_trained_timerange`, it will self retrain. ## Data anylsis techniques + ### Controlling the model learning process The user can define model settings for the data split `data_split_parameters` and learning parameters @@ -258,7 +264,7 @@ the user is asking for `labels` that are 24 candles in the future. ### Removing outliers with the Dissimilarity Index -The Dissimilarity Index (DI) aims to quantiy the uncertainty associated with each +The Dissimilarity Index (DI) aims to quantity the uncertainty associated with each prediction by the model. To do so, Freqai measures the distance between each training data point and all other training data points: @@ -310,11 +316,11 @@ Users can reduce the dimensionality of their features by activating the `princip ``` Which will perform PCA on the features and reduce the dimensionality of the data so that the explained -variance of the data set is >= 0.999. +variance of the data set is >= 0.999. ### Removing outliers based on feature statistical distributions -The user can tell Freqai to remove outlier data points from the trainig/test data sets by setting: +The user can tell Freqai to remove outlier data points from the training/test data sets by setting: ```json "freqai": { @@ -326,9 +332,10 @@ The user can tell Freqai to remove outlier data points from the trainig/test dat Freqai will check the statistical distributions of each feature (or component if the user activated `principal_component_analysis`) and remove any data point that sits more than 3 standard deviations away -from the mean. +from the mean. ## Additional information + ### Feature standardization The feature set created by the user is automatically standardized to the training @@ -337,5 +344,5 @@ data only. This includes all test data and unseen prediction data (dry/live/back ### File structure `user_data_dir/models/` contains all the data associated with the trainings and -backtestings. This file structure is heavily controlled and read by the `FreqaiDataKitchen()` -and should thus not be modified. +backtests. This file structure is heavily controlled and read by the `FreqaiDataKitchen()` +and should thus not be modified. diff --git a/freqtrade/freqai/freqai_interface.py b/freqtrade/freqai/freqai_interface.py index ae05ae33a..b7c879ff0 100644 --- a/freqtrade/freqai/freqai_interface.py +++ b/freqtrade/freqai/freqai_interface.py @@ -43,7 +43,7 @@ class IFreqaiModel(ABC): def start(self, dataframe: DataFrame, metadata: dict, strategy: IStrategy) -> DataFrame: """ Entry point to the FreqaiModel, it will train a new model if - necesssary before making the prediction. + necessary before making the prediction. The backtesting and training paradigm is a sliding training window with a following backtest window. Both windows slide according to the length of the backtest window. This function is not intended to be @@ -54,7 +54,7 @@ class IFreqaiModel(ABC): :dataframe: Full dataframe coming from strategy - it contains entire backtesting timerange + additional historical data necessary to train the model. - :metadata: pair metadataa coming from strategy. + :metadata: pair metadata coming from strategy. """ live = strategy.dp.runmode in (RunMode.DRY_RUN, RunMode.LIVE) @@ -71,7 +71,7 @@ class IFreqaiModel(ABC): logger.info("going to train %s timeranges", len(self.dh.training_timeranges)) - # Loop enforcing the sliding window training/backtesting paragigm + # Loop enforcing the sliding window training/backtesting paradigm # tr_train is the training time range e.g. 1 historical month # tr_backtest is the backtesting time range e.g. the week directly # following tr_train. Both of these windows slide through the diff --git a/freqtrade/templates/ExamplePredictionModel.py b/freqtrade/templates/ExamplePredictionModel.py index 796fb23ed..3d2b7a808 100644 --- a/freqtrade/templates/ExamplePredictionModel.py +++ b/freqtrade/templates/ExamplePredictionModel.py @@ -42,7 +42,7 @@ class ExamplePredictionModel(IFreqaiModel): def train(self, unfiltered_dataframe: DataFrame, metadata: dict) -> Tuple[DataFrame, DataFrame]: """ - Filter the training data and train a model to it. Train makes heavy use of the datahkitchen + Filter the training data and train a model to it. Train makes heavy use of the datakitchen for storing, saving, loading, and analyzing the data. :params: :unfiltered_dataframe: Full dataframe for the current training period diff --git a/mkdocs.yml b/mkdocs.yml index 64d78363d..18744e0d5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -35,8 +35,8 @@ nav: - Edge Positioning: edge.md - Advanced Strategy: strategy-advanced.md - Advanced Hyperopt: advanced-hyperopt.md - - Sandbox Testing: sandbox-testing.md - Freqai: freqai.md + - Sandbox Testing: sandbox-testing.md - FAQ: faq.md - SQL Cheat-sheet: sql_cheatsheet.md - Strategy migration: strategy_migration.md