Merge pull request #9152 from stash86/bt-metrics

Add recursive-analysis sub-command

docs/recursive-analysis.md

@@ -0,0 +1,89 @@
+# Recursive analysis
+
+This page explains how to validate your strategy for inaccuracies due to recursive issues with certain indicators.
+
+A recursive formula defines any term of a sequence relative to its preceding term(s). An example of a recursive formula is a<sub>n</sub> = a<sub>n-1</sub> + b.
+
+Why does this matter for Freqtrade? In backtesting, the bot will get full data of the pairs according to the timerange specified. But in a dry/live run, the bot will be limited by the amount of data each exchanges gives.
+
+For example, to calculate a very basic indicator called `steps`, the first row's value is always 0, while the following rows' values are equal to the value of the previous row plus 1. If I were to calculate it using the latest 1000 candles, then the `steps` value of the first row is 0, and the `steps` value at the last closed candle is 999.
+
+What happens if the calculation is using only the latest 500 candles? Then instead of 999, the `steps` value at last closed candle is 499. The difference of the value means your backtest result can differ from your dry/live run result.
+
+The `recursive-analysis` command requires historic data to be available. To learn how to get data for the pairs and exchange you're interested in,
+head over to the [Data Downloading](data-download.md) section of the documentation.
+
+This command is built upon preparing different lengths of data and calculates indicators based on them.
+This does not backtest the strategy itself, but rather only calculates the indicators. After calculating the indicators of different startup candle values (`startup_candle_count`) are done, the values of last rows across all specified `startup_candle_count` are compared to see how much variance they show compared to the base calculation.
+
+Command settings:
+
+- Use the `-p` option to set your desired pair to analyze. Since we are only looking at indicator values, using more than one pair is redundant. Preferably use a pair with a relatively high price and at least moderate volatility, such as BTC or ETH, to avoid rounding issues that can make the results inaccurate. If no pair is set on the command, the pair used for this analysis is the first pair in the whitelist.
+- It is recommended to set a long timerange (at least 5000 candles) so that the initial indicators' calculation that is going to be used as a benchmark has very small or no recursive issues itself. For example, for a 5m timeframe, a timerange of 5000 candles would be equal to 18 days.
+- `--cache` is forced to "none" to avoid loading previous indicators calculation automatically.
+
+In addition to the recursive formula check, this command also carries out a simple lookahead bias check on the indicator values only. For a full lookahead check, use [Lookahead-analysis](lookahead-analysis.md).
+
+## Recursive-analysis command reference
+
+```
+usage: freqtrade recursive-analysis [-h] [-v] [--logfile FILE] [-V] [-c PATH]
+                                    [-d PATH] [--userdir PATH] [-s NAME]
+                                    [--strategy-path PATH]
+                                    [--recursive-strategy-search]
+                                    [--freqaimodel NAME]
+                                    [--freqaimodel-path PATH] [-i TIMEFRAME]
+                                    [--timerange TIMERANGE]
+                                    [--data-format-ohlcv {json,jsongz,hdf5,feather,parquet}]
+                                    [-p PAIR]
+                                    [--freqai-backtest-live-models]
+                                    [--startup-candle STARTUP_CANDLES [STARTUP_CANDLES ...]]
+
+optional arguments:
+-p PAIR, --pairs PAIR
+                        Limit command to this pair.
+--startup-candle STARTUP_CANDLE [STARTUP_CANDLE ...]
+                        Provide a space-separated list of startup_candle_count to
+                        be checked. Default : `199 399 499 999 1999`.
+```
+
+### Why are odd-numbered default startup candles used?
+
+The default value for startup candles are odd numbers. When the bot fetches candle data from the exchange's API, the last candle is the one being checked by the bot and the rest of the data are the "startup candles".
+
+For example, Binance allows 1000 candles per API call. When the bot receives 1000 candles, the last candle is the "current candle", and the preceding 999 candles are the "startup candles". By setting the startup candle count as 1000 instead of 999, the bot will try to fetch 1001 candles instead. The exchange API will then send candle data in a paginated form, i.e. in case of the Binance API, this will be two groups- one of length 1000 and another of length 1. This results in the bot thinking the strategy needs 1001 candles of data, and so it will download 2000 candles worth of data instead, which means there will be 1 "current candle" and 1999 "startup candles".
+
+Furthermore, exchanges limit the number of consecutive bulk API calls, e.g. Binance allows 5 calls. In this case, only 5000 candles can be downloaded from Binance API without hitting the API rate limit, which means the max `startup_candle_count` you can have is 4999.
+
+Please note that this candle limit may be changed in the future by the exchanges without any prior notice.
+
+### How does the command work?
+
+- Firstly an initial indicator calculation is carried out using the supplied timerange to generate a benchmark for indicator values.
+- After setting the benchmark it will then carry out additional runs for each of the different startup candle count values.
+- The command will then compare the indicator values at the last candle rows and report the differences in a table.
+
+## Understanding the recursive-analysis output
+
+This is an example of an output results table where at least one indicator has a recursive formula issue:
+
+```
+| indicators   | 20      | 40      | 80     | 100    | 150     | 300     | 999    |
+|--------------+---------+---------+--------+--------+---------+---------+--------|
+| rsi_30       | nan%    | -6.025% | 0.612% | 0.828% | -0.140% | 0.000%  | 0.000% |
+| rsi_14       | 24.141% | -0.876% | 0.070% | 0.007% | -0.000% | -0.000% | -      |
+```
+
+The column headers indicate the different `startup_candle_count` used in the analysis. The values in the table indicate the variance of the calculated indicators compared to the benchmark value.
+
+`nan%` means the value of that indicator cannot be calculated due to lack of data. In this example, you cannot calculate RSI with length 30 with just 21 candles (1 current candle + 20 startup candles).
+
+Users should assess the table per indicator to decide if the specified `startup_candle_count` results in a sufficiently small variance so that the indicator does not have any effect on entries and/or exits.
+
+As such, aiming for absolute zero variance (shown by `-` value) might not be the best option, because some indicators might require you to use such a long `startup_candle_count` to have zero variance.
+
+## Caveats
+
+- `recursive-analysis` will only calculate and compare the indicator values at the last row. The output table reports the percentage differences between the different startup candle count calculations and the original benchmark calculation. Whether it has any actual impact on your entries and exits is not included.
+- The ideal scenario is that indicators will have no variance (or at least very close to 0%) despite the startup candle being varied. In reality, indicators such as EMA are using a recursive formula to calculate indicator values, so the goal is not necessarily to have zero percentage variance, but to have the variance low enough (and therefore `startup_candle_count` high enough) that the recursion inherent in the indicator will not have any real impact on trading decisions.
+- `recursive-analysis` will only run calculations on `populate_indicators` and `@informative` decorator(s). If you put any indicator calculation on `populate_entry_trend` or `populate_exit_trend`, it won't be calculated.

docs/strategy-customization.md

@@ -168,10 +168,12 @@ Most indicators have an instable startup period, in which they are either not av
 To account for this, the strategy can be assigned the `startup_candle_count` attribute.
 This should be set to the maximum number of candles that the strategy requires to calculate stable indicators. In the case where a user includes higher timeframes with informative pairs, the `startup_candle_count` does not necessarily change. The value is the maximum period (in candles) that any of the informatives timeframes need to compute stable indicators.
 
-In this example strategy, this should be set to 100 (`startup_candle_count = 100`), since the longest needed history is 100 candles.
+You can use [recursive-analysis](recursive-analysis.md) to check and find the correct `startup_candle_count` to be used.
+
+In this example strategy, this should be set to 400 (`startup_candle_count = 400`), since the minimum needed history for ema100 calculation to make sure the value is correct is 400 candles.
 
 ``` python
-    dataframe['ema100'] = ta.EMA(dataframe, timeperiod=100)
+    dataframe['ema100'] = ta.EMA(dataframe, timeperiod=400)
 ```
 
 By letting the bot know how much history is needed, backtest trades can start at the specified timerange during backtesting and hyperopt.
@@ -193,11 +195,11 @@ Let's try to backtest 1 month (January 2019) of 5m candles using an example stra
 freqtrade backtesting --timerange 20190101-20190201 --timeframe 5m
 ```
 
-Assuming `startup_candle_count` is set to 100, backtesting knows it needs 100 candles to generate valid buy signals. It will load data from `20190101 - (100 * 5m)` - which is ~2018-12-31 15:30:00.
+Assuming `startup_candle_count` is set to 400, backtesting knows it needs 400 candles to generate valid buy signals. It will load data from `20190101 - (400 * 5m)` - which is ~2018-12-30 11:40:00.
 If this data is available, indicators will be calculated with this extended timerange. The instable startup period (up to 2019-01-01 00:00:00) will then be removed before starting backtesting.
 
 !!! Note
-    If data for the startup period is not available, then the timerange will be adjusted to account for this startup period - so Backtesting would start at 2019-01-01 08:30:00.
+    If data for the startup period is not available, then the timerange will be adjusted to account for this startup period - so Backtesting would start at 2019-01-02 09:20:00.
 
 ### Entry signal rules
 

freqtrade/commands/__init__.py

@@ -20,7 +20,8 @@ from freqtrade.commands.list_commands import (start_list_exchanges, start_list_f
                                               start_list_timeframes, start_show_trades)
 from freqtrade.commands.optimize_commands import (start_backtesting, start_backtesting_show,
                                                   start_edge, start_hyperopt,
-                                                  start_lookahead_analysis)
+                                                  start_lookahead_analysis,
+                                                  start_recursive_analysis)
 from freqtrade.commands.pairlist_commands import start_test_pairlist
 from freqtrade.commands.plot_commands import start_plot_dataframe, start_plot_profit
 from freqtrade.commands.strategy_utils_commands import start_strategy_update

freqtrade/commands/arguments.py

@@ -122,6 +122,8 @@ ARGS_LOOKAHEAD_ANALYSIS = [
     a for a in ARGS_BACKTEST if a not in ("position_stacking", "use_max_market_positions", 'cache')
     ] + ["minimum_trade_amount", "targeted_trade_amount", "lookahead_analysis_exportfilename"]
 
+ARGS_RECURSIVE_ANALYSIS = ["timeframe", "timerange", "dataformat_ohlcv", "pairs", "startup_candle"]
+
 
 class Arguments:
     """
@@ -206,8 +208,9 @@ class Arguments:
                                         start_list_strategies, start_list_timeframes,
                                         start_lookahead_analysis, start_new_config,
                                         start_new_strategy, start_plot_dataframe, start_plot_profit,
-                                        start_show_trades, start_strategy_update,
-                                        start_test_pairlist, start_trading, start_webserver)
+                                        start_recursive_analysis, start_show_trades,
+                                        start_strategy_update, start_test_pairlist, start_trading,
+                                        start_webserver)
 
         subparsers = self.parser.add_subparsers(dest='command',
                                                 # Use custom message when no subhandler is added
@@ -467,3 +470,14 @@ class Arguments:
 
         self._build_args(optionlist=ARGS_LOOKAHEAD_ANALYSIS,
                          parser=lookahead_analayis_cmd)
+
+        # Add recursive_analysis subcommand
+        recursive_analayis_cmd = subparsers.add_parser(
+            'recursive-analysis',
+            help="Check for potential recursive formula issue.",
+            parents=[_common_parser, _strategy_parser])
+
+        recursive_analayis_cmd.set_defaults(func=start_recursive_analysis)
+
+        self._build_args(optionlist=ARGS_RECURSIVE_ANALYSIS,
+                         parser=recursive_analayis_cmd)

freqtrade/commands/cli_options.py

@@ -705,4 +705,9 @@ AVAILABLE_CLI_OPTIONS = {
         help="Use this csv-filename to store lookahead-analysis-results",
         type=str
     ),
+    "startup_candle": Arg(
+        '--startup-candle',
+        help='Specify startup candles to be checked (`199`, `499`, `999`, `1999`).',
+        nargs='+',
+    ),
 }

freqtrade/commands/optimize_commands.py

@@ -144,3 +144,15 @@ def start_lookahead_analysis(args: Dict[str, Any]) -> None:
 
     config = setup_utils_configuration(args, RunMode.UTIL_NO_EXCHANGE)
     LookaheadAnalysisSubFunctions.start(config)
+
+
+def start_recursive_analysis(args: Dict[str, Any]) -> None:
+    """
+    Start the backtest recursive tester script
+    :param args: Cli args from Arguments()
+    :return: None
+    """
+    from freqtrade.optimize.recursive_analysis_helpers import RecursiveAnalysisSubFunctions
+
+    config = setup_utils_configuration(args, RunMode.UTIL_NO_EXCHANGE)
+    RecursiveAnalysisSubFunctions.start(config)

freqtrade/configuration/configuration.py

@@ -490,6 +490,9 @@ class Configuration:
         self._args_to_config(config, argname='lookahead_analysis_exportfilename',
                              logstring='Path to store lookahead-analysis-results: {}')
 
+        self._args_to_config(config, argname='startup_candle',
+                             logstring='Startup candle to be used on recursive analysis: {}')
+
     def _process_runmode(self, config: Config) -> None:
 
         self._args_to_config(config, argname='dry_run',

freqtrade/constants.py

@@ -178,6 +178,11 @@ CONF_SCHEMA = {
         'minimum_trade_amount': {'type': 'number', 'default': 10},
         'targeted_trade_amount': {'type': 'number', 'default': 20},
         'lookahead_analysis_exportfilename': {'type': 'string'},
+        'startup_candle': {
+            'type': 'array',
+            'uniqueItems': True,
+            'default': [199, 399, 499, 999, 1999],
+        },
         'liquidation_buffer': {'type': 'number', 'minimum': 0.0, 'maximum': 0.99},
         'backtest_breakdown': {
             'type': 'array',

freqtrade/optimize/base_analysis.py

@@ -0,0 +1,66 @@
+import logging
+from copy import deepcopy
+from datetime import datetime, timezone
+from typing import Any, Dict, Optional
+
+from pandas import DataFrame
+
+from freqtrade.configuration import TimeRange
+
+
+logger = logging.getLogger(__name__)
+
+
+class VarHolder:
+    timerange: TimeRange
+    data: DataFrame
+    indicators: Dict[str, DataFrame]
+    result: DataFrame
+    compared: DataFrame
+    from_dt: datetime
+    to_dt: datetime
+    compared_dt: datetime
+    timeframe: str
+    startup_candle: int
+
+
+class BaseAnalysis:
+
+    def __init__(self, config: Dict[str, Any], strategy_obj: Dict):
+        self.failed_bias_check = True
+        self.full_varHolder = VarHolder()
+        self.exchange: Optional[Any] = None
+        self._fee = None
+
+        # pull variables the scope of the lookahead_analysis-instance
+        self.local_config = deepcopy(config)
+        self.local_config['strategy'] = strategy_obj['name']
+        self.strategy_obj = strategy_obj
+
+    @staticmethod
+    def dt_to_timestamp(dt: datetime):
+        timestamp = int(dt.replace(tzinfo=timezone.utc).timestamp())
+        return timestamp
+
+    def fill_full_varholder(self):
+        self.full_varHolder = VarHolder()
+
+        # define datetime in human-readable format
+        parsed_timerange = TimeRange.parse_timerange(self.local_config['timerange'])
+
+        if parsed_timerange.startdt is None:
+            self.full_varHolder.from_dt = datetime.fromtimestamp(0, tz=timezone.utc)
+        else:
+            self.full_varHolder.from_dt = parsed_timerange.startdt
+
+        if parsed_timerange.stopdt is None:
+            self.full_varHolder.to_dt = datetime.utcnow()
+        else:
+            self.full_varHolder.to_dt = parsed_timerange.stopdt
+
+        self.prepare_data(self.full_varHolder, self.local_config['pairs'])
+
+    def start(self) -> None:
+
+        # first make a single backtest
+        self.fill_full_varholder()

freqtrade/optimize/lookahead_analysis.py

@@ -1,35 +1,23 @@
 import logging
 import shutil
 from copy import deepcopy
-from datetime import datetime, timedelta, timezone
+from datetime import datetime, timedelta
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List
 
 from pandas import DataFrame
 
-from freqtrade.configuration import TimeRange
 from freqtrade.data.history import get_timerange
 from freqtrade.exchange import timeframe_to_minutes
 from freqtrade.loggers.set_log_levels import (reduce_verbosity_for_bias_tester,
                                               restore_verbosity_for_bias_tester)
 from freqtrade.optimize.backtesting import Backtesting
+from freqtrade.optimize.base_analysis import BaseAnalysis, VarHolder
 
 
 logger = logging.getLogger(__name__)
 
 
-class VarHolder:
-    timerange: TimeRange
-    data: DataFrame
-    indicators: Dict[str, DataFrame]
-    result: DataFrame
-    compared: DataFrame
-    from_dt: datetime
-    to_dt: datetime
-    compared_dt: datetime
-    timeframe: str
-
-
 class Analysis:
     def __init__(self) -> None:
         self.total_signals = 0
@@ -39,29 +27,18 @@ class Analysis:
         self.has_bias = False
 
 
-class LookaheadAnalysis:
+class LookaheadAnalysis(BaseAnalysis):
 
     def __init__(self, config: Dict[str, Any], strategy_obj: Dict):
-        self.failed_bias_check = True
-        self.full_varHolder = VarHolder()
+
+        super().__init__(config, strategy_obj)
 
         self.entry_varHolders: List[VarHolder] = []
         self.exit_varHolders: List[VarHolder] = []
-        self.exchange: Optional[Any] = None
-        self._fee = None
 
-        # pull variables the scope of the lookahead_analysis-instance
-        self.local_config = deepcopy(config)
-        self.local_config['strategy'] = strategy_obj['name']
         self.current_analysis = Analysis()
         self.minimum_trade_amount = config['minimum_trade_amount']
         self.targeted_trade_amount = config['targeted_trade_amount']
-        self.strategy_obj = strategy_obj
-
-    @staticmethod
-    def dt_to_timestamp(dt: datetime):
-        timestamp = int(dt.replace(tzinfo=timezone.utc).timestamp())
-        return timestamp
 
     @staticmethod
     def get_result(backtesting: Backtesting, processed: DataFrame):
@@ -162,24 +139,6 @@ class LookaheadAnalysis:
         varholder.indicators = backtesting.strategy.advise_all_indicators(varholder.data)
         varholder.result = self.get_result(backtesting, varholder.indicators)
 
-    def fill_full_varholder(self):
-        self.full_varHolder = VarHolder()
-
-        # define datetime in human-readable format
-        parsed_timerange = TimeRange.parse_timerange(self.local_config['timerange'])
-
-        if parsed_timerange.startdt is None:
-            self.full_varHolder.from_dt = datetime.fromtimestamp(0, tz=timezone.utc)
-        else:
-            self.full_varHolder.from_dt = parsed_timerange.startdt
-
-        if parsed_timerange.stopdt is None:
-            self.full_varHolder.to_dt = datetime.utcnow()
-        else:
-            self.full_varHolder.to_dt = parsed_timerange.stopdt
-
-        self.prepare_data(self.full_varHolder, self.local_config['pairs'])
-
     def fill_entry_and_exit_varHolders(self, result_row):
         # entry_varHolder
         entry_varHolder = VarHolder()
@@ -246,8 +205,7 @@ class LookaheadAnalysis:
 
     def start(self) -> None:
 
-        # first make a single backtest
-        self.fill_full_varholder()
+        super().start()
 
         reduce_verbosity_for_bias_tester()
 

freqtrade/optimize/lookahead_analysis_helpers.py

@@ -184,12 +184,12 @@ class LookaheadAnalysisSubFunctions:
 
         lookaheadAnalysis_instances = []
 
-        # unify --strategy and --strategy_list to one list
+        # unify --strategy and --strategy-list to one list
         if not (strategy_list := config.get('strategy_list', [])):
             if config.get('strategy') is None:
                 raise OperationalException(
                     "No Strategy specified. Please specify a strategy via --strategy or "
-                    "--strategy_list"
+                    "--strategy-list"
                 )
             strategy_list = [config['strategy']]
 
@@ -211,5 +211,5 @@ class LookaheadAnalysisSubFunctions:
         else:
             logger.error("There were no strategies specified neither through "
                          "--strategy nor through "
-                         "--strategy_list "
+                         "--strategy-list "
                          "or timeframe was not specified.")

freqtrade/optimize/recursive_analysis.py

@@ -0,0 +1,182 @@
+import logging
+import shutil
+from copy import deepcopy
+from datetime import timedelta
+from pathlib import Path
+from typing import Any, Dict, List
+
+from pandas import DataFrame
+
+from freqtrade.exchange import timeframe_to_minutes
+from freqtrade.loggers.set_log_levels import (reduce_verbosity_for_bias_tester,
+                                              restore_verbosity_for_bias_tester)
+from freqtrade.optimize.backtesting import Backtesting
+from freqtrade.optimize.base_analysis import BaseAnalysis, VarHolder
+
+
+logger = logging.getLogger(__name__)
+
+
+class RecursiveAnalysis(BaseAnalysis):
+
+    def __init__(self, config: Dict[str, Any], strategy_obj: Dict):
+
+        self._startup_candle = config.get('startup_candle', [199, 399, 499, 999, 1999])
+
+        super().__init__(config, strategy_obj)
+
+        self.partial_varHolder_array: List[VarHolder] = []
+        self.partial_varHolder_lookahead_array: List[VarHolder] = []
+
+        self.dict_recursive: Dict[str, Any] = dict()
+
+    # For recursive bias check
+    # analyzes two data frames with processed indicators and shows differences between them.
+    def analyze_indicators(self):
+
+        pair_to_check = self.local_config['pairs'][0]
+        logger.info("Start checking for recursive bias")
+
+        # check and report signals
+        base_last_row = self.full_varHolder.indicators[pair_to_check].iloc[-1]
+
+        for part in self.partial_varHolder_array:
+            part_last_row = part.indicators[pair_to_check].iloc[-1]
+
+            compare_df = base_last_row.compare(part_last_row)
+            if compare_df.shape[0] > 0:
+                # print(compare_df)
+                for col_name, values in compare_df.items():
+                    # print(col_name)
+                    if 'other' == col_name:
+                        continue
+                    indicators = values.index
+
+                    for indicator in indicators:
+                        if (indicator not in self.dict_recursive):
+                            self.dict_recursive[indicator] = {}
+
+                        values_diff = compare_df.loc[indicator]
+                        values_diff_self = values_diff.loc['self']
+                        values_diff_other = values_diff.loc['other']
+                        diff = (values_diff_other - values_diff_self) / values_diff_self * 100
+
+                        self.dict_recursive[indicator][part.startup_candle] = f"{diff:.3f}%"
+
+            else:
+                logger.info("No difference found. Stop the process.")
+                break
+
+    # For lookahead bias check
+    # analyzes two data frames with processed indicators and shows differences between them.
+    def analyze_indicators_lookahead(self):
+
+        pair_to_check = self.local_config['pairs'][0]
+        logger.info("Start checking for lookahead bias on indicators only")
+
+        part = self.partial_varHolder_lookahead_array[0]
+        part_last_row = part.indicators[pair_to_check].iloc[-1]
+        date_to_check = part_last_row['date']
+        index_to_get = (self.full_varHolder.indicators[pair_to_check]['date'] == date_to_check)
+        base_row_check = self.full_varHolder.indicators[pair_to_check].loc[index_to_get].iloc[-1]
+
+        check_time = part.to_dt.strftime('%Y-%m-%dT%H:%M:%S')
+
+        logger.info(f"Check indicators at {check_time}")
+        # logger.info(f"vs {part_timerange} with {part.startup_candle} startup candle")
+
+        compare_df = base_row_check.compare(part_last_row)
+        if compare_df.shape[0] > 0:
+            # print(compare_df)
+            for col_name, values in compare_df.items():
+                # print(col_name)
+                if 'other' == col_name:
+                    continue
+                indicators = values.index
+
+                for indicator in indicators:
+                    logger.info(f"=> found lookahead in indicator {indicator}")
+                    # logger.info("base value {:.5f}".format(values_diff_self))
+                    # logger.info("part value {:.5f}".format(values_diff_other))
+
+        else:
+            logger.info("No lookahead bias on indicators found. Stop the process.")
+
+    def prepare_data(self, varholder: VarHolder, pairs_to_load: List[DataFrame]):
+
+        if 'freqai' in self.local_config and 'identifier' in self.local_config['freqai']:
+            # purge previous data if the freqai model is defined
+            # (to be sure nothing is carried over from older backtests)
+            path_to_current_identifier = (
+                Path(f"{self.local_config['user_data_dir']}/models/"
+                     f"{self.local_config['freqai']['identifier']}").resolve())
+            # remove folder and its contents
+            if Path.exists(path_to_current_identifier):
+                shutil.rmtree(path_to_current_identifier)
+
+        prepare_data_config = deepcopy(self.local_config)
+        prepare_data_config['timerange'] = (str(self.dt_to_timestamp(varholder.from_dt)) + "-" +
+                                            str(self.dt_to_timestamp(varholder.to_dt)))
+        prepare_data_config['exchange']['pair_whitelist'] = pairs_to_load
+
+        backtesting = Backtesting(prepare_data_config, self.exchange)
+        backtesting._set_strategy(backtesting.strategylist[0])
+
+        varholder.data, varholder.timerange = backtesting.load_bt_data()
+        backtesting.load_bt_data_detail()
+        varholder.timeframe = backtesting.timeframe
+
+        varholder.indicators = backtesting.strategy.advise_all_indicators(varholder.data)
+
+    def fill_partial_varholder(self, start_date, startup_candle):
+        logger.info(f"Calculating indicators using startup candle of {startup_candle}.")
+        partial_varHolder = VarHolder()
+
+        partial_varHolder.from_dt = start_date
+        partial_varHolder.to_dt = self.full_varHolder.to_dt
+        partial_varHolder.startup_candle = startup_candle
+
+        self.local_config['startup_candle_count'] = startup_candle
+
+        self.prepare_data(partial_varHolder, self.local_config['pairs'])
+
+        self.partial_varHolder_array.append(partial_varHolder)
+
+    def fill_partial_varholder_lookahead(self, end_date):
+        logger.info("Calculating indicators to test lookahead on indicators.")
+
+        partial_varHolder = VarHolder()
+
+        partial_varHolder.from_dt = self.full_varHolder.from_dt
+        partial_varHolder.to_dt = end_date
+
+        self.prepare_data(partial_varHolder, self.local_config['pairs'])
+
+        self.partial_varHolder_lookahead_array.append(partial_varHolder)
+
+    def start(self) -> None:
+
+        super().start()
+
+        reduce_verbosity_for_bias_tester()
+        start_date_full = self.full_varHolder.from_dt
+        end_date_full = self.full_varHolder.to_dt
+
+        timeframe_minutes = timeframe_to_minutes(self.full_varHolder.timeframe)
+
+        end_date_partial = start_date_full + timedelta(minutes=int(timeframe_minutes * 10))
+
+        self.fill_partial_varholder_lookahead(end_date_partial)
+
+        # restore_verbosity_for_bias_tester()
+
+        start_date_partial = end_date_full - timedelta(minutes=int(timeframe_minutes))
+
+        for startup_candle in self._startup_candle:
+            self.fill_partial_varholder(start_date_partial, int(startup_candle))
+
+        # Restore verbosity, so it's not too quiet for the next strategy
+        restore_verbosity_for_bias_tester()
+
+        self.analyze_indicators()
+        self.analyze_indicators_lookahead()

freqtrade/optimize/recursive_analysis_helpers.py

@@ -0,0 +1,106 @@
+import logging
+import time
+from pathlib import Path
+from typing import Any, Dict, List
+
+from freqtrade.constants import Config
+from freqtrade.exceptions import OperationalException
+from freqtrade.optimize.recursive_analysis import RecursiveAnalysis
+from freqtrade.resolvers import StrategyResolver
+
+
+logger = logging.getLogger(__name__)
+
+
+class RecursiveAnalysisSubFunctions:
+
+    @staticmethod
+    def text_table_recursive_analysis_instances(
+            recursive_instances: List[RecursiveAnalysis]):
+        startups = recursive_instances[0]._startup_candle
+        headers = ['indicators']
+        for candle in startups:
+            headers.append(candle)
+
+        data = []
+        for inst in recursive_instances:
+            if len(inst.dict_recursive) > 0:
+                for indicator, values in inst.dict_recursive.items():
+                    temp_data = [indicator]
+                    for candle in startups:
+                        temp_data.append(values.get(int(candle), '-'))
+                    data.append(temp_data)
+
+        from tabulate import tabulate
+        table = tabulate(data, headers=headers, tablefmt="orgtbl")
+        print(table)
+        return table, headers, data
+
+    @staticmethod
+    def calculate_config_overrides(config: Config):
+        if 'timerange' not in config:
+            # setting a timerange is enforced here
+            raise OperationalException(
+                "Please set a timerange. "
+                "A timerange of 5000 candles are enough for recursive analysis."
+            )
+
+        if config.get('backtest_cache') is None:
+            config['backtest_cache'] = 'none'
+        elif config['backtest_cache'] != 'none':
+            logger.info(f"backtest_cache = "
+                        f"{config['backtest_cache']} detected. "
+                        f"Inside recursive-analysis it is enforced to be 'none'. "
+                        f"Changed it to 'none'")
+            config['backtest_cache'] = 'none'
+        return config
+
+    @staticmethod
+    def initialize_single_recursive_analysis(config: Config, strategy_obj: Dict[str, Any]):
+
+        logger.info(f"Recursive test of {Path(strategy_obj['location']).name} started.")
+        start = time.perf_counter()
+        current_instance = RecursiveAnalysis(config, strategy_obj)
+        current_instance.start()
+        elapsed = time.perf_counter() - start
+        logger.info(f"Checking recursive and indicator-only lookahead bias of indicators "
+                    f"of {Path(strategy_obj['location']).name} "
+                    f"took {elapsed:.0f} seconds.")
+        return current_instance
+
+    @staticmethod
+    def start(config: Config):
+        config = RecursiveAnalysisSubFunctions.calculate_config_overrides(config)
+
+        strategy_objs = StrategyResolver.search_all_objects(
+            config, enum_failed=False, recursive=config.get('recursive_strategy_search', False))
+
+        RecursiveAnalysis_instances = []
+
+        # unify --strategy and --strategy-list to one list
+        if not (strategy_list := config.get('strategy_list', [])):
+            if config.get('strategy') is None:
+                raise OperationalException(
+                    "No Strategy specified. Please specify a strategy via --strategy or "
+                    "--strategy-list"
+                )
+            strategy_list = [config['strategy']]
+
+        # check if strategies can be properly loaded, only check them if they can be.
+        for strat in strategy_list:
+            for strategy_obj in strategy_objs:
+                if strategy_obj['name'] == strat and strategy_obj not in strategy_list:
+                    RecursiveAnalysis_instances.append(
+                        RecursiveAnalysisSubFunctions.initialize_single_recursive_analysis(
+                            config, strategy_obj))
+                    break
+
+        # report the results
+        if RecursiveAnalysis_instances:
+            RecursiveAnalysisSubFunctions.text_table_recursive_analysis_instances(
+                RecursiveAnalysis_instances)
+        else:
+            logger.error("There were no strategies specified neither through "
+                         "--strategy nor through "
+                         "--strategy-list "
+                         "or timeframe was not specified.")

freqtrade/templates/sample_strategy.py

@@ -77,7 +77,7 @@ class SampleStrategy(IStrategy):
     exit_short_rsi = IntParameter(low=1, high=50, default=30, space='buy', optimize=True, load=True)
 
     # Number of candles the strategy requires before producing valid signals
-    startup_candle_count: int = 30
+    startup_candle_count: int = 200
 
     # Optional order type mapping.
     order_types = {

mkdocs.yml

@@ -22,7 +22,6 @@ nav:
         - Web Hook: webhook-config.md
     - Data Downloading: data-download.md
     - Backtesting: backtesting.md
-    - Lookahead analysis: lookahead-analysis.md
     - Hyperopt: hyperopt.md
     - FreqAI:
         - Introduction: freqai.md
@@ -43,6 +42,8 @@ nav:
     - Advanced Topics:
         - Advanced Post-installation Tasks: advanced-setup.md
         - Trade Object: trade-object.md
+        - Lookahead analysis: lookahead-analysis.md
+        - Recursive analysis: recursive-analysis.md
         - Advanced Strategy: strategy-advanced.md
         - Advanced Hyperopt: advanced-hyperopt.md
         - Producer/Consumer mode: producer-consumer.md

tests/commands/test_commands.py

@@ -1556,7 +1556,7 @@ def test_start_strategy_updater(mocker, tmpdir):
     pargs['config'] = None
     start_strategy_update(pargs)
     # Number of strategies in the test directory
-    assert sc_mock.call_count == 11
+    assert sc_mock.call_count == 12
 
     sc_mock.reset_mock()
     args = [

tests/optimize/test_recursive_analysis.py

@@ -0,0 +1,188 @@
+# pragma pylint: disable=missing-docstring, W0212, line-too-long, C0103, unused-argument
+from copy import deepcopy
+from pathlib import Path
+from unittest.mock import MagicMock, PropertyMock
+
+import pytest
+
+from freqtrade.commands.optimize_commands import start_recursive_analysis
+from freqtrade.data.history import get_timerange
+from freqtrade.exceptions import OperationalException
+from freqtrade.optimize.recursive_analysis import RecursiveAnalysis
+from freqtrade.optimize.recursive_analysis_helpers import RecursiveAnalysisSubFunctions
+from tests.conftest import get_args, log_has_re, patch_exchange
+
+
+@pytest.fixture
+def recursive_conf(default_conf_usdt):
+    default_conf_usdt['timerange'] = '20220101-20220501'
+
+    default_conf_usdt['strategy_path'] = str(
+        Path(__file__).parent.parent / "strategy/strats")
+    default_conf_usdt['strategy'] = 'strategy_test_v3_recursive_issue'
+    default_conf_usdt['pairs'] = ['UNITTEST/USDT']
+    default_conf_usdt['startup_candle'] = [100]
+    return default_conf_usdt
+
+
+def test_start_recursive_analysis(mocker):
+    single_mock = MagicMock()
+    text_table_mock = MagicMock()
+    mocker.patch.multiple(
+        'freqtrade.optimize.recursive_analysis_helpers.RecursiveAnalysisSubFunctions',
+        initialize_single_recursive_analysis=single_mock,
+        text_table_recursive_analysis_instances=text_table_mock,
+    )
+    args = [
+        "recursive-analysis",
+        "--strategy",
+        "strategy_test_v3_recursive_issue",
+        "--strategy-path",
+        str(Path(__file__).parent.parent / "strategy/strats"),
+        "--pairs",
+        "UNITTEST/BTC",
+        "--timerange",
+        "20220101-20220201"
+    ]
+    pargs = get_args(args)
+    pargs['config'] = None
+
+    start_recursive_analysis(pargs)
+    assert single_mock.call_count == 1
+    assert text_table_mock.call_count == 1
+
+    single_mock.reset_mock()
+
+    # Missing timerange
+    args = [
+        "recursive-analysis",
+        "--strategy",
+        "strategy_test_v3_with_recursive_bias",
+        "--strategy-path",
+        str(Path(__file__).parent.parent / "strategy/strats"),
+        "--pairs",
+        "UNITTEST/BTC"
+    ]
+    pargs = get_args(args)
+    pargs['config'] = None
+    with pytest.raises(OperationalException,
+                       match=r"Please set a timerange\..*"):
+        start_recursive_analysis(pargs)
+
+
+def test_recursive_helper_no_strategy_defined(recursive_conf):
+    conf = deepcopy(recursive_conf)
+    conf['pairs'] = ['UNITTEST/USDT']
+    del conf['strategy']
+    with pytest.raises(OperationalException,
+                       match=r"No Strategy specified"):
+        RecursiveAnalysisSubFunctions.start(conf)
+
+
+def test_recursive_helper_start(recursive_conf, mocker) -> None:
+    single_mock = MagicMock()
+    text_table_mock = MagicMock()
+    mocker.patch.multiple(
+        'freqtrade.optimize.recursive_analysis_helpers.RecursiveAnalysisSubFunctions',
+        initialize_single_recursive_analysis=single_mock,
+        text_table_recursive_analysis_instances=text_table_mock,
+    )
+    RecursiveAnalysisSubFunctions.start(recursive_conf)
+    assert single_mock.call_count == 1
+    assert text_table_mock.call_count == 1
+
+    single_mock.reset_mock()
+    text_table_mock.reset_mock()
+
+
+def test_recursive_helper_text_table_recursive_analysis_instances(recursive_conf):
+    dict_diff = dict()
+    dict_diff['rsi'] = {}
+    dict_diff['rsi'][100] = "0.078%"
+
+    strategy_obj = {
+        'name': "strategy_test_v3_recursive_issue",
+        'location': Path(recursive_conf['strategy_path'], f"{recursive_conf['strategy']}.py")
+    }
+
+    instance = RecursiveAnalysis(recursive_conf, strategy_obj)
+    instance.dict_recursive = dict_diff
+    table, headers, data = (RecursiveAnalysisSubFunctions.
+                            text_table_recursive_analysis_instances([instance]))
+
+    # check row contents for a try that has too few signals
+    assert data[0][0] == 'rsi'
+    assert data[0][1] == '0.078%'
+    assert len(data[0]) == 2
+
+    # now check when there is no issue
+    dict_diff = dict()
+    instance = RecursiveAnalysis(recursive_conf, strategy_obj)
+    instance.dict_recursive = dict_diff
+    table, headers, data = (RecursiveAnalysisSubFunctions.
+                            text_table_recursive_analysis_instances([instance]))
+    assert len(data) == 0
+
+
+def test_initialize_single_recursive_analysis(recursive_conf, mocker, caplog):
+    mocker.patch('freqtrade.data.history.get_timerange', get_timerange)
+    patch_exchange(mocker)
+    mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist',
+                 PropertyMock(return_value=['UNITTEST/BTC']))
+    recursive_conf['pairs'] = ['UNITTEST/BTC']
+
+    recursive_conf['timeframe'] = '5m'
+    recursive_conf['timerange'] = '20180119-20180122'
+    start_mock = mocker.patch('freqtrade.optimize.recursive_analysis.RecursiveAnalysis.start')
+    strategy_obj = {
+        'name': "strategy_test_v3_recursive_issue",
+        'location': Path(recursive_conf['strategy_path'], f"{recursive_conf['strategy']}.py")
+    }
+
+    instance = RecursiveAnalysisSubFunctions.initialize_single_recursive_analysis(
+        recursive_conf, strategy_obj)
+    assert log_has_re(r"Recursive test of .* started\.", caplog)
+    assert start_mock.call_count == 1
+
+    assert instance.strategy_obj['name'] == "strategy_test_v3_recursive_issue"
+
+
+@pytest.mark.parametrize('scenario', [
+    'no_bias', 'bias1', 'bias2'
+])
+def test_recursive_biased_strategy(recursive_conf, mocker, caplog, scenario) -> None:
+    mocker.patch('freqtrade.data.history.get_timerange', get_timerange)
+    patch_exchange(mocker)
+    mocker.patch('freqtrade.plugins.pairlistmanager.PairListManager.whitelist',
+                 PropertyMock(return_value=['UNITTEST/BTC']))
+    recursive_conf['pairs'] = ['UNITTEST/BTC']
+
+    recursive_conf['timeframe'] = '5m'
+    recursive_conf['timerange'] = '20180119-20180122'
+    recursive_conf['startup_candle'] = [100]
+
+    # Patch scenario Parameter to allow for easy selection
+    mocker.patch('freqtrade.strategy.hyper.HyperStrategyMixin.load_params_from_file',
+                 return_value={
+                     'params': {
+                         "buy": {
+                             "scenario": scenario
+                         }
+                     }
+                 })
+
+    strategy_obj = {'name': "strategy_test_v3_recursive_issue"}
+    instance = RecursiveAnalysis(recursive_conf, strategy_obj)
+    instance.start()
+    # Assert init correct
+    assert log_has_re(f"Strategy Parameter: scenario = {scenario}", caplog)
+
+    if scenario == "bias2":
+        assert log_has_re("=> found lookahead in indicator rsi", caplog)
+    diff_pct = abs(float(instance.dict_recursive['rsi'][100].replace("%", "")))
+    # check non-biased strategy
+    if scenario == "no_bias":
+        assert diff_pct < 0.01
+    # check biased strategy
+    elif scenario in ("bias1", "bias2"):
+        assert diff_pct >= 0.01

tests/rpc/test_rpc_apiserver.py

@@ -1636,7 +1636,8 @@ def test_api_strategies(botclient, tmpdir):
         'freqai_test_classifier',
         'freqai_test_multimodel_classifier_strat',
         'freqai_test_multimodel_strat',
-        'freqai_test_strat'
+        'freqai_test_strat',
+        'strategy_test_v3_recursive_issue'
     ]}
 
 

tests/strategy/strats/strategy_test_v3_recursive_issue.py

@@ -0,0 +1,46 @@
+# pragma pylint: disable=missing-docstring, invalid-name, pointless-string-statement
+import talib.abstract as ta
+from pandas import DataFrame
+
+from freqtrade.strategy import IStrategy
+from freqtrade.strategy.parameters import CategoricalParameter
+
+
+class strategy_test_v3_recursive_issue(IStrategy):
+    INTERFACE_VERSION = 3
+
+    # Minimal ROI designed for the strategy
+    minimal_roi = {
+        "0": 0.04
+    }
+
+    # Optimal stoploss designed for the strategy
+    stoploss = -0.10
+
+    # Optimal timeframe for the strategy
+    timeframe = '5m'
+    scenario = CategoricalParameter(['no_bias', 'bias1', 'bias2'], default='bias1', space="buy")
+
+    # Number of candles the strategy requires before producing valid signals
+    startup_candle_count: int = 100
+
+    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+        # bias is introduced here
+        if self.scenario.value == 'no_bias':
+            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=14)
+        else:
+            dataframe['rsi'] = ta.RSI(dataframe, timeperiod=50)
+
+        if self.scenario.value == 'bias2':
+            # Has both bias1 and bias2
+            dataframe['rsi_lookahead'] = ta.RSI(dataframe, timeperiod=50).shift(-1)
+
+        return dataframe
+
+    def populate_entry_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+
+        return dataframe
+
+    def populate_exit_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
+
+        return dataframe

tests/strategy/test_strategy_loading.py

@@ -35,7 +35,7 @@ def test_search_all_strategies_no_failed():
     directory = Path(__file__).parent / "strats"
     strategies = StrategyResolver._search_all_objects(directory, enum_failed=False)
     assert isinstance(strategies, list)
-    assert len(strategies) == 12
+    assert len(strategies) == 13
     assert isinstance(strategies[0], dict)
 
 
@@ -43,10 +43,10 @@ def test_search_all_strategies_with_failed():
     directory = Path(__file__).parent / "strats"
     strategies = StrategyResolver._search_all_objects(directory, enum_failed=True)
     assert isinstance(strategies, list)
-    assert len(strategies) == 13
+    assert len(strategies) == 14
     # with enum_failed=True search_all_objects() shall find 2 good strategies
     # and 1 which fails to load
-    assert len([x for x in strategies if x['class'] is not None]) == 12
+    assert len([x for x in strategies if x['class'] is not None]) == 13
 
     assert len([x for x in strategies if x['class'] is None]) == 1