lychiyu/freqtrade_origin
1
0
Fork 0
mirror of https://github.com/freqtrade/freqtrade.git synced 2024-11-10 10:21:59 +00:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Merge pull request #2067 from freqtrade/align_userdata

Browse Source 
Align userdata usage
This commit is contained in:
Matthias 2019-08-21 19:55:42 +02:00 committed by GitHub
commit e52d5e32aa
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
27 changed files with 360 additions and 184 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/backtesting.md
View File

@ -248,7 +248,7 @@ All listed Strategies need to be in the same directory.
freqtrade backtesting --timerange 20180401-20180410 --ticker-interval 5m --strategy-list Strategy001 Strategy002 --export trades
```
This will save the results to `user_data/backtest_data/backtest-result-<strategy>.json`, injecting the strategy-name into the target filename.
This will save the results to `user_data/backtest_results/backtest-result-<strategy>.json`, injecting the strategy-name into the target filename.
There will be an additional table comparing win/losses of the different strategies (identical to the "Total" row in the first table).
Detailed output for all strategies one after the other will be available, so make sure to scroll up.

60
docs/bot-usage.md
View File

@ -9,38 +9,43 @@ This page explains the different parameters of the bot and how to run it.
## Bot commands
```
usage: freqtrade [-h] [-v] [--logfile FILE] [--version] [-c PATH] [-d PATH]
[-s NAME] [--strategy-path PATH] [--db-url PATH]
[--sd-notify]
{backtesting,edge,hyperopt} ...
usage: freqtrade [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH]
[--userdir PATH] [-s NAME] [--strategy-path PATH]
[--db-url PATH] [--sd-notify]
{backtesting,edge,hyperopt,create-userdir,list-exchanges} ...
Free, open source crypto trading bot
positional arguments:
{backtesting,edge,hyperopt}
{backtesting,edge,hyperopt,create-userdir,list-exchanges}
backtesting Backtesting module.
edge Edge module.
hyperopt Hyperopt module.
create-userdir Create user-data directory.
list-exchanges Print available exchanges.
optional arguments:
-h, --help show this help message and exit
-v, --verbose Verbose mode (-vv for more, -vvv to get all messages).
--logfile FILE Log to the file specified
--logfile FILE Log to the file specified.
-V, --version show program's version number and exit
-c PATH, --config PATH
Specify configuration file (default: None). Multiple
--config options may be used. Can be set to '-' to
read config from stdin.
Specify configuration file (default: `config.json`).
Multiple --config options may be used. Can be set to
`-` to read config from stdin.
-d PATH, --datadir PATH
Path to backtest data.
Path to directory with historical backtesting data.
--userdir PATH, --user-data-dir PATH
Path to userdata directory.
-s NAME, --strategy NAME
Specify strategy class name (default:
DefaultStrategy).
`DefaultStrategy`).
--strategy-path PATH Specify additional strategy lookup path.
--db-url PATH Override trades database URL, this is useful if
dry_run is enabled or in custom deployments (default:
None).
--db-url PATH Override trades database URL, this is useful in custom
deployments (default: `sqlite:///tradesv3.sqlite` for
Live Run mode, `sqlite://` for Dry Run).
--sd-notify Notify systemd service manager.
```
### How to specify which configuration file be used?
@ -85,6 +90,29 @@ of your configuration in the project issues or in the Internet.
See more details on this technique with examples in the documentation page on
[configuration](configuration.md).
### Where to store custom data
Freqtrade allows the creation of a user-data directory using `freqtrade create-userdir --userdir someDirectory`.
This directory will look as follows:
```
user_data/
├── backtest_results
├── data
├── hyperopts
├── hyperopts_results
├── plot
└── strategies
```
You can add the entry "user_data_dir" setting to your configuration, to always point your bot to this directory.
Alternatively, pass in `--userdir` to every command.
The bot will fail to start if the directory does not exist, but will create necessary subdirectories.
This directory should contain your custom strategies, custom hyperopts and hyperopt loss functions, backtesting historical data (downloaded using either backtesting command or the download script) and plot outputs.
It is recommended to use version control to keep track of changes to your strategies.
### How to use **--strategy**?
This parameter will allow you to load your custom strategy class.
@ -179,8 +207,8 @@ optional arguments:
--export-filename PATH
Save backtest results to this filename requires
--export to be set as well Example --export-
filename=user_data/backtest_data/backtest_today.json
(default: user_data/backtest_data/backtest-
filename=user_data/backtest_results/backtest_today.json
(default: user_data/backtest_results/backtest-
result.json)
```

1
docs/configuration.md
View File

@ -91,6 +91,7 @@ Mandatory parameters are marked as **Required**.
| `internals.process_throttle_secs` | 5 | **Required.** Set the process throttle. Value in second.
| `internals.sd_notify` | false | Enables use of the sd_notify protocol to tell systemd service manager about changes in the bot state and issue keep-alive pings. See [here](installation.md#7-optional-configure-freqtrade-as-a-systemd-service) for more details.
| `logfile` | | Specify Logfile. Uses a rolling strategy of 10 files, with 1Mb per file.
| `user_data_dir` | cwd()/user_data | Directory containing user data. Defaults to `./user_data/`.
### Parameters in the strategy

2
docs/data-analysis.md
View File

@ -11,7 +11,7 @@ You can analyze the results of backtests and trading history easily using Jupyte
```python
from freqtrade.data.btanalysis import load_backtest_data
# Load backtest results
df = load_backtest_data("user_data/backtest_data/backtest-result.json")
df = load_backtest_data("user_data/backtest_results/backtest-result.json")
# Show value-counts per pair
df.groupby("pair")["sell_reason"].value_counts()

2
docs/plotting.md
View File

@ -64,7 +64,7 @@ python3 scripts/plot_dataframe.py --db-url sqlite:///tradesv3.dry_run.sqlite -p
To plot trades from a backtesting result, use `--export-filename <filename>`
``` bash
python3 scripts/plot_dataframe.py --export-filename user_data/backtest_data/backtest-result.json -p BTC/ETH
python3 scripts/plot_dataframe.py --export-filename user_data/backtest_results/backtest-result.json -p BTC/ETH
```
To plot a custom strategy the strategy should have first be backtested.

11
freqtrade/configuration/arguments.py
View File

@ -7,7 +7,7 @@ from typing import List, Optional
from freqtrade.configuration.cli_options import AVAILABLE_CLI_OPTIONS
from freqtrade import constants
ARGS_COMMON = ["verbosity", "logfile", "version", "config", "datadir"]
ARGS_COMMON = ["verbosity", "logfile", "version", "config", "datadir", "user_data_dir"]
ARGS_STRATEGY = ["strategy", "strategy_path"]
@ -30,6 +30,8 @@ ARGS_EDGE = ARGS_COMMON_OPTIMIZE + ["stoploss_range"]
ARGS_LIST_EXCHANGES = ["print_one_column"]
ARGS_CREATE_USERDIR = ["user_data_dir"]
ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "exchange", "timeframes", "erase"]
ARGS_PLOT_DATAFRAME = (ARGS_COMMON + ARGS_STRATEGY +
@ -98,7 +100,7 @@ class Arguments(object):
:return: None
"""
from freqtrade.optimize import start_backtesting, start_hyperopt, start_edge
from freqtrade.utils import start_download_data, start_list_exchanges
from freqtrade.utils import start_create_userdir, start_download_data, start_list_exchanges
subparsers = self.parser.add_subparsers(dest='subparser')
@ -117,6 +119,11 @@ class Arguments(object):
hyperopt_cmd.set_defaults(func=start_hyperopt)
self._build_args(optionlist=ARGS_HYPEROPT, parser=hyperopt_cmd)
create_userdir_cmd = subparsers.add_parser('create-userdir',
help="Create user-data directory.")
create_userdir_cmd.set_defaults(func=start_create_userdir)
self._build_args(optionlist=ARGS_CREATE_USERDIR, parser=create_userdir_cmd)
# Add list-exchanges subcommand
list_exchanges_cmd = subparsers.add_parser(
'list-exchanges',

11
freqtrade/configuration/cli_options.py
View File

@ -55,7 +55,12 @@ AVAILABLE_CLI_OPTIONS = {
),
"datadir": Arg(
'-d', '--datadir',
help='Path to backtest data.',
help='Path to directory with historical backtesting data.',
metavar='PATH',
),
"user_data_dir": Arg(
'--userdir', '--user-data-dir',
help='Path to userdata directory.',
metavar='PATH',
),
# Main options
@ -146,9 +151,9 @@ AVAILABLE_CLI_OPTIONS = {
'--export-filename',
help='Save backtest results to the file with this filename (default: `%(default)s`). '
'Requires `--export` to be set as well. '
'Example: `--export-filename=user_data/backtest_data/backtest_today.json`',
'Example: `--export-filename=user_data/backtest_results/backtest_today.json`',
metavar='PATH',
default=os.path.join('user_data', 'backtest_data',
default=os.path.join('user_data', 'backtest_results',
'backtest-result.json'),
),
# Edge

33
freqtrade/configuration/configuration.py
View File

@ -7,11 +7,12 @@ from argparse import Namespace
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional
from freqtrade import constants, OperationalException
from freqtrade import OperationalException, constants
from freqtrade.configuration.check_exchange import check_exchange
from freqtrade.configuration.create_datadir import create_datadir
from freqtrade.configuration.config_validation import (validate_config_schema,
validate_config_consistency)
from freqtrade.configuration.config_validation import (
validate_config_consistency, validate_config_schema)
from freqtrade.configuration.directory_operations import (create_datadir,
create_userdata_dir)
from freqtrade.configuration.load_config import load_config_file
from freqtrade.loggers import setup_logging
from freqtrade.misc import deep_merge_dicts, json_load
@ -115,7 +116,9 @@ class Configuration(object):
setup_logging(config)
def _process_strategy_options(self, config: Dict[str, Any]) -> None:
def _process_common_options(self, config: Dict[str, Any]) -> None:
self._process_logging_options(config)
# Set strategy if not specified in config and or if it's non default
if self.args.strategy != constants.DEFAULT_STRATEGY or not config.get('strategy'):
@ -124,11 +127,6 @@ class Configuration(object):
self._args_to_config(config, argname='strategy_path',
logstring='Using additional Strategy lookup path: {}')
def _process_common_options(self, config: Dict[str, Any]) -> None:
self._process_logging_options(config)
self._process_strategy_options(config)
if ('db_url' in self.args and self.args.db_url and
self.args.db_url != constants.DEFAULT_DB_PROD_URL):
config.update({'db_url': self.args.db_url})
@ -159,9 +157,19 @@ class Configuration(object):
def _process_datadir_options(self, config: Dict[str, Any]) -> None:
"""
Extract information for sys.argv and load datadir configuration:
the --datadir option
Extract information for sys.argv and load directory configurations
--user-data, --datadir
"""
if 'user_data_dir' in self.args and self.args.user_data_dir:
config.update({'user_data_dir': self.args.user_data_dir})
elif 'user_data_dir' not in config:
# Default to cwd/user_data (legacy option ...)
config.update({'user_data_dir': str(Path.cwd() / "user_data")})
# reset to user_data_dir so this contains the absolute path.
config['user_data_dir'] = create_userdata_dir(config['user_data_dir'], create_dir=False)
logger.info('Using user-data directory: %s ...', config['user_data_dir'])
if 'datadir' in self.args and self.args.datadir:
config.update({'datadir': create_datadir(config, self.args.datadir)})
else:
@ -357,7 +365,6 @@ class Configuration(object):
else:
# Fall back to /dl_path/pairs.json
pairs_file = Path(config['datadir']) / config['exchange']['name'].lower() / "pairs.json"
print(config['datadir'])
if pairs_file.exists():
with pairs_file.open('r') as f:
config['pairs'] = json_load(f)

20
freqtrade/configuration/create_datadir.py
View File

@ -1,20 +0,0 @@
import logging
from typing import Any, Dict, Optional
from pathlib import Path
logger = logging.getLogger(__name__)
def create_datadir(config: Dict[str, Any], datadir: Optional[str] = None) -> str:
folder = Path(datadir) if datadir else Path('user_data/data')
if not datadir:
# set datadir
exchange_name = config.get('exchange', {}).get('name').lower()
folder = folder.joinpath(exchange_name)
if not folder.is_dir():
folder.mkdir(parents=True)
logger.info(f'Created data directory: {datadir}')
return str(folder)

50
freqtrade/configuration/directory_operations.py Normal file
View File

@ -0,0 +1,50 @@
import logging
from typing import Any, Dict, Optional
from pathlib import Path
from freqtrade import OperationalException
logger = logging.getLogger(__name__)
def create_datadir(config: Dict[str, Any], datadir: Optional[str] = None) -> str:
folder = Path(datadir) if datadir else Path(f"{config['user_data_dir']}/data")
if not datadir:
# set datadir
exchange_name = config.get('exchange', {}).get('name').lower()
folder = folder.joinpath(exchange_name)
if not folder.is_dir():
folder.mkdir(parents=True)
logger.info(f'Created data directory: {datadir}')
return str(folder)
def create_userdata_dir(directory: str, create_dir=False) -> Path:
"""
Create userdata directory structure.
if create_dir is True, then the parent-directory will be created if it does not exist.
Sub-directories will always be created if the parent directory exists.
Raises OperationalException if given a non-existing directory.
:param directory: Directory to check
:param create_dir: Create directory if it does not exist.
:return: Path object containing the directory
"""
sub_dirs = ["backtest_results", "data", "hyperopts", "hyperopt_results", "plot", "strategies", ]
folder = Path(directory)
if not folder.is_dir():
if create_dir:
folder.mkdir(parents=True)
logger.info(f'Created user-data directory: {folder}')
else:
raise OperationalException(
f"Directory `{folder}` does not exist. "
"Please use `freqtrade create-userdir` to create a user directory")
# Create required subdirectories
for f in sub_dirs:
subfolder = folder / f
if not subfolder.is_dir():
subfolder.mkdir(parents=False)
return folder

4
freqtrade/optimize/__init__.py
View File

@ -64,14 +64,14 @@ def start_hyperopt(args: Namespace) -> None:
:return: None
"""
# Import here to avoid loading hyperopt module when it's not used
from freqtrade.optimize.hyperopt import Hyperopt, HYPEROPT_LOCKFILE
from freqtrade.optimize.hyperopt import Hyperopt
# Initialize configuration
config = setup_configuration(args, RunMode.HYPEROPT)
logger.info('Starting freqtrade in Hyperopt mode')
lock = FileLock(HYPEROPT_LOCKFILE)
lock = FileLock(Hyperopt.get_lock_filename(config))
try:
with lock.acquire(timeout=1):

25
freqtrade/optimize/hyperopt.py
View File

@ -5,7 +5,6 @@ This module contains the hyperopt logic
"""
import logging
import os
import sys
from collections import OrderedDict
@ -36,9 +35,6 @@ logger = logging.getLogger(__name__)
INITIAL_POINTS = 30
MAX_LOSS = 100000 # just a big enough number to be bad result in loss optimization
TICKERDATA_PICKLE = os.path.join('user_data', 'hyperopt_tickerdata.pkl')
TRIALSDATA_PICKLE = os.path.join('user_data', 'hyperopt_results.pickle')
HYPEROPT_LOCKFILE = os.path.join('user_data', 'hyperopt.lock')
class Hyperopt(Backtesting):
@ -56,7 +52,12 @@ class Hyperopt(Backtesting):
self.custom_hyperoptloss = HyperOptLossResolver(self.config).hyperoptloss
self.calculate_loss = self.custom_hyperoptloss.hyperopt_loss_function
self.trials_file = (self.config['user_data_dir'] /
'hyperopt_results' / 'hyperopt_results.pickle')
self.tickerdata_pickle = (self.config['user_data_dir'] /
'hyperopt_results' / 'hyperopt_tickerdata.pkl')
self.total_epochs = config.get('epochs', 0)
self.current_best_loss = 100
if not self.config.get('hyperopt_continue'):
@ -65,7 +66,6 @@ class Hyperopt(Backtesting):
logger.info("Continuing on previous hyperopt results.")
# Previous evaluations
self.trials_file = TRIALSDATA_PICKLE
self.trials: List = []
# Populate functions here (hasattr is slow so should not be run during "regular" operations)
@ -89,11 +89,16 @@ class Hyperopt(Backtesting):
self.config['experimental'] = {}
self.config['experimental']['use_sell_signal'] = True
@staticmethod
def get_lock_filename(config) -> str:
return str(config['user_data_dir'] / 'hyperopt.lock')
def clean_hyperopt(self):
"""
Remove hyperopt pickle files to restart hyperopt.
"""
for f in [TICKERDATA_PICKLE, TRIALSDATA_PICKLE]:
for f in [self.tickerdata_pickle, self.trials_file]:
p = Path(f)
if p.is_file():
logger.info(f"Removing `{p}`.")
@ -126,7 +131,7 @@ class Hyperopt(Backtesting):
"""
logger.info('Reading Trials from \'%s\'', self.trials_file)
trials = load(self.trials_file)
os.remove(self.trials_file)
self.trials_file.unlink()
return trials
def log_trials_result(self) -> None:
@ -255,7 +260,7 @@ class Hyperopt(Backtesting):
if self.has_space('stoploss'):
self.strategy.stoploss = params['stoploss']
processed = load(TICKERDATA_PICKLE)
processed = load(self.tickerdata_pickle)
min_date, max_date = get_timeframe(processed)
@ -327,7 +332,7 @@ class Hyperopt(Backtesting):
def load_previous_results(self):
""" read trials file if we have one """
if os.path.exists(self.trials_file) and os.path.getsize(self.trials_file) > 0:
if self.trials_file.is_file() and self.trials_file.stat().st_size > 0:
self.trials = self.read_trials()
logger.info(
'Loaded %d previous evaluations from disk.',
@ -364,7 +369,7 @@ class Hyperopt(Backtesting):
preprocessed = self.strategy.tickerdata_to_dataframe(data)
dump(preprocessed, TICKERDATA_PICKLE)
dump(preprocessed, self.tickerdata_pickle)
# We don't need exchange instance anymore while running hyperopt
self.exchange = None # type: ignore

6
freqtrade/plot/plotting.py
View File

@ -308,7 +308,7 @@ def generate_plot_filename(pair, ticker_interval) -> str:
return file_name
def store_plot_file(fig, filename: str, auto_open: bool = False) -> None:
def store_plot_file(fig, filename: str, directory: Path, auto_open: bool = False) -> None:
"""
Generate a plot html file from pre populated fig plotly object
:param fig: Plotly Figure to plot
@ -316,9 +316,9 @@ def store_plot_file(fig, filename: str, auto_open: bool = False) -> None:
:param ticker_interval: Used as part of the filename
:return: None
"""
directory.mkdir(parents=True, exist_ok=True)
Path("user_data/plots").mkdir(parents=True, exist_ok=True)
_filename = Path('user_data/plots').joinpath(filename)
_filename = directory.joinpath(filename)
plot(fig, filename=str(_filename),
auto_open=auto_open)
logger.info(f"Stored plot as {_filename}")

18
freqtrade/resolvers/hyperopt_resolver.py
View File

@ -31,7 +31,8 @@ class HyperOptResolver(IResolver):
# Verify the hyperopt is in the configuration, otherwise fallback to the default hyperopt
hyperopt_name = config.get('hyperopt') or DEFAULT_HYPEROPT
self.hyperopt = self._load_hyperopt(hyperopt_name, extra_dir=config.get('hyperopt_path'))
self.hyperopt = self._load_hyperopt(hyperopt_name, config,
extra_dir=config.get('hyperopt_path'))
# Assign ticker_interval to be used in hyperopt
self.hyperopt.__class__.ticker_interval = str(config['ticker_interval'])
@ -44,17 +45,18 @@ class HyperOptResolver(IResolver):
"Using populate_sell_trend from DefaultStrategy.")
def _load_hyperopt(
self, hyperopt_name: str, extra_dir: Optional[str] = None) -> IHyperOpt:
self, hyperopt_name: str, config: Dict, extra_dir: Optional[str] = None) -> IHyperOpt:
"""
Search and loads the specified hyperopt.
:param hyperopt_name: name of the module to import
:param config: configuration dictionary
:param extra_dir: additional directory to search for the given hyperopt
:return: HyperOpt instance or None
"""
current_path = Path(__file__).parent.parent.joinpath('optimize').resolve()
abs_paths = [
Path.cwd().joinpath('user_data/hyperopts'),
config['user_data_dir'].joinpath('hyperopts'),
current_path,
]
@ -79,7 +81,7 @@ class HyperOptLossResolver(IResolver):
__slots__ = ['hyperoptloss']
def __init__(self, config: Optional[Dict] = None) -> None:
def __init__(self, config: Dict = None) -> None:
"""
Load the custom class from config parameter
:param config: configuration dictionary or None
@ -89,7 +91,7 @@ class HyperOptLossResolver(IResolver):
# Verify the hyperopt is in the configuration, otherwise fallback to the default hyperopt
hyperopt_name = config.get('hyperopt_loss') or DEFAULT_HYPEROPT_LOSS
self.hyperoptloss = self._load_hyperoptloss(
hyperopt_name, extra_dir=config.get('hyperopt_path'))
hyperopt_name, config, extra_dir=config.get('hyperopt_path'))
# Assign ticker_interval to be used in hyperopt
self.hyperoptloss.__class__.ticker_interval = str(config['ticker_interval'])
@ -99,17 +101,19 @@ class HyperOptLossResolver(IResolver):
f"Found hyperopt {hyperopt_name} does not implement `hyperopt_loss_function`.")
def _load_hyperoptloss(
self, hyper_loss_name: str, extra_dir: Optional[str] = None) -> IHyperOptLoss:
self, hyper_loss_name: str, config: Dict,
extra_dir: Optional[str] = None) -> IHyperOptLoss:
"""
Search and loads the specified hyperopt loss class.
:param hyper_loss_name: name of the module to import
:param config: configuration dictionary
:param extra_dir: additional directory to search for the given hyperopt
:return: HyperOptLoss instance or None
"""
current_path = Path(__file__).parent.parent.joinpath('optimize').resolve()
abs_paths = [
Path.cwd().joinpath('user_data/hyperopts'),
config['user_data_dir'].joinpath('hyperopts'),
current_path,
]

2
freqtrade/resolvers/iresolver.py
View File

@ -58,7 +58,7 @@ class IResolver(object):
if not str(entry).endswith('.py'):
logger.debug('Ignoring %s', entry)
continue
module_path = Path.resolve(directory.joinpath(entry))
module_path = entry.resolve()
obj = IResolver._get_valid_object(
object_type, module_path, object_name
)

9
freqtrade/resolvers/pairlist_resolver.py
View File

@ -25,21 +25,22 @@ class PairListResolver(IResolver):
Load the custom class from config parameter
:param config: configuration dictionary or None
"""
self.pairlist = self._load_pairlist(pairlist_name, kwargs={'freqtrade': freqtrade,
'config': config})
self.pairlist = self._load_pairlist(pairlist_name, config, kwargs={'freqtrade': freqtrade,
'config': config})
def _load_pairlist(
self, pairlist_name: str, kwargs: dict) -> IPairList:
self, pairlist_name: str, config: dict, kwargs: dict) -> IPairList:
"""
Search and loads the specified pairlist.
:param pairlist_name: name of the module to import
:param config: configuration dictionary
:param extra_dir: additional directory to search for the given pairlist
:return: PairList instance or None
"""
current_path = Path(__file__).parent.parent.joinpath('pairlist').resolve()
abs_paths = [
Path.cwd().joinpath('user_data/pairlist'),
config['user_data_dir'].joinpath('pairlist'),
current_path,
]

2
freqtrade/resolvers/strategy_resolver.py
View File

@ -123,7 +123,7 @@ class StrategyResolver(IResolver):
current_path = Path(__file__).parent.parent.joinpath('strategy').resolve()
abs_paths = [
Path.cwd().joinpath('user_data/strategies'),
config['user_data_dir'].joinpath('strategies'),
current_path,
]

1
freqtrade/tests/conftest.py
View File

@ -239,6 +239,7 @@ def default_conf():
},
"initial_state": "running",
"db_url": "sqlite://",
"user_data_dir": Path("user_data"),
"verbosity": 3,
}
return configuration

23
freqtrade/tests/optimize/test_hyperopt.py
View File

@ -1,12 +1,13 @@
# pragma pylint: disable=missing-docstring,W0212,C0103
import os
from datetime import datetime
from unittest.mock import MagicMock
from unittest.mock import MagicMock, PropertyMock
import pandas as pd
import pytest
from arrow import Arrow
from filelock import Timeout
from pathlib import Path
from freqtrade import DependencyException, OperationalException
from freqtrade.data.converter import parse_ticker_dataframe
@ -14,8 +15,7 @@ from freqtrade.data.history import load_tickerdata_file
from freqtrade.optimize import setup_configuration, start_hyperopt
from freqtrade.optimize.default_hyperopt import DefaultHyperOpts
from freqtrade.optimize.default_hyperopt_loss import DefaultHyperOptLoss
from freqtrade.optimize.hyperopt import (HYPEROPT_LOCKFILE, TICKERDATA_PICKLE,
Hyperopt)
from freqtrade.optimize.hyperopt import Hyperopt
from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver, HyperOptLossResolver
from freqtrade.state import RunMode
from freqtrade.strategy.interface import SellType
@ -54,11 +54,14 @@ def create_trials(mocker, hyperopt) -> None:
- we might have a pickle'd file so make sure that we return
false when looking for it
"""
hyperopt.trials_file = os.path.join('freqtrade', 'tests', 'optimize', 'ut_trials.pickle')
hyperopt.trials_file = Path('freqtrade/tests/optimize/ut_trials.pickle')
mocker.patch('freqtrade.optimize.hyperopt.os.path.exists', return_value=False)
mocker.patch('freqtrade.optimize.hyperopt.os.path.getsize', return_value=1)
mocker.patch('freqtrade.optimize.hyperopt.os.remove', return_value=True)
mocker.patch.object(Path, "is_file", MagicMock(return_value=False))
stat_mock = MagicMock()
stat_mock.st_size = PropertyMock(return_value=1)
mocker.patch.object(Path, "stat", MagicMock(return_value=False))
mocker.patch.object(Path, "unlink", MagicMock(return_value=True))
mocker.patch('freqtrade.optimize.hyperopt.dump', return_value=None)
return [{'loss': 1, 'result': 'foo', 'params': {}}]
@ -264,7 +267,7 @@ def test_start_failure(mocker, default_conf, caplog) -> None:
def test_start_filelock(mocker, default_conf, caplog) -> None:
start_mock = MagicMock(side_effect=Timeout(HYPEROPT_LOCKFILE))
start_mock = MagicMock(side_effect=Timeout(Hyperopt.get_lock_filename(default_conf)))
patched_configuration_load_config_file(mocker, default_conf)
mocker.patch('freqtrade.optimize.hyperopt.Hyperopt.start', start_mock)
patch_exchange(mocker)
@ -597,10 +600,10 @@ def test_clean_hyperopt(mocker, default_conf, caplog):
})
mocker.patch("freqtrade.optimize.hyperopt.Path.is_file", MagicMock(return_value=True))
unlinkmock = mocker.patch("freqtrade.optimize.hyperopt.Path.unlink", MagicMock())
Hyperopt(default_conf)
h = Hyperopt(default_conf)
assert unlinkmock.call_count == 2
assert log_has(f"Removing `{TICKERDATA_PICKLE}`.", caplog)
assert log_has(f"Removing `{h.tickerdata_pickle}`.", caplog)
def test_continue_hyperopt(mocker, default_conf, caplog):

166
freqtrade/tests/strategy/test_strategy.py
View File

@ -60,62 +60,65 @@ def test_search_strategy():
assert s is None
def test_load_strategy(result):
resolver = StrategyResolver({'strategy': 'TestStrategy'})
def test_load_strategy(default_conf, result):
default_conf.update({'strategy': 'TestStrategy'})
resolver = StrategyResolver(default_conf)
assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'})
def test_load_strategy_base64(result, caplog):
def test_load_strategy_base64(result, caplog, default_conf):
with open("user_data/strategies/test_strategy.py", "rb") as file:
encoded_string = urlsafe_b64encode(file.read()).decode("utf-8")
resolver = StrategyResolver({'strategy': 'TestStrategy:{}'.format(encoded_string)})
default_conf.update({'strategy': 'TestStrategy:{}'.format(encoded_string)})
resolver = StrategyResolver(default_conf)
assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'})
# Make sure strategy was loaded from base64 (using temp directory)!!
assert log_has_re(r"Using resolved strategy TestStrategy from '"
+ tempfile.gettempdir() + r"/.*/TestStrategy\.py'\.\.\.", caplog)
def test_load_strategy_invalid_directory(result, caplog):
resolver = StrategyResolver()
def test_load_strategy_invalid_directory(result, caplog, default_conf):
resolver = StrategyResolver(default_conf)
extra_dir = Path.cwd() / 'some/path'
resolver._load_strategy('TestStrategy', config={}, extra_dir=extra_dir)
resolver._load_strategy('TestStrategy', config=default_conf, extra_dir=extra_dir)
assert log_has_re(r'Path .*' + r'some.*path.*' + r'.* does not exist', caplog)
assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'})
def test_load_not_found_strategy():
strategy = StrategyResolver()
def test_load_not_found_strategy(default_conf):
strategy = StrategyResolver(default_conf)
with pytest.raises(OperationalException,
match=r"Impossible to load Strategy 'NotFoundStrategy'. "
r"This class does not exist or contains Python code errors."):
strategy._load_strategy(strategy_name='NotFoundStrategy', config={})
strategy._load_strategy(strategy_name='NotFoundStrategy', config=default_conf)
def test_load_staticmethod_importerror(mocker, caplog):
def test_load_staticmethod_importerror(mocker, caplog, default_conf):
mocker.patch("freqtrade.resolvers.strategy_resolver.import_strategy", Mock(
side_effect=TypeError("can't pickle staticmethod objects")))
with pytest.raises(OperationalException,
match=r"Impossible to load Strategy 'DefaultStrategy'. "
r"This class does not exist or contains Python code errors."):
StrategyResolver()
StrategyResolver(default_conf)
assert log_has_re(r".*Error: can't pickle staticmethod objects", caplog)
def test_strategy(result):
config = {'strategy': 'DefaultStrategy'}
def test_strategy(result, default_conf):
default_conf.update({'strategy': 'DefaultStrategy'})
resolver = StrategyResolver(config)
resolver = StrategyResolver(default_conf)
metadata = {'pair': 'ETH/BTC'}
assert resolver.strategy.minimal_roi[0] == 0.04
assert config["minimal_roi"]['0'] == 0.04
assert default_conf["minimal_roi"]['0'] == 0.04
assert resolver.strategy.stoploss == -0.10
assert config['stoploss'] == -0.10
assert default_conf['stoploss'] == -0.10
assert resolver.strategy.ticker_interval == '5m'
assert config['ticker_interval'] == '5m'
assert default_conf['ticker_interval'] == '5m'
df_indicators = resolver.strategy.advise_indicators(result, metadata=metadata)
assert 'adx' in df_indicators
@ -127,54 +130,54 @@ def test_strategy(result):
assert 'sell' in dataframe.columns
def test_strategy_override_minimal_roi(caplog):
def test_strategy_override_minimal_roi(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'minimal_roi': {
"0": 0.5
}
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.minimal_roi[0] == 0.5
assert log_has("Override strategy 'minimal_roi' with value in config file: {'0': 0.5}.", caplog)
def test_strategy_override_stoploss(caplog):
def test_strategy_override_stoploss(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'stoploss': -0.5
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.stoploss == -0.5
assert log_has("Override strategy 'stoploss' with value in config file: -0.5.", caplog)
def test_strategy_override_trailing_stop(caplog):
def test_strategy_override_trailing_stop(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'trailing_stop': True
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.trailing_stop
assert isinstance(resolver.strategy.trailing_stop, bool)
assert log_has("Override strategy 'trailing_stop' with value in config file: True.", caplog)
def test_strategy_override_trailing_stop_positive(caplog):
def test_strategy_override_trailing_stop_positive(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'trailing_stop_positive': -0.1,
'trailing_stop_positive_offset': -0.2
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.trailing_stop_positive == -0.1
assert log_has("Override strategy 'trailing_stop_positive' with value in config file: -0.1.",
@ -185,15 +188,15 @@ def test_strategy_override_trailing_stop_positive(caplog):
caplog)
def test_strategy_override_ticker_interval(caplog):
def test_strategy_override_ticker_interval(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'ticker_interval': 60,
'stake_currency': 'ETH'
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.ticker_interval == 60
assert resolver.strategy.stake_currency == 'ETH'
@ -201,21 +204,21 @@ def test_strategy_override_ticker_interval(caplog):
caplog)
def test_strategy_override_process_only_new_candles(caplog):
def test_strategy_override_process_only_new_candles(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'process_only_new_candles': True
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.process_only_new_candles
assert log_has("Override strategy 'process_only_new_candles' with value in config file: True.",
caplog)
def test_strategy_override_order_types(caplog):
def test_strategy_override_order_types(caplog, default_conf):
caplog.set_level(logging.INFO)
order_types = {
@ -224,12 +227,11 @@ def test_strategy_override_order_types(caplog):
'stoploss': 'limit',
'stoploss_on_exchange': True,
}
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'order_types': order_types
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.order_types
for method in ['buy', 'sell', 'stoploss', 'stoploss_on_exchange']:
@ -239,18 +241,18 @@ def test_strategy_override_order_types(caplog):
" {'buy': 'market', 'sell': 'limit', 'stoploss': 'limit',"
" 'stoploss_on_exchange': True}.", caplog)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'order_types': {'buy': 'market'}
}
})
# Raise error for invalid configuration
with pytest.raises(ImportError,
match=r"Impossible to load Strategy 'DefaultStrategy'. "
r"Order-types mapping is incomplete."):
StrategyResolver(config)
StrategyResolver(default_conf)
def test_strategy_override_order_tif(caplog):
def test_strategy_override_order_tif(caplog, default_conf):
caplog.set_level(logging.INFO)
order_time_in_force = {
@ -258,11 +260,11 @@ def test_strategy_override_order_tif(caplog):
'sell': 'gtc',
}
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'order_time_in_force': order_time_in_force
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.order_time_in_force
for method in ['buy', 'sell']:
@ -271,61 +273,61 @@ def test_strategy_override_order_tif(caplog):
assert log_has("Override strategy 'order_time_in_force' with value in config file:"
" {'buy': 'fok', 'sell': 'gtc'}.", caplog)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'order_time_in_force': {'buy': 'fok'}
}
})
# Raise error for invalid configuration
with pytest.raises(ImportError,
match=r"Impossible to load Strategy 'DefaultStrategy'. "
r"Order-time-in-force mapping is incomplete."):
StrategyResolver(config)
StrategyResolver(default_conf)
def test_strategy_override_use_sell_signal(caplog):
def test_strategy_override_use_sell_signal(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert not resolver.strategy.use_sell_signal
assert isinstance(resolver.strategy.use_sell_signal, bool)
# must be inserted to configuration
assert 'use_sell_signal' in config['experimental']
assert not config['experimental']['use_sell_signal']
assert 'use_sell_signal' in default_conf['experimental']
assert not default_conf['experimental']['use_sell_signal']
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'experimental': {
'use_sell_signal': True,
},
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.use_sell_signal
assert isinstance(resolver.strategy.use_sell_signal, bool)
assert log_has("Override strategy 'use_sell_signal' with value in config file: True.", caplog)
def test_strategy_override_use_sell_profit_only(caplog):
def test_strategy_override_use_sell_profit_only(caplog, default_conf):
caplog.set_level(logging.INFO)
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert not resolver.strategy.sell_profit_only
assert isinstance(resolver.strategy.sell_profit_only, bool)
# must be inserted to configuration
assert 'sell_profit_only' in config['experimental']
assert not config['experimental']['sell_profit_only']
assert 'sell_profit_only' in default_conf['experimental']
assert not default_conf['experimental']['sell_profit_only']
config = {
default_conf.update({
'strategy': 'DefaultStrategy',
'experimental': {
'sell_profit_only': True,
},
}
resolver = StrategyResolver(config)
})
resolver = StrategyResolver(default_conf)
assert resolver.strategy.sell_profit_only
assert isinstance(resolver.strategy.sell_profit_only, bool)
@ -333,10 +335,11 @@ def test_strategy_override_use_sell_profit_only(caplog):
@pytest.mark.filterwarnings("ignore:deprecated")
def test_deprecate_populate_indicators(result):
def test_deprecate_populate_indicators(result, default_conf):
default_location = path.join(path.dirname(path.realpath(__file__)))
resolver = StrategyResolver({'strategy': 'TestStrategyLegacy',
'strategy_path': default_location})
default_conf.update({'strategy': 'TestStrategyLegacy',
'strategy_path': default_location})
resolver = StrategyResolver(default_conf)
with warnings.catch_warnings(record=True) as w:
# Cause all warnings to always be triggered.
warnings.simplefilter("always")
@ -366,10 +369,11 @@ def test_deprecate_populate_indicators(result):
@pytest.mark.filterwarnings("ignore:deprecated")
def test_call_deprecated_function(result, monkeypatch):
def test_call_deprecated_function(result, monkeypatch, default_conf):
default_location = path.join(path.dirname(path.realpath(__file__)))
resolver = StrategyResolver({'strategy': 'TestStrategyLegacy',
'strategy_path': default_location})
default_conf.update({'strategy': 'TestStrategyLegacy',
'strategy_path': default_location})
resolver = StrategyResolver(default_conf)
metadata = {'pair': 'ETH/BTC'}
# Make sure we are using a legacy function

45
freqtrade/tests/test_configuration.py
View File

@ -13,7 +13,8 @@ from freqtrade import OperationalException, constants
from freqtrade.configuration import Arguments, Configuration, validate_config_consistency
from freqtrade.configuration.check_exchange import check_exchange
from freqtrade.configuration.config_validation import validate_config_schema
from freqtrade.configuration.create_datadir import create_datadir
from freqtrade.configuration.directory_operations import (create_datadir,
create_userdata_dir)
from freqtrade.configuration.load_config import load_config_file
from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL
from freqtrade.loggers import _set_loggers
@ -52,6 +53,7 @@ def test_load_config_incorrect_stake_amount(default_conf) -> None:
def test_load_config_file(default_conf, mocker, caplog) -> None:
del default_conf['user_data_dir']
file_mock = mocker.patch('freqtrade.configuration.load_config.open', mocker.mock_open(
read_data=json.dumps(default_conf)
))
@ -331,6 +333,7 @@ def test_setup_configuration_without_arguments(mocker, default_conf, caplog) ->
assert 'exchange' in config
assert 'pair_whitelist' in config['exchange']
assert 'datadir' in config
assert 'user_data_dir' in config
assert log_has('Using data directory: {} ...'.format(config['datadir']), caplog)
assert 'ticker_interval' in config
assert not log_has('Parameter -i/--ticker-interval detected ...', caplog)
@ -355,11 +358,15 @@ def test_setup_configuration_with_arguments(mocker, default_conf, caplog) -> Non
'freqtrade.configuration.configuration.create_datadir',
lambda c, x: x
)
mocker.patch(
'freqtrade.configuration.configuration.create_userdata_dir',
lambda x, *args, **kwargs: Path(x)
)
arglist = [
'--config', 'config.json',
'--strategy', 'DefaultStrategy',
'--datadir', '/foo/bar',
'--userdir', "/tmp/freqtrade",
'backtesting',
'--ticker-interval', '1m',
'--live',
@ -380,7 +387,10 @@ def test_setup_configuration_with_arguments(mocker, default_conf, caplog) -> Non
assert 'exchange' in config
assert 'pair_whitelist' in config['exchange']
assert 'datadir' in config
assert log_has('Using data directory: {} ...'.format(config['datadir']), caplog)
assert log_has('Using data directory: {} ...'.format("/foo/bar"), caplog)
assert log_has('Using user-data directory: {} ...'.format("/tmp/freqtrade"), caplog)
assert 'user_data_dir' in config
assert 'ticker_interval' in config
assert log_has('Parameter -i/--ticker-interval detected ... Using ticker_interval: 1m ...',
caplog)
@ -615,6 +625,35 @@ def test_create_datadir(mocker, default_conf, caplog) -> None:
assert log_has('Created data directory: /foo/bar', caplog)
def test_create_userdata_dir(mocker, default_conf, caplog) -> None:
mocker.patch.object(Path, "is_dir", MagicMock(return_value=False))
md = mocker.patch.object(Path, 'mkdir', MagicMock())
x = create_userdata_dir('/tmp/bar', create_dir=True)
assert md.call_count == 7
assert md.call_args[1]['parents'] is False
assert log_has('Created user-data directory: /tmp/bar', caplog)
assert isinstance(x, Path)
assert str(x) == "/tmp/bar"
def test_create_userdata_dir_exists(mocker, default_conf, caplog) -> None:
mocker.patch.object(Path, "is_dir", MagicMock(return_value=True))
md = mocker.patch.object(Path, 'mkdir', MagicMock())
create_userdata_dir('/tmp/bar')
assert md.call_count == 0
def test_create_userdata_dir_exists_exception(mocker, default_conf, caplog) -> None:
mocker.patch.object(Path, "is_dir", MagicMock(return_value=False))
md = mocker.patch.object(Path, 'mkdir', MagicMock())
with pytest.raises(OperationalException, match=r'Directory `/tmp/bar` does not exist.*'):
create_userdata_dir('/tmp/bar', create_dir=False)
assert md.call_count == 0
def test_validate_tsl(default_conf):
default_conf['trailing_stop'] = True
default_conf['trailing_stop_positive'] = 0

4
freqtrade/tests/test_plotting.py
View File

@ -1,5 +1,6 @@
from copy import deepcopy
from pathlib import Path
from unittest.mock import MagicMock
import plotly.graph_objects as go
@ -209,7 +210,8 @@ def test_generate_Plot_filename():
def test_generate_plot_file(mocker, caplog):
fig = generage_empty_figure()
plot_mock = mocker.patch("freqtrade.plot.plotting.plot", MagicMock())
store_plot_file(fig, filename="freqtrade-plot-UNITTEST_BTC-5m.html")
store_plot_file(fig, filename="freqtrade-plot-UNITTEST_BTC-5m.html",
directory=Path("user_data/plots"))
assert plot_mock.call_count == 1
assert plot_mock.call_args[0][0] == fig

27
freqtrade/tests/test_utils.py
View File

@ -6,8 +6,8 @@ import pytest
from freqtrade.state import RunMode
from freqtrade.tests.conftest import get_args, log_has, patch_exchange
from freqtrade.utils import (setup_utils_configuration, start_download_data,
start_list_exchanges)
from freqtrade.utils import (setup_utils_configuration, start_create_userdir,
start_download_data, start_list_exchanges)
def test_setup_utils_configuration():
@ -47,6 +47,29 @@ def test_list_exchanges(capsys):
assert re.search(r"^bittrex$", captured.out, re.MULTILINE)
def test_create_datadir_failed(caplog):
args = [
"create-userdir",
]
with pytest.raises(SystemExit):
start_create_userdir(get_args(args))
assert log_has("`create-userdir` requires --userdir to be set.", caplog)
def test_create_datadir(caplog, mocker):
cud = mocker.patch("freqtrade.utils.create_userdata_dir", MagicMock())
args = [
"create-userdir",
"--userdir",
"/temp/freqtrade/test"
]
start_create_userdir(get_args(args))
assert cud.call_count == 1
assert len(caplog.record_tuples) == 0
def test_download_data(mocker, markets, caplog):
dl_mock = mocker.patch('freqtrade.utils.download_pair_history', MagicMock())
patch_exchange(mocker)

14
freqtrade/utils.py
View File

@ -7,6 +7,7 @@ from typing import Any, Dict
import arrow
from freqtrade.configuration import Configuration, TimeRange
from freqtrade.configuration.directory_operations import create_userdata_dir
from freqtrade.data.history import download_pair_history
from freqtrade.exchange import available_exchanges
from freqtrade.resolvers import ExchangeResolver
@ -46,6 +47,19 @@ def start_list_exchanges(args: Namespace) -> None:
f"{', '.join(available_exchanges())}")
def start_create_userdir(args: Namespace) -> None:
"""
Create "user_data" directory to contain user data strategies, hyperopts, ...)
:param args: Cli args from Arguments()
:return: None
"""
if "user_data_dir" in args and args.user_data_dir:
create_userdata_dir(args.user_data_dir, create_dir=True)
else:
logger.warning("`create-userdir` requires --userdir to be set.")
sys.exit(1)
def start_download_data(args: Namespace) -> None:
"""
Download data (former download_backtest_data.py script)

3
scripts/plot_dataframe.py
View File

@ -63,7 +63,8 @@ def analyse_and_plot_pairs(config: Dict[str, Any]):
indicators2=config["indicators2"].split(",")
)
store_plot_file(fig, generate_plot_filename(pair, config['ticker_interval']))
store_plot_file(fig, filename=generate_plot_filename(pair, config['ticker_interval']),
directory=config['user_data_dir'] / "plot")
logger.info('End of ploting process %s plots generated', pair_counter)

3
scripts/plot_profit.py
View File

@ -32,7 +32,8 @@ def plot_profit(config: Dict[str, Any]) -> None:
# Create an average close price of all the pairs that were involved.
# this could be useful to gauge the overall market trend
fig = generate_profit_graph(plot_elements["pairs"], plot_elements["tickers"], trades)
store_plot_file(fig, filename='freqtrade-profit-plot.html', auto_open=True)
store_plot_file(fig, filename='freqtrade-profit-plot.html',
directory=config['user_data_dir'] / "plot", auto_open=True)
def plot_parse_args(args: List[str]) -> Dict[str, Any]:

0
user_data/backtest_results/.gitkeep Normal file
View File