Merge branch 'develop' into feat/new_args_system

docs/backtesting.md

@@ -201,6 +201,8 @@ Since backtesting lacks some detailed information about what happens within a ca
 Taking these assumptions, backtesting tries to mirror real trading as closely as possible. However, backtesting will **never** replace running a strategy in dry-run mode.
 Also, keep in mind that past results don't guarantee future success.
 
+In addition to the above assumptions, strategy authors should carefully read the [Common Mistakes](strategy-customization.md#common-mistakes-when-developing-strategies) section, to avoid using data in backtesting which is not available in real market conditions.
+
 ### Further backtest-result analysis
 
 To further analyze your backtest results, you can [export the trades](#exporting-trades-to-file).

docs/strategy-customization.md

@@ -60,8 +60,7 @@ file as reference.**
 !!! Warning Using future data
     Since backtesting passes the full time interval to the `populate_*()` methods, the strategy author
     needs to take care to avoid having the strategy utilize data from the future.
-    Samples for usage of future data are `dataframe.shift(-1)`, `dataframe.resample("1h")` (this uses the left border of the interval, so moves data from an hour to the start of the hour).
+    Some common patterns for this are listed in the [Common Mistakes](#common-mistakes-when-developing-strategies) section of this document.
-    They all use data which is not available during regular operations, so these strategies will perform well during backtesting, but will fail / perform badly in dry-runs.
 
 ### Customize Indicators
 
@@ -399,10 +398,10 @@ def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
 
 Printing more than a few rows is also possible (simply use  `print(dataframe)` instead of `print(dataframe.tail())`), however not recommended, as that will be very verbose (~500 lines per pair every 5 seconds).
 
-### Where is the default strategy?
+### Where can i find a strategy template?
 
-The default buy strategy is located in the file
+The strategy template is located in the file
-[freqtrade/default_strategy.py](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/strategy/default_strategy.py).
+[user_data/strategies/sample_strategy.py](https://github.com/freqtrade/freqtrade/blob/develop/user_data/strategies/sample_strategy.py).
 
 ### Specify custom strategy location
 
@@ -412,6 +411,18 @@ If you want to use a strategy from a different directory you can pass `--strateg
 freqtrade trade --strategy AwesomeStrategy --strategy-path /some/directory
 ```
 
+### Common mistakes when developing strategies
+
+Backtesting analyzes the whole time-range at once for performance reasons. Because of this, strategy authors need to make sure that strategies do not look-ahead into the future.
+This is a common pain-point, which can cause huge differences between backtesting and dry/live run methods, since they all use data which is not available during dry/live runs, so these strategies will perform well during backtesting, but will fail / perform badly in real conditions.
+
+The following lists some common patterns which should be avoided to prevent frustration:
+
+- don't use `shift(-1)`. This uses data from the future, which is not available.
+- don't use `.iloc[-1]` or any other absolute position in the dataframe, this will be different between dry-run and backtesting.
+- don't use `dataframe['volume'].mean()`. This uses the full DataFrame for backtesting, including data from the future. Use `dataframe['volume'].rolling(<window>).mean()` instead
+- don't use `.resample('1h')`. This uses the left border of the interval, so moves data from an hour to the start of the hour. Use `.resample('1h', label='right')` instead.
+
 ### Further strategy ideas
 
 To get additional Ideas for strategies, head over to our [strategy repository](https://github.com/freqtrade/freqtrade-strategies). Feel free to use them as they are - but results will depend on the current market situation, pairs used etc. - therefore please backtest the strategy for your exchange/desired pairs first, evaluate carefully, use at your own risk.

docs/utils.md

@@ -54,3 +54,73 @@ Timeframes available for the exchange `binance`: 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4
 ```
 $ for i in `freqtrade list-exchanges -1`; do freqtrade list-timeframes --exchange $i; done
 ```
+
+## List pairs/list markets
+
+The `list-pairs` and `list-markets` subcommands allow to see the pairs/markets available on exchange.
+
+Pairs are markets with the '/' character between the base currency part and the quote currency part in the market symbol.
+For example, in the 'ETH/BTC' pair 'ETH' is the base currency, while 'BTC' is the quote currency.
+
+For pairs traded by Freqtrade the pair quote currency is defined by the value of the `stake_currency` configuration setting.
+
+You can print info about any pair/market with these subcommands - and you can filter output by quote-currency using `--quote BTC`, or by base-currency using `--base ETH` options correspondingly.
+
+These subcommands have same usage and same set of available options:
+
+```
+usage: freqtrade list-markets [-h] [--exchange EXCHANGE] [--print-list]
+                              [--print-json] [-1] [--print-csv]
+                              [--base BASE_CURRENCY [BASE_CURRENCY ...]]
+                              [--quote QUOTE_CURRENCY [QUOTE_CURRENCY ...]]
+                              [-a]
+
+usage: freqtrade list-pairs [-h] [--exchange EXCHANGE] [--print-list]
+                            [--print-json] [-1] [--print-csv]
+                            [--base BASE_CURRENCY [BASE_CURRENCY ...]]
+                            [--quote QUOTE_CURRENCY [QUOTE_CURRENCY ...]] [-a]
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --exchange EXCHANGE   Exchange name (default: `bittrex`). Only valid if no
+                        config is provided.
+  --print-list          Print list of pairs or market symbols. By default data
+                        is printed in the tabular format.
+  --print-json          Print list of pairs or market symbols in JSON format.
+  -1, --one-column      Print output in one column.
+  --print-csv           Print exchange pair or market data in the csv format.
+  --base BASE_CURRENCY [BASE_CURRENCY ...]
+                        Specify base currency(-ies). Space-separated list.
+  --quote QUOTE_CURRENCY [QUOTE_CURRENCY ...]
+                        Specify quote currency(-ies). Space-separated list.
+  -a, --all             Print all pairs or market symbols. By default only
+                        active ones are shown.
+```
+
+By default, only active pairs/markets are shown. Active pairs/markets are those that can currently be traded
+on the exchange. The see the list of all pairs/markets (not only the active ones), use the `-a`/`-all` option.
+
+Pairs/markets are sorted by its symbol string in the printed output.
+
+### Examples
+
+* Print the list of active pairs with quote currency USD on exchange, specified in the default
+configuration file (i.e. pairs on the "Bittrex" exchange) in JSON format:
+
+```
+$ freqtrade list-pairs --quote USD --print-json
+```
+
+* Print the list of all pairs on the exchange, specified in the `config_binance.json` configuration file
+(i.e. on the "Binance" exchange) with base currencies BTC or ETH and quote currencies USDT or USD, as the
+human-readable list with summary:
+
+```
+$ freqtrade -c config_binance.json list-pairs --all --base BTC ETH --quote USDT USD --print-list
+```
+
+* Print all markets on exchange "Kraken", in the tabular format:
+
+```
+$ freqtrade list-markets --exchange kraken --all
+```

freqtrade/configuration/arguments.py

@@ -2,6 +2,7 @@
 This module contains the argument manager class
 """
 import argparse
+from functools import partial
 from pathlib import Path
 from typing import Any, Dict, List, Optional
 
@@ -33,6 +34,9 @@ ARGS_LIST_EXCHANGES = ["print_one_column", "list_exchanges_all"]
 
 ARGS_LIST_TIMEFRAMES = ["exchange", "print_one_column"]
 
+ARGS_LIST_PAIRS = ["exchange", "print_list", "list_pairs_print_json", "print_one_column",
+                   "print_csv", "base_currencies", "quote_currencies", "list_pairs_all"]
+
 ARGS_CREATE_USERDIR = ["user_data_dir"]
 
 ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "download_trades", "exchange",
@@ -45,7 +49,8 @@ ARGS_PLOT_DATAFRAME = ["pairs", "indicators1", "indicators2", "plot_limit",
 ARGS_PLOT_PROFIT = ["pairs", "timerange", "export", "exportfilename", "db_url",
                     "trade_source", "ticker_interval"]
 
-NO_CONF_REQURIED = ["download-data", "list-timeframes", "plot-dataframe", "plot-profit"]
+NO_CONF_REQURIED = ["download-data", "list-timeframes", "list-markets", "list-pairs",
+                    "plot-dataframe", "plot-profit"]
 
 NO_CONF_ALLOWED = ["create-userdir", "list-exchanges"]
 
@@ -111,7 +116,8 @@ class Arguments:
 
         from freqtrade.optimize import start_backtesting, start_hyperopt, start_edge
         from freqtrade.utils import (start_create_userdir, start_download_data,
-                                     start_list_exchanges, start_list_timeframes, start_trading)
+                                     start_list_exchanges, start_list_markets,
+                                     start_list_timeframes, start_trading)
         from freqtrade.plot.plot_utils import start_plot_dataframe, start_plot_profit
 
         subparsers = self.parser.add_subparsers(dest='command',
@@ -170,6 +176,22 @@ class Arguments:
         list_timeframes_cmd.set_defaults(func=start_list_timeframes)
         self._build_args(optionlist=ARGS_LIST_TIMEFRAMES, parser=list_timeframes_cmd)
 
+        # Add list-markets subcommand
+        list_markets_cmd = subparsers.add_parser(
+            'list-markets',
+            help='Print markets on exchange.'
+        )
+        list_markets_cmd.set_defaults(func=partial(start_list_markets, pairs_only=False))
+        self._build_args(optionlist=ARGS_LIST_PAIRS, parser=list_markets_cmd)
+
+        # Add list-pairs subcommand
+        list_pairs_cmd = subparsers.add_parser(
+            'list-pairs',
+            help='Print pairs on exchange.'
+        )
+        list_pairs_cmd.set_defaults(func=partial(start_list_markets, pairs_only=True))
+        self._build_args(optionlist=ARGS_LIST_PAIRS, parser=list_pairs_cmd)
+
         # Add download-data subcommand
         download_data_cmd = subparsers.add_parser(
             'download-data',

freqtrade/configuration/cli_options.py

@@ -255,6 +255,42 @@ AVAILABLE_CLI_OPTIONS = {
         help='Print all exchanges known to the ccxt library.',
         action='store_true',
     ),
+    # List pairs / markets
+    "list_pairs_all": Arg(
+        '-a', '--all',
+        help='Print all pairs or market symbols. By default only active '
+             'ones are shown.',
+        action='store_true',
+    ),
+    "print_list": Arg(
+        '--print-list',
+        help='Print list of pairs or market symbols. By default data is '
+             'printed in the tabular format.',
+        action='store_true',
+    ),
+    "list_pairs_print_json": Arg(
+        '--print-json',
+        help='Print list of pairs or market symbols in JSON format.',
+        action='store_true',
+        default=False,
+    ),
+    "print_csv": Arg(
+        '--print-csv',
+        help='Print exchange pair or market data in the csv format.',
+        action='store_true',
+    ),
+    "quote_currencies": Arg(
+        '--quote',
+        help='Specify quote currency(-ies). Space-separated list.',
+        nargs='+',
+        metavar='QUOTE_CURRENCY',
+    ),
+    "base_currencies": Arg(
+        '--base',
+        help='Specify base currency(-ies). Space-separated list.',
+        nargs='+',
+        metavar='BASE_CURRENCY',
+    ),
     # Script options
     "pairs": Arg(
         '-p', '--pairs',

freqtrade/exchange/__init__.py

@@ -10,5 +10,7 @@ from freqtrade.exchange.exchange import (timeframe_to_seconds,  # noqa: F401
                                          timeframe_to_msecs,
                                          timeframe_to_next_date,
                                          timeframe_to_prev_date)
+from freqtrade.exchange.exchange import (market_is_active,  # noqa: F401
+                                         symbol_is_pair)
 from freqtrade.exchange.kraken import Kraken  # noqa: F401
 from freqtrade.exchange.binance import Binance  # noqa: F401

freqtrade/exchange/exchange.py

@@ -22,6 +22,7 @@ from freqtrade import (DependencyException, InvalidOrderException,
 from freqtrade.data.converter import parse_ticker_dataframe
 from freqtrade.misc import deep_merge_dicts
 
+
 logger = logging.getLogger(__name__)
 
 
@@ -165,7 +166,7 @@ class Exchange:
     }
     _ft_has: Dict = {}
 
-    def __init__(self, config: dict) -> None:
+    def __init__(self, config: dict, validate: bool = True) -> None:
         """
         Initializes this module with the given config,
         it does basic validation whether the specified exchange and pairs are valid.
@@ -216,19 +217,21 @@ class Exchange:
 
         logger.info('Using Exchange "%s"', self.name)
 
-        # Check if timeframe is available
+        if validate:
-        self.validate_timeframes(config.get('ticker_interval'))
+            # Check if timeframe is available
+            self.validate_timeframes(config.get('ticker_interval'))
+
+            # Initial markets load
+            self._load_markets()
+
+            # Check if all pairs are available
+            self.validate_pairs(config['exchange']['pair_whitelist'])
+            self.validate_ordertypes(config.get('order_types', {}))
+            self.validate_order_time_in_force(config.get('order_time_in_force', {}))
 
         # Converts the interval provided in minutes in config to seconds
         self.markets_refresh_interval: int = exchange_config.get(
             "markets_refresh_interval", 60) * 60
-        # Initial markets load
-        self._load_markets()
-
-        # Check if all pairs are available
-        self.validate_pairs(config['exchange']['pair_whitelist'])
-        self.validate_ordertypes(config.get('order_types', {}))
-        self.validate_order_time_in_force(config.get('order_time_in_force', {}))
 
     def __del__(self):
         """
@@ -293,6 +296,28 @@ class Exchange:
             self._load_markets()
         return self._api.markets
 
+    def get_markets(self, base_currencies: List[str] = None, quote_currencies: List[str] = None,
+                    pairs_only: bool = False, active_only: bool = False) -> Dict:
+        """
+        Return exchange ccxt markets, filtered out by base currency and quote currency
+        if this was requested in parameters.
+
+        TODO: consider moving it to the Dataprovider
+        """
+        markets = self.markets
+        if not markets:
+            raise OperationalException("Markets were not loaded.")
+
+        if base_currencies:
+            markets = {k: v for k, v in markets.items() if v['base'] in base_currencies}
+        if quote_currencies:
+            markets = {k: v for k, v in markets.items() if v['quote'] in quote_currencies}
+        if pairs_only:
+            markets = {k: v for k, v in markets.items() if symbol_is_pair(v['symbol'])}
+        if active_only:
+            markets = {k: v for k, v in markets.items() if market_is_active(v)}
+        return markets
+
     def klines(self, pair_interval: Tuple[str, str], copy=True) -> DataFrame:
         if pair_interval in self._klines:
             return self._klines[pair_interval].copy() if copy else self._klines[pair_interval]
@@ -1074,3 +1099,27 @@ def timeframe_to_next_date(timeframe: str, date: datetime = None) -> datetime:
     new_timestamp = ccxt.Exchange.round_timeframe(timeframe, date.timestamp() * 1000,
                                                   ROUND_UP) // 1000
     return datetime.fromtimestamp(new_timestamp, tz=timezone.utc)
+
+
+def symbol_is_pair(market_symbol: str, base_currency: str = None, quote_currency: str = None):
+    """
+    Check if the market symbol is a pair, i.e. that its symbol consists of the base currency and the
+    quote currency separated by '/' character. If base_currency and/or quote_currency is passed,
+    it also checks that the symbol contains appropriate base and/or quote currency part before
+    and after the separating character correspondingly.
+    """
+    symbol_parts = market_symbol.split('/')
+    return (len(symbol_parts) == 2 and
+            (symbol_parts[0] == base_currency if base_currency else len(symbol_parts[0]) > 0) and
+            (symbol_parts[1] == quote_currency if quote_currency else len(symbol_parts[1]) > 0))
+
+
+def market_is_active(market):
+    """
+    Return True if the market is active.
+    """
+    # "It's active, if the active flag isn't explicitly set to false. If it's missing or
+    # true then it's true. If it's undefined, then it's most likely true, but not 100% )"
+    # See https://github.com/ccxt/ccxt/issues/4874,
+    # https://github.com/ccxt/ccxt/issues/4075#issuecomment-434760520
+    return market.get('active', True) is not False

freqtrade/misc.py

@@ -123,3 +123,7 @@ def round_dict(d, n):
     Rounds float values in the dict to n digits after the decimal point.
     """
     return {k: (round(v, n) if isinstance(v, float) else v) for k, v in d.items()}
+
+
+def plural(num, singular: str, plural: str = None) -> str:
+    return singular if (num == 1 or num == -1) else plural or singular + 's'

freqtrade/pairlist/IPairList.py

@@ -8,6 +8,9 @@ import logging
 from abc import ABC, abstractmethod
 from typing import List
 
+from freqtrade.exchange import market_is_active
+
+
 logger = logging.getLogger(__name__)
 
 
@@ -77,7 +80,7 @@ class IPairList(ABC):
                 continue
             # Check if market is active
             market = markets[pair]
-            if not market['active']:
+            if not market_is_active(market):
                 logger.info(f"Ignoring {pair} from whitelist. Market is not active.")
                 continue
             sanitized_whitelist.add(pair)

freqtrade/resolvers/exchange_resolver.py

@@ -17,7 +17,7 @@ class ExchangeResolver(IResolver):
 
     __slots__ = ['exchange']
 
-    def __init__(self, exchange_name: str, config: dict) -> None:
+    def __init__(self, exchange_name: str, config: dict, validate: bool = True) -> None:
         """
         Load the custom class from config parameter
         :param config: configuration dictionary
@@ -26,12 +26,13 @@ class ExchangeResolver(IResolver):
         exchange_name = MAP_EXCHANGE_CHILDCLASS.get(exchange_name, exchange_name)
         exchange_name = exchange_name.title()
         try:
-            self.exchange = self._load_exchange(exchange_name, kwargs={'config': config})
+            self.exchange = self._load_exchange(exchange_name, kwargs={'config': config,
+                                                                       'validate': validate})
         except ImportError:
             logger.info(
                 f"No {exchange_name} specific subclass found. Using the generic class instead.")
         if not hasattr(self, "exchange"):
-            self.exchange = Exchange(config)
+            self.exchange = Exchange(config, validate=validate)
 
     def _load_exchange(
             self, exchange_name: str, kwargs: dict) -> Exchange:
@@ -45,7 +46,7 @@ class ExchangeResolver(IResolver):
         try:
             ex_class = getattr(exchanges, exchange_name)
 
-            exchange = ex_class(kwargs['config'])
+            exchange = ex_class(**kwargs)
             if exchange:
                 logger.info(f"Using resolved exchange '{exchange_name}'...")
                 return exchange

freqtrade/rpc/api_server.py

@@ -2,7 +2,7 @@ import logging
 import threading
 from datetime import date, datetime
 from ipaddress import IPv4Address
-from typing import Dict
+from typing import Dict, Callable, Any
 
 from arrow import Arrow
 from flask import Flask, jsonify, request
@@ -34,41 +34,45 @@ class ArrowJSONEncoder(JSONEncoder):
         return JSONEncoder.default(self, obj)
 
 
+# Type should really be Callable[[ApiServer, Any], Any], but that will create a circular dependency
+def require_login(func: Callable[[Any, Any], Any]):
+
+    def func_wrapper(obj, *args, **kwargs):
+
+        auth = request.authorization
+        if auth and obj.check_auth(auth.username, auth.password):
+            return func(obj, *args, **kwargs)
+        else:
+            return jsonify({"error": "Unauthorized"}), 401
+
+    return func_wrapper
+
+
+# Type should really be Callable[[ApiServer], Any], but that will create a circular dependency
+def rpc_catch_errors(func: Callable[[Any], Any]):
+
+    def func_wrapper(obj, *args, **kwargs):
+
+        try:
+            return func(obj, *args, **kwargs)
+        except RPCException as e:
+            logger.exception("API Error calling %s: %s", func.__name__, e)
+            return obj.rest_error(f"Error querying {func.__name__}: {e}")
+
+    return func_wrapper
+
+
 class ApiServer(RPC):
     """
     This class runs api server and provides rpc.rpc functionality to it
 
-    This class starts a none blocking thread the api server runs within
+    This class starts a non-blocking thread the api server runs within
     """
 
-    def rpc_catch_errors(func):
-
-        def func_wrapper(self, *args, **kwargs):
-
-            try:
-                return func(self, *args, **kwargs)
-            except RPCException as e:
-                logger.exception("API Error calling %s: %s", func.__name__, e)
-                return self.rest_error(f"Error querying {func.__name__}: {e}")
-
-        return func_wrapper
-
     def check_auth(self, username, password):
         return (username == self._config['api_server'].get('username') and
                 password == self._config['api_server'].get('password'))
 
-    def require_login(func):
-
-        def func_wrapper(self, *args, **kwargs):
-
-            auth = request.authorization
-            if auth and self.check_auth(auth.username, auth.password):
-                return func(self, *args, **kwargs)
-            else:
-                return jsonify({"error": "Unauthorized"}), 401
-
-        return func_wrapper
-
     def __init__(self, freqtrade) -> None:
         """
         Init the api server, and init the super class RPC

freqtrade/utils.py

@@ -1,9 +1,13 @@
 import logging
 import sys
+from collections import OrderedDict
 from pathlib import Path
 from typing import Any, Dict, List
 
 import arrow
+import csv
+import rapidjson
+from tabulate import tabulate
 
 from freqtrade import OperationalException
 from freqtrade.configuration import Configuration, TimeRange
@@ -11,7 +15,9 @@ from freqtrade.configuration.directory_operations import create_userdata_dir
 from freqtrade.data.history import (convert_trades_to_ohlcv,
                                     refresh_backtest_ohlcv_data,
                                     refresh_backtest_trades_data)
-from freqtrade.exchange import available_exchanges, ccxt_exchanges
+from freqtrade.exchange import (available_exchanges, ccxt_exchanges, market_is_active,
+                                symbol_is_pair)
+from freqtrade.misc import plural
 from freqtrade.resolvers import ExchangeResolver
 from freqtrade.state import RunMode
 
@@ -133,10 +139,94 @@ def start_list_timeframes(args: Dict[str, Any]) -> None:
     config['ticker_interval'] = None
 
     # Init exchange
-    exchange = ExchangeResolver(config['exchange']['name'], config).exchange
+    exchange = ExchangeResolver(config['exchange']['name'], config, validate=False).exchange
 
     if args['print_one_column']:
         print('\n'.join(exchange.timeframes))
     else:
         print(f"Timeframes available for the exchange `{config['exchange']['name']}`: "
-              f"{', '.join(exchange.timeframes)}")
+              f"{', '.join(exchange.timeframes)}.")
+
+
+def start_list_markets(args: Dict[str, Any], pairs_only: bool = False) -> None:
+    """
+    Print pairs/markets on the exchange
+    :param args: Cli args from Arguments()
+    :param pairs_only: if True print only pairs, otherwise print all instruments (markets)
+    :return: None
+    """
+    config = setup_utils_configuration(args, RunMode.OTHER)
+
+    # Init exchange
+    exchange = ExchangeResolver(config['exchange']['name'], config, validate=False).exchange
+
+    # By default only active pairs/markets are to be shown
+    active_only = not args.get('list_pairs_all', False)
+
+    base_currencies = args.get('base_currencies', [])
+    quote_currencies = args.get('quote_currencies', [])
+
+    try:
+        pairs = exchange.get_markets(base_currencies=base_currencies,
+                                     quote_currencies=quote_currencies,
+                                     pairs_only=pairs_only,
+                                     active_only=active_only)
+        # Sort the pairs/markets by symbol
+        pairs = OrderedDict(sorted(pairs.items()))
+    except Exception as e:
+        raise OperationalException(f"Cannot get markets. Reason: {e}") from e
+
+    else:
+        summary_str = ((f"Exchange {exchange.name} has {len(pairs)} ") +
+                       ("active " if active_only else "") +
+                       (plural(len(pairs), "pair" if pairs_only else "market")) +
+                       (f" with {', '.join(base_currencies)} as base "
+                        f"{plural(len(base_currencies), 'currency', 'currencies')}"
+                        if base_currencies else "") +
+                       (" and" if base_currencies and quote_currencies else "") +
+                       (f" with {', '.join(quote_currencies)} as quote "
+                        f"{plural(len(quote_currencies), 'currency', 'currencies')}"
+                        if quote_currencies else ""))
+
+        headers = ["Id", "Symbol", "Base", "Quote", "Active",
+                   *(['Is pair'] if not pairs_only else [])]
+
+        tabular_data = []
+        for _, v in pairs.items():
+            tabular_data.append({'Id': v['id'], 'Symbol': v['symbol'],
+                                 'Base': v['base'], 'Quote': v['quote'],
+                                 'Active': market_is_active(v),
+                                 **({'Is pair': symbol_is_pair(v['symbol'])}
+                                    if not pairs_only else {})})
+
+        if (args.get('print_one_column', False) or
+                args.get('list_pairs_print_json', False) or
+                args.get('print_csv', False)):
+            # Print summary string in the log in case of machine-readable
+            # regular formats.
+            logger.info(f"{summary_str}.")
+        else:
+            # Print empty string separating leading logs and output in case of
+            # human-readable formats.
+            print()
+
+        if len(pairs):
+            if args.get('print_list', False):
+                # print data as a list, with human-readable summary
+                print(f"{summary_str}: {', '.join(pairs.keys())}.")
+            elif args.get('print_one_column', False):
+                print('\n'.join(pairs.keys()))
+            elif args.get('list_pairs_print_json', False):
+                print(rapidjson.dumps(list(pairs.keys()), default=str))
+            elif args.get('print_csv', False):
+                writer = csv.DictWriter(sys.stdout, fieldnames=headers)
+                writer.writeheader()
+                writer.writerows(tabular_data)
+            else:
+                # print data as a table, with the human-readable summary
+                print(f"{summary_str}:")
+                print(tabulate(tabular_data, headers='keys', tablefmt='pipe'))
+        elif not (args.get('print_one_column', False) or
+                  args.get('list_pairs_print_json', False) or
+                  args.get('print_csv', False)):
+            print(f"{summary_str}.")

requirements-common.txt

@@ -1,8 +1,8 @@
 # requirements without requirements installable via conda
 # mainly used for Raspberry pi installs
-ccxt==1.18.1260
+ccxt==1.18.1306
 SQLAlchemy==1.3.10
-python-telegram-bot==12.1.1
+python-telegram-bot==12.2.0
 arrow==0.15.2
 cachetools==3.1.1
 requests==2.22.0

requirements-dev.txt

@@ -6,8 +6,8 @@
 coveralls==1.8.2
 flake8==3.7.8
 flake8-type-annotations==0.1.0
-flake8-tidy-imports==2.0.0
+flake8-tidy-imports==3.0.0
-mypy==0.730
+mypy==0.740
 pytest==5.2.1
 pytest-asyncio==0.10.0
 pytest-cov==2.8.1

requirements-plot.txt

@@ -1,5 +1,5 @@
 # Include all requirements to run the bot.
 -r requirements.txt
 
-plotly==4.1.1
+plotly==4.2.1
 

requirements.txt

@@ -1,5 +1,5 @@
 # Load common requirements
 -r requirements-common.txt
 
-numpy==1.17.2
+numpy==1.17.3
-pandas==0.25.1
+pandas==0.25.2

tests/conftest.py

@@ -319,7 +319,8 @@ def markets():
             'symbol': 'TKN/BTC',
             'base': 'TKN',
             'quote': 'BTC',
-            'active': True,
+            # According to ccxt, markets without active item set are also active
+            # 'active': True,
             'precision': {
                 'price': 8,
                 'amount': 8,
@@ -510,6 +511,50 @@ def markets():
                 }
             },
             'info': {},
+        },
+        'LTC/USD': {
+            'id': 'USD-LTC',
+            'symbol': 'LTC/USD',
+            'base': 'LTC',
+            'quote': 'USD',
+            'active': True,
+            'precision': {
+                'amount': 8,
+                'price': 8
+            },
+            'limits': {
+                'amount': {
+                    'min': 0.06646786,
+                    'max': None
+                },
+                'price': {
+                    'min': 1e-08,
+                    'max': None
+                }
+            },
+            'info': {},
+        },
+        'XLTCUSDT': {
+            'id': 'xLTCUSDT',
+            'symbol': 'XLTCUSDT',
+            'base': 'LTC',
+            'quote': 'USDT',
+            'active': True,
+            'precision': {
+                'amount': 8,
+                'price': 8
+            },
+            'limits': {
+                'amount': {
+                    'min': 0.06646786,
+                    'max': None
+                },
+                'price': {
+                    'min': 1e-08,
+                    'max': None
+                }
+            },
+            'info': {},
         }
     }
 

tests/exchange/test_exchange.py

@@ -18,7 +18,9 @@ from freqtrade.exchange.exchange import (API_RETRY_COUNT, timeframe_to_minutes,
                                          timeframe_to_msecs,
                                          timeframe_to_next_date,
                                          timeframe_to_prev_date,
-                                         timeframe_to_seconds)
+                                         timeframe_to_seconds,
+                                         symbol_is_pair,
+                                         market_is_active)
 from freqtrade.resolvers.exchange_resolver import ExchangeResolver
 from tests.conftest import get_patched_exchange, log_has, log_has_re
 
@@ -929,17 +931,17 @@ def test_get_balances_prod(default_conf, mocker, exchange_name):
 def test_get_tickers(default_conf, mocker, exchange_name):
     api_mock = MagicMock()
     tick = {'ETH/BTC': {
-          'symbol': 'ETH/BTC',
+        'symbol': 'ETH/BTC',
-          'bid': 0.5,
+        'bid': 0.5,
-          'ask': 1,
+        'ask': 1,
-          'last': 42,
+        'last': 42,
-      }, 'BCH/BTC': {
+    }, 'BCH/BTC': {
-          'symbol': 'BCH/BTC',
+        'symbol': 'BCH/BTC',
-          'bid': 0.6,
+        'bid': 0.6,
-          'ask': 0.5,
+        'ask': 0.5,
-          'last': 41,
+        'last': 41,
-      }
+    }
-      }
+    }
     api_mock.fetch_tickers = MagicMock(return_value=tick)
     exchange = get_patched_exchange(mocker, default_conf, api_mock, id=exchange_name)
     # retrieve original ticker
@@ -1693,6 +1695,74 @@ def test_get_valid_pair_combination(default_conf, mocker, markets):
         ex.get_valid_pair_combination("NOPAIR", "ETH")
 
 
+@pytest.mark.parametrize(
+    "base_currencies, quote_currencies, pairs_only, active_only, expected_keys", [
+        # Testing markets (in conftest.py):
+        # 'BLK/BTC':  'active': True
+        # 'BTT/BTC':  'active': True
+        # 'ETH/BTC':  'active': True
+        # 'ETH/USDT': 'active': True
+        # 'LTC/BTC':  'active': False
+        # 'LTC/USD':  'active': True
+        # 'LTC/USDT': 'active': True
+        # 'NEO/BTC':  'active': False
+        # 'TKN/BTC':  'active'  not set
+        # 'XLTCUSDT': 'active': True, not a pair
+        # 'XRP/BTC':  'active': False
+        # all markets
+        ([], [], False, False,
+         ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/BTC', 'LTC/USD',
+          'LTC/USDT', 'NEO/BTC', 'TKN/BTC', 'XLTCUSDT', 'XRP/BTC']),
+        # active markets
+        ([], [], False, True,
+         ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/USD', 'LTC/USDT',
+          'TKN/BTC', 'XLTCUSDT']),
+        # all pairs
+        ([], [], True, False,
+         ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/BTC', 'LTC/USD',
+          'LTC/USDT', 'NEO/BTC', 'TKN/BTC', 'XRP/BTC']),
+        # active pairs
+        ([], [], True, True,
+         ['BLK/BTC', 'BTT/BTC', 'ETH/BTC', 'ETH/USDT', 'LTC/USD', 'LTC/USDT', 'TKN/BTC']),
+        # all markets, base=ETH, LTC
+        (['ETH', 'LTC'], [], False, False,
+         ['ETH/BTC', 'ETH/USDT', 'LTC/BTC', 'LTC/USD', 'LTC/USDT', 'XLTCUSDT']),
+        # all markets, base=LTC
+        (['LTC'], [], False, False,
+         ['LTC/BTC', 'LTC/USD', 'LTC/USDT', 'XLTCUSDT']),
+        # all markets, quote=USDT
+        ([], ['USDT'], False, False,
+         ['ETH/USDT', 'LTC/USDT', 'XLTCUSDT']),
+        # all markets, quote=USDT, USD
+        ([], ['USDT', 'USD'], False, False,
+         ['ETH/USDT', 'LTC/USD', 'LTC/USDT', 'XLTCUSDT']),
+        # all markets, base=LTC, quote=USDT
+        (['LTC'], ['USDT'], False, False,
+         ['LTC/USDT', 'XLTCUSDT']),
+        # all pairs, base=LTC, quote=USDT
+        (['LTC'], ['USDT'], True, False,
+         ['LTC/USDT']),
+        # all markets, base=LTC, quote=USDT, NONEXISTENT
+        (['LTC'], ['USDT', 'NONEXISTENT'], False, False,
+         ['LTC/USDT', 'XLTCUSDT']),
+        # all markets, base=LTC, quote=NONEXISTENT
+        (['LTC'], ['NONEXISTENT'], False, False,
+         []),
+    ])
+def test_get_markets(default_conf, mocker, markets,
+                     base_currencies, quote_currencies, pairs_only, active_only,
+                     expected_keys):
+    mocker.patch.multiple('freqtrade.exchange.Exchange',
+                          _init_ccxt=MagicMock(return_value=MagicMock()),
+                          _load_async_markets=MagicMock(),
+                          validate_pairs=MagicMock(),
+                          validate_timeframes=MagicMock(),
+                          markets=PropertyMock(return_value=markets))
+    ex = Exchange(default_conf)
+    pairs = ex.get_markets(base_currencies, quote_currencies, pairs_only, active_only)
+    assert sorted(pairs.keys()) == sorted(expected_keys)
+
+
 def test_timeframe_to_minutes():
     assert timeframe_to_minutes("5m") == 5
     assert timeframe_to_minutes("10m") == 10
@@ -1762,3 +1832,33 @@ def test_timeframe_to_next_date():
 
     date = datetime.now(tz=timezone.utc)
     assert timeframe_to_next_date("5m") > date
+
+
+@pytest.mark.parametrize("market_symbol,base_currency,quote_currency,expected_result", [
+    ("BTC/USDT", None, None, True),
+    ("USDT/BTC", None, None, True),
+    ("BTCUSDT", None, None, False),
+    ("BTC/USDT", None, "USDT", True),
+    ("USDT/BTC", None, "USDT", False),
+    ("BTCUSDT", None, "USDT", False),
+    ("BTC/USDT", "BTC", None, True),
+    ("USDT/BTC", "BTC", None, False),
+    ("BTCUSDT", "BTC", None, False),
+    ("BTC/USDT", "BTC", "USDT", True),
+    ("BTC/USDT", "USDT", "BTC", False),
+    ("BTC/USDT", "BTC", "USD", False),
+    ("BTCUSDT", "BTC", "USDT", False),
+    ("BTC/", None, None, False),
+    ("/USDT", None, None, False),
+])
+def test_symbol_is_pair(market_symbol, base_currency, quote_currency, expected_result) -> None:
+    assert symbol_is_pair(market_symbol, base_currency, quote_currency) == expected_result
+
+
+@pytest.mark.parametrize("market,expected_result", [
+    ({'symbol': 'ETH/BTC', 'active': True}, True),
+    ({'symbol': 'ETH/BTC', 'active': False}, False),
+    ({'symbol': 'ETH/BTC', }, True),
+])
+def test_market_is_active(market, expected_result) -> None:
+    assert market_is_active(market) == expected_result

tests/test_misc.py

@@ -7,7 +7,7 @@ from unittest.mock import MagicMock
 from freqtrade.data.converter import parse_ticker_dataframe
 from freqtrade.data.history import pair_data_filename
 from freqtrade.misc import (datesarray_to_datetimearray, file_dump_json,
-                            file_load_json, format_ms_time, shorten_date)
+                            file_load_json, format_ms_time, plural, shorten_date)
 
 
 def test_shorten_date() -> None:
@@ -69,3 +69,35 @@ def test_format_ms_time() -> None:
     # Date 2017-12-13 08:02:01
     date_in_epoch_ms = 1513152121000
     assert format_ms_time(date_in_epoch_ms) == res.astimezone(None).strftime('%Y-%m-%dT%H:%M:%S')
+
+
+def test_plural() -> None:
+    assert plural(0, "page") == "pages"
+    assert plural(0.0, "page") == "pages"
+    assert plural(1, "page") == "page"
+    assert plural(1.0, "page") == "page"
+    assert plural(2, "page") == "pages"
+    assert plural(2.0, "page") == "pages"
+    assert plural(-1, "page") == "page"
+    assert plural(-1.0, "page") == "page"
+    assert plural(-2, "page") == "pages"
+    assert plural(-2.0, "page") == "pages"
+    assert plural(0.5, "page") == "pages"
+    assert plural(1.5, "page") == "pages"
+    assert plural(-0.5, "page") == "pages"
+    assert plural(-1.5, "page") == "pages"
+
+    assert plural(0, "ox", "oxen") == "oxen"
+    assert plural(0.0, "ox", "oxen") == "oxen"
+    assert plural(1, "ox", "oxen") == "ox"
+    assert plural(1.0, "ox", "oxen") == "ox"
+    assert plural(2, "ox", "oxen") == "oxen"
+    assert plural(2.0, "ox", "oxen") == "oxen"
+    assert plural(-1, "ox", "oxen") == "ox"
+    assert plural(-1.0, "ox", "oxen") == "ox"
+    assert plural(-2, "ox", "oxen") == "oxen"
+    assert plural(-2.0, "ox", "oxen") == "oxen"
+    assert plural(0.5, "ox", "oxen") == "oxen"
+    assert plural(1.5, "ox", "oxen") == "oxen"
+    assert plural(-0.5, "ox", "oxen") == "oxen"
+    assert plural(-1.5, "ox", "oxen") == "oxen"

tests/test_utils.py

@@ -8,7 +8,7 @@ from freqtrade import OperationalException
 from freqtrade.state import RunMode
 from freqtrade.utils import (setup_utils_configuration, start_create_userdir,
                              start_download_data, start_list_exchanges,
-                             start_list_timeframes)
+                             start_list_markets, start_list_timeframes)
 from tests.conftest import get_args, log_has, patch_exchange
 
 
@@ -164,6 +164,247 @@ def test_list_timeframes(mocker, capsys):
     assert re.search(r"^1d$", captured.out, re.MULTILINE)
 
 
+def test_list_markets(mocker, markets, capsys):
+
+    api_mock = MagicMock()
+    api_mock.markets = markets
+    patch_exchange(mocker, api_mock=api_mock)
+
+    # Test with no --config
+    args = [
+        "list-markets",
+    ]
+    pargs = get_args(args)
+    pargs['config'] = None
+    with pytest.raises(OperationalException,
+                       match=r"This command requires a configured exchange.*"):
+        start_list_markets(pargs, False)
+
+    # Test with --config config.json.example
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 8 active markets: "
+            "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/USD, LTC/USDT, TKN/BTC, XLTCUSDT.\n"
+            in captured.out)
+
+    patch_exchange(mocker, api_mock=api_mock, id="binance")
+    # Test with --exchange
+    args = [
+        "list-markets",
+        "--exchange", "binance"
+    ]
+    pargs = get_args(args)
+    pargs['config'] = None
+    start_list_markets(pargs, False)
+    captured = capsys.readouterr()
+    assert re.match("\nExchange Binance has 8 active markets:\n",
+                    captured.out)
+
+    patch_exchange(mocker, api_mock=api_mock, id="bittrex")
+    # Test with --all: all markets
+    args = [
+        '--config', 'config.json.example',
+        "list-markets", "--all",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 11 markets: "
+            "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/BTC, LTC/USD, LTC/USDT, NEO/BTC, "
+            "TKN/BTC, XLTCUSDT, XRP/BTC.\n"
+            in captured.out)
+
+    # Test list-pairs subcommand: active pairs
+    args = [
+        '--config', 'config.json.example',
+        "list-pairs",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), True)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 7 active pairs: "
+            "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/USD, LTC/USDT, TKN/BTC.\n"
+            in captured.out)
+
+    # Test list-pairs subcommand with --all: all pairs
+    args = [
+        '--config', 'config.json.example',
+        "list-pairs", "--all",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), True)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 10 pairs: "
+            "BLK/BTC, BTT/BTC, ETH/BTC, ETH/USDT, LTC/BTC, LTC/USD, LTC/USDT, NEO/BTC, "
+            "TKN/BTC, XRP/BTC.\n"
+            in captured.out)
+
+    # active markets, base=ETH, LTC
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--base", "ETH", "LTC",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 5 active markets with ETH, LTC as base currencies: "
+            "ETH/BTC, ETH/USDT, LTC/USD, LTC/USDT, XLTCUSDT.\n"
+            in captured.out)
+
+    # active markets, base=LTC
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--base", "LTC",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 3 active markets with LTC as base currency: "
+            "LTC/USD, LTC/USDT, XLTCUSDT.\n"
+            in captured.out)
+
+    # active markets, quote=USDT, USD
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--quote", "USDT", "USD",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 4 active markets with USDT, USD as quote currencies: "
+            "ETH/USDT, LTC/USD, LTC/USDT, XLTCUSDT.\n"
+            in captured.out)
+
+    # active markets, quote=USDT
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--quote", "USDT",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 3 active markets with USDT as quote currency: "
+            "ETH/USDT, LTC/USDT, XLTCUSDT.\n"
+            in captured.out)
+
+    # active markets, base=LTC, quote=USDT
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--base", "LTC", "--quote", "USDT",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 2 active markets with LTC as base currency and "
+            "with USDT as quote currency: LTC/USDT, XLTCUSDT.\n"
+            in captured.out)
+
+    # active pairs, base=LTC, quote=USDT
+    args = [
+        '--config', 'config.json.example',
+        "list-pairs",
+        "--base", "LTC", "--quote", "USDT",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), True)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 1 active pair with LTC as base currency and "
+            "with USDT as quote currency: LTC/USDT.\n"
+            in captured.out)
+
+    # active markets, base=LTC, quote=USDT, NONEXISTENT
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--base", "LTC", "--quote", "USDT", "NONEXISTENT",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 2 active markets with LTC as base currency and "
+            "with USDT, NONEXISTENT as quote currencies: LTC/USDT, XLTCUSDT.\n"
+            in captured.out)
+
+    # active markets, base=LTC, quote=NONEXISTENT
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--base", "LTC", "--quote", "NONEXISTENT",
+        "--print-list",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 0 active markets with LTC as base currency and "
+            "with NONEXISTENT as quote currency.\n"
+            in captured.out)
+
+    # Test tabular output
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 8 active markets:\n"
+            in captured.out)
+
+    # Test tabular output, no markets found
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--base", "LTC", "--quote", "NONEXISTENT",
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Exchange Bittrex has 0 active markets with LTC as base currency and "
+            "with NONEXISTENT as quote currency.\n"
+            in captured.out)
+
+    # Test --print-json
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--print-json"
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ('["BLK/BTC","BTT/BTC","ETH/BTC","ETH/USDT","LTC/USD","LTC/USDT","TKN/BTC","XLTCUSDT"]'
+            in captured.out)
+
+    # Test --print-csv
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--print-csv"
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert ("Id,Symbol,Base,Quote,Active,Is pair" in captured.out)
+    assert ("blkbtc,BLK/BTC,BLK,BTC,True,True" in captured.out)
+    assert ("BTTBTC,BTT/BTC,BTT,BTC,True,True" in captured.out)
+
+    # Test --one-column
+    args = [
+        '--config', 'config.json.example',
+        "list-markets",
+        "--one-column"
+    ]
+    start_list_markets(get_args(args), False)
+    captured = capsys.readouterr()
+    assert re.search(r"^BLK/BTC$", captured.out, re.MULTILINE)
+    assert re.search(r"^BTT/BTC$", captured.out, re.MULTILINE)
+
+
 def test_create_datadir_failed(caplog):
 
     args = [