Merge pull request #4285 from freqtrade/ui_deploy

Deploy FreqUI into webserver
This commit is contained in:
Matthias 2021-02-03 20:09:31 +01:00 committed by GitHub
commit 024849d844
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 355 additions and 44 deletions

1
.gitignore vendored
View File

@ -8,6 +8,7 @@ user_data/*
user_data/notebooks/*
freqtrade-plot.html
freqtrade-profit-plot.html
freqtrade/rpc/api_server/ui/*
# Byte-compiled / optimized / DLL files
__pycache__/

View File

@ -40,7 +40,9 @@ COPY --from=python-deps /root/.local /root/.local
# Install and execute
COPY . /freqtrade/
RUN pip install -e . --no-cache-dir \
&& mkdir /freqtrade/user_data/
&& mkdir /freqtrade/user_data/ \
&& freqtrade install-ui
ENTRYPOINT ["freqtrade"]
# Default to trade mode
CMD [ "trade" ]

View File

@ -41,7 +41,9 @@ COPY --from=python-deps /root/.local /root/.local
# Install and execute
COPY . /freqtrade/
RUN pip install -e . --no-cache-dir
RUN pip install -e . --no-cache-dir \
&& freqtrade install-ui
ENTRYPOINT ["freqtrade"]
# Default to trade mode
CMD [ "trade" ]

View File

@ -2,3 +2,5 @@ include LICENSE
include README.md
recursive-include freqtrade *.py
recursive-include freqtrade/templates/ *.j2 *.ipynb
include freqtrade/rpc/api_server/ui/fallback_file.html
include freqtrade/rpc/api_server/ui/favicon.ico

View File

@ -14,6 +14,11 @@ services:
container_name: freqtrade
volumes:
- "./user_data:/freqtrade/user_data"
# Expose api on port 8080 (localhost only)
# Please read the https://www.freqtrade.io/en/latest/rest-api/ documentation
# before enabling this.
# ports:
# - "127.0.0.1:8080:8080"
# Default command used when running `docker compose up`
command: >
trade

View File

@ -1,4 +1,19 @@
# REST API Usage
# REST API & FreqUI
## FreqUI
Freqtrade provides a builtin webserver, which can serve [FreqUI](https://github.com/freqtrade/frequi), the freqtrade UI.
By default, the UI is not included in the installation (except for docker images), and must be installed explicitly with `freqtrade install-ui`.
This same command can also be used to update freqUI, should there be a new release.
Once the bot is started in trade / dry-run mode (with `freqtrade trade`) - the UI will be available under the configured port below (usually `http://127.0.0.1:8080`).
!!! info "Alpha release"
FreqUI is still considered an alpha release - if you encounter bugs or inconsistencies please open a [FreqUI issue](https://github.com/freqtrade/frequi/issues/new/choose).
!!! Note "developers"
Developers should not use this method, but instead use the method described in the [freqUI repository](https://github.com/freqtrade/frequi) to get the source-code of freqUI.
## Configuration
@ -23,9 +38,6 @@ Sample configuration:
!!! Danger "Security warning"
By default, the configuration listens on localhost only (so it's not reachable from other systems). We strongly recommend to not expose this API to the internet and choose a strong, unique password, since others will potentially be able to control your bot.
!!! Danger "Password selection"
Please make sure to select a very strong, unique password to protect your bot from unauthorized access.
You can then access the API by going to `http://127.0.0.1:8080/api/v1/ping` in a browser to check if the API is running correctly.
This should return the response:
@ -35,16 +47,22 @@ This should return the response:
All other endpoints return sensitive info and require authentication and are therefore not available through a web browser.
To generate a secure password, either use a password manager, or use the below code snipped.
### Security
To generate a secure password, best use a password manager, or use the below code.
``` python
import secrets
secrets.token_hex()
```
!!! Hint
!!! Hint "JWT token"
Use the same method to also generate a JWT secret key (`jwt_secret_key`).
!!! Danger "Password selection"
Please make sure to select a very strong, unique password to protect your bot from unauthorized access.
Also change `jwt_secret_key` to something random (no need to remember this, but it'll be used to encrypt your session, so it better be something unique!).
### Configuration with docker
If you run your bot using docker, you'll need to have the bot listen to incoming connections. The security is then handled by docker.
@ -57,28 +75,20 @@ If you run your bot using docker, you'll need to have the bot listen to incoming
},
```
Add the following to your docker command:
Uncomment the following from your docker-compose file:
``` bash
-p 127.0.0.1:8080:8080
```
A complete sample-command may then look as follows:
```bash
docker run -d \
--name freqtrade \
-v ~/.freqtrade/config.json:/freqtrade/config.json \
-v ~/.freqtrade/user_data/:/freqtrade/user_data \
-v ~/.freqtrade/tradesv3.sqlite:/freqtrade/tradesv3.sqlite \
-p 127.0.0.1:8080:8080 \
freqtrade trade --db-url sqlite:///tradesv3.sqlite --strategy MyAwesomeStrategy
```yml
ports:
- "127.0.0.1:8080:8080"
```
!!! Danger "Security warning"
By using `-p 8080:8080` the API is available to everyone connecting to the server under the correct port, so others may be able to control your bot.
By using `8080:8080` in the docker port mapping, the API will be available to everyone connecting to the server under the correct port, so others may be able to control your bot.
## Consuming the API
## Rest API
### Consuming the API
You can consume the API by using the script `scripts/rest_client.py`.
The client script only requires the `requests` module, so Freqtrade does not need to be installed on the system.
@ -89,7 +99,7 @@ python3 scripts/rest_client.py <command> [optional parameters]
By default, the script assumes `127.0.0.1` (localhost) and port `8080` to be used, however you can specify a configuration file to override this behaviour.
### Minimalistic client config
#### Minimalistic client config
``` json
{
@ -105,7 +115,7 @@ By default, the script assumes `127.0.0.1` (localhost) and port `8080` to be use
python3 scripts/rest_client.py --config rest_config.json <command> [optional parameters]
```
## Available endpoints
### Available endpoints
| Command | Description |
|----------|-------------|
@ -264,12 +274,12 @@ whitelist
```
## OpenAPI interface
### OpenAPI interface
To enable the builtin openAPI interface (Swagger UI), specify `"enable_openapi": true` in the api_server configuration.
This will enable the Swagger UI at the `/docs` endpoint. By default, that's running at http://localhost:8080/docs/ - but it'll depend on your settings.
## Advanced API usage using JWT tokens
### Advanced API usage using JWT tokens
!!! Note
The below should be done in an application (a Freqtrade REST API client, which fetches info via API), and is not intended to be used on a regular basis.
@ -294,9 +304,9 @@ Since the access token has a short timeout (15 min) - the `token/refresh` reques
{"access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1ODkxMTk5NzQsIm5iZiI6MTU4OTExOTk3NCwianRpIjoiMDBjNTlhMWUtMjBmYS00ZTk0LTliZjAtNWQwNTg2MTdiZDIyIiwiZXhwIjoxNTg5MTIwODc0LCJpZGVudGl0eSI6eyJ1IjoiRnJlcXRyYWRlciJ9LCJmcmVzaCI6ZmFsc2UsInR5cGUiOiJhY2Nlc3MifQ.1seHlII3WprjjclY6DpRhen0rqdF4j6jbvxIhUFaSbs"}
```
## CORS
### CORS
All web-based frontends are subject to [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) - Cross-Origin Resource Sharing.
All web-based front-ends are subject to [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) - Cross-Origin Resource Sharing.
Since most of the requests to the Freqtrade API must be authenticated, a proper CORS policy is key to avoid security problems.
Also, the standard disallows `*` CORS policies for requests with credentials, so this setting must be set appropriately.

View File

@ -10,8 +10,8 @@ from freqtrade.commands.arguments import Arguments
from freqtrade.commands.build_config_commands import start_new_config
from freqtrade.commands.data_commands import (start_convert_data, start_download_data,
start_list_data)
from freqtrade.commands.deploy_commands import (start_create_userdir, start_new_hyperopt,
start_new_strategy)
from freqtrade.commands.deploy_commands import (start_create_userdir, start_install_ui,
start_new_hyperopt, start_new_strategy)
from freqtrade.commands.hyperopt_commands import start_hyperopt_list, start_hyperopt_show
from freqtrade.commands.list_commands import (start_list_exchanges, start_list_hyperopts,
start_list_markets, start_list_strategies,

View File

@ -70,6 +70,8 @@ ARGS_PLOT_DATAFRAME = ["pairs", "indicators1", "indicators2", "plot_limit",
ARGS_PLOT_PROFIT = ["pairs", "timerange", "export", "exportfilename", "db_url",
"trade_source", "timeframe"]
ARGS_INSTALL_UI = ["erase_ui_only"]
ARGS_SHOW_TRADES = ["db_url", "trade_ids", "print_json"]
ARGS_HYPEROPT_LIST = ["hyperopt_list_best", "hyperopt_list_profitable",
@ -167,8 +169,8 @@ class Arguments:
from freqtrade.commands import (start_backtesting, start_convert_data, start_create_userdir,
start_download_data, start_edge, start_hyperopt,
start_hyperopt_list, start_hyperopt_show, start_list_data,
start_list_exchanges, start_list_hyperopts,
start_hyperopt_list, start_hyperopt_show, start_install_ui,
start_list_data, start_list_exchanges, start_list_hyperopts,
start_list_markets, start_list_strategies,
start_list_timeframes, start_new_config, start_new_hyperopt,
start_new_strategy, start_plot_dataframe, start_plot_profit,
@ -355,6 +357,14 @@ class Arguments:
test_pairlist_cmd.set_defaults(func=start_test_pairlist)
self._build_args(optionlist=ARGS_TEST_PAIRLIST, parser=test_pairlist_cmd)
# Add install-ui subcommand
install_ui_cmd = subparsers.add_parser(
'install-ui',
help='Install FreqUI',
)
install_ui_cmd.set_defaults(func=start_install_ui)
self._build_args(optionlist=ARGS_INSTALL_UI, parser=install_ui_cmd)
# Add Plotting subcommand
plot_dataframe_cmd = subparsers.add_parser(
'plot-dataframe',

View File

@ -387,6 +387,12 @@ AVAILABLE_CLI_OPTIONS = {
help='Clean all existing data for the selected exchange/pairs/timeframes.',
action='store_true',
),
"erase_ui_only": Arg(
'--erase',
help="Clean UI folder, don't download new version.",
action='store_true',
default=False,
),
# Templating options
"template": Arg(
'--template',

View File

@ -1,7 +1,9 @@
import logging
import sys
from pathlib import Path
from typing import Any, Dict
from typing import Any, Dict, Optional, Tuple
import requests
from freqtrade.configuration import setup_utils_configuration
from freqtrade.configuration.directory_operations import copy_sample_files, create_userdata_dir
@ -137,3 +139,87 @@ def start_new_hyperopt(args: Dict[str, Any]) -> None:
deploy_new_hyperopt(args['hyperopt'], new_path, args['template'])
else:
raise OperationalException("`new-hyperopt` requires --hyperopt to be set.")
def clean_ui_subdir(directory: Path):
if directory.is_dir():
logger.info("Removing UI directory content.")
for p in reversed(list(directory.glob('**/*'))): # iterate contents from leaves to root
if p.name in ('.gitkeep', 'fallback_file.html'):
continue
if p.is_file():
p.unlink()
elif p.is_dir():
p.rmdir()
def read_ui_version(dest_folder: Path) -> Optional[str]:
file = dest_folder / '.uiversion'
if not file.is_file():
return None
with file.open('r') as f:
return f.read()
def download_and_install_ui(dest_folder: Path, dl_url: str, version: str):
from io import BytesIO
from zipfile import ZipFile
logger.info(f"Downloading {dl_url}")
resp = requests.get(dl_url).content
dest_folder.mkdir(parents=True, exist_ok=True)
with ZipFile(BytesIO(resp)) as zf:
for fn in zf.filelist:
with zf.open(fn) as x:
destfile = dest_folder / fn.filename
if fn.is_dir():
destfile.mkdir(exist_ok=True)
else:
destfile.write_bytes(x.read())
with (dest_folder / '.uiversion').open('w') as f:
f.write(version)
def get_ui_download_url() -> Tuple[str, str]:
base_url = 'https://api.github.com/repos/freqtrade/frequi/'
# Get base UI Repo path
resp = requests.get(f"{base_url}releases")
resp.raise_for_status()
r = resp.json()
latest_version = r[0]['name']
assets = r[0].get('assets', [])
dl_url = ''
if assets and len(assets) > 0:
dl_url = assets[0]['browser_download_url']
# URL not found - try assets url
if not dl_url:
assets = r[0]['assets_url']
resp = requests.get(assets)
r = resp.json()
dl_url = r[0]['browser_download_url']
return dl_url, latest_version
def start_install_ui(args: Dict[str, Any]) -> None:
dest_folder = Path(__file__).parents[1] / 'rpc/api_server/ui/installed/'
# First make sure the assets are removed.
dl_url, latest_version = get_ui_download_url()
curr_version = read_ui_version(dest_folder)
if curr_version == latest_version and not args.get('erase_ui_only'):
logger.info(f"UI already up-to-date, FreqUI Version {curr_version}.")
return
clean_ui_subdir(dest_folder)
if args.get('erase_ui_only'):
logger.info("Erased UI directory content. Not downloading new version.")
else:
# Download a new version
download_and_install_ui(dest_folder, dl_url, latest_version)

View File

@ -0,0 +1,31 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Freqtrade UI</title>
<style>
body {
background-color: #3c3c3c;
color: #dedede;
text-align: center;
}
.main-container {
display: flex;
align-items: center;
justify-content: center;
flex-direction: column;
}
h1 {
margin-top: 10rem;
}
</style>
</head>
<body>
<div class="main-container">
<h1>Freqtrade UI not installed.</h1>
<p>Please run `freqtrade install-ui` in your terminal to install the UI files and restart your bot.</p>
<p>You can then refresh this page and you should see the UI.</p>
</div>
</body>
</html>

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

View File

@ -0,0 +1,31 @@
from pathlib import Path
from fastapi import APIRouter
from fastapi.exceptions import HTTPException
from starlette.responses import FileResponse
router_ui = APIRouter()
@router_ui.get('/favicon.ico', include_in_schema=False)
async def favicon():
return FileResponse(Path(__file__).parent / 'ui/favicon.ico')
@router_ui.get('/{rest_of_path:path}', include_in_schema=False)
async def index_html(rest_of_path: str):
"""
Emulate path fallback to index.html.
"""
if rest_of_path.startswith('api') or rest_of_path.startswith('.'):
raise HTTPException(status_code=404, detail="Not Found")
uibase = Path(__file__).parent / 'ui/installed/'
if (uibase / rest_of_path).is_file():
return FileResponse(str(uibase / rest_of_path))
index_file = uibase / 'index.html'
if not index_file.is_file():
return FileResponse(str(uibase.parent / 'fallback_file.html'))
# Fall back to index.html, as indicated by vue router docs
return FileResponse(str(index_file))

View File

@ -57,12 +57,16 @@ class ApiServer(RPCHandler):
from freqtrade.rpc.api_server.api_auth import http_basic_or_jwt_token, router_login
from freqtrade.rpc.api_server.api_v1 import router as api_v1
from freqtrade.rpc.api_server.api_v1 import router_public as api_v1_public
from freqtrade.rpc.api_server.web_ui import router_ui
app.include_router(api_v1_public, prefix="/api/v1")
app.include_router(api_v1, prefix="/api/v1",
dependencies=[Depends(http_basic_or_jwt_token)],
)
app.include_router(router_login, prefix="/api/v1", tags=["auth"])
# UI Router MUST be last!
app.include_router(router_ui, prefix='')
app.add_middleware(
CORSMiddleware,

View File

@ -14,7 +14,7 @@ nav:
- Control the bot:
- Telegram: telegram-usage.md
- Web Hook: webhook-config.md
- REST API: rest-api.md
- REST API & FreqUI: rest-api.md
- Data Downloading: data-download.md
- Backtesting: backtesting.md
- Hyperopt: hyperopt.md

View File

@ -31,6 +31,7 @@ sdnotify==0.3.2
fastapi==0.63.0
uvicorn==0.13.3
pyjwt==2.0.1
aiofiles==0.6.0
# Support for colorized terminal output
colorama==0.4.4

View File

@ -1,16 +1,20 @@
import re
from io import BytesIO
from pathlib import Path
from unittest.mock import MagicMock, PropertyMock
from zipfile import ZipFile
import arrow
import pytest
from freqtrade.commands import (start_convert_data, start_create_userdir, start_download_data,
start_hyperopt_list, start_hyperopt_show, start_list_data,
start_list_exchanges, start_list_hyperopts, start_list_markets,
start_list_strategies, start_list_timeframes, start_new_hyperopt,
start_new_strategy, start_show_trades, start_test_pairlist,
start_trading)
start_hyperopt_list, start_hyperopt_show, start_install_ui,
start_list_data, start_list_exchanges, start_list_hyperopts,
start_list_markets, start_list_strategies, start_list_timeframes,
start_new_hyperopt, start_new_strategy, start_show_trades,
start_test_pairlist, start_trading)
from freqtrade.commands.deploy_commands import (clean_ui_subdir, download_and_install_ui,
get_ui_download_url, read_ui_version)
from freqtrade.configuration import setup_utils_configuration
from freqtrade.exceptions import OperationalException
from freqtrade.state import RunMode
@ -546,7 +550,7 @@ def test_start_new_hyperopt_DefaultHyperopt(mocker, caplog):
start_new_hyperopt(get_args(args))
def test_start_new_hyperopt_no_arg(mocker, caplog):
def test_start_new_hyperopt_no_arg(mocker):
args = [
"new-hyperopt",
]
@ -555,6 +559,107 @@ def test_start_new_hyperopt_no_arg(mocker, caplog):
start_new_hyperopt(get_args(args))
def test_start_install_ui(mocker):
clean_mock = mocker.patch('freqtrade.commands.deploy_commands.clean_ui_subdir')
get_url_mock = mocker.patch('freqtrade.commands.deploy_commands.get_ui_download_url',
return_value=('https://example.com/whatever', '0.0.1'))
download_mock = mocker.patch('freqtrade.commands.deploy_commands.download_and_install_ui')
mocker.patch('freqtrade.commands.deploy_commands.read_ui_version', return_value=None)
args = [
"install-ui",
]
start_install_ui(get_args(args))
assert clean_mock.call_count == 1
assert get_url_mock.call_count == 1
assert download_mock.call_count == 1
clean_mock.reset_mock()
get_url_mock.reset_mock()
download_mock.reset_mock()
args = [
"install-ui",
"--erase",
]
start_install_ui(get_args(args))
assert clean_mock.call_count == 1
assert get_url_mock.call_count == 1
assert download_mock.call_count == 0
def test_clean_ui_subdir(mocker, tmpdir, caplog):
mocker.patch("freqtrade.commands.deploy_commands.Path.is_dir",
side_effect=[True, True])
mocker.patch("freqtrade.commands.deploy_commands.Path.is_file",
side_effect=[False, True])
rd_mock = mocker.patch("freqtrade.commands.deploy_commands.Path.rmdir")
ul_mock = mocker.patch("freqtrade.commands.deploy_commands.Path.unlink")
mocker.patch("freqtrade.commands.deploy_commands.Path.glob",
return_value=[Path('test1'), Path('test2'), Path('.gitkeep')])
folder = Path(tmpdir) / "uitests"
clean_ui_subdir(folder)
assert log_has("Removing UI directory content.", caplog)
assert rd_mock.call_count == 1
assert ul_mock.call_count == 1
def test_download_and_install_ui(mocker, tmpdir):
# Create zipfile
requests_mock = MagicMock()
file_like_object = BytesIO()
with ZipFile(file_like_object, mode='w') as zipfile:
for file in ('test1.txt', 'hello/', 'test2.txt'):
zipfile.writestr(file, file)
file_like_object.seek(0)
requests_mock.content = file_like_object.read()
mocker.patch("freqtrade.commands.deploy_commands.requests.get", return_value=requests_mock)
mocker.patch("freqtrade.commands.deploy_commands.Path.is_dir",
side_effect=[True, False])
wb_mock = mocker.patch("freqtrade.commands.deploy_commands.Path.write_bytes")
folder = Path(tmpdir) / "uitests_dl"
folder.mkdir(exist_ok=True)
assert read_ui_version(folder) is None
download_and_install_ui(folder, 'http://whatever.xxx/download/file.zip', '22')
assert wb_mock.call_count == 2
assert read_ui_version(folder) == '22'
def test_get_ui_download_url(mocker):
response = MagicMock()
response.json = MagicMock(
side_effect=[[{'assets_url': 'http://whatever.json', 'name': '0.0.1'}],
[{'browser_download_url': 'http://download.zip'}]])
get_mock = mocker.patch("freqtrade.commands.deploy_commands.requests.get",
return_value=response)
x, last_version = get_ui_download_url()
assert get_mock.call_count == 2
assert last_version == '0.0.1'
assert x == 'http://download.zip'
def test_get_ui_download_url_direct(mocker):
response = MagicMock()
response.json = MagicMock(
side_effect=[[{
'assets_url': 'http://whatever.json',
'name': '0.0.1',
'assets': [{'browser_download_url': 'http://download11.zip'}]}]])
get_mock = mocker.patch("freqtrade.commands.deploy_commands.requests.get",
return_value=response)
x, last_version = get_ui_download_url()
assert get_mock.call_count == 1
assert last_version == '0.0.1'
assert x == 'http://download11.zip'
def test_download_data_keyboardInterrupt(mocker, caplog, markets):
dl_mock = mocker.patch('freqtrade.commands.data_commands.refresh_backtest_ohlcv_data',
MagicMock(side_effect=KeyboardInterrupt))

View File

@ -83,11 +83,26 @@ def assert_response(response, expected_code=200, needs_cors=True):
def test_api_not_found(botclient):
ftbot, client = botclient
rc = client_post(client, f"{BASE_URI}/invalid_url")
rc = client_get(client, f"{BASE_URI}/invalid_url")
assert_response(rc, 404)
assert rc.json() == {"detail": "Not Found"}
def test_api_ui_fallback(botclient):
ftbot, client = botclient
rc = client_get(client, "/favicon.ico")
assert rc.status_code == 200
rc = client_get(client, "/fallback_file.html")
assert rc.status_code == 200
assert '`freqtrade install-ui`' in rc.text
# Forwarded to fallback_html or index.html (depending if it's installed or not)
rc = client_get(client, "/something")
assert rc.status_code == 200
def test_api_auth():
with pytest.raises(ValueError):
create_token({'identity': {'u': 'Freqtrade'}}, 'secret1234', token_type="NotATokenType")