mirror of
https://github.com/c9s/bbgo.git
synced 2024-11-22 06:53:52 +00:00
Merge pull request #407 from c9s/c9s/update-helm-chart
feature: update helm chart for metrics and webserver option
This commit is contained in:
Yo-An Lin
GitHub
commit
8bcc9ca655
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -38,3 +38,5 @@ bbgo.sqlite3
|
|||
node_modules
|
||||
|
||||
otp*png
|
||||
|
||||
/.deploy
|
||||
|
|
|
|||
147
README.md
147
README.md
|
|
@ -21,7 +21,10 @@ A trading bot framework written in Go. The name bbgo comes from the BB8 bot in t
|
|||
- Many built-in strategies.
|
||||
- Multi-exchange session support: you can connect to more than 2 exchanges with different accounts or subaccounts.
|
||||
- Standard indicators, e.g., SMA, EMA, BOLL, VMA, MACD...
|
||||
- React-powered Web Dashboard
|
||||
- React-powered Web Dashboard.
|
||||
- Docker image ready.
|
||||
- Kubernetes support.
|
||||
- Helm chart ready.
|
||||
|
||||
## Screenshots
|
||||
|
||||
|
|
@ -474,88 +477,15 @@ streambook := types.NewStreamBook(symbol)
|
|||
streambook.BindStream(stream)
|
||||
```
|
||||
|
||||
## How To Add A New Exchange
|
||||
## Deployment
|
||||
|
||||
(TBD)
|
||||
|
||||
## Helm Chart
|
||||
|
||||
If you need redis:
|
||||
|
||||
```sh
|
||||
helm repo add bitnami https://charts.bitnami.com/bitnami
|
||||
helm install redis bitnami/redis
|
||||
```
|
||||
|
||||
To get the dynamically generated redis password, you can use the following command:
|
||||
|
||||
```sh
|
||||
export REDIS_PASSWORD=$(kubectl get secret --namespace bbgo redis -o jsonpath="{.data.redis-password}" | base64 --decode)
|
||||
```
|
||||
|
||||
Prepare your docker image locally (you can also use the docker image from docker hub):
|
||||
|
||||
```sh
|
||||
make docker DOCKER_TAG=1.16.0
|
||||
```
|
||||
|
||||
The docker tag version number is from the file [Chart.yaml](charts/bbgo/Chart.yaml)
|
||||
|
||||
Choose your instance name:
|
||||
|
||||
```sh
|
||||
export INSTANCE=grid
|
||||
```
|
||||
|
||||
Prepare your secret:
|
||||
|
||||
```sh
|
||||
kubectl create secret generic bbgo-$INSTANCE --from-env-file .env.local
|
||||
```
|
||||
|
||||
Configure your config file, the chart defaults to read config/bbgo.yaml to create a configmap:
|
||||
|
||||
```sh
|
||||
cp config/grid.yaml bbgo-$INSTANCE.yaml
|
||||
vim bbgo-$INSTANCE.yaml
|
||||
```
|
||||
|
||||
Prepare your configmap:
|
||||
|
||||
```sh
|
||||
kubectl create configmap bbgo-$INSTANCE --from-file=bbgo.yaml=bbgo-$INSTANCE.yaml
|
||||
```
|
||||
|
||||
Install chart with the preferred release name, the release name maps to the previous secret we just created, that
|
||||
is, `bbgo-grid`:
|
||||
|
||||
```sh
|
||||
helm install --set existingConfigmap=bbgo-$INSTANCE bbgo-$INSTANCE ./charts/bbgo
|
||||
```
|
||||
|
||||
To use the latest version:
|
||||
|
||||
```sh
|
||||
helm install --set existingConfigmap=bbgo-$INSTANCE --set image.tag=latest bbgo-$INSTANCE ./charts/bbgo
|
||||
```
|
||||
|
||||
To upgrade:
|
||||
|
||||
```sh
|
||||
helm upgrade bbgo-$INSTANCE ./charts/bbgo
|
||||
helm upgrade --set image.tag=1.15.2 bbgo-$INSTANCE ./charts/bbgo
|
||||
```
|
||||
|
||||
Delete chart:
|
||||
|
||||
```sh
|
||||
helm delete bbgo-$INSTANCE
|
||||
```
|
||||
- [Helm Chart](./doc/deployment/helm-chart.md)
|
||||
- Baremetal machine or a VPS
|
||||
|
||||
## Development
|
||||
|
||||
The overview function flow at bbgo
|
||||
|
||||
- [Adding New Exchange](./doc/development/adding-new-exchange.md)
|
||||
- [Migration](./doc/development/migration.md)
|
||||
|
||||
### Setting up your local repository
|
||||
|
||||
|
|
@ -567,65 +497,6 @@ The overview function flow at bbgo
|
|||
6. Push your changes to your fork.
|
||||
7. Send a pull request.
|
||||
|
||||
### Adding new migration
|
||||
|
||||
1. The project used rockerhopper for db migration.
|
||||
https://github.com/c9s/rockhopper
|
||||
|
||||
|
||||
2. Create migration files
|
||||
|
||||
```sh
|
||||
rockhopper --config rockhopper_sqlite.yaml create --type sql add_pnl_column
|
||||
rockhopper --config rockhopper_mysql.yaml create --type sql add_pnl_column
|
||||
```
|
||||
|
||||
or you can use the util script:
|
||||
|
||||
```
|
||||
bash utils/generate-new-migration.sh add_pnl_column
|
||||
```
|
||||
|
||||
Be sure to edit both sqlite3 and mysql migration files. ( [Sample](migrations/mysql/20210531234123_add_kline_taker_buy_columns.sql) )
|
||||
|
||||
|
||||
To test the drivers, you have to update the rockhopper_mysql.yaml file to connect your database,
|
||||
then do:
|
||||
|
||||
```sh
|
||||
rockhopper --config rockhopper_sqlite.yaml up
|
||||
rockhopper --config rockhopper_mysql.yaml up
|
||||
```
|
||||
|
||||
Then run the following command to compile the migration files into go files:
|
||||
|
||||
```shell
|
||||
make migrations
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```shell
|
||||
rockhopper compile --config rockhopper_mysql.yaml --output pkg/migrations/mysql
|
||||
rockhopper compile --config rockhopper_sqlite.yaml --output pkg/migrations/sqlite3
|
||||
git add -v pkg/migrations && git commit -m "compile and update migration package" pkg/migrations || true
|
||||
```
|
||||
|
||||
|
||||
If you want to override the DSN and the Driver defined in the YAML config file, you can add some env vars in your dotenv file like this:
|
||||
|
||||
```shell
|
||||
ROCKHOPPER_DRIVER=mysql
|
||||
ROCKHOPPER_DIALECT=mysql
|
||||
ROCKHOPPER_DSN="root:123123@unix(/opt/local/var/run/mysql57/mysqld.sock)/bbgo"
|
||||
```
|
||||
|
||||
And then, run:
|
||||
|
||||
```shell
|
||||
dotenv -f .env.local -- rockhopper --config rockhopper_mysql.yaml up
|
||||
```
|
||||
|
||||
### Setup frontend development environment
|
||||
|
||||
```sh
|
||||
|
|
|
|||
|
|
@ -15,9 +15,9 @@ type: application
|
|||
# This is the chart version. This version number should be incremented each time you make changes
|
||||
# to the chart and its templates, including the app version.
|
||||
# Versions are expected to follow Semantic Versioning (https://semver.org/)
|
||||
version: 0.2.0
|
||||
version: 0.3.0
|
||||
|
||||
# This is the version number of the application being deployed. This version number should be
|
||||
# incremented each time you make changes to the application. Versions are not expected to
|
||||
# follow Semantic Versioning. They should reflect the version the application is using.
|
||||
appVersion: 1.15.0
|
||||
appVersion: 1.21.0
|
||||
|
|
|
|||
|
|
@ -33,14 +33,30 @@ spec:
|
|||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
# ports:
|
||||
# - name: http
|
||||
# containerPort: 80
|
||||
# protocol: TCP
|
||||
# livenessProbe:
|
||||
# httpGet:
|
||||
# path: /
|
||||
# port: http
|
||||
args:
|
||||
{{- if .Values.metrics.enabled }}
|
||||
- "--metrics"
|
||||
- "--metrics-port"
|
||||
- {{ .Values.metrics.port | quote }}
|
||||
{{- end }}
|
||||
- "run"
|
||||
- "--config"
|
||||
- "/config/bbgo.yaml"
|
||||
- "--no-compile"
|
||||
{{- if .Values.webserver.enabled }}
|
||||
- "--enable-webserver"
|
||||
{{- end }}
|
||||
|
||||
ports:
|
||||
{{- if .Values.metrics.enabled }}
|
||||
- name: metrics
|
||||
containerPort: 9090
|
||||
protocol: TCP
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /metrics
|
||||
port: metrics
|
||||
{{- end }}
|
||||
# readinessProbe:
|
||||
# httpGet:
|
||||
# path: /
|
||||
|
|
|
|||
15
charts/bbgo/templates/podmonitor.yaml
Normal file
15
charts/bbgo/templates/podmonitor.yaml
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{{- if .Values.metrics.enabled }}
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: {{ include "bbgo.fullname" . }}
|
||||
labels:
|
||||
{{- include "bbgo.labels" . | nindent 4 }}
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "bbgo.selectorLabels" . | nindent 6 }}
|
||||
podMetricsEndpoints:
|
||||
- port: metrics
|
||||
{{- end }}
|
||||
|
|
@ -16,12 +16,15 @@ nameOverride: ""
|
|||
fullnameOverride: ""
|
||||
|
||||
# existingConfigMap is used for loading the existing configmap
|
||||
# optional, if you have a configmap rather than the default configmap name
|
||||
existingConfigMap:
|
||||
|
||||
# configmap could be a string pointing to the local file in the chart
|
||||
# or a object with inlined bbgo yaml config
|
||||
configmap:
|
||||
|
||||
# dotenv defines environment variables, which are loaded from a k8s secret.
|
||||
# if not defined, then release name will be used to load the secret.
|
||||
dotenv:
|
||||
secret: null
|
||||
|
||||
|
|
@ -64,6 +67,13 @@ ingress:
|
|||
# hosts:
|
||||
# - chart-example.local
|
||||
|
||||
webserver:
|
||||
enabled: false
|
||||
|
||||
metrics:
|
||||
enabled: false
|
||||
port: 9090
|
||||
|
||||
resources:
|
||||
# We usually recommend not to specify default resources and to leave this as a conscious
|
||||
# choice for the user. This also increases chances charts run on environments with little
|
||||
|
|
|
|||
88
doc/deployment/helm-chart.md
Normal file
88
doc/deployment/helm-chart.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# Helm Chart
|
||||
|
||||
## Requirement
|
||||
|
||||
- redis (optional, if you need persistence)
|
||||
- docker image (you can use the image from docker hub or build one by yourself)
|
||||
|
||||
## Install
|
||||
|
||||
If you need redis:
|
||||
|
||||
```sh
|
||||
helm repo add bitnami https://charts.bitnami.com/bitnami
|
||||
helm install redis bitnami/redis
|
||||
```
|
||||
|
||||
To get the dynamically generated redis password, you can use the following command:
|
||||
|
||||
```sh
|
||||
export REDIS_PASSWORD=$(kubectl get secret --namespace bbgo redis -o jsonpath="{.data.redis-password}" | base64 --decode)
|
||||
```
|
||||
|
||||
Prepare your docker image locally (you can also use the docker image from docker hub):
|
||||
|
||||
```sh
|
||||
make docker DOCKER_TAG=1.16.0
|
||||
```
|
||||
|
||||
The docker tag version number is from the file [Chart.yaml](charts/bbgo/Chart.yaml)
|
||||
|
||||
Choose your instance name:
|
||||
|
||||
```sh
|
||||
export INSTANCE=grid
|
||||
```
|
||||
|
||||
Prepare your secret:
|
||||
|
||||
```sh
|
||||
kubectl create secret generic bbgo-$INSTANCE --from-env-file .env.local
|
||||
```
|
||||
|
||||
Configure your config file, the chart defaults to read config/bbgo.yaml to create a configmap:
|
||||
|
||||
```sh
|
||||
cp config/grid.yaml bbgo-$INSTANCE.yaml
|
||||
vim bbgo-$INSTANCE.yaml
|
||||
```
|
||||
|
||||
Prepare your configmap:
|
||||
|
||||
```sh
|
||||
kubectl create configmap bbgo-$INSTANCE --from-file=bbgo.yaml=bbgo-$INSTANCE.yaml
|
||||
```
|
||||
|
||||
Install chart with the preferred release name, the release name maps to the previous secret we just created, that
|
||||
is, `bbgo-grid`:
|
||||
|
||||
```sh
|
||||
helm install bbgo-$INSTANCE ./charts/bbgo
|
||||
```
|
||||
|
||||
By default, the helm chart uses configmap and dotenv secret by the release name,
|
||||
if you have an existing configmap that is not named `bbgo-$INSTANCE`, you can specify the configmap via
|
||||
the `existingConfigmap` option:
|
||||
|
||||
```sh
|
||||
helm install --set existingConfigmap=bbgo-$INSTANCE bbgo-$INSTANCE ./charts/bbgo
|
||||
```
|
||||
|
||||
To use the latest version:
|
||||
|
||||
```sh
|
||||
helm install --set existingConfigmap=bbgo-$INSTANCE --set image.tag=latest bbgo-$INSTANCE ./charts/bbgo
|
||||
```
|
||||
|
||||
To upgrade:
|
||||
|
||||
```sh
|
||||
helm upgrade bbgo-$INSTANCE ./charts/bbgo
|
||||
helm upgrade --set image.tag=1.15.2 bbgo-$INSTANCE ./charts/bbgo
|
||||
```
|
||||
|
||||
## Delete an installed chart
|
||||
|
||||
```sh
|
||||
helm delete bbgo-$INSTANCE
|
||||
```
|
||||
|
|
@ -8,14 +8,23 @@ You should send multiple small pull request to implement them.
|
|||
|
||||
## Checklist
|
||||
|
||||
Exchange Interface (minimum)
|
||||
Exchange Interface - minimum requirement for trading
|
||||
|
||||
- [ ] QueryMarkets
|
||||
- [ ] QueryKLines
|
||||
- [ ] QueryTickers
|
||||
- [ ] QueryOrders
|
||||
- [ ] QueryTrades
|
||||
- [ ] QueryOpenOrders
|
||||
- [ ] SubmitOrders
|
||||
- [ ] CancelOrders
|
||||
- [ ] NewStream
|
||||
|
||||
Trading History Service Interface - used for syncing user trading data
|
||||
|
||||
- [ ] QueryClosedOrders
|
||||
- [ ] QueryTrades
|
||||
|
||||
Back-testing service - kline data is used for back-testing
|
||||
|
||||
- [ ] QueryKLines
|
||||
|
||||
Convert functions:
|
||||
|
||||
60
doc/development/migration.md
Normal file
60
doc/development/migration.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# Migration
|
||||
|
||||
### Adding new migration
|
||||
|
||||
1. The project used rockerhopper for db migration.
|
||||
https://github.com/c9s/rockhopper
|
||||
|
||||
|
||||
2. Create migration files
|
||||
|
||||
```sh
|
||||
rockhopper --config rockhopper_sqlite.yaml create --type sql add_pnl_column
|
||||
rockhopper --config rockhopper_mysql.yaml create --type sql add_pnl_column
|
||||
```
|
||||
|
||||
or you can use the util script:
|
||||
|
||||
```
|
||||
bash utils/generate-new-migration.sh add_pnl_column
|
||||
```
|
||||
|
||||
Be sure to edit both sqlite3 and mysql migration files. ( [Sample](migrations/mysql/20210531234123_add_kline_taker_buy_columns.sql) )
|
||||
|
||||
|
||||
To test the drivers, you have to update the rockhopper_mysql.yaml file to connect your database,
|
||||
then do:
|
||||
|
||||
```sh
|
||||
rockhopper --config rockhopper_sqlite.yaml up
|
||||
rockhopper --config rockhopper_mysql.yaml up
|
||||
```
|
||||
|
||||
Then run the following command to compile the migration files into go files:
|
||||
|
||||
```shell
|
||||
make migrations
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```shell
|
||||
rockhopper compile --config rockhopper_mysql.yaml --output pkg/migrations/mysql
|
||||
rockhopper compile --config rockhopper_sqlite.yaml --output pkg/migrations/sqlite3
|
||||
git add -v pkg/migrations && git commit -m "compile and update migration package" pkg/migrations || true
|
||||
```
|
||||
|
||||
|
||||
If you want to override the DSN and the Driver defined in the YAML config file, you can add some env vars in your dotenv file like this:
|
||||
|
||||
```shell
|
||||
ROCKHOPPER_DRIVER=mysql
|
||||
ROCKHOPPER_DIALECT=mysql
|
||||
ROCKHOPPER_DSN="root:123123@unix(/opt/local/var/run/mysql57/mysqld.sock)/bbgo"
|
||||
```
|
||||
|
||||
And then, run:
|
||||
|
||||
```shell
|
||||
dotenv -f .env.local -- rockhopper --config rockhopper_mysql.yaml up
|
||||
```
|
||||
|
|
@ -810,7 +810,7 @@ func (session *ExchangeSession) metricsTradeUpdater(trade types.Trade) {
|
|||
"symbol": trade.Symbol,
|
||||
"liquidity": trade.Liquidity(),
|
||||
}
|
||||
metricsTradingVolume.With(labels).Add(trade.Quantity)
|
||||
metricsTradingVolume.With(labels).Add(trade.Quantity * trade.Price)
|
||||
metricsTradesTotal.With(labels).Inc()
|
||||
}
|
||||
|
||||
|
|
@ -818,4 +818,18 @@ func (session *ExchangeSession) bindUserDataStreamMetrics(stream types.Stream) {
|
|||
stream.OnBalanceUpdate(session.metricsBalancesUpdater)
|
||||
stream.OnBalanceSnapshot(session.metricsBalancesUpdater)
|
||||
stream.OnTradeUpdate(session.metricsTradeUpdater)
|
||||
stream.OnDisconnect(func() {
|
||||
metricsConnectionStatus.With(prometheus.Labels{
|
||||
"exchange": session.ExchangeName.String(),
|
||||
"margin": session.MarginType(),
|
||||
"symbol": session.IsolatedMarginSymbol,
|
||||
}).Set(0.0)
|
||||
})
|
||||
stream.OnConnect(func() {
|
||||
metricsConnectionStatus.With(prometheus.Labels{
|
||||
"exchange": session.ExchangeName.String(),
|
||||
"margin": session.MarginType(),
|
||||
"symbol": session.IsolatedMarginSymbol,
|
||||
}).Set(1.0)
|
||||
})
|
||||
}
|
||||
|
|
|
|||
58
utils/changelog.sh
Normal file
58
utils/changelog.sh
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
#!/bin/bash
|
||||
# Generate a Markdown change log of pull requests from commits between two tags
|
||||
# Author: Russell Heimlich
|
||||
# URL: https://gist.github.com/kingkool68/09a201a35c83e43af08fcbacee5c315a
|
||||
|
||||
# HOW TO USE
|
||||
# Copy this script to a directory under Git version control
|
||||
# Make the script executable i.e. chmod +x changelog.sh
|
||||
# Run it! ./changelog.sh
|
||||
# Check CHANGELOG.md to see your results
|
||||
|
||||
# Repo URL to base links off of
|
||||
REPOSITORY_URL=https://github.com/c9s/bbgo
|
||||
|
||||
# Get a list of all tags in reverse order
|
||||
# Assumes the tags are in version format like v1.2.3
|
||||
GIT_TAGS=$(git tag -l --sort=-version:refname)
|
||||
|
||||
# Make the tags an array
|
||||
TAGS=($GIT_TAGS)
|
||||
LATEST_TAG=${TAGS[0]}
|
||||
PREVIOUS_TAG=${TAGS[1]}
|
||||
|
||||
# If you want to specify your own two tags to compare, uncomment and enter them below
|
||||
# LATEST_TAG=v0.23.1
|
||||
# PREVIOUS_TAG=v0.22.0
|
||||
|
||||
# Get a log of commits that occured between two tags
|
||||
# We only get the commit hash so we don't have to deal with a bunch of ugly parsing
|
||||
# See Pretty format placeholders at https://git-scm.com/docs/pretty-formats
|
||||
COMMITS=$(git log $PREVIOUS_TAG..$LATEST_TAG --pretty=format:"%H")
|
||||
|
||||
# Store our changelog in a variable to be saved to a file at the end
|
||||
MARKDOWN="[Full Changelog]($REPOSITORY_URL/compare/$PREVIOUS_TAG...$LATEST_TAG)"
|
||||
MARKDOWN+='\n'
|
||||
|
||||
# Loop over each commit and look for merged pull requests
|
||||
for COMMIT in $COMMITS; do
|
||||
# Get the subject of the current commit
|
||||
SUBJECT=$(git log -1 ${COMMIT} --pretty=format:"%s")
|
||||
|
||||
# If the subject contains "Merge pull request #xxxxx" then it is deemed a pull request
|
||||
PULL_REQUEST=$( grep -Eo "Merge pull request #[[:digit:]]+" <<< "$SUBJECT" )
|
||||
if [[ $PULL_REQUEST ]]; then
|
||||
# Perform a substring operation so we're left with just the digits of the pull request
|
||||
PULL_NUM=${PULL_REQUEST#"Merge pull request #"}
|
||||
# AUTHOR_NAME=$(git log -1 ${COMMIT} --pretty=format:"%an")
|
||||
# AUTHOR_EMAIL=$(git log -1 ${COMMIT} --pretty=format:"%ae")
|
||||
|
||||
# Get the body of the commit
|
||||
BODY=$(git log -1 ${COMMIT} --pretty=format:"%b")
|
||||
MARKDOWN+='\n'
|
||||
MARKDOWN+=" - [#$PULL_NUM]($REPOSITORY_URL/pull/$PULL_NUM): $BODY"
|
||||
fi
|
||||
done
|
||||
|
||||
# Save our markdown to a file
|
||||
echo -e $MARKDOWN
|
||||
Loading…
Reference in New Issue
Block a user