Upgrades and kill mysql everywhere

This commit is contained in:
Nemo 2018-07-18 18:17:57 +05:30
parent 1353fd2c61
commit 3ab14e79e5
15 changed files with 76 additions and 214 deletions

131
README.md
View File

@ -2,102 +2,113 @@
![Nebula header image](https://cdn.spacetelescope.org/archives/images/thumb700x/heic0707a.jpg) ![Nebula header image](https://cdn.spacetelescope.org/archives/images/thumb700x/heic0707a.jpg)
>Where stars are born. > Where stars are born.
Manages the local infrastructure of my home server. I'm also doing blog posts around the same: Manages the local infrastructure of my home server. I'm also doing blog posts around the same:
1. [Part 1, Hardware](https://captnemo.in/blog/2017/09/17/home-server-build/) 1. [Part 1, Hardware](https://captnemo.in/blog/2017/09/17/home-server-build/)
2. [Part 2, Terraform/Docker](https://captnemo.in/blog/2017/11/09/home-server-update/) 2. [Part 2, Terraform/Docker](https://captnemo.in/blog/2017/11/09/home-server-update/)
3. [Part 3, Learnings](https://captnemo.in/blog/2017/12/18/home-server-learnings/) 3. [Part 3, Learnings](https://captnemo.in/blog/2017/12/18/home-server-learnings/)
4. [Part 4, Migrating from Google (and more)](https://captnemo.in/blog/2017/12/31/migrating-from-google/) 4. [Part 4, Migrating from Google (and more)](https://captnemo.in/blog/2017/12/31/migrating-from-google/)
5. [Part 5, Networking](https://captnemo.in/blog/2018/04/22/home-server-networking/) 5. [Part 5, Networking](https://captnemo.in/blog/2018/04/22/home-server-networking/)
The canonical URL for this repo is https://git.captnemo.in/nemo/nebula/. A mirror is maintained on GitHub at <https://github.com/captn3m0/nebula> The canonical URL for this repo is https://git.captnemo.in/nemo/nebula/. A mirror is maintained on GitHub at <https://github.com/captn3m0/nebula>
# modules # modules
1. docker: to actually run the services. Catch-all for miscellaneous containers 1. docker: to actually run the services. Catch-all for miscellaneous containers
2. cloudflare: to manage the DNS. 2. cloudflare: to manage the DNS.
3. mysql: to create mysql users and databases. 3. mysql: to create mysql users and databases.
4. media: Media related containers (Jackett, Lidarr, Radarr, Sonarr) 4. media: Media related containers (Jackett, Lidarr, Radarr, Sonarr)
5. Monitoring: Monitoring related resources (Cadvisor, Grafana, NodeExporter, Prometheus, Transmission-Exporter) 5. Monitoring: Monitoring related resources (Cadvisor, Grafana, NodeExporter, Prometheus, Transmission-Exporter)
6. Gitea: Just git.captnemo.in 6. Gitea: Just git.captnemo.in
7. tt-rss: Tiny-Tiny RSS Web reader 7. tt-rss: Tiny-Tiny RSS Web reader
8. Radicale: CardDav/CalDav webserver 8. Radicale: CardDav/CalDav webserver
Self-learning project for terraform/docker. Self-learning project for terraform/docker.
# Planned # Planned
1. ~Setup DigitalOcean~ 1. ~Setup DigitalOcean~
2. Add DO infrastructure via ansible 2. Add DO infrastructure via ansible
3. ~Add traefik for proper proxying~ 3. ~Add traefik for proper proxying~
4. Maybe add docker swarm (or k8s?) across both the servers. Might setup the k8s API on the Raspberry Pi. 4. Maybe add docker swarm (or k8s?) across both the servers. Might setup the k8s API on the Raspberry Pi.
# Service List # Service List
Currently running the following (all links are to the `store.docker.com` links for the docker images that I'm using: Currently running the following (all links are to the `store.docker.com` links for the docker images that I'm using:
| image | tag | size | category/module | | image | tag | module/link |
|--------------------------------|---------|------|-----------------| | -------------------------------- | ---------- | ---------------------------------------------------- |
| prom/node-exporter | v0.15.2 | 22.8 | monitoring | | bleenco/abstruse | latest | ci |
| redis | alpine | 27.8 | gitea | | captn3m0/opml-gen | latest | https://opml.bb8.fun |
| linuxserver/transmission | latest | 43.9 | media | | captn3m0/prometheus-act-exporter | latest | https://git.captnemo.in/nemo/prometheus-act-exporter |
| traefik | 1.6 | 51.8 | docker | | captn3m0/rss-bridge | latest | https://github.com/RSS-Bridge/rss-bridge |
| google/cadvisor | latest | 62.2 | monitoring | | captn3m0/speedtest-exporter | alpine | https://github.com/stefanwalther/speedtest-exporter |
| odarriba/timemachine | latest | 77.2 | backup | | emby/embyserver | latest | https://emby.media |
| gitea/gitea | 1.4 | 77.4 | gitea | | gitea/gitea | 1.5.0-rc1 | services |
| linuxserver/heimdall | latest | 101 | general | | google/cadvisor | latest | monitoring |
| linuxserver/tt-rss | latest | 108 | tt-rss | | grafana/grafana | latest | monitoring |
| prom/prometheus | latest | 113 | monitoring | | jankysolutions/requestbin | latest | tools |
| linuxserver/ubooquity | latest | 114 | docker | | linuxserver/airsonic | latest | media |
| captn3m0/speedtest-exporter | alpine | 115 | monitoring | | linuxserver/heimdall | latest | tools |
| tomsquest/docker-radicale | latest | 130 | radicale | | linuxserver/jackett | latest | media |
| linuxserver/lychee | latest | 154 | lychee | | linuxserver/lidarr | latest | media |
| linuxserver/resilio-sync | latest | 167 | resilio | | linuxserver/lychee | latest | media |
| emby/embyserver | latest | 202 | media | | linuxserver/radarr | latest | media |
| linuxserver/airsonic | latest | 239 | media | | linuxserver/resilio-sync | latest | sync |
| grafana/grafana | latest | 301 | monitoring | | linuxserver/sonarr | latest | media |
| requarks/wiki | latest | 317 | wiki | | linuxserver/transmission | latest | media |
| percona/percona-server-mongodb | latest | 321 | wiki | | linuxserver/tt-rss | latest | tools |
| mariadb | 10.3 | 402 | db | | linuxserver/ubooquity | latest | media |
| linuxserver/jackett | latest | 556 | media | | miniflux/miniflux | 2.0.9 | tools |
| linuxserver/sonarr | latest | 562 | media | | monicahq/monicahq | latest | services |
| linuxserver/radarr | latest | 566 | media | | odarriba/timemachine | latest | tools |
| linuxserver/lidarr | latest | 574 | media | | percona/percona-server-mongodb | 3.4 | database |
| postgres | 10-alpine | database |
| prom/node-exporter | v0.15.2 | monitoring |
| prom/prometheus | latest | monitoring |
| requarks/wiki | latest | services |
| serjs/go-socks5-proxy | latest | tools |
| tocttou/gotviz | latest | na |
| tomsquest/docker-radicale | latest | services |
| traefik | 1.6-alpine | plumbing |
## Docker Notes ## Docker Notes
- Lots of the above images are from the excellent [LinuxServer.io](https://www.linuxserver.io), and they're doing great work :+1: - Lots of the above images are from the excellent [LinuxServer.io](https://www.linuxserver.io), and they're doing great work :+1:
- Most images are running the latest beta (if available) or stable versions. - Most images are running the latest beta (if available) or stable versions.
- Traefik is running with wildcard certificates. - Traefik is running with wildcard certificates.
## Upstream ## Upstream
Issues I've faced/reported as a result of this project: I've been using this as a contributing opportunity and reporting/fixing issues upstream:
1. Airsonic HTTPS proxying is broken. Reported: https://github.com/airsonic/airsonic/issues/641. Turned out to be a known issue: https://github.com/airsonic/airsonic/issues/594. Now fixed. 1. Airsonic HTTPS proxying is broken. Reported: https://github.com/airsonic/airsonic/issues/641. Turned out to be a known issue: https://github.com/airsonic/airsonic/issues/594. Now fixed.
2. Traefik docker backend security headers were broken with dashes. I [reported it here](https://github.com/containous/traefik/issues/2493), and fixed by https://github.com/containous/traefik/pull/2496 :white_check_mark: 2. Traefik docker backend security headers were broken with dashes. I [reported it here](https://github.com/containous/traefik/issues/2493), and fixed by https://github.com/containous/traefik/pull/2496 :white_check_mark:
3. Headphones dies repeatedly with no error logs. Yet-to-report. (Already reported, fails due to classical artists) 3. Headphones dies repeatedly with no error logs. Yet-to-report. (Already reported, fails due to classical artists)
4. Terraform doesn't parse mariadb version numbers. Report: https://github.com/terraform-providers/terraform-provider-mysql/issues/6. Filed a [PR to fix](https://github.com/hashicorp/go-version/pull/34) and [to bump the go-version dependency](https://github.com/terraform-providers/terraform-provider-mysql/pull/27) :white_check_mark: 4. Terraform doesn't parse mariadb version numbers. Report: https://github.com/terraform-providers/terraform-provider-mysql/issues/6. Filed a [PR to fix](https://github.com/hashicorp/go-version/pull/34) and [to bump the go-version dependency](https://github.com/terraform-providers/terraform-provider-mysql/pull/27) :white_check_mark:
5. `elibsrv` didn't support ebook-convert, only mobigen. PR is at https://github.com/captn3m0/elibsrv/pull/1. Merged to `elibsrv` trunk, will be part of next release. 5. `elibsrv` didn't support ebook-convert, only mobigen. PR is at https://github.com/captn3m0/elibsrv/pull/1. Merged to `elibsrv` trunk, will be part of next release.
6. `ubooquity` docker container doesn't let you set admin password: https://github.com/linuxserver/docker-ubooquity/issues/17. (Couldn't reproduce, closed) :white_check_mark: 6. `ubooquity` docker container doesn't let you set admin password: https://github.com/linuxserver/docker-ubooquity/issues/17. (Couldn't reproduce, closed) :white_check_mark:
7. Traefik customresponseheaders can't contain colons on the docker backend: https://github.com/containous/traefik/issues/2517. Fixed with https://github.com/containous/traefik/pull/2509 :white_check_mark: 7. Traefik customresponseheaders can't contain colons on the docker backend: https://github.com/containous/traefik/issues/2517. Fixed with https://github.com/containous/traefik/pull/2509 :white_check_mark:
8. Traefik Security headers don't overwrite upstream headers: https://github.com/containous/traefik/issues/2618 :white_check_mark: 8. Traefik Security headers don't overwrite upstream headers: https://github.com/containous/traefik/issues/2618 :white_check_mark:
9. Transmission exporter broke with different data types while unmarshalling JSON in go. I filed a PR https://github.com/metalmatze/transmission-exporter/pull/2 :white_check_mark: 9. Transmission exporter broke with different data types while unmarshalling JSON in go. I filed a PR https://github.com/metalmatze/transmission-exporter/pull/2 :white_check_mark:
10. Radarr official docker container was [running a very old `mediainfo`](https://github.com/Radarr/Radarr/issues/2668#issuecomment-376310514). [Filed a fix to upgrade `mediainfo` on the official radarr image](https://github.com/linuxserver/docker-baseimage-mono/pull/3) :white_check_mark: 10. Radarr official docker container was [running a very old `mediainfo`](https://github.com/Radarr/Radarr/issues/2668#issuecomment-376310514). [Filed a fix to upgrade `mediainfo` on the official radarr image](https://github.com/linuxserver/docker-baseimage-mono/pull/3) :white_check_mark:
11. Patched the [speedtest-exporter](https://github.com/stefanwalther/speedtest-exporter/pull/7) to use Alpine and upgraded Node.JS for a smaller updated build. 11. Patched the [speedtest-exporter](https://github.com/stefanwalther/speedtest-exporter/pull/7) to use Alpine and upgraded Node.JS for a smaller updated build.
12. Faced (4) above again because mariadb decided to add `:` in the version response. [Workaround was to force set `--version=10.3-mariadb`](https://git.captnemo.in/nemo/nebula/commit/5f47a08bb55eea2c708c41668657ac1efa84c72a) 12. Faced (4) above again because mariadb decided to add `:` in the version response. [Workaround was to force set `--version=10.3-mariadb`](https://git.captnemo.in/nemo/nebula/commit/5f47a08bb55eea2c708c41668657ac1efa84c72a)
13. Reported [2 critical security issues in Abstruse CI](https://github.com/bleenco/abstruse/issues/363). :white_check_mark: 13. Reported [2 critical security issues in Abstruse CI](https://github.com/bleenco/abstruse/issues/363). :white_check_mark:
14. Faced (13) above again with postgres, thankfully [someone already fixed version parsing](https://github.com/terraform-providers/terraform-provider-postgresql/pull/31) :white_check_mark:
15. RSS Bridge was missing an official Docker Image. [I Filed a PR](https://github.com/RSS-Bridge/rss-bridge/pull/720) :white_check_mark:
# Plumbing # Plumbing
Their is a lot of additional infrastructure that is _not-yet_ part of this repo. This includes: Their is a lot of additional infrastructure that is _not-yet_ part of this repo. This includes:
1. The Digital Ocean droplet running DNSCrypt and simpleproxy to proxy over a openvpn connection to this box. 1. The Digital Ocean droplet running DNSCrypt and simpleproxy to proxy over a openvpn connection to this box.
2. openbox, kodi configuration to run on boot along with the Steam Controller for the HTPC setup 2. openbox, kodi configuration to run on boot along with the Steam Controller for the HTPC setup
3. Docker main configuration with half-baked CA setup 3. Docker main configuration with half-baked CA setup
4. btrfs-backed subvolumes and snapshotting for most things in /mnt/xwing/ (in-progress) 4. btrfs-backed subvolumes and snapshotting for most things in /mnt/xwing/ (in-progress)
5. User-creation on the main server. (I'm using a common user for media applications and specific users for other applications) 5. User-creation on the main server. (I'm using a common user for media applications and specific users for other applications)
# License # License

View File

@ -1,49 +0,0 @@
resource "docker_container" "mariadb" {
name = "mariadb"
image = "${docker_image.mariadb.latest}"
volumes {
volume_name = "${docker_volume.mariadb_volume.name}"
container_path = "/var/lib/mysql"
host_path = "${docker_volume.mariadb_volume.mountpoint}"
}
// This is so that other host-only services can share this
ports {
internal = 3306
external = 3306
ip = "${var.ips["eth0"]}"
}
// This is a not-so-great idea
// TODO: Figure out a better way to make terraform SSH and then connect to localhost
ports {
internal = 3306
external = 3306
ip = "${var.ips["tun0"]}"
}
memory = 512
restart = "unless-stopped"
destroy_grace_seconds = 10
must_run = true
env = [
"MYSQL_ROOT_PASSWORD=${var.mysql_root_password}",
]
command = [
"--version=${var.mariadb-version}-MariaDB",
]
networks = ["${docker_network.mariadb.id}"]
}
resource "docker_image" "mariadb" {
name = "${data.docker_registry_image.mariadb.name}"
pull_triggers = ["${data.docker_registry_image.mariadb.sha256_digest}"]
}
data "docker_registry_image" "mariadb" {
name = "mariadb:${var.mariadb-version}"
}

View File

@ -1,14 +1,3 @@
resource "docker_network" "mariadb" {
name = "mariadb"
driver = "bridge"
internal = true
ipam_config {
subnet = "172.19.0.0/28"
gateway = "172.19.0.1"
}
}
resource "docker_network" "mongorocks" { resource "docker_network" "mongorocks" {
name = "mongorocks" name = "mongorocks"
driver = "bridge" driver = "bridge"

View File

@ -1,7 +1,3 @@
output "names-mariadb" {
value = "${docker_container.mariadb.name}"
}
output "networks-mongorocks" { output "networks-mongorocks" {
value = "${docker_network.mongorocks.name}" value = "${docker_network.mongorocks.name}"
} }

View File

@ -1,8 +1,3 @@
variable "mariadb-version" {
description = "mariadb version to use for fetching the docker image"
default = "10.2.14"
}
variable "postgres-version" { variable "postgres-version" {
description = "postgres version to use for fetching the docker image" description = "postgres version to use for fetching the docker image"
default = "10-alpine" default = "10-alpine"
@ -12,5 +7,4 @@ variable "ips" {
type = "map" type = "map"
} }
variable "mysql_root_password" {}
variable "postgres-root-password" {} variable "postgres-root-password" {}

View File

@ -1,7 +1,3 @@
resource "docker_volume" "mariadb_volume" {
name = "mariadb_volume"
}
resource "docker_volume" "postgres_volume" { resource "docker_volume" "postgres_volume" {
name = "postgres_volume" name = "postgres_volume"
} }

View File

@ -1,7 +1,7 @@
data "docker_registry_image" "traefik" { data "docker_registry_image" "traefik" {
# Critical and I like upgrading it # Critical and I like upgrading it
# for updating config for new features # for updating config for new features
name = "traefik:1.6-alpine" name = "traefik:1.7-alpine"
} }
data "docker_registry_image" "wikijs" { data "docker_registry_image" "wikijs" {

View File

@ -1,4 +1,4 @@
resource "docker_image" "traefik16" { resource "docker_image" "traefik17" {
name = "${data.docker_registry_image.traefik.name}" name = "${data.docker_registry_image.traefik.name}"
pull_triggers = ["${data.docker_registry_image.traefik.sha256_digest}"] pull_triggers = ["${data.docker_registry_image.traefik.sha256_digest}"]
} }

View File

@ -1,6 +1,6 @@
resource "docker_container" "traefik" { resource "docker_container" "traefik" {
name = "traefik" name = "traefik"
image = "${docker_image.traefik16.latest}" image = "${docker_image.traefik17.latest}"
# Admin Backend # Admin Backend
ports { ports {

17
main.tf
View File

@ -4,15 +4,6 @@ module "cloudflare" {
ips = "${var.ips}" ips = "${var.ips}"
} }
# module "mysql" {
# source = "mysql"
# mysql_root_password = "${var.mysql_root_password}"
# mysql_lychee_password = "${var.mysql_lychee_password}"
# mysql_airsonic_password = "${var.mysql_airsonic_password}"
# mysql_kodi_password = "${var.mysql_kodi_password}"
# lychee_ip = "${module.docker.lychee-ip}"
# }
module "docker" { module "docker" {
source = "docker" source = "docker"
web_username = "${var.web_username}" web_username = "${var.web_username}"
@ -27,7 +18,6 @@ module "docker" {
module "db" { module "db" {
source = "db" source = "db"
mysql_root_password = "${var.mysql_root_password}"
postgres-root-password = "${var.postgres-root-password}" postgres-root-password = "${var.postgres-root-password}"
ips = "${var.ips}" ips = "${var.ips}"
} }
@ -120,13 +110,10 @@ module "heimdall" {
} }
module "media" { module "media" {
source = "media" source = "media"
domain = "bb8.fun" domain = "bb8.fun"
# links-mariadb = "${module.db.names-mariadb}"
traefik-labels = "${var.traefik-common-labels}" traefik-labels = "${var.traefik-common-labels}"
airsonic-smtp-password = "${var.airsonic-smtp-password}" airsonic-smtp-password = "${var.airsonic-smtp-password}"
airsonic-db-password = "${var.mysql_airsonic_password}"
ips = "${var.ips}" ips = "${var.ips}"
traefik-network-id = "${module.docker.traefik-network-id}" traefik-network-id = "${module.docker.traefik-network-id}"
} }

View File

@ -2,11 +2,8 @@ variable "domain" {
type = "string" type = "string"
} }
# variable "links-mariadb" {}
variable "airsonic-smtp-password" {} variable "airsonic-smtp-password" {}
variable "airsonic-db-password" {}
variable "traefik-labels" { variable "traefik-labels" {
type = "map" type = "map"
} }

View File

@ -1,16 +0,0 @@
resource "mysql_database" "airsonic" {
name = "airsonic"
}
resource "mysql_user" "airsonic" {
user = "airsonic"
host = "%"
plaintext_password = "${var.mysql_airsonic_password}"
}
resource "mysql_grant" "airsonic" {
user = "${mysql_user.airsonic.user}"
host = "${mysql_user.airsonic.host}"
database = "${mysql_database.airsonic.name}"
privileges = ["ALL"]
}

View File

@ -1,16 +0,0 @@
resource "mysql_database" "lychee" {
name = "lychee"
}
resource "mysql_user" "lychee" {
user = "lychee"
host = "%"
plaintext_password = "${var.mysql_lychee_password}"
}
resource "mysql_grant" "lychee" {
user = "${mysql_user.lychee.user}"
host = "${mysql_user.lychee.host}"
database = "${mysql_database.lychee.name}"
privileges = ["ALL"]
}

View File

@ -1,17 +0,0 @@
variable "mysql_root_password" {
type = "string"
}
variable "mysql_lychee_password" {
type = "string"
}
variable "mysql_airsonic_password" {
type = "string"
}
variable "mysql_kodi_password" {
type = "string"
}
variable "lychee_ip" {}

View File

@ -11,20 +11,10 @@ variable "web_password" {
type = "string" type = "string"
} }
variable "mysql_root_password" {
type = "string"
}
variable "postgres-root-password" { variable "postgres-root-password" {
type = "string" type = "string"
} }
variable "mysql_lychee_password" {}
variable "mysql_airsonic_password" {}
variable "mysql_kodi_password" {}
variable "mysql-ttrss-password" {} variable "mysql-ttrss-password" {}
variable "gitea-mysql-password" {} variable "gitea-mysql-password" {}