Antikythera

Antikythera 103: Installing Services

By January 13, 2019 No Comments

Please note: the entire docker-compose.yml file can be found for reference at the bottom of this post.

Welcome to the third installation of the Antikythera series. Thus far, what we’ve accomplished, while paving the way for a stable and secure web server, we haven’t really done anything particularly useful with it yet. In this tutorial, we are going to install our first service: Nextcloud. It is probably the most powerful self-hosted web application available. It can replace Drop Box/Google Drive as a file hosting service, has online file editing tools, can host a private calendar and contacts available for access on open standards, and much much more.

However, we are also going to keep two things in mind while setting up our server and applications:

  1. We want to continue to employ good security practices.
  2. We want to future proof and install applications in such a way that they won’t interfere with other applications and services on our server.

For this reason, we are going to start by installing two other supporting services:

  1. Docker – a container management service
  2. Traefik – a reverse proxy

Once we have Docker and Traefik set up, we will use them to install and set up Nextcloud in a secure and easily managed fashion. This tutorial might take a while, so make a cup of coffee or grab a beer, and settle in.

Installing Docker

One of the problems when using a single server for multiple applications is that many applications have overlapping dependencies. For instance, if you wanted to host a website and a Nextcloud installation, both of those services might rely on the Apache web server. That web server might need a specific configuration for each service, and those configurations may conflict with each other. Or maybe you have two services installed, using PHP 5, but then one updates and requires PHP 7, while the other doesn’t. Then you are forced to run old, outdated software while you wait for the second service to be updated. Or maybe one service you are running goes haywire, or gets hacked, and since it is just sitting in the middle of your Linux installation, it can take down everything with it.

One of the traditional ways to address this problem was to run virtual machines. Virtual machines are quite powerful. They are an operating system within an operating system. Many companies will have beefy hardware setups, and run multiple virtual machines – one for each service. However, virtual machines have a lot of overhead. They require a full, or nearly full installation of the OS and all its dependencies. Then, they need to be run on hardware that supports virtualization. And any type of interfacing must be patched through to the virtual machine. They can easily become quite large and power intensive processes. They quite quickly become inefficient for a self-hoster like us.

Our solution, then, is going to be similar to a virtual machine, but not quite. We are going to use Docker to solve this problem. Docker is a container service. It runs on our Linux installation, and creates containers for each application we use. Each container contains the application, and the dependencies needed to run it, usually preconfigured. There is no real installation to perform, all of our applications are neatly organized and, well, contained. They all have their own version of whatever dependencies they need, so they won’t need to share services, and they’re extremely lightweight, with almost no processing overhead. Docker is nothing short of a miracle, and while it can have a small learning curve, getting started with it right away will save you loads of time and effort when you inevitably decide that you need it in the future.

To install Docker, the first thing we want to do is log into our server and update our repositories:

sudo apt update

Because we want to have the most recent version of Docker, we are going to install it directly from Docker’s repository, not Ubuntu’s. To do this, we must first tell Ubuntu to check Docker’s repository. However, we also want to verify that it’s actually Docker’s repository we’re communicating with. So we will start by verifying that with their official “GPG” key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

The curl command stands for cURL. cURL is a common way to download files from the internet in a Linux terminal. We’re essentially grabbing their GPG key and adding it to our repositories so that we can download directly from Docker, and verify that it’s them we are communicating with it.

Now we need to actually add their repository:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Now that it’s in our repository, we’ll update again:

sudo apt update

And then it’s just a simple apt install to download Docker:

sudo apt install docker-ce

Congratulations, Docker is installed. Now that we have Docker, we just want to do a couple of quick housekeeping things. The first is that we want to be able to run Docker commands without having to say “sudo” first. The easiest way to do this is to add our user to the Docker group.

sudo usermod -aG docker username

where “username” is your username.

Next, we want to install Docker Compose. Docker Compose is a Docker service that will let us set up our Docker containers in a static configuration file, and then load that configuration file to install, update, and run them. This is advantageous because with many services, running everything through the terminal becomes very difficult to keep track of. To install Docker Compose, simply run:

sudo apt install docker-compose

And that’s it. Docker is installed and ready to rock.

Installing Traefik

One of the other problems with hosting multiple services from one server is that most web services communicate over port 80 for unencrypted traffic, and port 443 for encrypted traffic. If we have two services relying on the same port, when the communication comes to our server requesting port 80, our server won’t know which service to direct it to. We can solve this with a reverse proxy, which essentially handles all the requests on port 80 and port 443, and uses metadata attached to the requests to figure out what service to direct it to. One nice thing that a reverse proxy will do, as well, is employ HTTPS encryption for all the requests it receives, helping to secure your connection to your web services. Traefik is an extremely modern reverse proxy, made specifically with Docker containers in mind. In fact, it even ships in a Docker container, so we can very conveniently install it and manage it. However, it’s worth noting Traefik can be a little tricky to set up, and their documentation isn’t always the most detailed, so you may find this part very tedious, but thankfully we only have to set it up once.

The first thing we want to do is make our Docker Compose file to tell Docker to install Traefik. We’re going to make the Docker Compose file in a directory that we can use for everything we install with Docker. So go ahead an make a directory in /etc/ from which we can base our Docker setup:

sudo mkdir /etc/selfhosted

Next, we are going to navigate to that directory with:

cd /etc/selfhosted

Now, go ahead and create your Docker Compose file:

sudo nano docker-compose.yml

Docker Compose files must be named docker-compose.yml to work. YML is a configuration file format. At the top of this file, we’re going to put our Traefik configuration. You can copy and paste this if you like, but please read the explanation of what each command is doing, as you will need to use it later. Also, please be mindful of the indentation, as the configuration file will result in errors if the syntax and indentation isn’t precisely followed:

version: "3"
services:
#Traefik

   traefik:
   container_name: traefik
   image: traefik:alpine
   ports:
     - 80:80
     - 443:443
     - 8080:8080
   restart: unless_stopped
   volumes:
     - /etc/selfhosted/traefik:/etc/traefik
     - /var/run/docker.sock:/var/run/docker.sock

Here is the breakdown of this configuration:

  • The first line, version, tells Docker what version of the Docker-Compose format we’re using.
  • Services declares what Docker services we are running. We will add more to this later, but for now, we’re just running Traefik.
  • The “traefik:” line is just what we have decided to name our Traefik service.
  • The “container_name” is how it will appear in Docker.
  • The “image” is the actual program we will be running. This image is downloaded from Docker’s repository. The “:alpine” appended to it tells Docker to download the alpine image, which is an image the Traefik developers have made available that is extremely lightweight.
  • The “ports” section tells Docker what ports to forward to this container. The port on the left is the port the container listens on, the port on the right is the one that Docker will forward to it.
  • “restart” tells Docker when to restart this container. If something goes wrong that causes it to crash, it will automatically restart with this setting.
  • Volumes is a little more complicated. If the Docker container crashes, all the data inside it will disappear. This is bad if we have data we don’t want to lose. Also, accessing the inside of a container is very difficult while it is running. The volumes line allows us to tell the container to store certain files outside of the container, so that if it crashes those files don’t disappear, and it also makes some files available for easy editing. The line on the left is the path outside of the container, and the line on the right is the path on the inside of the container. We’re telling the Docker container that everything stored in /etc/traefik should actually be stored outside the container inside /etc/selfhosted/traefik

OK, now all we need to do is make that directory that we told Docker to store the files in, as well as make sure some other files are in the right spots, and we should be all set to start Traefik up. We told Docker to store files in /etc/selfhosted/traefik, so go ahead and make that directory:

sudo mkdir traefik

Next, we need to generate a configuration file for Traefik. This file is by default stored in the /etc/traefik folder, so it’ll be in our /etc/selfhosted/traefik folder outside of the container. So go ahead and make that by typing:

sudo nano traefik/traefik.toml

This tells the server to make the file traefik.toml inside the traefik directory, inside the /etc/selfhosted directory that we’re currently in. In the traefik.toml file, enter the following:

defaultEntryPoints = ["http", "https"]

[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[entryPoints.api]
address = ":8080"

[api]
entryPoint = "api"

[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "yourdomain.com"
exposedByDefault = false

[acme]
email = "youremail@email.com"
entryPoint = "https"
storage = "/etc/traefik/acme/acme.json"
onHostRule = true

[acme.httpChallenge]
entryPoint = "http"
[[acme.domains]]
main = "yourdomain.com"

OK, let’s go over this configuration file:

  • “defaultEntryPoints” is declaring to Traefik where to listen for traffic. These entry points are further declared below, specifying what ports they are, and what type of encryption to use. We are using HTTPS because it encrypts HTTP traffic over the web, preventing people from intercepting it and seeing things like passwords and credit card numbers. HTTPS relies on other parties issuing the certificates, which we will address further down.
  • The “[api]” section simply is telling Traefik to run a small web UI that will allow us to investigate what it is currently detecting.
  • The “[docker]” section is basically allowing it to plug into Docker. Be sure to replace “yourdomain.com” with the domain name you registered and pointed toward your web server in the first tutorial. The “exposedByDefault” setting allows new Docker containers to be automatically exposed. We have turned that off as we will manually expose them, because we don’t want all of them exposed.
  • The “[acme]” section is how we will generate HTTPS certificates. This uses a service called “Lets Encrypt” which is a free certificate authority. Essentially, what they will do is test to see if the server requesting the certificate is actually the one that the domain name is owned by. Then, if it is, they will issue a certificate verifying this. When your web browser sees this certificate, it knows that it has the right web server for the domain name, and can securely communicate with it. Be sure to set the email to your email that you used to register the domain name, and be sure to set the domain name to the one you purchased.

Go ahead and save this file. Make sure you are in the /etc/selfhosted directory. Now it’s time to boot up our Docker and get Traefik installed. Run the command for this:

docker-compose up -d

Hopefully, everything will boot up properly. If not, check your configuration files, make sure that the syntax matches what is listed here, and that all your email addresses and domain names are inputted properly. If this doesn’t work, feel free to post your error message in the comments and we will try to help you out. To test out if Traefik is working, navigate to your port 8080 on your domain name in your web browser: mydomain.com:8080

You should see a screen similar to this:

Congratulations! The hard part is over. Now we can install pretty much as many apps and services as we like, in a well organized manner. So what are we waiting for? Lets get Nextcloud installed.

Installing Nextcloud

Nextcloud is a phenomenal piece of software. It nearly single-handedly makes self hosting a viable option, as it replaces so many services and has so many extensions. It’s practically operable as its own cloud-based operating system. We are most interested its ability to store and synchronize files and other data. To have a fully functioning Nextcloud, we need two pieces:

  1. The Nextcloud software.
  2. Some sort of SQL database.

Fortunately, we can install and manage both of these with Docker. We will start by checking out Nextcloud on Docker’s image hub. The official Nextcloud image has a variety of Nextcloud containers we can deploy. We are just going to use the default one, however. Additionally, it tells us that we can either use Postgre, MySQL, or MariaDB for our SQL database. We’re going to use MariaDB. It’s functionally equivalent to MySQL, but unlike MySQL, it isn’t owned by Oracle. Oracle has engaged in lots of anti-competitive business practices in the past, so we’re going to avoid their software whenever possible.

OK, so let’s get installing. Just like with Traefik, this is going to go into our docker-compose.yml file. So go ahead an nano that:

sudo nano /etc/selfhosted/docker-compose.yml

At the end of the file, we’re going to enter the necessary information for the Nextcloud install:

#NextCloud
  nextcloud:
    image: nextcloud
    container_name: nextcloud
    networks:
      - traefik
      - backend
    links:
      - nextcloud_db
    volumes:
      - /etc/selfhosted/cloud/nextcloud:/var/www/html
    restart: always
    labels:
      - traefik.enable=true
      - traefik.frontend.rule=Host:mydomain.com
      - traefik.frontend.redirect.entryPoint=https

Let’s break down what’s going on here:

  • nextcloud: – This is just our domain for the docker-compose file
  • image: – This is the image we’re downloading from the Docker hub. We’re just downloading the default nextcloud image.
  • container_name: – This is what we will call it in Docker. This is what we will reference if we need to manipulate it from the terminal, later. You can name it whatever you want to.
  • networks: – We have two networks defined here, traefik, and backend. We haven’t actually made our networks yet. We will do this from the terminal later. The traefik network will be an exposed network used by Traefik to point services to incoming web traffic. The backend one will be used to connect services that are web-facing to services that don’t need to be web facing (such as the MariaDB database).
  • There’s no ports for this container because we will let Traefik handle that automatically.
  • links: – This is going to enable the Nextcloud Docker to talk to the MariaDB Docker without having to go through any routing. We will name our MariaDB Docker “nextcloud_db”
  • volumes: – This is where we will store the files. Here we are saying that /var/www/html inside the Docker images should actually be stored outside of the Docker container, in /etc/selfhosted/cloud/nextcloud. This is a very important step. If you don’t set the volume to be external to the Docker, anytime the container restarts, you will lose all of your files.
  • restart: – If something accidentally kills this container, we are telling it to automatically restart. An alternative here is unless_stopped, which will restart it unless you manually stopped it.
  • labels: – This is where we tell the Docker container to use Traefik. Here’s what each of the labels is doing:
    • traefik.enable – This is telling the Docker container to have Traefik manage its routing.
    • traefik.frontend.rule=Host: – This is telling Traefik that any incoming traffic on that sub-domain should be redirected to this Docker container. Set this to the domain name you put in the traefik.toml file.
    • traefik.frontend.redirect.entrypoint=https – This is telling Traefik to redirect any HTTP (unsecured) traffic to HTTPS (secured) for this container.

That’s a lot to unpack. Don’t worry if it doesn’t make a lot of sense. Eventually, it will become easier to decipher as you get more familiar with how traffic moves across the internet and servers in general. Now, we need to install MariaDB, then we will do just a couple of finishing touches, and fire up all this software.

Installing MariaDB

Now, we need to install MariaDB to act as our database for NextCloud. We could go with the built-in SQLite database, but that’s not excellent in terms of performance. To get started with MariaDB, we need to look up the image name and documentation on the Docker Hub. You can access MariaDB’s Docker Hub page here.

OK, so now let’s type out the configuration:

  nextcloud_db:
    image: mariadb
    container_name: nextcloud_db
    labels:
      - traefik.enable=false
    networks:
      - traefik
      - backend
    restart: always
    volumes:
      - /etc/dockers/cloud/database:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=rootpassword
      - MYSQL_PASSWORD=databasepassword
      - MYSQL_DATABASE=databasename
      - MYSQL_USER=databaseuser

Once again, let’s break all this down:

  • The name is just how we are going to reference this container in the docker-compose.yml file. In the NextCloud configuration, we said we were going to link to “nextcloud_db”, so we have to be consistent and use that name here.
  • The image we grabbed from the Docker Hub. No fancy image names here, we’re just going to go with the normal mariadb image.
  • container_name is pretty self-explanatory. This is what Docker will name the container, and how we will reference it on the command line if we have to.
  • labels is different this time. We need to tell Traefik to ignore this container because we don’t want any direct internet traffic to it. Any and all traffic to this database will be facilitated by the internet-facing NextCloud container.
  • For networks, well, this brushes up against the limits of my knowledge. We need it to be on the backend network, however, it will not function properly unless it’s on the traefik network for some reason. Obviously both networks are redundant, but I’m going to leave backend in there in the hopes I can figure out why this isn’t working properly. However, for now you could probably get away with just the traefik network if you like.
  • We’re going to tell the container to always restart if something ends it.
  • Volumes is where we’re telling it to store everything inside the /var/lib/mysql folder of the container in the /etc/selfhosted/cloud/database folder on the host. This will keep the data persistent should the container restart.
  • The environment section is where we can set certain environmental variables to be pre-configured when we start the container. We’re setting the root password of the MariaDB instance, as well as the user and password and database name that the NextCloud container will be accessing. In another tutorial, I will go over how to store these values more securely, but for now, we’re just going to dump them in plain text here in the docker-compose.yml file.

You may notice that we reference MySQL frequently in MariaDB. This is not a typo. They are extremely similar pieces of software. MariaDB was just forked from MySQL because of perceived bad behavior by Oracle, the company that purchased MySQL.

Finishing Up

We just have to do a few more things before we can fire all this up. First off is the networks. We need to add the following code to the bottom of our docker-compose.yml file:

##Networks##

networks:
  traefik:
    external: true
  backend:
    external: true

Please note: the “external” does not mean that the network is web-facing, but rather just exists outside of the docker-compose file as well.

The last edit to the docker-compose.yml file we will make is up top, under the Traefik configuration. Add the following to the bottom of your Traefik configuration:

   networks: 
     - traefik
     - backend

Now press Ctrl+X to save the file.

Now we need to create the networks in Docker, since we said they’re external. Enter the following command:

docker network create traefik && docker network create backend

One more thing we have to do is create the directories and properly set their permissions. We said that we are going to store data from the containers on our host OS so that we won’t lose it if the containers crash. Let’s go ahead and run the following command to do that:

sudo mkdir /etc/selfhosted/cloud && sudo mkdir /etc/selfhosted/cloud/nextcloud && sudo mkdir /etc/selfhosted/cloud/database

Finally, we need to set permissions for the folders we just made. We’re going to run the following command to do this:

sudo chmod -R 755 /etc/selfhosted/cloud

This is basically telling the OS to allow the owner to read, write, and execute anything in the folders, while other users and guests can only read and execute things in those folders. The “-R” is making it a recursive command, applying it to the cloud directory, and all directories within.

Go Time

In the directory that contains your docker-compose.yml file (if you followed the tutorial, it should be /etc/selfhosted), simply run the following command and wait:

sudo docker-compose up -d

Then, you should be able to access your NextCloud instance via your web browser by navigating to your website. You should be greeted by the setup screen:

NextCloud Setup Screen

Congratulations! You’ve made it to the NextCloud setup screen!

Now for the last little bit until you actually get access to your NextCloud installation:

  • Choose the username and password you’d like for the install.
  • Select “storage and database” if it isn’t pre-selected. Don’t bother with the “Data folder”, as the default value should be fine, but we need to make additional changes to the database setup, so select the “MySQL/MariaDB” option
  • The username is the username you declared in the “environment” section of the nextcloud_db configuration. You’ll want to also set the database name to the one you put there as well, and set the password to the password you chose for that (NOTE: this is not the root password, but rather the second one).
  • The host name is the name of the Docker instance running our database. Despite NextCloud’s warning of adding a port number, we actually don’t need to since we linked them together with Docker.

After that, click “Finish Setup”. It may take a minute or two to finish the setup, and then it should bring you to the NextCloud main page. Congratulations! You now have access to a comprehensive cloud system you can use to store files, contacts, calendars, and more. Check future articles for how to maximize this setup and continue to de-Google yourself!

As always, please post in the comments with any problems or suggestions.

Full docker-compose.yml file:

version: "3"

services:

#Traefik
  traefik:
    container_name: traefik
    image: traefik:alpine
    networks:
      - traefik
      - backend
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    restart: always
    volumes:
      - /etc/selfhosted/traefik:/etc/traefik
      - /var/run/docker.sock:/var/run/docker.sock


#NextCloud
  nextcloud:
    image: nextcloud
    container_name: nextcloud
    networks:
      - traefik
      - backend
    links:
      - nextcloud_db
    volumes:
      - /etc/selfhosted/cloud/nextcloud:/var/www/html
    restart: always
    labels:
      - traefik.enable=true
      - traefik.frontend.rule=Host:mydomain.com
      - traefik.frontend.redirect.entryPoint=https

  nextcloud_db:
    image: mariadb
    container_name: nextcloud_db
    labels:
      - traefik.enable=false
    networks:
      - traefik
      - backend
    restart: always
    volumes:
      - /etc/dockers/cloud/database:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=rootpassword
      - MYSQL_PASSWORD=databasepassword
      - MYSQL_DATABASE=databasename
      - MYSQL_USER=databaseuser

##Networks##

networks:
  traefik:
    external: true
  backend:
    external: true

Leave a Reply