Simplest HTTPS setup for your new Linux Server: Nginx Reverse Proxy+ Letsencrypt+ AWS Cloud + Docker
I needed a set of steps that would help me setup a server for any new project.
My projects would start by pointing a domain to my web-app running on a Linux VM in cloud(AWS, Azure, Oracle/OCI etc). Traffic should be served over https, with free Letsencrypt SSL certificates. Setup certificate auto-renewal. Use Nginx reverse proxy.
So I wrote down these minimum, easy to follow steps to get it done in less than 5 min ⌛. This helped me and anyone in my team quickly setup a completely new backend project. I have tested this setup with both Cloudflare and Namecheap DNS.
Hope this will save your precious time 😀… consider hitting 👏 at the end. !! You can appreciate my efforts… buy me a tea ☕.
Before starting, I’ll assume you already have following things ready:
- You have a server/VM (doesn’t have to be AWS) with a public IP.
- The ports 80 and 443 are open for all TCP inbound traffic. (80 for http and 443 for https)
- Related to above, make sure you have tested it’s really open as some OS come with firewall by default and you need to open it to accept traffic on above ports.
- Docker and docker-compose is installed (for ubuntu use this doc, or follow these minimal steps)
- DNS setup: domain/sub-domain points to the IP address of server(check respective setup instructions for your domain registrar or DNS provider)
For any issues, jump to FAQ section at end.
*If you intend to get the job done quick, without getting into the details, just jump over to this github repo and follow the minimal steps mentioned there.
**Note: Now compose is a part of docker and hence all docker-compose
commands can be written as docker compose
i.e. without the -
. To be able to run using docker-compose
, the compose standalone should be installed first.
Step-1: Create project structure
Either clone this repo or follow the steps. If you clone the repo, it already has all files with default content.
We need to create files and folder structure like this:
nginx_https_docker
|
|-- config
| |-- nginx.conf
|
|-- docker
| |-- nginx.Dockerfile
|
|-- docker-compose.yaml
|-- docker-compose-le.yaml
Run these commands
mkdir nginx_https_docker && cd nginx_https_docker
mkdir config
mkdir docker
touch config/nginx.conf docker/nginx.Dockerfile
touch docker-compose.yaml docker-compose-le.yaml
nginx.conf
: config for nginx reverse proxynginx.Dockerfile
: for nginx docker containerdocker-compose.yaml
: compose file for our nginx server. Put any other services that are part of your application here.docker-compose-le.yaml
: for the first time setup and certificate auto-renewal
Creating the files one by one can be useful if you already have your project repo with code. Then you can just add stuff to your docker-compose etc. and things should work. Or else, you can create the setup separately and then copy stuff from new setup to your existing codebase. I prefer latter.
Step-2: Configure for first time setup
Open config/nginx.conf
and add below config.
**Replace test.leangaurav.dev
with the domain you wish to configure. It can be subdomain like mine or apex domain.
server {
listen 80;
listen [::]:80;
server_name test.leangaurav.dev;
location ~ /.well-known/acme-challenge {
allow all;
root /tmp/acme_challenge;
}
}
We need this config only for the first time to let certbot be able to get ssl certificates. Certbot places a token to validate the certificate request in the directory
/.well-known/acme-challenge
. Then certbot validates your domain by requesting the token at<yourdomain>/.well-known/acme-challenge/<some token>
Hence we have setup the location block to serve the challenge requests from/tmp/acme_challenge
. This same location we will be mounting into our nginx container as well later.
In the docker-compose.yml
paste below content as it is. For existing setup, just add the nginx service part to it. Remember to change the network name app
with whatever you have. Otherwise nginx won’t be able to communicate with your application.
version: "3.3"
services:
nginx:
container_name: 'nginx-service'
build:
context: .
dockerfile: docker/nginx.Dockerfile
ports:
- 80:80
- 443:443
volumes:
- ./config:/config
- /etc/letsencrypt:/etc/letsencrypt:ro
- /tmp/acme_challenge:/tmp/acme_challenge
networks:
- app
restart: always
networks:
app:
driver: bridge
Now we need the docker/nginx.Dockerfile
. This is pretty simple. Feel free to change the nginx version later.
FROM nginx:1.21.1-alpine
RUN rm /etc/nginx/conf.d/default.conf
COPY /config/nginx.conf /etc/nginx/conf.d
Last thing we need is the docker-compose-le.yaml
. Again, here you need to replace your.email@email.com
with the email where you wish to receive certificate renewal notifications and also replacetest.leangaurav.com
with your domain name.
The
expand
option is optional. It helps if you add another domain to the same command at a later point.
Multiple domains can be added with a comma separator liketest.leangaurav.com,about.leangaurav.com
without any spaces.
version: "3.3"
services:
letsencrypt:
container_name: 'certbot-service'
image: certbot/certbot:v1.17.0
command: sh -c "certbot certonly --expand --webroot -w /tmp/acme_challenge -d test.leangaurav.com --text --agree-tos --email your.email@email.com --rsa-key-size 4096 --verbose --keep-until-expiring --preferred-challenges=http"
entrypoint: ""
volumes:
- "/etc/letsencrypt:/etc/letsencrypt"
- "/tmp/acme_challenge:/tmp/acme_challenge"
environment:
- TERM=xterm
If you noticed, we have done a bind mount for the volume
/tmp/acme_challenge
in both the nginx and certbot docker-compose .The two dockerfiles:
1. docker-compose.yml: contains nginx service. you can add other services which need to share network with nginx, or copy this service into your main docker-compose.yml
2. docker-compose-le.yml: runs the certbot image with our config. this will be needed later in the auto-renew step.
Step-3: The first run
From the nginx_https_docker
folder on your server, run the command
docker compose up --build nginx
this will start our nginx server. Leave it running and you will also be able to see the logs here. Probably try accessing your server using the IP address or using http://yourdomain
. Make sure you use http when using both. This should show you the nginx home page or a 404 not found page.
If you don’t see this, check the common issues section about this and how to fix this.
If at this point you get an error like
ERROR: for nginx Cannot create container for service nginx: Conflict. The container name “/nginx-service” is already in use by container “31bd6a36f4ecc7a43f1ea991c7686ae60a4ddb13c749b9c5b848cbacb69e89d5”. You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.then run this command to remove old container:
docker stop nginx-service && docker container rm nginx-service
Now in another terminal window go to the folder nginx_https_docker
and run the letsencrypt container with the below command
docker compose -f docker-compose-le.yaml up --build
if things work well you will see something like this
Successfully received certificate.
letsencrypt_1 | Certificate is saved at: /etc/letsencrypt/live/yourdomain.com/fullchain.pem
letsencrypt_1 | Key is saved at: /etc/letsencrypt/live/yourdomain.com/privkey.pem
letsencrypt_1 | This certificate expires on 2021-10-20.
letsencrypt_1 | These files will be updated when the certificate renews.
letsencrypt_1 |
letsencrypt_1 | NEXT STEPS:
letsencrypt_1 | - The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.
letsencrypt_1 |
letsencrypt_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
letsencrypt_1 | If you like Certbot, please consider supporting our work by:
letsencrypt_1 | * Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
letsencrypt_1 | * Donating to EFF: https://eff.org/donate-le
letsencrypt_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Generated certificates will be available under `/etc/letsencrypt/<your domain name>` directory on your machine.
The above container stops itself after installing the certificates. For the other container running in the first terminal, stop it by pressing Ctrl+C or CMD+C.
Step-4: Enable https
Open the config/nginx.conf
again and delete everything. Paste below contents and remember again to remember to replacetest.leangaurav.com
with your domain name at all the four places.
server {
listen 80;
listen [::]:80;
server_name test.leangaurav.dev;
location / {
return 301 https://$host$request_uri;
}
location ~ /.well-known/acme-challenge {
allow all;
root /tmp/acme_challenge;
}
}
server {
listen 443 ssl;
listen [::]:443 ssl http2;
server_name test.leangaurav.dev;
ssl_certificate /etc/letsencrypt/live/test.leangaurav.dev/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/test.leangaurav.dev/privkey.pem;
}
Now run the nginx
container again but with the -d
flag this time.
docker compose up --build -d nginx
Now navigate to your domain and you should find an nginx 404 page served over https.
Congratulations 🎉. You are 99% done.
Step-4.1: Proxy requests(optional)
Next step would be telling nginx to proxy pass. See the bold items below
upstream default_app {
server webserver:8000;
}server {
listen 80;
listen [::]:80;
server_name test.leangaurav.dev;
location / {
return 301 https://$host$request_uri;
} location ~ /.well-known/acme-challenge {
allow all;
root /tmp/acme_challenge;
}
}server {
listen 443 ssl;
listen [::]:443 ssl http2;
server_name test.leangaurav.dev;
ssl_certificate /etc/letsencrypt/live/test.leangaurav.dev/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/test.leangaurav.dev/privkey.pem;
location / {
proxy_pass http://default_app;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
} location /static/ {
alias /static/;
}
}
The first block sets up a set of upstream request handler by the tag default_app
. One interesting thing here is the webserver:8000
part. Since I’m running nginx inside a container, I wanted it to access another container which actually handles any incoming request. The container was tagged with the name webserver
and it handles requests on port 8000
.
For this to work, you need to ensure that both the nginx
and the webserver
containers are part of the same network. There’s another way to handle this as well. If your other container had it’s ports mapped to host or specify a port mapping, then you can use localhost:8000
as well.
The second server now contains a location block specifying the proxy settings. And also another separate location block for handing static.
All these things can be customized according to your needs. What I have provided is just an example.
Step-5: Setup certificate auto-renewal
One last and important step is to setup the cron-job to renew the ssl certificates.
The cron job should do two things:
- Run the letsencrypt certbot container.
docker compose -f <absolute path to folder>/docker-compose-le.yaml up
- Tell nginx server to reload newly installed certificates
docker exec -it nginx-service nginx -s reload
Couple of things to notice:
- You need to give an absolute path to the docker-compose-le.yaml.
- The container name is
nginx-service
which is the same as what was used indocker-compose.yaml
If you did any changes there, use the same name here.
The command to launch the letsencrypt container in the docker-compose-le.yaml
had the flag — keep-until-expiring
. This will renew the certificates only when they are about to expire. Letsencrypt issues certificates valid for 90 days and certificates are marked for renewal 30 days before expiry.
We will run the crontab on 1st of every month at 12:00 GMT. We want nginx to reload only if renewal happened. I haven’t yet spent time to get that working, so we’ll just reload irrespective.
For the crontab, we will combine the two commands into one. Run the crontab -e
command and paste the below thing (adding the correct path). Save and exit.
0 12 1 * * docker compose -f <absolute path to folder>/docker-compose-le.yaml up && docker exec -it nginx-service nginx -s reload
To understand the cron tab, check these:
- https://crontab.guru/#0_0_*_*_0
- https://stackoverflow.com/questions/350047/how-to-instruct-cron-to-execute-a-job-every-second-week
Optinally you can pipe the crontab logs to a specific file for debugging by adding this >> <path to log file> 2>&1
to thee
So complete crontab can look like this
0 12 1 * * docker compose -f /home/ubuntu/nginx_https_docker/docker-compose-le.yaml up && docker exec -it nginx-service nginx -s reload >> /home/ubuntu/crontab.log 2>&1
Thats it you are all set !!
Consider hitting 👏. You can find me on Linkedin and say 👋. Saved a few days of effort ? Buy me a couple coffee ☕
Common Issues
Re-directed too many times when using cloudflare DNS.
While installing certificate in steps 1 and 2 the cloudflare DNS setting should be set to Flexible
.
After installation it can be either Full
or Full(strict)
, otherwise you get a too many redirect error.
Nginx home page or an nginx 404 error page doesn’t show up.
There can be multiple reasons. Go step by steps:
- check if the server IP address is public or not and also ports 80 and 443 should be open.
Test this by running the nginx server with default config and then visitinghttp://<server ip>:80
. This should display the nginx default home page
OR
Just run a simple python file server on port 80 usingsudo python3 -m http.server 80
and then accesshttp://<server ip>:80
. Sudo is needed to bind to port 80. - Make sure you have already added an
A record
do your DNS provider pointing to the server where you are setting upnginx
. - Check firewall of the instance. For example this helped solve my problem on Oracle cloud
Basically ran following commands
sudo iptables -I INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo netfilter-persistent save
Exec Format error
⠋ letsencrypt The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
Attaching to certbot-service
certbot-service | exec /bin/sh: exec format error
certbot-service exited with code 1
Replace the certbot image tag in docker-compose-le.yaml
with platform specific build like:
image: certbot/certbot:arm64v8-v1.32.2
Unable to connect to upstream service from nginx with network_mode host
If you use host network mode in nginx container and wish to connect to another service from nginx, then use 127.0.0.1
instead of localhost
upstream default_app {
server 127.0.0.1:8000;
}
See this answer for more details.