How we generate and renew SSL certs for arbitrary custom domains using LetsEncrypt!

Last month we decided to do something interesting and allow Hashnode users to publish articles under a free hashnode.dev subdomain or any custom domain of their choice. It required a bit of work on our part, but the most challenging task was enabling SSL for arbitrary custom domains. Thankfully, we have LetsEncrypt which lets you generate SSL for any domain for free. However, it's still a challenge to generate and serve the certs on demand using nginx. While building hashnode.dev we faced that challenge. In the spirit of blogging, I am going to share the steps we took to overcome this.

For those of you who are unaware, hashnode.dev lets developers start a personal blog for free. All you need is a hashnode.com account. Fun fact: this blog is powered by hashnode.dev. You can find more details here.

So, what options did we really have? Let me outline those here.

Option 1

Our backend is a Node.js server. So, I thought of using something like node-greenlock to generate and serve SSL certs. But that would require us to expose our Node.js server on port 80/443 and we will lose the benefits of nignx. I didn't want that. So, I kept digging more.

Option 2

I kept researching for couple of more days, but every solution was complex and demanded time. Since we were done with all the aspects of hashnode.dev (except SSL part), I decided not to delay the launch of alpha preview. Then I had an idea -- Why not create a Amazon Cloudfront distribution temporarily and use it as a reverse proxy to <username>.hashnode.dev? AWS offers free SSL certificates as long as you want to use them with CF distributions. This was certainly not a long term solution because:

  • Number of cloudfront distributions/certs are limited per account. You have to request AWS to increase the limit periodically.
  • Cost of the CDN usage.
  • The process was complex. Our users had to validate ownership of their domain first which took several hours in some cases. Then they had to update their CNAME to CF distribution URL. But they couldn't use it at root level of their domain because of CNAME flattening issue. Only those who used a registrar that supported CNAME flattening (e.g. Cloudflare) were able add the record at root.

Clearly, I didn't want the above solution. I wanted something that just works with minimal efforts from customers. I anyway went ahead with this solution and decided to keep looking for a better solution.

Option 3

This is the best part. I came across OpenResty which is apparently one of the most used web servers. I felt like I was living under a rock. As I researched further I realized that OpenResty bundles nginx core and lets you embed logic into your server using Lua. 😍

To my surprise I also discovered a module called lua-resty-auto-ssl which generates and renews SSL certs automatically inside OpenResty. I was overjoyed. In the coming few days my teammate Aravind did a small PoC and confirmed that it's indeed working. So, I decided to give this a try.

Step 1

The first step was obviously installing OpenResty. I followed this simple guide from DigitalOcean and successfully installed OpenResty.

I did only one modification though which is enabling http2 during the configuration:

./configure -j2 --with-pcre-jit --with-ipv6 --with-http_v2_module

Step 2

Installed LuaRocks package manager so that I could install lua-resty-auto-ssl package. I am on latest Ubuntu.

sudo apt-get install unzip
wget http://luarocks.org/releases/luarocks-2.0.13.tar.gz
tar -xzvf luarocks-2.0.13.tar.gz
cd luarocks-2.0.13/
./configure --prefix=/usr/local/openresty/luajit \
    --with-lua=/usr/local/openresty/luajit/ \
    --lua-suffix=jit \
    --with-lua-include=/usr/local/openresty/luajit/include/luajit-2.1
make
sudo make install

Then I installed lua-resty-auto-ssl and set up the directory:

sudo luarocks install lua-resty-auto-ssl
sudo mkdir /etc/resty-auto-ssl
sudo chown sandeep /etc/resty-auto-ssl

Step 3

Then I added configuration for our web server. It goes into /usr/local/openresty/nginx/conf/nginx.conf:

user sandeep www;
events {
  worker_connections 1024;
}

http {
  lua_shared_dict auto_ssl 1m;
  lua_shared_dict auto_ssl_settings 64k;
  resolver 8.8.8.8 ipv6=off;

  init_by_lua_block {
    auto_ssl = (require "resty.auto-ssl").new()
    auto_ssl:set("allow_domain", function(domain)
      return true // Allow all the domains
    end)
    auto_ssl:init()
  }

  init_worker_by_lua_block {
    auto_ssl:init_worker()
  }

  server {
    listen 443 ssl http2;
    ssl_certificate_by_lua_block {
      auto_ssl:ssl_certificate()
    }
    ssl_certificate /etc/letsencrypt/live/hashnode.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/hashnode.com/privkey.pem;

    location / {
      proxy_pass http://localhost:3000;
      # Other configs  
    }
  }

  server {
    listen 80;
    location /.well-known/acme-challenge/ {
      content_by_lua_block {
        auto_ssl:challenge_server()
      }
    }
  }

  server {
    listen 127.0.0.1:8999;
    client_body_buffer_size 128k;
    client_max_body_size 128k;

    location / {
      content_by_lua_block {
        auto_ssl:hook_server()
      }
    }
  }
}

SSL cert is provided by ssl_certificate_by_lua_block. We still need ssl_certificate and ssl_certificate_key directives to serve SSL certs for domains *.hashnode.dev and *.hashnode.com. We exclude these two cases and non-whitelisted domains in init_by_lua_block as following:

init_by_lua_block {
    auto_ssl = (require "resty.auto-ssl").new()
    auto_ssl:set("allow_domain", function(domain)

       -- we don't want to generate SSL for hashnode.com
       if domain == "hashnode.com" then
          return false
       end

       -- we don't want to generate SSL for *.hashnode.dev as well
       if string.find(domain, ".hashnode.dev") == nil then 
         local http = require("resty.http")
         local httpc = http.new()

         httpc:set_timeout(3000)

         local uri = "<API_ON_HASHNODE>"

         local res, err = httpc:request_uri(uri, {
           ssl_verify = false,
           method = "GET"
         })

         if not res then
           return false
         end

         if res.status == 200 then
           return true
         end

         if res.status == 404 then
           return false
         end
       else
         return false
       end
    end)
   --auto_ssl:set("ca", "https://acme-staging.api.letsencrypt.org/directory")
   auto_ssl:init()
  }

This simple solution pings our endpoint to check if a domain is whitelisted for SSL. If not, it just skips the process.

Make sure to use auto_ssl:set("ca", "https://acme-staging.api.letsencrypt.org/directory") > during development otherwise your account may hit rate limiting issues.

Step 4

With a few other configurations, I could make this work successfully. Now we simply ask our users to add an A record that points to our IP and everything just works. If SSL cert doesn't exist for a domain, it's generated and served automatically when you request the URL for the first time. Naturally, there is a latency of ~10s for the very first request. Next time onwards it's blazing fast since the certs are cached. Since LE certs are valid for 90 days, renewals happen automatically if the expiry date is < 30 days.

This is the message I sent to our alpha testers after I deployed the change:

Screenshot 2019-03-21 at 1.01.49 AM.png


I hope this guide will help you if you are going to create a multi-tenant SaaS app that relies on SSL. Special thanks to LetsEncrypt, OpenResty and auto-ssl package. It wouldn't have been possible without these tools/services.

If you are a developer and are looking to start a personal blog, please show your interest by visiting hashnode.dev.

Comments (8)

Add a comment
Richard Uie's photo

Dude, this is not merely a wonderful story of ingenuity, but a master class in how-to-investigate-and-optimize-design-for-all-stakeholders. Huzzah, HUZZAH, HUZZAH (three cheers)!

sivaram's photo

We are also using the same lua-resty-auto-ssl it works great:). By the way, there are few rate limits imposed by letsencrypt like you can create 50 certificates per week. Here you can check it out letsencrypt.org/docs/rate-limits. Just curious How you gonna tackle this thing?

Show all replies
Amit Lamba's photo

Oh, I just read the LE rate limit link above. So, if I understand correctly, 300 new orders (300 registered domains) per account (hashnode account) per 3 hours translates to every 3 hours you can create 300 custom domain registration SSL certs.

Vito Botta's photo

Hi, very useful. I am looking for some alternatives I could use with Kubernetes in an automated way, but haven't had much success so far. Perhaps I could have the app create an ingress whenever a user adds a custom domain, and let cert-manager handle the certificate for me, but I am not sure yet if this is the best approach with Kubernetes. Anyway, what I wanted to ask you is if you have run into any limits not just with LetsEncrypt, but with the OpenResty solution. How many certificates can be handled with the Lua thing and by OpenResty/Nginx itself? Would this scale to thousands or 100s of thousands users if the app is successful? If scalability is not a huge issue with this solution I might try to adapt it to Kubernetes by using SSL passthrough from ingress controller to a customised instance of OpenResty. Thanks in advance!

sivaram's photo

How do you guys force a redirect to https for an arbitrary domain? Am stuck in it Facing too many redirects

  server {
    listen 443 ssl;
    ssl_certificate_by_lua_block {
      auto_ssl:ssl_certificate()
    }

  location / {

        proxy_pass http://localhost:4444;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
        #try_files $uri $uri/ /;        
    }
         location @rewrites {
         rewrite ^(.+)$ / last;
  }
    ssl_certificate /etc/letsencrypt/live/mydomain/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mydomain/privkey.pem;
  }
  server {
    listen 80;
    location /.well-known/acme-challenge/ {
      content_by_lua_block {
        auto_ssl:challenge_server()
      }
    }
      location / {
         return 301 https://$host$request_uri; 
   }    
}
Ramiro Berrelleza's photo

If you're running in Kubernetes (or at some point decide to move this), Bitnami recently released a runtime that automates this by using NGINX's Ingress Controller, External-DNS and Cert-Manager.