Multiple service instances with NixOS Containers

You have a NixOS node. You want to run multiple instances of a service on this machine. You noticed the service module definition doesn't support this use case.

If you are reading this, you share the same pain I once felt. I have been bit by the Forgejo service module.

Yes, this is something that should be designed at the module definition level. However, this is a volunteer-based project and people have limited time to contribute and support a big module change as this on currently working modules1.

Well then, good news! You most definitely can run multiple instances with the help of NixOS Containers.

In this post I'm going to setup two or more forgejo instances, but the idea is similar to a multitude of services. Even more so if they are also web applications.

First, the requirements:

  • The instances should share the same database instance (backups are easier)
  • The instances are publicly accessable through HTTPS

For brevity, I won't address secret management nor individual settings for each individual service.

Let's get started by declaring a set of configurations for each instance:

instances = {
  community = {
    domain = "community.example.com";
  };
  social = {
    domain = "social.example.com";
  };
};

Then let's enrich the instances with auto generated IP addresses. This way we don't need to manually set them:

instancesList = lib.attrNames instances;
instancesWithAddr = lib.listToAttrs (lib.imap0 (i: name: {
  inherit name;
  value = instances.${name} // {
    hostAddress = "192.168.100.1";
    localAddress = "192.168.100.${toString (11 + i)}";
  };
}) instancesList);

And to finish the booststrapping, let's create a function that generates the containers for us:

# host's postgresql socket we'll mount on the containers
pgSocketDir = config.services.postgresql.settings.unix_socket_directories or "/run/postgresql";

mkContainer = name: cfg: {
  privateNetwork = true;
  inherit (cfg) hostAddress localAddress;

  bindMounts.${pgSocketDir} = {
    hostPath = pgSocketDir;
    isReadOnly = false;
  };

  config = { pkgs, ... }: {
    services.forgejo = {
      enable = true;
      settings = lib.recursiveUpdate {
        server = {
          DOMAIN = cfg.domain;
          ROOT_URL = "https://${cfg.domain}/";
          HTTP_ADDR = "0.0.0.0";
        };
      } (cfg.settings or {});
      database = {
        type = "postgres";
        socket = pgSocketDir;
        inherit name;
        user = name;
      };
    };
    networking.firewall.allowedTCPPorts = [ 3000 ];
  };
};

Now we just need to configure our host leveraging the values we defined above:

services.postgresql = {
  enable = true;
  ensureDatabases = lib.attrNames instances;
  ensureUsers = map (name: {
    inherit name;
    ensureDBOwnership = true;
  }) (lib.attrNames instances);
};

services.caddy = {
  enable = true;
  virtualHosts = lib.mapAttrs' (name: cfg: {
    name = cfg.domain;
    value.extraConfig = "reverse_proxy ${name}:3000";
  }) instances;
};

containers = lib.mapAttrs mkContainer instances;

  1. I might bite the bullet and try to introduce instances to the Forgejo service module. Who knows? 😎↩