Deploying Gitlab with Podman
- Added how to bind to privileged port with rootless container. Thanks Baptiste for the suggestion!
For my private projects I run a self-hosted GitLab instance deployed with the official Community Edition Docker image. In addition to Git repository management GitLab comes packed with a lot of features such as Continuous Integration/Deployment, Wikis, Kubernetes cluster integration and much more. Those looking for a minimal Git solution should probably look elsewhere. I have heard a lot of positive about Gitea.
Feature heavy as it is, and realistically speaking, GitLab is a bit too excessive for my own use cases. I could easily replace it with managed solutions such as GitLab.com, GitHub or Azure DevOps. But where’s the fun in that?
Table of contents
- GitLab CE with Podman
- Prepare the host
- GitLab Configuration
- podman run
- Manage container lifecyle with systemd
GitLab CE with Podman
When it comes to deploying and running the containerized version of GitLab I use Podman as the container engine. Podman is really evolving into a great alternative to Docker. One of the top benefits is that it runs daemon-less which allows for true rootless mode out of the box.
Please note that in the following example I do in fact run the container as root. The only reason for this is that I have not (yet) setup a proxy in front of GitLab that would eliminate the need to expose privileged ports from the container, which requires it to run as root.
Update: Bind to privileged port
More recent Linux kernel versions does in fact come with the capability to lower the unpriviliged port range. For example to bind a rootless container to port 80 you can either add
/etc/sysctl.conf or run command:
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
VM or bare-metal running Linux/amd64 (there’s also unofficial images for arm64, which I have not tried, but they should in theory work fine even on Raspberry Pi’s. Especially Pi4 4GB or 8GB versions.).
2GB — 4GB available RAM. The official recommendation is 4GB but I have managed to run it with as low as 2GB, although on some occasions Out-Of-Memory related restarts would be triggered. The sweet-spot that has been working well for me is 3GB total RAM to the host VM with 2.5GB dedicated to the container.
Podman (latest version 2.2.1 at the time of writing).
Prepare the host
GitLab requires a location on the host system to store persistent data in-between container restarts and version upgrades. In this example I use
/opt/gitlab as the base directory.
- First export the base directory path to environment variable as it will be used in further configuration.
- Create the base directory
mkdir -p $GITLAB_HOME
- Now create sub-directories for storing application data, configuration and log files.
mkdir $GITLAB_HOME/data $GITLAB_HOME/config $GITLAB_HOME/logs
The base directory and sub-directories should now have the following structure:
gitlab ├── config └── data └── log
There are a lot of parameters available to configure in GitLab. For a minimal working configuration only external URL and SSH port (if a custom port other than 22 is used) needs to be defined in order to have GitLab generate correct Git URLs. External URL is the IP or FQDN that GitLab should be reachable with. For example if GitLab should only be reachable on the internal network the host IP can be used.
Configuration can be read from environment variables or file
$GITLAB_HOME/config/gitlab.rb. I use the latter approach and template the configuration in
gitlab.rb file and keep it version controlled.
- Define external URL and custom SSH port and create
export GITLAB_EXTERNAL_URL=https://192.168.1.10 GITLAB_CUSTOM_SSH=2222 cat << EOF > $GITLAB_HOME/config/gitlab.rb external_url '$GITLAB_EXTERNAL_URL' gitlab_rails['gitlab_shell_ssh_port'] = $GITLAB_CUSTOM_SSH ## Uncomment to disable Let's encrypt # letsencrypt['enable'] = false EOF
See below for an example of a more extensive configuration where outbound SMTP, custom SSL certificates, disabling not used features and some performance tweaks are included:
View extended gitlab.rb
external_url '$GITLAB_EXTERNAL_URL' gitlab_rails['gitlab_shell_ssh_port'] = $GITLAB_CUSTOM_SSH gitlab_rails['time_zone'] = 'Europe/Stockholm' gitlab_rails['incoming_email_enabled'] = false gitlab_rails['initial_root_password'] = '$GITLAB_ROOT_PASSWORD' gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = '$EXTERNAL_SMTP_SERVER' gitlab_rails['smtp_port'] = $EXTERNAL_SMTP_PORT gitlab_rails['smtp_user_name'] = '$EXTERNAL_SMTP_USERNAME' gitlab_rails['smtp_password'] = '$EXTERNAL_SMTP_PASSWORD' gitlab_rails['smtp_domain'] = '$EXTERNAL_SMTP_SERVER' gitlab_rails['smtp_authentication'] = 'login' gitlab_rails['smtp_tls'] = true gitlab_rails['gitlab_email_from'] = '$EXTERNAL_SMTP_ADDRESS' registry['enable'] = false letsencrypt['enable'] = false nginx['ssl_certificate'] = '/etc/gitlab/ssl/server.crt' nginx['ssl_certificate_key'] = '/etc/gitlab/ssl/server.key' ## Optimizations puma['worker_processes'] = 2 puma['per_worker_max_memory_mb'] = 250 sidekiq['concurrency'] = 9 postgresql['shared_buffers'] = "256MB" ## Disable Grafana & Prometheus for now prometheus_monitoring['enable'] = false grafana['enable'] = false
Note: By default GitLab will try to issue a certificate via Let’s Encrypt Staging CA. GitLab will throw an error during startup if an IP in the private RFC1918-block is used as external URL:
Error executing action `create` on resource 'acme_certificate[staging]'
To resolve either use a FQDN in
external_url or add
letsencrypt['enable'] = false in
gitlab.rb to disable Let’s Encrypt
- Initial run of GitLab container:
sudo podman run -d --name gitlab \ --publish 443:443 --publish 80:80 --publish $GITLAB_CUSTOM_SSH:22 \ --memory=2560m \ --hostname $GITLAB_EXTERNAL_URL \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ gitlab/gitlab-ce:13.7.4-ce.0
At the time of writing the latest version of GitLab CE was 13.7.4-ce.0 — feel free to change this to a different version
- Tail the log with:
podman logs -f gitlab
First time starting up will take around 5-10 minutes. Consecutive container runs will start up much more quickly.
When a similar message is displayed startup is completed:
Running handlers: Running handlers complete Chef Infra Client finished, 345/1429 resources updated in 02 minutes 14 seconds gitlab Reconfigured!
If container startup fails with:
cat: /var/opt/gitlab/gitlab-rails/VERSION: Permission denied Installing gitlab.rb config... cp: failed to access '/etc/gitlab/gitlab.rb': Permission denied
SELinux might be enabled on the host OS. To grant the container permissions to mounted volumes append
:Z to the container directory in the run command:
sudo podman run -d --name gitlab \ --publish 443:443 --publish 80:80 \ --publish $GITLAB_CUSTOM_SSH:22 \ --memory=2560m \ --hostname $GITLAB_EXTERNAL_URL \ --volume $GITLAB_HOME/config:/etc/gitlab:Z \ --volume $GITLAB_HOME/logs:/var/log/gitlab:Z \ --volume $GITLAB_HOME/data:/var/opt/gitlab:Z \ gitlab/gitlab-ce:13.7.4-ce.0
https://192.168.1.10(IP/FQDN that was used with
$GITLAB_EXTERNAL_URL). Ignore the TLS Insecure error and verify that the login page is visible.
While there take the opportunity to set a default root password for the instance.
Manage container lifecyle with systemd
So far the GitLab instance is running within the container but it will not start up automatically if the host is rebooted or perform clean-up of previous container references if stopped manually.
That responsibility can be delegated to systemd by defining a service for the GitLab container.
Podman comes with a neat feature that generates the required systemd unit file.
- Generate the unit file:
podman generate systemd gitlab \ --files --restart-policy=always \ --new --name --time 60
For definitions of the arguments reference Podman offical documentation
- Copy the generated unit file to /etc/systemd/system:
sudo cp container-gitlab.service /etc/systemd/system
- By default
KillMode=noneis specified in the unit file. This is being deprecated by systemd. Let’s fix that by replacing it with
sudo vim /etc/systemd/system/container-gitlab.service
Set the value to 70 seconds to ensure it does not override
ExecStop timeout value that is set to 60 seconds.
# container-gitlab.service # autogenerated by Podman 2.2.1 # Tue Jan 19 21:21:47 UTC 2021 [Unit] Description=Podman container-gitlab.service Documentation=man:podman-generate-systemd(1) Wants=network.target After=network-online.target [Service] Environment=PODMAN_SYSTEMD_UNIT=%n Restart=always ExecStartPre=/bin/rm -f %t/container-gitlab.pid %t/container-gitlab.ctr-id ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-gitlab.pid --cidfile %t/container-gitlab.ctr-id --cgroups=no-conmon --replace -d --name gitlab --publish 443:443 --publish 80:80 --publish 2222:22 --memory=2560m --hostname https://192.168.1.10 --volume /opt/gitlab/config:/etc/gitlab:Z --volume /opt/gitlab/logs:/var/log/gitlab:Z --volume /opt/gitlab/data:/var/opt/gitlab:Z gitlab/gitlab-ce:13.7.4-ce.0 ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-gitlab.ctr-id -t 60 ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-gitlab.ctr-id PIDFile=%t/container-gitlab.pid - KillMode=none + TimeoutStopSec=70 Type=forking [Install] WantedBy=multi-user.target default.target
- Now enable and start the service:
systemctl enable --now container-gitlab.service
- Verify that the service started:
systemctl status container-gitlab.service
If everything went well it should be active (running):
● container-gitlab.service - Podman container-gitlab.service Loaded: loaded (/etc/systemd/system/container-gitlab.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2021-01-19 22:04:29 UTC; 5min ago
GitLab has now successfully been deployed with Podman!
Questions? Errors spotted? Feel free to
comment below contact me!