Why Apache webserver is refusing to serve symlinks to /tmp

TL;DR:

• Apache web server may be running with its own, ~empty /tmp directory due to PrivateTmp=true in the (e.g., Debian and derivatives) apache2.service systemd unit file.

• Broken by this was (a non-default configuration of) lacme ACME client, which can be made to work again by running it under systemd, too! (with JoinsNamespaceOf=apache2.service)

• When reading the full article, you might learn about Debian code search and Debian Sources website, and get example systemd service/timer unit files to run lacme with. It should be full of explanations and reference links, too!

Introduction

Hi, today I'm going to fill in on a problem I could find plenty on Google searches, but no solution so far¹: Apache webserver, when given a symlink, e.g., in the document root configured with Options FollowSymLinks, pointing to a file in /tmp, it'll often just refuse to serve it (HTTP status code 403 Forbidden), with a cryptic message in the VHost's error log:

AH00037: Symbolic link not allowed or link target not accessible: /srv/www/foo/testfile.txt

Normally ...

Normally, one would now have a look at the webserver source code to learn what's really going on. (For this, you can use the Debian Code Search, though you'll have to drop the AH part from the error message identifier, and don't expect the error message format string to be presented as a single line in the source code (it isn't, but uses preprocessor merging of adjacent string literals, instead...) The resulting search would be package:apache2 00037, resulting in a hit in apache2_2.4.43-1/server/request.c, at this time. You'd then go to Debian Sources, enter apache2 as package to search, click on the resulting single apache2 result link, then on the exact version you're running, ... et voila, -> server -> request.c, and let your web browser search for the 00037, again.

..., but in this case ...

But, as stated before, this is in this case useless as the real problem can't be found by looking at the apache2 source code.) In this case (-- note that, in other cases, it can also be an SELinux issue, but in my case wasn't --), the problem rather lies with the apache2.service systemd unit file. In Debian (Debian 9 "stretch" or newer), it contains a PrivateTmp=true, which instructs systemd to start the webserver with its own, initially empty set of /tmp, /var/tmp directories; so, as far as Apache is concerned, the symlink target simply isn't there, as its /tmp is ~empty! It's then refusing to serve a dangling symlink, which (a bit strangely) fulfills the "or link target not accessible" part of the logged error message. (Why doesn't it possibly give a 404 Not Found, in this case? ... Could have saved hours of debugging.)

Workaround

As my initial problem was to get the Perl-based, minimal Let's Encrypt "ACME" client lacme running again, (which in my setup created symlinks from a /.well-known backing directory to a temporary directory in /tmp to serve the prove of domain ownership to LE, which of course just failed with 403 Forbidden for them all the time...) Well, the simplest solution was to run lacme under systemd, too, using JoinsNamespaceOf=apache2.service!

Resulting systemd unit files for lacme ACME client

This is what the systemd unit file to run lacme newOrder as looks like: (To be placed into /etc/systemd/system/lacme-newOrder.service, then the usual systemctl daemon-reload; systemctl start ... to run it once. Note there is no [Install] section, it'll just be started from/via associated timer unit, see below.)

[Unit]
Description=lacme Let's Encrypt client new order
After=apache2.service
Requisite=apache2.service
JoinsNamespaceOf=apache2.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/lacme newOrder
PrivateTmp=true

View raw systemd service unit file

For completeness, here's also the timer unit needed for cron-like operation: (To be placed into /etc/systemd/system/lacme-newOrder.timer, then the usual systemctl daemon-reload, this time also systemctl enable --now lacme-newOrder.timer, which here works as there is an [Install] section, for registering under timers.target. If all went well, use systemctl list-timers to verify scheduling.)

[Unit]
Description=Daily run of lacme Let's Encrypt client new order
# Based on apt-daily-upgrade.timer

[Timer]
# Previous cron.daily run:
#OnCalendar=*-*-* 6:44
# Now, with leeway:
OnCalendar=*-*-* 5:44
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target

View raw systemd timer unit file

(Created Tue 28 Jul 2020 02:34:50 CEST, published around Tue 28 Jul 2020 02:52:00 CEST; filed/ordered under when the idea was made.)