Why Apache webserver is refusing to serve symlinks to /tmp

TL;DR:

• Apache web server may be running with its own, ~empty /tmp directory due to PrivateTmp=true in the (e.g., Debian and derivatives) apache2.service systemd unit file.

• Broken by this was (a non-default configuration of) lacme ACME client, which can be made to work again by running it under systemd, too! (with JoinsNamespaceOf=apache2.service)

• When reading the full article, you might learn about Debian code search and Debian Sources website, and get example systemd service/timer unit files to run lacme with. It should be full of explanations and reference links, too!

Introduction

Hi, today I'm going to fill in on a problem I could find plenty on Google searches, but no solution so far¹: Apache webserver, when given a symlink, e.g., in the document root configured with Options FollowSymLinks, pointing to a file in /tmp, it'll often just refuse to serve it (HTTP status code 403 Forbidden), with a cryptic message in the VHost's error log:

AH00037: Symbolic link not allowed or link target not accessible: /srv/www/foo/testfile.txt

Normally ...

Normally, one would now have a look at the webserver source code to learn what's really going on. (For this, you can use the Debian Code Search, though you'll have to drop the AH part from the error message identifier, and don't expect the error message format string to be presented as a single line in the source code (it isn't, but uses preprocessor merging of adjacent string literals, instead...) The resulting search would be package:apache2 00037, resulting in a hit in apache2_2.4.43-1/server/request.c, at this time. You'd then go to Debian Sources, enter apache2 as package to search, click on the resulting single apache2 result link, then on the exact version you're running, ... et voila, -> server -> request.c, and let your web browser search for the 00037, again.

..., but in this case ...

But, as stated before, this is in this case useless as the real problem can't be found by looking at the apache2 source code.) In this case (-- note that, in other cases, it can also be an SELinux issue, but in my case wasn't --), the problem rather lies with the apache2.service systemd unit file. In Debian (Debian 9 "stretch" or newer), it contains a PrivateTmp=true, which instructs systemd to start the webserver with its own, initially empty set of /tmp, /var/tmp directories; so, as far as Apache is concerned, the symlink target simply isn't there, as its /tmp is ~empty! It's then refusing to serve a dangling symlink, which (a bit strangely) fulfills the "or link target not accessible" part of the logged error message. (Why doesn't it possibly give a 404 Not Found, in this case? ... Could have saved hours of debugging.)

Workaround

As my initial problem was to get the Perl-based, minimal Let's Encrypt "ACME" client lacme running again, (which in my setup created symlinks from a /.well-known backing directory to a temporary directory in /tmp to serve the prove of domain ownership to LE, which of course just failed with 403 Forbidden for them all the time...) Well, the simplest solution was to run lacme under systemd, too, using JoinsNamespaceOf=apache2.service!

Resulting systemd unit files for lacme ACME client

This is what the systemd unit file to run lacme newOrder as looks like: (To be placed into /etc/systemd/system/lacme-newOrder.service, then the usual systemctl daemon-reload; systemctl start ... to run it once. Note there is no [Install] section, it'll just be started from/via associated timer unit, see below.)

[Unit]
Description=lacme Let's Encrypt client new order
After=apache2.service
Requisite=apache2.service
JoinsNamespaceOf=apache2.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/lacme newOrder
PrivateTmp=true

View raw systemd service unit file

For completeness, here's also the timer unit needed for cron-like operation: (To be placed into /etc/systemd/system/lacme-newOrder.timer, then the usual systemctl daemon-reload, this time also systemctl enable --now lacme-newOrder.timer, which here works as there is an [Install] section, for registering under timers.target. If all went well, use systemctl list-timers to verify scheduling.)

[Unit]
Description=Daily run of lacme Let's Encrypt client new order
# Based on apt-daily-upgrade.timer

[Timer]
# Previous cron.daily run:
#OnCalendar=*-*-* 6:44
# Now, with leeway:
OnCalendar=*-*-* 5:44
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target

View raw systemd timer unit file

(Created Tue 28 Jul 2020 02:34:50 CEST, published around Tue 28 Jul 2020 02:52:00 CEST; filed/ordered under when the idea was made.)

Restarting provisioning on FritzBox modem-router

Today¹ was the day my Internet connection's upgrade should land, but I spent the day looking at the walled garden of my Internet Service Provider (ISP), instead.

The CWMP auto-configuration & provisioning for some reason didn't work, so I was trapped in the Internet-like environment my ISP set up for contacting the CWMP provisioning server and/or accessing some of the ISP-hosted services. The problem was, I didn't know it, and initially believed it to be some testing environment where my line had been put to measure its stability after the bandwidth upgrade. After several hours of staring and waiting, calling the ISP's hotline seemed a saner solution than to still try and wait. The person there took me through assessing the situation and put me on my way to restart the modem-router's auto-provisioning process; so that's what this blog post is about.

Auto-provisioning

In modern practice, telling an AVM FritzBox DSL modem-router what ISP one is using should be enough to get it going; no typing in secret usernames and passwords anymore (for the Internet line, the telephone line, ...), it'll just pull all the necessary(? :V) settings from a CWMP/TR-069 "ACS" ("auto configuration server"). A friend even told me he didn't even need to tell the device what ISP he was using, but that may be because they may be pre-provisioning/pre-configuring it, as it's specially branded for that ISP anyway.

The process of pulling the configuration from the respective ACS on the FritzBox (FRITZ!Box 7430 with firmware 07.12) seems to be: Logging in into the web interface, selecting the right page; then the ISP (or ISP category + ISP) has to be chosen and we run for apply. (form button) ... This will take some seconds for the box to apply the change, then up to some minutes while the ACS settings/directions are retrieved & applied.

After that, the previous standard/default username/password, with which the walled garden / ACS can be reached, will be replaced with a personalized username/password (of which only the username can be seen).

Failing the provisioning step

As I was contacting my FritzBox via https, and it keeps its self-signed certificate changing and changing again whenever the box gets a new IP address, the auto-provisioning didn't finish because the web app running in my browser couldn't contact the FritzBox anymore after it changed its certificate again and I needed to ack it. (Or at least that's what I'm telling myself. Maybe I just wasn't patient enough and skipped the final part of the provisioning early, just assuming it would hang on the cert question again. Nevertheless,) when it fails, it stays failed and doesn't ever retry the initial provisioning.

It always stays with the default username/password, and keeps me in the ISP's walled garden.

Restarting provisioning

Now comes in the ISP's hotline. They told me to temporarily change to some (possibly completely unrelated) different ISP profile in the FritzBox's list of providers, "complete" that setup by running apply (form button); then, when that (obviously) failed, go back to the "right" ISP setting and hit apply again. This, then, starts the initial auto-provisioning process anew, without needing the otherwise often-mentioned factory reset.

When I was (on an iteration later ...) running with http (not https, due to the cert problem described above,) and the box' IP address (not the beautiful hostname from my internal DNS, as it would reject the hostname as a protection against DNS rebinding attacks, and changing the ISP always resets the exemption configured in another part of the web interface), I finally got the initial auto-provisioning to complete and got my shiny new username/password!

(Created Mon 27 Jul 2020 23:30:24 CEST, published around Tue 28 Jul 2020 00:38:00 CEST.)