findmnt

One of my customers uses cPanel to administer their internet facing servers.

It has an interesting virtual filesystem setup for sandboxing user accounts: cPanel creates, per user, a new root filesystem for that user, and then chroot into that before running user code (shells, php, ...).

To create that file system, cPanel uses bind mounts to make the root-jail file system look very much like the real root file system.

bind mounts are a thing that appeared after the 1997-era of me spending a lot of time learning Linux. (if it was invented post 2000, lol I've never heard of it)

In the intervening years, isolation techniques like this have been becoming more mainstream - for example, my main use of docker (via my tool cue) has been to prepare and use different root file systems.

Anyway, back to cPanel. I was trying to figure out how this virtual filesystem was constructed. Bind mounts don't appear in the output of mount or df or /proc/mounts with all the information I wanted: the mount just shows are being from the same device that its target is, without saying where that target is.

For example, I can see that /home/virtfs/x/usr/sbin goes to somewhere in the filesystem on s_os-lv_root but not where. (I can guess it's /usr/sbin in this case).

/dev/mapper/vg_os-lv_root   50G   21G   30G  41% /home/virtfs/x/usr/sbin
/dev/mapper/vg_os-lv_root   50G   21G   30G  41% /home/virtfs/x/var/spool
/dev/mapper/vg_os-lv_root   50G   21G   30G  41% /home/virtfs/x/etc/apache2
/dev/mapper/vg_os-lv_root   50G   21G   30G  41% /home/virtfs/y/usr/sbin

Anyway, surely this can't be the way things are??

So along comes findmnt, which gives me this info:

...
│ │ ├─/home/virtfs/x/usr/sbin     /dev/mapper/vg_os-lv_root[/usr/sbin]
      xfs         ro,relatime,seclabel,attr2,inode64,sunit=512,swidth=512,usrquota
...

... which tells me that yes that really is mounted from /usr/sbin.

Anyway, a nice new command to discover, around since only util-linux v2.18 in mid 2010.

fcgi, haskell, cpanel, php, drupal

I played with fastcgi, which is like CGI but doesn't have to spawn a new process each time.

The initial motivation for this was a server which has a bunch of drupal websites. It was previously running in plain CGI mode, which forks a PHP process for every page request (about 15 spawns per second on this server), with each site's PHP running under a different user account. (The other mode we've tried with this is using mod_php, which runs things much faster but doesn't provide as much isolation between web sites as using CGI as everything runs as the www-data unix user, rather than as a per-site user).

I thought I'd have to do more compiling, but it turns out fastcgi support for both apache and for PHP was already available. On my dev server I needed to apt-get the fastcgi apache module; on the production server, which uses cpanel, fastcgi support was already installed and switching it on was a single mouse click.

Here's a plot of the server CPU load before and after the switch:

There's a clearly visible daily cycle, using up almost 8 cores worth of CPU before the change. At the end of the 30th, I switched on fastcgi, and woo, the load drops right down and stays down. That's just what I wanted.

Reading more, cpanel disrecommends using fastcgi, and recommends somethign else - ruid2 - which looks like it does something similar but different. That recommendation seems mostly because fastcgi has a lot of tweakables that are hard to get right. see this thread.

caveats

I discovered a few interesting things during deployment:

Firstly, a potential attack on directories that have the ExecCGI option enabled - this is discussed in the context of the nginx web server here.

Another was a bug with a specific version of mod_fcgid and the specific configuration I set up, which resulted in a new PHP process being spawned for every page request, and then staying resident (!). Other people have experienced this and it was straightforward to tweak it so that it didn't happen.

haskell

I have a few apps for my own use written in Haskell, and one (a photo ranking app) struggles when called through the regular CGI interface, due to loading the vote/photo database each time. I've considered putting that into snap, a haskell framework, but it seemed interesting to see if I could get fastcgi running under Haskell.

apt-get install libcfgi-dev; cabal install fcgi got me the modules installed. I had some trouble running the hello-world app here

that came down to me not compiling with the -threaded option.

(I also tried the haskell direct-fastcgi module, but the home page for it is gone, and there is no example code so I rapidly gave up)

barwen.ch

I made an fcgi-bin directory available to all barwen.ch users, running FastCGI code under user accounts. There isn't much CGI going on on barwen.ch, but it seemed easy enough to deploy and make available, and is one more feature for the feature list.

suPHP vs CVE-2012-1823

I needed to investigate CVE-2012-1823 for a few sites that I help look after.

They all use suPHP (some of them via cPanel, some directly configured).

I couldn't find anything in Google about whether CVE-2012-1823 affects suPHP - they all talk about php-cgi, and suPHP does something very similar, but with a bit more functionality.

As far as I can tell, the exploit comes specifically from CGI handling; and relates to how a URL turns into an invocation of PHP.

From looking at the suPHP source code, it looks like that exploit path is not available. The arguments to pass to PHP seem to be formed totally differently in suPHP compared to a CGI execution.

I'd love to hear anyone else's opinion though...

https in cpanel

working with someone who has a cpanel server. they want https on it. cpanel doesn't do that by default. google doesn't reveal much in the way of tutorials for this, so here's a note for people to find.

1. generate a key pair and certificate using the Generate a SSL Certificate & Signing Request page. Copy the certificate onto your clipboard.
2. go to the Install a SSL Certificate and Setup the Domain page. Paste in the certificate. click fetch on the key text field and it should populate that field for you. Set the username to nobody so that all users can use this key pair.
3. When you save that page, apache will reload and you'll get https service on port 443, with a self-signed certificate (and so with consequent certificate mismatch error messages). But your existing domains won't work on that server - they'll go to the default cpanel parking page - cpanel only configures its virtual hosts on port 80... grr
4. So next I made an apache mod_rewrite rule in the VirtualHost directive for the port 443 virtual server. That causes all the internal sites appear on port 443.

       RewriteEngine on
       RewriteRule   ^(.+)          http://%{HTTP_HOST}$1 [P]
   

   That's an awkward hack to have to add to cpanel's generated config, but it seems to work (modulo invalid certificate warnings that all users ignore anyway)...

There's also a hole in the way that that rewrite rule is implemented: with a custom http client, you can probably make this server act as an arbitrary proxy for you, depending on mod_proxy configuration.