NFSv4: Interop, ACLs & Automount

NFSv4 has been around for a long time but it still seems a bit foreign to me. The following is a quick rundown of things I recent learned related to NFSv4 from limited experience in implementing it.


Is it possible to setup NFSv4 along side NFSv3 on the same server, serving the same volumes? Of course. However, it might not always work exactly as expected with legacy clients.

A normal /etc/exports for NFSv3/v4 interoperability might look like so:

/export         ,no_subtree_check,fsid=0)

With this configuration, we have the “virtual root” export (fsid=0), the namespace export (for mounting the whole namespace with one mount) and the individual “share” exports (for mounting individual shares, most likely with automount). The NFSv4 clients can perform mounts using the servername:/namepace syntax and the NFSv3 clients can mount the whole root, namepace or individual “shares” with servername:/export, servername:/export/namespace or servername:/export/namespace/share1.

All is well in the NFS world… or so it seems at first. It turns out that an older SunOS does not entirely like how this RHEL 6 NFS server is exporting the file systems:

hostname% cd /namespace
hostname% ls
share1     share2     share3     share4
hostname% pwd
ubcpetnxi% cd share1
ubcpetnxi% pwd

Notice the final line. I was just in /namespace then I changed into /namespace/share1. Now pwd tells me the path is only /share1. I was expecting /namespace/share1. It looks to me like the SunOS NFS client is not behaving well with how the NFS server exporting the file systems and/or how the bind mounts are setup locally on the server to map the storage into the NFSv4 “virtual root”.

Please leave a comment to if you know of a different /etc/exports and/or mount configuration that would alleviate the SunOS NFS client issues noted here!

Access Control Lists

NFSv4 defines a model for Access Control Lists (ACLs) that has similarities to that of Microsoft’s NTFS. But don’t worry about interoperability: NFSv4 translates your existing “POSIX” ACLs on ext3,ext4,xfs,etc. to NFSv4 ACLs automatically.

The main gotcha with exporting a filesystem with “POSIX” ACLs with the NFSv4 server is that the normal getfacl and setfacl tools don’t seem to work on the NFS client side! Because the NFSv4 server only presents the translated NFSv4 ACLs to the clients, the nfs4-progs package must be installed and the nfs4_getfacl and nfs4_setfacl commands used instead to view and manipulate the ACLs on NFSv4 clients.

Also, the little + at the end of the rwxrwxrwx permissions listing you can see with some variant of ls -l, the symbol that normally indicates the presence of an ACL, it simply doesn’t appear on a (Linux?) NFSv4 mount where ACLs exist. Sadness.


Automount on RHEL 6 (and clones) appears to have a bug related to bind mounts. NFSv4 exports cannot (trivially?) be mounted locally on the NFSv4 server on itself with bind mounts as is possible with NFSv3 (or lower) exports. I have read that this inability is due to the “virtual root” abstraction that NFSv4 employs. Instead, automount should be performing true NFSv4 mounts when operating locally on the server… but it doesn’t do that on CentOS 6 (and in my experience RHEL 6):


The workaround is to specify port=2049 in the NFS mount options of the automount map in use (where 2049 is the port the NFS server is listening on). This appears to cause automount to immediately attempt an NFS mount, bypassing the (failing) attempt at a bind mount.


When using Syncrepl…

Quick OpenLDAP tip boys & girls…

When using syncrepl to replicate from a master LDAP server to a slave LDAP server, always remember to configure the ACLs on the master LDAP server to allow the “sync dn” to read everything.

I know it sounds entirely obvious but today I realized that the order in which I had defined the ACLs on the master LDAP server was preventing the sync dn from reading the “userPassword” attribute and thus also preventing it from syncing it to the slave. The consequence of which was that users would not be able to authenticate against the slave server! Shit!

Of course, since everything else was syncing properly, all the NSS (lookup) stuff worked fine but anything authentication related like PAM wouldn’t work because the user bind would fail with “Invalid credentials” in /var/log/secure. It had a been some time since I tested authentication so I must never have actually tested authentication against the slave (whoops!) and thus didn’t notice until now. I know I tested lookups but testing authentication must have slipped by somehow. Grrr, testing.

Good thing I caught the problem early and it never escalated into a problem, that really could have sucked down the line.

Don’t make the same mistake I did.

POSIX Default ACLs, umask and Project Directories

I’ve recently come across a situation where the inherent design of POSIX ACLs has left me scratching my head for a solution to the problem of setting up a “project” or “group share” directory on Linux. The problem is as follows: We have several different projects or groups that desire a directory where any and every file created, copied or moved to said directory will become owned by a particular group and have group read/write permissions set automatically.

Most of the problem is solved through age-old UNIX techniques. For group ownership, all we need to do is setup the top-level directory to be owned by the “project” or “group share” group and setgid the directory:

$ mkdir project1
$ chown .projgroup project1
$ chmod g+s project1

This effectively forces every file created, moved or copied into the “project1” directory to be owned by group “projgroup”. So far, so good. The difficulties begin when we attempt to use default ACLs to enforce the permissions of any files created, moved or copied into the directory.

The POSIX ACL standard defines “default” ACLs which can be applied to a directory, which are in turn inherited by newly created/copied/moved child files and directories. While the default ACLs are inherited properly, the ACL mask when applied to files copied into the group share directory WITHOUT previous group write set prevents the files from being group writable!

$ getfacl project1
# file: project1
# owner: root
# group: projgroup

So far so good, right?

$ ls -alh test
-rw-r--r--  1 user1 user 0 Apr 23 15:10 test
$ cp test project1
$ ls -alh project1/test
-rw-r--r--+ 1 user1 projgroup 0 Apr 23 15:10 project1/test

What the… ?!?! No group write? Noooooo!

$ getfacl project1
# file: project1/test
# owner: user1
# group: projgroup
group::rw-			#effective:r--
group:projgroup:rw-		#effective:r--

And so we have the great POSIX ACL mask problem, which is by design in fact. Still looking for a complete solution that doesn’t involve global trying to force a specific umask on every account… It would be nice if I could ensure that every file had group write set before it was copied into the group share directory but alas, I cannot. Telling users to manually check and change permissions is also a pain. Cron jobs to change group write recursively is also ugly. Please, someone provide me with the solution.