NFS: Network File System

NFS stands for Network File System.

It is a protocol that allows one computer to share directories with other computers over a network.

The important idea is:

A remote directory can appear like a local directory.

For example, a server may export:

/opt/shared

A client may mount it at:

/mnt/nfs_shared

Then users on the client can access remote files as if they were local:

ls /mnt/nfs_shared

Big Picture

NFS has two main sides:

Diagram:

Client                       Network                      Server
+--------+                                            +--------+
| User   |    file open/read/write requests           | NFS    |
| Space  |  <------------------------------->         | Server |
+--------+                                            +--------+
    |                                                     |
+--------+                                            +--------+
| NFS    |                                            | NFS    |
| Client |                                            | Daemon |
+--------+                                            +--------+
    |                                                     |
+--------+                                            +--------+
| Mount  |                                            | Local  |
| Point  |                                            | Export |
+--------+                                            +--------+
    |                                                     |
+--------+                                            +--------+
| Apps   |                                            | Disk   |
+--------+                                            +--------+

When a client reads or writes a file under the NFS mount point, the NFS client code sends network requests to the server.

The server receives those requests and performs file operations on its own local filesystem.

Why Use NFS?

NFS is useful when several systems need shared access to files.

Common reasons include:

Example use cases:

NFS is common in Linux and Unix environments, but clients also exist for macOS and Windows.

NFS Server and Client Roles

The NFS server owns and exports the real directory.

The NFS client mounts that exported directory.

Diagram:

Server filesystem:
/
└── opt
    └── shared
        ├── file1.txt
        └── file2.txt

Client filesystem:
/
└── mnt
    └── nfs_shared
        ├── file1.txt
        └── file2.txt

The files are physically stored on the server, but visible through the client mount point.

Important NFS Components

Common NFS-related components include:

NFSv3 often depends on several RPC services and ports.

NFSv4 simplifies firewalling because it mainly uses TCP port 2049.

NFS Versions

NFS has several versions.

NFSv2

NFSv2 is old and rarely used today.

Limitations include:

NFSv3

NFSv3 is still widely used.

It introduced improvements such as:

NFSv3 is stable and common, but it often requires more firewall considerations because supporting services may use multiple ports.

NFSv4

NFSv4 is newer and usually preferred for modern environments.

Advantages include:

For secure or enterprise environments, NFSv4 with Kerberos is often preferred.

Basic Server Setup

The NFS server needs:

The basic flow is:

  1. Install NFS packages
  2. Create shared directory
  3. Set ownership and permissions
  4. Edit /etc/exports
  5. Apply exports with exportfs
  6. Start and enable NFS service
  7. Open firewall
  8. Verify export

Installing NFS Packages

On RHEL, CentOS, Rocky, AlmaLinux, or Fedora-style systems:

sudo dnf install nfs-utils

On older CentOS 7 systems:

sudo yum install nfs-utils

On Debian or Ubuntu systems:

sudo apt install nfs-kernel-server nfs-common

Package names vary slightly by distribution, but the main idea is the same:

Creating a Shared Directory

Example server directory:

sudo mkdir -p /opt/shared

Add a test file:

echo "hello from NFS server" | sudo tee /opt/shared/hello.txt

Set ownership and permissions based on your use case.

For a simple lab:

sudo chmod 755 /opt/shared

For a shared writable directory, you may use a group:

sudo groupadd nfsusers
sudo chgrp nfsusers /opt/shared
sudo chmod 2775 /opt/shared

The 2 in 2775 sets the setgid bit so new files tend to inherit the directory group.

/etc/exports

The server uses /etc/exports to define what directories are shared and who can access them.

Example:

/opt/shared 192.168.1.0/24(rw,sync,root_squash)

Meaning:

A more restrictive example:

/opt/shared 192.168.1.50(ro,sync,root_squash)

This allows only one client and gives read-only access.

Common Export Options

A common safe default:

/opt/shared 192.168.1.0/24(rw,sync,root_squash,no_subtree_check)

Important warning:

Applying Export Changes

After editing /etc/exports, apply changes with:

sudo exportfs -r

View exports:

sudo exportfs -v

Example output:

/opt/shared  192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,root_squash,no_all_squash)

Interpretation:

Starting NFS Services

On systemd systems:

sudo systemctl enable --now nfs-server

Check status:

systemctl status nfs-server

Example output:

● nfs-server.service - NFS server and services
     Loaded: loaded
     Active: active (exited)

Interpretation:

On older setups, you may also see or manage:

sudo systemctl enable --now rpcbind
sudo systemctl enable --now nfs-idmapd

Firewall Rules for NFS

For NFSv4, TCP port 2049 is the main port.

For NFSv3, additional RPC services such as mountd and rpcbind may be needed.

With firewalld:

sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --permanent --add-service=mountd
sudo firewall-cmd --permanent --add-service=rpc-bind
sudo firewall-cmd --reload

Check:

sudo firewall-cmd --list-services

Example output:

ssh dhcpv6-client nfs mountd rpc-bind

Interpretation:

With UFW, a simple NFSv4 example:

sudo ufw allow from 192.168.1.0/24 to any port 2049 proto tcp

Client Setup

The NFS client needs:

Install packages.

RHEL-style:

sudo dnf install nfs-utils

Debian/Ubuntu:

sudo apt install nfs-common

Create mount point:

sudo mkdir -p /mnt/nfs_shared

Mount:

sudo mount -t nfs 192.168.1.100:/opt/shared /mnt/nfs_shared

For NFSv4 explicitly:

sudo mount -t nfs4 192.168.1.100:/opt/shared /mnt/nfs_shared

Check:

mount | grep nfs

Example output:

192.168.1.100:/opt/shared on /mnt/nfs_shared type nfs4 (rw,relatime,vers=4.2,addr=192.168.1.100)

Interpretation:

Verifying the NFS Mount

List files:

ls -l /mnt/nfs_shared

Example output:

-rw-r--r-- 1 root root 22 Jun 1 12:00 hello.txt

Read the test file:

cat /mnt/nfs_shared/hello.txt

Example output:

hello from NFS server

Create a file if write access is allowed:

touch /mnt/nfs_shared/client-test.txt

If this works, the mount is writable for your user.

If it fails with permission denied, check Unix permissions, UID/GID mapping, root squashing, and export options.

Persistent NFS Mounts with /etc/fstab

To mount automatically at boot, add an entry on the client.

Example:

192.168.1.100:/opt/shared /mnt/nfs_shared nfs defaults,_netdev 0 0

For NFSv4:

192.168.1.100:/opt/shared /mnt/nfs_shared nfs4 defaults,_netdev 0 0

Important option:

_netdev means this mount depends on the network.

For systems where the NFS server may not always be available, consider:

192.168.1.100:/opt/shared /mnt/nfs_shared nfs4 defaults,_netdev,nofail,x-systemd.automount 0 0

Meaning:

Test fstab:

sudo mount -a

Check:

findmnt /mnt/nfs_shared

UID and GID Mapping

NFS permissions are based heavily on numeric user IDs and group IDs.

This is a major concept.

Linux file ownership is stored as numbers:

Example:

id alice

Output:

uid=1000(alice) gid=1000(alice) groups=1000(alice)

If Alice has UID 1000 on the client but UID 2000 on the server, permissions may not behave as expected.

This can cause:

Strategies for UID and GID Consistency

Common strategies include:

In small labs, manually matching UIDs may be enough.

In larger environments, use centralized identity management.

NFSv4 and idmapd

NFSv4 can use name-based identity mapping through idmapd.

The client and server should use the same domain in:

/etc/idmapd.conf

Example:

[General]
Domain = example.com

Restart relevant services after changes.

Example:

sudo systemctl restart nfs-idmapd
sudo systemctl restart nfs-server

Check identity mapping issues if files appear owned by:

These often indicate mapping problems.

Root Squashing

Root squashing protects the server from root users on clients.

With root_squash, a request from UID 0 on the client is mapped to an anonymous user on the server.

Client root UID 0
      |
      v
NFS server maps it to anonymous user
      |
      v
Usually nfsnobody or nobody

This prevents client root from automatically having root privileges on the server export.

Example export:

/opt/shared 192.168.1.0/24(rw,sync,root_squash)

Dangerous option:

/opt/shared 192.168.1.0/24(rw,sync,no_root_squash)

Use no_root_squash only in carefully controlled environments.

all_squash

The all_squash option maps all client users to the anonymous user.

Example:

/opt/public 192.168.1.0/24(rw,sync,all_squash)

This can be useful for simple public drop-box style shares where all access should use one server-side identity.

Common related options:

Example:

/opt/public 192.168.1.0/24(rw,sync,all_squash,anonuid=2000,anongid=2000)

This maps all access to UID 2000 and GID 2000.

Security Considerations

NFS is powerful, but it must be configured carefully.

Good practices:

A safe export is usually specific:

/srv/project 192.168.10.0/24(rw,sync,root_squash,no_subtree_check)

A risky export is broad:

/  *(rw,no_root_squash)

Avoid broad exports like that.

Performance Considerations

NFS performance depends on:

Common mount options include:

Example:

sudo mount -t nfs -o rsize=8192,wsize=8192 192.168.1.100:/opt/shared /mnt/nfs_shared

Modern systems often negotiate good defaults automatically. Tune only after measuring.

Important safety note:

Managing Exports with exportfs

View exports:

sudo exportfs -v

Reload exports:

sudo exportfs -r

Unexport one directory:

sudo exportfs -u 192.168.1.0/24:/opt/shared

Unexport all:

sudo exportfs -ua

Re-export all from /etc/exports:

sudo exportfs -a

Scenario 1: Create a Basic NFS Share

Set up a server export and mount it from a client.

Server Steps

Install packages:

sudo dnf install nfs-utils

Create directory:

sudo mkdir -p /opt/shared
echo "hello from server" | sudo tee /opt/shared/hello.txt
sudo chmod 755 /opt/shared

Edit /etc/exports:

/opt/shared 192.168.1.0/24(rw,sync,root_squash,no_subtree_check)

Apply:

sudo exportfs -r
sudo systemctl enable --now nfs-server
sudo exportfs -v

Example output:

/opt/shared 192.168.1.0/24(sync,wdelay,no_subtree_check,sec=sys,rw,root_squash,no_all_squash)

Client Steps

Install client tools:

sudo dnf install nfs-utils

Create mount point:

sudo mkdir -p /mnt/nfs_shared

Mount:

sudo mount -t nfs4 192.168.1.100:/opt/shared /mnt/nfs_shared

Check:

findmnt /mnt/nfs_shared
cat /mnt/nfs_shared/hello.txt

Example output:

hello from server

Interpretation:

Scenario 2: Simulate “Access Denied by Server”

Show what happens when the client IP is not allowed by /etc/exports.

Simulate Problem

On the server, restrict the export to the wrong network:

/opt/shared 10.10.10.0/24(rw,sync,root_squash)

Apply:

sudo exportfs -r

On the client:

sudo mount -t nfs 192.168.1.100:/opt/shared /mnt/nfs_shared

Example output:

mount.nfs: access denied by server while mounting 192.168.1.100:/opt/shared

Check on Server

sudo exportfs -v

Example output:

/opt/shared 10.10.10.0/24(rw,sync,root_squash)

Interpretation:

Fix

Use the correct client subnet or IP:

/opt/shared 192.168.1.0/24(rw,sync,root_squash,no_subtree_check)

Apply:

sudo exportfs -r

Scenario 3: Simulate NFS Blocked by Firewall

Diagnose when the export is correct but the client cannot reach NFS services.

Simulate Problem

On the server, remove NFS firewall services:

sudo firewall-cmd --permanent --remove-service=nfs
sudo firewall-cmd --permanent --remove-service=mountd
sudo firewall-cmd --permanent --remove-service=rpc-bind
sudo firewall-cmd --reload

On the client:

sudo mount -t nfs 192.168.1.100:/opt/shared /mnt/nfs_shared

Possible output:

mount.nfs: Connection timed out

Check Connectivity

nc -vz 192.168.1.100 2049

Example output:

nc: connect to 192.168.1.100 port 2049 (tcp) timed out

Check Server Firewall

sudo firewall-cmd --list-services

Example output:

ssh dhcpv6-client

Interpretation:

Fix

sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --permanent --add-service=mountd
sudo firewall-cmd --permanent --add-service=rpc-bind
sudo firewall-cmd --reload

Retest:

nc -vz 192.168.1.100 2049

Expected:

Connection to 192.168.1.100 2049 port [tcp/nfs] succeeded!

Scenario 4: Simulate Permission Denied from UID/GID Mismatch

Show why matching usernames is not enough if numeric UIDs differ.

Situation

On server:

alice UID = 1001

On client:

alice UID = 1002

The server directory is owned by UID 1001:

ls -ln /opt/shared

Example output on server:

drwxr-x--- 2 1001 1001 4096 Jun 1 12:00 /opt/shared

On client, Alice tries:

touch /mnt/nfs_shared/test.txt

Example output:

touch: cannot touch '/mnt/nfs_shared/test.txt': Permission denied

Check IDs

On client:

id alice

Example:

uid=1002(alice) gid=1002(alice)

On server:

id alice

Example:

uid=1001(alice) gid=1001(alice)

Interpretation:

Fix Options

Scenario 5: Simulate Root Squash Behavior

Show why root on the client may not have root power on the NFS export.

Server Export

/opt/shared 192.168.1.0/24(rw,sync,root_squash)

Apply:

sudo exportfs -r

On client as root:

sudo touch /mnt/nfs_shared/root-created.txt

Possible output:

touch: cannot touch '/mnt/nfs_shared/root-created.txt': Permission denied

Or if the directory allows anonymous writes, check ownership:

ls -ln /mnt/nfs_shared/root-created.txt

Example output:

-rw-r--r-- 1 65534 65534 0 Jun 1 12:30 root-created.txt

Interpretation:

Unsafe Alternative

/opt/shared 192.168.1.0/24(rw,sync,no_root_squash)

Warning:

Scenario 6: Simulate a Stale NFS File Handle

Understand what happens when the server-side exported directory changes while clients still have old references.

Simulate

Client mounts:

sudo mount -t nfs 192.168.1.100:/opt/shared /mnt/nfs_shared
cd /mnt/nfs_shared

On the server, rename and recreate the export directory:

sudo mv /opt/shared /opt/shared.old
sudo mkdir /opt/shared
sudo exportfs -r

On the client:

ls

Possible output:

ls: cannot access '.': Stale file handle

Interpretation:

Fix

On the client:

cd /
sudo umount /mnt/nfs_shared
sudo mount /mnt/nfs_shared

If unmount is busy:

sudo lsof +f -- /mnt/nfs_shared
sudo fuser -vm /mnt/nfs_shared

Then stop the using process or move out of the directory.

Scenario 7: Simulate Boot Hang from NFS in /etc/fstab

Show why NFS mounts should be configured carefully for boot.

Problem fstab Entry

192.168.1.100:/opt/shared /mnt/nfs_shared nfs defaults 0 0

If the NFS server is down during boot, the client may wait for a long time.

Better Entry

192.168.1.100:/opt/shared /mnt/nfs_shared nfs4 defaults,_netdev,nofail,x-systemd.automount 0 0

Apply

sudo systemctl daemon-reload
sudo mount -a

Check systemd mount units:

systemctl list-units | grep nfs_shared

Example output:

mnt-nfs_shared.automount loaded active waiting /mnt/nfs_shared

Interpretation:

Scenario 8: Simulate Read-Only Export

Show how export options override client expectations.

Server Export

/opt/shared 192.168.1.0/24(ro,sync,root_squash)

Apply:

sudo exportfs -r

Client remount:

sudo umount /mnt/nfs_shared
sudo mount -t nfs 192.168.1.100:/opt/shared /mnt/nfs_shared

Try write:

touch /mnt/nfs_shared/test.txt

Example output:

touch: cannot touch '/mnt/nfs_shared/test.txt': Read-only file system

Check Mount

findmnt /mnt/nfs_shared

Example output:

TARGET          SOURCE                    FSTYPE OPTIONS
/mnt/nfs_shared 192.168.1.100:/opt/shared nfs4   ro,relatime,vers=4.2

Interpretation:

Scenario 9: Measure NFS Performance

Check whether NFS is slow and where the bottleneck may be.

Write Test

On the client:

dd if=/dev/zero of=/mnt/nfs_shared/testfile bs=1M count=512 conv=fdatasync

Example output:

536870912 bytes copied, 8.2 s, 65.5 MB/s

Read Test

dd if=/mnt/nfs_shared/testfile of=/dev/null bs=1M

Example output:

536870912 bytes copied, 4.1 s, 130 MB/s

Check Mount Stats

nfsiostat 1

Example output:

op/s    rpc bklog
120.00  0.00

read:  avg RTT  4.0 ms   avg exe  5.0 ms
write: avg RTT 12.0 ms   avg exe 15.0 ms

Interpretation:

Other Checks

On client:

mount | grep nfs
nfsstat -c

On server:

nfsstat -s
iostat -xz 1

Scenario 10: Troubleshoot “NFS Server Is Not Responding”

Diagnose a client that hangs or reports server not responding.

Symptom

Client log or terminal shows:

nfs: server 192.168.1.100 not responding, still trying

Check Network

ping 192.168.1.100

Check NFS port:

nc -vz 192.168.1.100 2049

Example failure:

nc: connect to 192.168.1.100 port 2049 failed: No route to host

Check Server Service

On server:

systemctl status nfs-server
sudo ss -tulnp | grep 2049

Example output:

tcp LISTEN 0 64 0.0.0.0:2049 0.0.0.0:*

Interpretation:

Scenario 11: Use showmount to Inspect Exports

Check what the server appears to export.

On client:

showmount -e 192.168.1.100

Example output:

Export list for 192.168.1.100:
/opt/shared 192.168.1.0/24

Interpretation:

Important note:

Scenario 12: Unexport a Shared Directory

Stop sharing a directory without editing many files manually.

Check current exports:

sudo exportfs -v

Unexport:

sudo exportfs -u 192.168.1.0/24:/opt/shared

Check again:

sudo exportfs -v

Interpretation:

For a permanent stop, remove or comment out the line in /etc/exports.

Common NFS Problems and Fixes

Problem: Access Denied by Server

Symptoms:

mount.nfs: access denied by server

Check:

sudo exportfs -v
cat /etc/exports
showmount -e SERVER

Likely causes:

Fix:

sudo exportfs -r

and correct /etc/exports.

Problem: Connection Timed Out

Symptoms:

mount.nfs: Connection timed out

Check:

ping SERVER
nc -vz SERVER 2049
systemctl status nfs-server
sudo firewall-cmd --list-services

Likely causes:

Problem: Permission Denied While Writing

Symptoms:

touch: Permission denied

Check:

id
ls -ln /mnt/nfs_shared
ls -ln /opt/shared
sudo exportfs -v

Likely causes:

Problem: Files Owned by nobody

Symptoms:

-rw-r--r-- 1 nobody nobody file.txt

or numeric:

4294967294

Likely causes:

Check:

cat /etc/idmapd.conf
id username
nfsidmap -l

Problem: Stale File Handle

Symptoms:

Stale file handle

Likely causes:

Fix:

cd /
sudo umount /mnt/nfs_shared
sudo mount /mnt/nfs_shared

Problem: Boot Delays Because NFS Is Unavailable

Symptoms:

Fix fstab with:

Example:

192.168.1.100:/opt/shared /mnt/nfs_shared nfs4 defaults,_netdev,nofail,x-systemd.automount 0 0

NFS Troubleshooting Workflow

When NFS fails, troubleshoot in layers.

  1. Is the server reachable?
  2. Is NFS service running?
  3. Is port 2049 reachable?
  4. Is the export listed?
  5. Is the client allowed by /etc/exports?
  6. Is the firewall open?
  7. Is the mount command correct?
  8. Are Unix permissions correct?
  9. Are UID/GID mappings correct?
  10. Are logs showing NFS errors?

Useful commands:

ping SERVER
nc -vz SERVER 2049
systemctl status nfs-server
sudo exportfs -v
showmount -e SERVER
mount | grep nfs
findmnt /mnt/nfs_shared
id
ls -ln
journalctl -u nfs-server -b
dmesg -T | grep -i nfs

Useful Command Summary

Server setup:

sudo dnf install nfs-utils
sudo mkdir -p /opt/shared
sudo vi /etc/exports
sudo exportfs -r
sudo exportfs -v
sudo systemctl enable --now nfs-server

Client setup:

sudo dnf install nfs-utils
sudo mkdir -p /mnt/nfs_shared
sudo mount -t nfs4 SERVER:/opt/shared /mnt/nfs_shared
findmnt /mnt/nfs_shared

Firewall:

sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --permanent --add-service=mountd
sudo firewall-cmd --permanent --add-service=rpc-bind
sudo firewall-cmd --reload

NFS inspection:

sudo exportfs -v
showmount -e SERVER
nfsstat -s
nfsstat -c
nfsiostat 1
mount | grep nfs

Persistent mount:

SERVER:/opt/shared /mnt/nfs_shared nfs4 defaults,_netdev,nofail,x-systemd.automount 0 0

Unmount:

sudo umount /mnt/nfs_shared

Force investigation if busy:

sudo lsof +f -- /mnt/nfs_shared
sudo fuser -vm /mnt/nfs_shared

Safe Lab Cleanup

On the client:

cd /
sudo umount /mnt/nfs_shared 2>/dev/null
sudo rmdir /mnt/nfs_shared 2>/dev/null

Remove fstab test entry if added:

sudo vi /etc/fstab
sudo systemctl daemon-reload

On the server, remove export line from /etc/exports, then:

sudo exportfs -r
sudo exportfs -v

Optionally remove test directory:

sudo rm -rf /opt/shared

Challenges

  1. Set up an NFS server that exports /opt/shared to one trusted client IP.
  2. Mount the export from a client at /mnt/nfs_shared and verify it with findmnt.
  3. Add a file on the server and confirm it appears on the client.
  4. Create a file on the client and confirm it appears on the server.
  5. Change the export from rw to ro, reload exports, remount on the client, and explain the write failure.
  6. Simulate an incorrect client subnet in /etc/exports and diagnose the resulting access denied by server error.
  7. Block NFS with the firewall and confirm that the client cannot connect to port 2049.
  8. Compare UID and GID values for the same user on client and server. Explain how mismatches affect NFS permissions.
  9. Demonstrate root squashing by trying to write as root from the client and inspecting ownership on the server.
  10. Add an NFS mount to /etc/fstab using _netdev,nofail,x-systemd.automount, then test it with mount -a.
  11. Use nfsstat or nfsiostat to observe NFS activity during a file copy.
  12. Write a troubleshooting report for one NFS failure. Include symptom, command used, output, interpretation, and fix.