Category: NFS

“It is a mistake to think you can solve any major problems just with potatoes.”
Douglas Adams (1952-2001, English humorist, writer and dramatist)

While there have been many advances and improvements to NFS over the last decade, some IT organizations have elected to continue with NFSv3 – like potatoes, it’s the staple filesystem protocol that just about any UNIX administrator understands.

Although adequate for many purposes and a familiar and well understood protocol, choosing and continuing to deploy NFSv3 has become increasingly difficult to justify in a modern datacenter. For example, NFSv3 makes promiscuous use of ports, something that is unsuitable for a variety of security reasons for use over a wide area network (WAN); plus increased server & client bandwidth demands and improved functionality of Network Attached Storage (NAS) arrays have outstripped NFSv3’s ability to deliver high throughput.
NFSv4 and the minor versions that follow it are designed to address many of the issues that NFSv3 poses. NFSv4 also includes features intended to enable its use in global wide area networks (WANs), and to improve the performance and resilience of NAS (Network Attached Storage):

• Firewall-friendly single port operations
• Internationalization support
• Replication and migration facilities
• Mandatory use of strong RPC security flavors that depend on cryptography, with support of access control that is compatible with both UNIX ® and Windows ®
• Use of character strings instead of integers to represent user and group identifiers
• Advanced and aggressive cache management features with delegations
• (with NFSv4.1 pNFS, or parallel NFS) Trunking

In April 2003, the Network File System (NFS) version 4 Protocol was ratified as an Internet standard, described in RFC-3530, which superseded NFS Version 3 (NFSv3, specified in RFC-1813). Since the ratification of NFSv4, further advances have been made to the standard, notably NFSv4.1 (as described in RFC-5661, ratified in January 2010) that included several new features such as parallel NFS (pNFS). And further work is currently underway in the IETF for NFSv4.2.

Delegations with NFSv4

In NFSv3, clients have to function as if there is contention for the files they have opened, even though this is often not the case. As a result of this conservative approach to file locking, there are frequently many unneeded requests from the client to the server to find out whether an open file has been modified by some other client. Even worse, all write I/O in this scenario is required to be synchronous, further impacting client-side performance.
NFSv4 differs by allowing the server to delegate specific actions on a file to the client; this enables more aggressive client caching of data and the locks. A server temporarily cedes control of file updates and the locking state to a client via a delegation, and promises to notify the client if other clients are accessing the file. Once the client holds a delegation, it can perform operations on files with data has been cached locally, and thereby avoid network latency and optimize its use of I/O.

Trunking with pNFS

Many additional enhancements to NFSv4 are available with NFSv4.1, of which pNFS is a part. pNFS adds the capability to perform trunking at the NFS level by adding a session layer. The client establishes a session with an NFSv4.1 server, and can then create multiple TCP connections to the NFSv4.1 server, each potentially going over a different network interface on the client, and arriving on a different interface on the NFSv4.1 server. Now different requests sent over the same session identifier can go over different network paths, dramatically improving latency and increasing bandwidth.
Although client and server implementations of NFSv4.1 are available, they are in early stages of implementation and adoption. However, to take advantage of them in the future, it is important to plan now for the move to NFSv4 and beyond – and there are many servers and clients available now that support NFSv4. NFSv4 is a mature and stable protocol with many advantages in its own right over its predecessors NFSv3 and NFSv2.

Potatoes and Beyond

Now is the time to make the switchover; there really is no justification for not pursuing NFSv4 as the first NFS protocol version of choice. Although migrating from earlier versions of NFS requires some planning as there are significant differences between the two protocols, the benefits are impressive. To ensure a smooth migration to NFSv4 and beyond, the SNIA Ethernet Storage Forum NFS Special Interest Group has recently published an overview white paper “Migrating to NFSv4“. This covers internationalization support, automatic mounting of NFSv4 filesystems on demand, TCP protocol support amongst other considerations.
NFSv4 and NFSv4.1 have been developed for a reason; and NFSv4.2 is on the horizon. Like the potato, NFSv3 is a staple of the network Filesystem world. But as Douglas Adams said; “It is a mistake to think you can solve any major problems just with potatoes.” NFSv4 fixes many of NFSv3’s deficiencies, and represents a major advance that brings improved availability, performance and security; all the check-list items beyond potatoes that today’s users of network attached storage demand.

Few weeks ago I was invited to present the state of the pNFS to the Purdue University. They are interested to be one of the early adopters and I jumped on that opportunity to promote pNFS. The presentation included the deep dive in the protocol and the need for scalability and I continued with the current state of the protocol and the initial client implementations in Linux and Open Solaris.

The presentation and the discussion that followed addressed some basic questions that I expected around why should users trust that NFSv4.1/pNFS will not have the same faith as NFSv4.0. This is a legitimate question that often the pNFS developers in Linux ask themselves and the answer that I gave was same as the developers; pNFS will address many of the HPC needs. After additional details on scalability, performance, availability all the people in the room agreed that it is worth to look closer at pNFS. I recommended them to start looking at the current Fedora distribution that has both server and client pNFS file layout.

As Purdue is a heavy Lustre user, they further asked how would Lustre support pNFS. If you didn’t know, there are patches available for Lustre to support pNFS. I introduced Purdue team to the chief developer of Lustre so they can be the first to test the prototype patches.

This will help the promotion of pNFS in academia.

We need to continue our team effort to promote everywhere in universities pNFS.

Last week I attended the annual Massive Storage System Technologies 2010 Symposium in Tahoe. Most of the audience was from the National Labs, NASA, and Supercomputing centers. The discussions were mostly focused on the next exascale generation of storage and I was invited to present the pNFS as one of the major future technologies that will address the scalability issues related to the high I/O throughput requirements of HPC and SC. My presentation was very well received and many participants were not new to the pNFS and NFSv4.1. During the discussions after the presentation I asked the audience if anybody used already pNFS and I got some 4 votes. But when I asked who is considering using pNFS in the near future I got some 80 hands (out of 140 in the room). More interesting about the interest in pNFS was the fact that one of the storage managers of CERN presented unsolicited his latest use of pNFS by CERN scientists. It was the first time a user not close to the pNFS community presented to the researcher the use of pNFS.

Additional discussion was related to the reasons why NFSv4.1 will not have the same faith of NFSv4.0 which was supposed to bring new features to the NFSv3 and didn’t become a replacement of NFSv3. I just want to mention that this HPC community was always “hating” NFSv3 but they couldn’t live without it. The next question was what took us so long to get the NFSv4.1 and pNFS out of the door. Of course my answer was that we ensured that NFSv4.1 will have enough value added so it will not have the same faith as NFSv4.0. I am not sure that I was very convincing so perhaps we can try to organize a panel at SNIA to try to address this issue.