• libnfsclient

libnfsclient

libnfsclient is a userland, NFS client ops library. It can be used by user space programs to send NFS client operations like lookup, mount, read, write, etc. This is done by providing a layer over the Sockets interface thereby avoiding the need to go through the file system. Among other usage scenarios, I use it for my NFS benchmarking tool called nfsreplay.

The NFS benchmarking project page is here: NFSBenchmarking

I can be reached at <shehjart AT gelato DOT NO SPAM unsw DOT edu GREEBLIES DOT au>

News

• March 13, 2008, Interface stabilized months back.

• April 5, 2007, nfsreplay svn is up

• March 31, 2007 libnfsclient is still pre-alpha. Expect breakage due to interface changes.

Interface

See AsyncRPC also since libnfsclient is a layer on top of that library.

Client Context: nfs_ctx

Connection to each NFS server is represented by a structure called nfs_ctx. This is opaque to the users and is to be used only as a handle or identifier for:

• The particular mount, if the connection was initiated by a MOUNT
• or the TCP connection, if the connection was initiated without performing a MOUNT call.

All interface functions need to be passed the nfs_ctx handle that connects to a particular server.

Initializing the nfs client context is always the first step.

nfs_init

#include <nfsclient.h>

nfs_ctx *nfs_init(struct sockaddr_in *srv, int proto, int connflags);

Returns the handle for the connection to the server in srv, NULL on failure.

• srv - Pointer to the socket which contains the server IP address. The port member in struct sockaddr_in is optional. If srv->sin_port is 0, the portmapper is internally used to look up the NFS port on the server in srv.

  proto - The underlying transport layer protocol. The two values are:

  • IPPROTO_TCP - Transmission Control Protocol, the only protocol currently supported.

    IPPROTO_UDP - User Datagram Protocol, not supported at present..

  connflags - Flags to customize this TCP connection to the server. Flags supported at present:

  • NFSC_CFL_NONBLOCKING - Create a non-blocking connection to the server. This ensures that the transmission of nfs requests and reception replies does not block in the system calls. In case the system call will block, libnfsclient makes copies of the buffers so they can transmitted later.
    

    NFSC_CFL_BLOCKING - Create a blocking socket so that transmission and reception is synchronous.
    

    NFSC_CFL_DISABLE_NAGLE - Disable Nagle's algorithm for this connection.
    

Operations

At present, libnfsclient only implements NFSv3 and MOUNT protocol. The interface is described in nfs3Interface.

Error Handling

#include <nfsclient.h>

char * nfsstat3_strerror(int stat);

• nfsstat3_strerror is a helper function that allow translating NFS 3 status numbers into the corresponding status messages.
  stat is the value of the status in NFS 3 response structures. See test/nfs_errno.c.

XDR Translation

The message structures passed to the nfs3_* functions described in nfs3Interface, are converted internally into XDR format before being transmitted.

The message buffers, read from the socket and passed to the callbacks are in XDR format. These have to be first converted into the required message-specific structure. This implies that each callback somehow know what type of reply it will be processing. Again, see examples of how reply messages are converted into message structures, in the callbacks sources in the test/ directory.

For the interface for translating reply message buffers in XDR format to reply-message specific structures, see XDRInterface.

MOUNT and NFS integration

Generally, a client looks up the file handle of an exported file system(..by connecting to the mountd service..) on a server before it proceeds to operate on it. In cases, where the exported file system's filehandle is already known, clients can proceed directly to connect to the NFS service. Both modes are supported.

The Mount protocol interface is synchronous and returns only after a reply is received. This is not true for the NFS 3 protocol. There,a call function will return after atleast sending the request. Before returning, it'll check if a reply was received and if so, the callback for it will be executed, otherwise it just returns. So yes, there is no guarantee that the callback is executed before returning.

Though, this is the default behaviour, it can be changed, i.e. made blocking by setting the flag argument to nfs_init as NFSC_CFL_BLOCKING. In this case, the call functions will behave like the mount interface, i.e. return only after the reply was received and its callback executed.

Handling Callbacks

Callback should be of the following type.

typedef void (*user_cb)(void *msg_buf, int bufsz, void *priv);

• msg_buf - Message in XDR format.

  bufsz - Size of message buffer.

  priv - is the pointer to private data, passed to nfs3_* call functions.

Callbacks are called by the library whenever a complete reply is received. The message buffers passed to the callbacks are freed after the callback returns so copy anything out of the buffer if persistence is needed. See AsyncRPC for more info.

Completion Notification

In case, user programs need to explicitly have the pending socket buffers processed and any callbacks executed, the following function can be used.

int nfs_complete(nfs_ctx * ctx, int flag);

• ctx - The nfs client context.

  flag - Specifies the behaviour of the call. The two values for flag are:

  • RPC_BLOCKING_WAIT - Blocks on the read to a socket, even if the socket was created as non-blocking using the connection flag. If any replies are received the corresponding callbacks are run.

    RPC_NONBLOCKING_WAIT - Attempts a read of the socket. If there are pending replies, processes them and runs the corresponding callbacks, otherwise, returns to the user program.

  • The interaction between flag and connection flag is shown in the table below:

    • 		Flag
      		RPC_BLOCKING_WAIT	RPC_NONBLOCK_WAIT
      Socket Type	Non-blocking	BLOCKING read() Blocks till atleast 1 reply is completely received. More replies can be processed if larger buffers were available.	NON-BLOCKING read() Processes at least 1 reply only if the socket does not return EAGAIN or insufficient data while reading from the socket.
      	Blocking	BLOCKING read() Blocks till atleast 1 reply is completely received. More replies can be processed if larger buffers were available.	BLOCKING read() Blocks till atleast 1 reply is completely received. More replies can be processed if larger buffers were available.

    nfs_complete returns the count of callbacks executed. This count can also use used as the number of replies processed in this call. nfs_complete behaviour depends on two cases:
    • When there are pending replies(..libnfsclient keeps a count of these..). In this case, nfs_complete will behave according to the matrix above. Keep in mind that the number of replies processed,i.e. the number of callbacks executed by libnfsclient depends on the amount of data available in the socket buffers and not on the number of replies pending.
    • When there are no pending replies, it will simply return with a count of zero without blocking on the socket.

Multi-Threading with libnfsclient

All libnfsclient operations take place on the specified nfs client context so separate threads operating on their own nfs_ctxs will work just fine. A synchronisation mechanism will need to be implemented by the user program if sending requests from multiple threads., over a shared nfs_ctx.

Code

libnfsclient and AsyncRPC are part of the nfsreplay source package. See nfsreplay page for instructions.

Building

Building instructions are in the BUILD file in nfsreplay package.

Examples

For examples on usage of the library, see the sources in test/ subdirectory of the nfsreplay source.

Support

Use nfsreplay lists for support and discussion.