After filling a CornFS volume for a couple of days now, I found a few problems that really begged for another release.

I'm still building cornfs with debug flags and under gdb to catch any segfaults in the new caching code. Sure enough, it found a segfault or two that I needed to cleanup my pointer handling a bit. Cacheinsert() now works for a rather huge cacheinventory() run without incident.

There was also a bug with the statfs() setup in the cache upload function. Instead of statfs()ing the /data/cornfs/import/{servername} directory, it was handily using /data/cornfs/import. All of the servers appeared to have the same remaining free space, which caused the last two servers to fill to the brim.

Like I said, there will likely be some rapid releases this week as I stumble upon more nits to pick.

For now you can download cornfs-v0.0.5.1.tar.bz2 and have at it.

I've been working on cornfs this weekend a bit to speed things up.

With the help of gprof and gcc -pg, I found that the caching routines were causing a huge performance hit. Every read() and write() was doing a linear linked list search through every cached entry. This is fixed.

Along the way, I found it difficult to debug things with one huge cornfs.c source file. So I've split that up into numerous .c source files to fix this.

I also updated the Makefile to build on its own without building under fuse/examples as before. It is now 2.5.2 friendly, and compiles with 22 ABI compatibility. I'll see about adding the 25 ABI functions shortly.

So, download cornfs-v0.0.5.0.tar.bz2, extract, and build with make.

A few folks have mentioned they were playing with cornfs via private email. With this latest version, and NFS over ssh, NKS is finally running this in a production environment.

Look forward to some rapid updates here in the near future.

I've had serious problems using shfs, sshfs, and sfs. The first two fall apart under load, and the latter is a nightmare to get working in our environment (a PAM nightmare, that is).

Rather than dealing with something crazy, I decided to go back to a faithful old standard: NFS. As the remote storage nodes are accessible only via ssh, ssh was the ideal transport for the NFS mounts.

How do you do this? With a little port trickery and some inittab craziness to hold the tunnels up.

NFS v3 and newer have a TCP transport mode that make it possible to tunnel using ssh. Older versions of NFS use a UDP based ONC RPC transport. Make sure you have kernel support for TCP and NFS v3 before you continue.

On the remote nodes, install NFS:

 # apt-get install nfs-kernel-server nfs-common portmap

Then setup an exports file sharing something to localhost:

 # echo "/exports localhost(rw,async,insecure,no_root_squash)" >> /etc/exports

We need to have mountd start on a known port to setup the ssh tunnel from the master. The "-p" flag is used for this. Debian keeps the RPCMOUNTDOPTS flags in /etc/default/nfs-kernel/server, easily updated with this perl one-liner:

 # perl -pi -e 's/^(RPCMOUNTDOPTS)=.*$/$1="-p 32767"/' /etc/default/nfs-kernel-server

It's also a good idea to block portmap request from anything but localhost with tcpwrappers, just in case your firewall rules happen to be down for some reason.

 # echo "portmap: LOCAL" >> /etc/hosts.allow

Now restart things and make sure the mountpoint is being exported:

 # /etc/init.d/nfs-kernel-server stop
 # /etc/init.d/nfs-common stop
 # /etc/init.d/portmap stop
 # /etc/init.d/portmap start
 # /etc/init.d/nfs-common start
 # /etc/init.d/nfs-kernel-server start
 # rpcinfo -p localhost
 # showmount -e localhost

The remote server is now ready to mount. Return to your central master cornfs server that will act as the client and setup an ssh tunnel.

Step 1: Install nfs-client

 # apt-get install nfs-client

Step 2: Setup key trust with the remote server:

 # ssh-keygen -f ~/.ssh/id_dsa-cornfs -P'' -t dsa -b 1024
 # cat ~/.ssh/id_dsa-cornfs.pub | ssh remoteserver 'mkdir ~/.ssh; cat - >> ~/.ssh/authorized_keys'

Step 3: Setup the SSH tunnel with an inittab respawn

 # echo 'N0:23:respawn:/usr/bin/ssh -c blowfish -L 10000:localhost:2049 -L 11000:localhost:32767 remoteserver vmstat 300' >> /etc/inittab
 # telinit q

Now you should see an ssh tunnel running in a process listing. Check your system logs to see if there are any problems.

Step 4: Add fstab entries for NFS:

 # echo 'localhost:/export /data/cornfs/import/remoteserver nfs rw,bg,soft,port=10000,mountport=11000,tcp 0 0' >> /etc/fstab
 # mount /data/cornfs/import/remoteserver

You should now see your remote server /export filesystem mounted under /data/cornfs/import.

Each remote server will need to have a unique nfs and mountd port assignment. Repeat steps 3 and 4 for each.

I started at 10000 and 11000 and worked my way up from there. The next server's port assignments are 10001 and 11001, etc.

This works suprisingly well, and appears to be quite stable (far more stable than the other alternatives).

That's not to say things are as fast as they could possibly be, but it works.

The Nutch Distributed Filesystem (NDFS) has been renamed the Hadoop Distributed Filesystem (HDFS).

Follow along with the Nutch Hadoop Tutorial.

I actually started writing a FUSE frontend for NDFS for an internal project that looks like it is going the direction of Lustre instead... should probably finish it someday.