Search:
Where I Work
NKS
Subscribe
Add to Google
RSS 0.91
RSS 1.0
RSS 2.0
ATOM 1.0
RSS 2.0 and ATOM
Network
View Ian's profile on LinkedIn
Archives
2007 April (1)
2007 February (1)
2007 January (4)
2006 December (2)
2006 November (2)
2006 September (5)
2006 August (4)
2006 July (1)
2006 June (3)
2006 May (2)
2006 March (4)
2006 February (4)
2006 January (1)
2005 December (8)
2005 November (26)
2005 October (10)
2005 September (17)
2005 August (87)
2005 July (48)
2005 June (34)
2005 May (24)
2005 April (243)
2004 April (1)
2004 February (3)
2003 August (2)
2003 June (2)
2003 May (8)
2003 January (1)
2002 September (1)
2002 July (4)
2002 June (2)
2002 May (5)
2002 April (15)
2002 March (15)
Projects
CornFS
DENSO NAV
Rage Powered
Tampa Bay
TampaBad
SLUG
ob-buttons
Creative Commons OpenSource Linux Individual-i GeoURL Linux Speakeasy Speed Test
Twitter

follow icblenke at http://twitter.com
Google
Ian's shared items in Google Reader (subscribe)

iPad SDK 3.2 Beta 4 Clears Up Facts About iPad Camera And Give Some Gestures TO Developers.

OCZ drops SSDs to below $100

Google Maps Adds Biking Directions

App Engine joins the Google over IPv6 Program

Good Artists Copy, Great Artists Steal

Tech Tour: Cambridge Innovation Center

Nippon Oil and Hitachi aim at mass-producing microbe-derived biofuel

Get your Jetpack soon!

exmwSxv7XJI

(title unknown)

SCALE8x, OpenVZ goodies, and new kernels (including 2.6.32)

Strategy: Planning for a Power Outage Google Style

The island phone system adventure… « Baby is 60 – Tim Panton on voice and computers

Frameless laptop screens expected soon

The blind camera shows you someone else’s pictures

Princeton TPM-ICN series Bluetooth bracelet.

YouTube Blog: The Future Will Be Captioned: Improving Accessibility on YouTube

Put a Spark into your Presentations with Ignite

muCPjK4nGY4&hl=en_US&fs=1&

Google Code Blog: Google PowerMeter API introduced for device manufacturers

Geektalk

Sparkfun free day tomorrow: 1/7

websockets

C thulu ftagn recursion

Need a recursive DNS server? Use 8.8.8.8 and 8.8.4.4

Google Public DNS

JIQL - Java JDBC wrapper for Google DataStore

OpenNebula

Trillions

ZFS L2ARC ZIL on SSD

Swimming in OpenCL

Unicorn == Mongrel delayed_job

Remus - Transparent HA for Xen

Go

What DNS is not

Crossbow Virtual Wire Demo Tool

Banner ads on flies

PoolParty

Eucalyptus MySQL SOLR RabbitMQ Varnish == Nebula.nasa.gov

Nebula.nasa.org

Ubuntu Enterprise Cloud (UEC)

Evernote

Apple drops ZFS due to legal concerns

Peering disputes between Cogent and Hurricane Electric

Equinix to acquire Switch and Data for $689 million

We Are All Connected

Project kxen renamed project HXEN

Pomegranate Phone

Lessconf Jacksonville - followed the next day by Barcamp

Stick-figure guide to advanced AES crypto

Why you should pay attention to Google Wave

rails-primer - how to easily host rails projects on appengine

AppEngine-JRuby on google code

Ruby on Google AppEngine: appengine-jruby video

Dataliberation.org - The Data Liberation Front - a group concerned with moving data in and out of google

Detecting Spammers with SNARE: Spatio-temporal Network-level Automatic Reputation Engine

Proxmox VE - OpenVZ KVM Cluster appliance management

Sun/Oracle kill of SXCE: Sysadmins everywhere cry in horror.

Essentials of Metaheuristics

making water drinkable through nano-filtration

Pigin 2.6.1 adds Xmpp voice and video support

Opera Unite

Setting up a Layer-3 tunnel VPN using ssh 4.3 and -w option tun devices

shadowserver.org - botnet hunting resources

OpenBSC - a Siemens BS-11 microBTS or a ip.access nanoBTS == your own GSM tower

Voxbone's 883 country code

Apple keyboard firmware hack

Karesansui Project - a Xen management harness from Japan

eunicycle

Pygowave Server - Run your own Google Wave server

Happy Sysadmin Day!

Bokode

Bass cannon

Xen clocksource0 time went backwards

Internet vs World Population stats

BBC article on sat-3 cut

sat-3 cut

iPeak - RAIN

Asankya - RAIN

Apple pulls Google Voice app from iPhone - AT&T's fault

HadoopDB

live-android boot ISO - very neat

How to update your GeoIP information in addition to SWIPping

EATR

Google Wave hackathon on 20th/21st, if you happen to be in Mountainview

Did I mention OTOY here before?

NeatX - NX for Ganeti

STuPiD - STUN/TURN using PHP in Dispair

Aviary.com

Browser based Server-side 3D gaming from OTOY

Cisco's replacement for the WRT54GL is the WRT160NL

Spinn3r.com - Index the blogosphere

Team ARIN

Parts of galaxy Messier 87 are missing

DRAEGER ALCOTEST 7110 MKIII-C Evaluation of Breathalizer Source Code

Cyclops

Google's AJAX playground

How Michael Osinski Helped Build the Bomb That Blew Up Wallstreet

Bruce Perens - A Cyber-Attach on an American City

How Google and Facebook are using R

adito - the new gpl fork of the old sslexplorer project

A date idea: forklift sunset

Psytechnics - VVoiP QoE

r1soft cdp

IP Address geolocation for free

Shapeways - $50 "3-D poem rings" until the end of the month

GrandCentral to become Google Voice

Wolframalpha is coming

Hosted Xen Project

VirtualGL X11 transport

TurboVNC VirtualGL == FAST network GL

Ben Rockwood's presentation at the OpenSolaris Storage Summit: ZFS in the trenches

The Crisis of Credit Visualized on Vimeo

10gen - a java based app hosting infrastructure

Engineyard Vertebra - another cloud infrastructure management harness

Eucalyptus - an opensource EC2 compatible hosting infrastructure

asciicasts.com

railsbrain.com <-- ajaxified rdoc

AP IMPACT: SWAT Teams Deployed in 911 fraud

Lessons learned by people who have quit Google

Makwana indicted for Fanny Mae malware

"physicalized" servers

Zentific svn repo: alpha available

Holographic Space-Time ?

DACS - Distribution and Configuration System - version 2.0

Video of Cisco IOS attack talk at Chaos Computer Conference

Cosmic radio background noise 6 times higher than expected

We get a leap second tonight

Grow your own bioluminescent algae

Johnson and Ruby/Javascript

Two turntables and a git repo

Quartz Composer and Cruise Control status

Truthy and stupid.rb

The nature of truth

Get2Human

Sunay Tripathi's Solaris Networking Blog

Merry Christmas from XKCD

Merry Christmas from Chiron Beta Prime

Prius Emergency Generator

German folk tune Jazz improv

Memcached speed improvements

FSF sues Cisco

Asterisk Vishing Alert

Google's Native Client... the next ActiveX?

Waterballs

YAGNI development assistant

HA-xVM demo video posted

Kemari 1.0 released - HA Xen

The Decline and Fall of Agile

Zone Alarm 2009 Free Tomorrow

kenai.com - xVM Server Project site

58% Spam Drop from one colo shutdown

Xenomips - a Xen friendly domU version of Dynamips - Emulate a Cisco 7200

Debian and Android dual-boot on the G1

Sipper (SIPr) - a SIP testing framework in ruby

DBslayer - a SQL abstraction layer using JSON

Clojure - JVM based LISP dialect with immutable persistent data structures that are inherently thread safe

Fingerworks keyboard in a MacBookPro

NfSen - Netflow Sensor

The Phoenix BIOS hypervisor is Xen

Do you live in a Constitution-Free zone?

Puppet presentation at NYCOSUG this month

Kemari - Xen lock-step HA

XenSmartIO - Infiniband IO for Xen

Starting with b100, OpenSolaris has virtual consoles

OpenSolaris testfarm build server interface now available

Firefox M9 Fenric - Maemo alpha

SystemZ - aka Sirius - a port of OpenSolaris to IBM System Z mainframe OS running in z/VM mode

40.8% efficient solar cell

FREDNET

World sunlight map

Solaris and ZFS on a Dell 2950, tweaking notes

Logstalgia

Early Access Windows PV drivers for xVM

Economics: The Theory of Interstellar Trade

COMSTAR Admin Guide PDF file

The Financial Crisis: What Happened and What's Next?

3.5" DIY SSD drive

Microsoft usurping ODF

Cisco to run Windows 2008 on their appliance virtually for services

Packetfence: an OpenSource Network Access Control system

Public.resource.org

persist.js - an alternative to gears

Chinese building "impossible" EM drive

Supertinykeyboard

COMSTAR SMTF - solaris FC, SAS, and iSCSI targets

Flexiscale - yet another control panel?

RightScale - cloud control panels?

GoGrid, a servepath company.

OSCON in 37 minutes

Criticial ESXi remote vulnerability in openwsman

Parasitic power

Microsoft FUD on VMWare: vmwarecostswaytoomuch.com

nmap builds zenmap topology maps

Don't forget about BarCampTampaBay

RubyConf08, In Orlando

The LHC accelerates, and that's what it's all about.

Fun with mechanical turk

Sun's launch of xVM, live webinar

Microsoft to give away Hyper-V for free, live migration by 2010

Ubuntu's Intrepid Ibex will be followed by Jaunty Jackalope

Why Xen traps negative segment offsets

Rails 2.1.1 more REXML bug fixes

ISO torrent for OS2008.11

Indiana OS2008.03 RN3 released - based on nv_b96

Skype Mobile Phone (Not in the US)

Youtube gets closed captioning support

Getting xVM to work on OpenSolaris 2008.05

Xen Memory Overcommit

Algae farming for biofuels

Mozilla Ubiquity

How a VoIP E911 call is handled

A critique of RDMA

MonetDB - a column based RDBMS, ideal for time series data

BarcampTampaBay

Intel's programmable matter

Nexenta Hackathon

The value of side projects

VMfaq's comparison of virtual storage IO

Xen 3.3 released

USB3.0 cables

Intel wireless power.

Xen and Solaris, a log of experience.

Adeona.cs.washington.edu

OpenSolaris CR#6654713 - 32G limit bug stemmed from bad USB hardware? Perhaps fixed?

Xen CPUID example config

OpenSolaris CommonArrayManager

Multiple zero capacity quantum communication channels can actually transmit non-zero amounts of data thanks to entanglement

Sharity-Light - smbfs derived samba clone

Drizzle, a thin mysql, generating buzz

VMWare to offer ESX hypervisor for free

Veedeeeyes

Dr Horrible's Sing-Along-Blog

Fan, the programming language.

Blackberry Thunder with Haptics keyboard

iPhone App Store Live Walkthrough now available

Google Protocol Buffers

Time to patch your DNS

Recent JVM benchmarks

Overclocking tool for the Mac Pro

ADO.NET Entity Framework (Microsoft's new ORM) given a non-confidence vote by beta testers

Ruby interpreter flaws make the case for JRuby

The Stalled Server Room

AdvFS - Tru64 filesystem ported to Linux

OpenSolaris 2005.05 repository update to b91 - follow these instructions carefully

SXCE can ZFS install as of b90

Vertebra: EngineYard's Next Generation Cloud Computing Platform

Skype 4.0 beta overhauls video chat

Mozilla org receives traditional IE cake

Toyota Prius to go entirely Electric

Bill Gates steps down permanently for philanthropic activities

Men write code from Mars, Women write more helpful code from Venus

SproutCore - a GUI event driven model javascript web development platform, rails based by the looks of it.

Finding ARPANET

DRBD LVM Xen = Bug. A rather nasty one at that.

Intel unveils Ct as an extension for C/C to encourage threaded programming for multiple cores

VMWare ThinApp - Run any Windows app on any version of Windows

JDBC adapter for HBase

JRuby-Rack <-- a JRuby port of Rack

Rack <-- a lighter cousin to Merb, fully threaded and no Mutex.

Datamapper.org <-- ActiveRecord like, with no need to do migrations, it just kind of handles that by itself internally automagically.

Solaris Cluster Express (SCX) 6/08 released.

a-i-studio.com/cmd

CMDLogParserDemo

Changing solaris' default password hashing

Texas based service provider explosion affects 9,000 servers and 7,500 customers.

Jruby on Rails on Tomcat deployed as as WAR file

Rubinius

Milkfish.org SIP Router

42 more of the best Linux games

42 of the best Linux games

XenWindowsGplPv drivers

Use Google's cached ajax libraries

Arduino microcontroller with OS/X

The metasploit page describing the full impact of the poor RNG.

Holger Bert's blog post on the openssl RNG fiasco

Cayac - Cherokee MySQL PHP5 phpMyAdmin

ZFS very slow under an xVM kernel

VMWare's review-board.org

Google DocType

Dynamically editing libvirt xml configs while a VM is running to redefine reboot flags.

Chronoton - the time travelling robot who's best friend is a talking pie game

Endace DAG

Your pizza is done

Rietveld - Google's code review tool

Opensource multitouch displays

RTL8139 drivers supporting QEMU tcp segmentation offloading (XP's default driver does not) - doubles networking speed of Xen HVM networking without using the GPLPV drivers

Corporate map.

Ono - an efficient way to locate nearby peers

Solaris CIFS integrated AD with ZFS acls

Samba Winbind and ZFS acl working together

Why's unholy Ruby to Python .pyc compiler

Zentific poll daemon 1.0 beta

Solaris SAM-QFS NFS and OS/X

OpenSolaris 2008.05 final ISO image

Mon, 25 Jul 2005

Please excuse this brain dump. As ideas come up, I continue to edit this node. Eventually, some structure will be enforced.

Inspired by SSHFS and SHFS, what would it take to make a filesystem that spans a cluster of servers and exposes aggregate diskspace while still mirroring data?

Exposing a filesystem with FUSE on a master node would be ideal, with some form of WebDAV network access (using something as simple as Apache mod_dav) for client access.

Most distributed filesystems have the idea of a "master" for metadata:

  • Google's Filesystem has a master model with distributed "chunk servers" for the data. Not OpenSource. Also not POSIX, it's a programming API interface, you can't "mount" it AFAIK. They could probably throw a FUSE filesystem together in short order if they really wanted to.
  • HDFS (previously NDFS), or the Hadoop (Nutch) Distributed Filesystem is a Java knockoff of the Google Filesystem. As a backend for the Apache Lucene Nutch project, it is a programmatic API inteface filesystem. While you can't mount it, writing a FUSE frontend wouldn't be hard.
  • PVFS v1 has one master, v2 has multiple masters, but no mirroring - meant for high-IO scientific clusters.
  • OpenAFS has many servers, and mirrors at the volume level, but requires a complex kerberos infrastruture and much manual volume creation to balance the layout. There is only one read/write volume, the rest of the volume replicas are read-only. Don't think I'm not temped by OpenAFS, it just doesn't solve the need we have at the moment (long story).
  • CODA (sometimes referred to as AFSv3) offers disconnected roaming, but mirrors at the server level - not at a volume level.
  • Lustre has a master model, but mirrors on a volume level.
  • Intermezzo was Peter J. Braam's predecessor to Lustre. Ideal for straight mirroring, not distributing files throughout a cluster.
  • both GFS and OpenGFS use a DLM cluster arrangement with shared storage to present a shared filesystem. CLVM mirroring is very young (lvmcreate -m is undocumented at best, allocation is impossible to specify, and you can't have more than one mirror log volume yet). Boy was this fun to play with.
  • CXFS is SGI's Clustered XFS. Very similar to GFS, only cross platform and very scalable.
  • OpenSSI's CFS is little more than network mirroring across whatever underlying filesystem to present a unified root image for the OpenSSI cluster. Not what we're looking for.
  • MFS and DFSA are from Mosix / Openmosix. MFS is the feature of openMosix that enables you access to remote filesystems as if those filesystems were locally mounted. With DFSA enabled, system calls will be executed on the remote node without migrating the process back to it's home node

There are others, but these are the "big boys" that I can think of.

There are a couple of distributed filesystems that run without a master server. This isn't trivial to implement:

  • GPFS is IBM's General Parallel File System. What is claims is downright nirvana. I've not have the time (or money) to play with it. Seriously, read this page. I want a copy. Not OpenSource. ;)
  • xFS is Berkeley's Serverless Network File Service. Basically, a log based network striped filesystem with metadata "map" servers that trade "write tokens" to update files between each other.

Storage servers in the cluster might each have some space set aside to this purpose. The easiest way would be to create and mount a loopback file filesystem with the space to be shared:

storage-node$ mkdir -p /data/cornfs/spool/ /data/cornfs/export/
storage-node$ dd if=/dev/zero of=/data/cornfs/spool/storage_fs bs=1M count=5k
storage-node$ mke2fs -f /data/cornfs/spool/storage_fs
storage-node$ mount -o loop /data/cornfs/spool/storage_fs /data/cornfs/export/storage

On the Master, each storage server's remote filesystem would be mounted based on the master's config (which is modeled likewise in a filesystem tree):

master-node$ mkdir -p /data/cornfs/cfgs/nodes
master-node$ cd /data/cornfs/cfgs/nodes
master-node$ echo /data/cornfs/export/storage > storage-node1
master-node$ echo /data/cornfs/export/storage > storage-node2

master-node$ mkdir -p /data/cornfs/import
master-node$ for node in * ; do mkdir -p /data/cornfs/import/$node ; shfsmount $node:`cat $node` /data/cornfs/import/$node ; done

The beauty of this is that shfs caches files and works with pretty much any host you can ssh into (including Windows via Cygwin). There are some shortcomings to shfs: "df -i" doesn't work, extended attributes aren't maintained, and it only works from linux kernels (were there only a Mac port ;)

Each file in the master tree will have a FILE pathname, including the filename.

Ideally, each file would have at least two copies. For our purposes, I'll suggest that this filesystem should endeavor to track two mirrors for every file, and clean up any "extra" copies.

The Master itself should have a few trees for the metadata. This leaves us with a few directory trees:

/data/cornfs/metadata/state/FILE
- the FILE has the same owner, group, permissions, ctime/atime/mtime, and size as the actual FILE (as a sparse file). 
- Extended attributes make a great storage for things like the primary and secondary mirror server names (setxattr/getxattr).

/data/cornfs/import/SERVER/FILE
- contains the actual file, if SERVER is one of the FILE mirrors.

/data/cornfs/metadata/SERVER/FILE
- this is a sparse version of the above file, used as a sanity check and for regenerating a SERVER from scratch. 
- This local metadata replica of a remote server is the masters opinion of what the server actually holds. 
- If something does not exist in this copy, but exists on the server, it should be removed from that server. 
- If something exists in this copy but not on the server, corruption has occurred.

/data/cornfs/metadata/cache/FILE
- a directory tree containing the past N days worth of accessed FILEs (pruned via cron)

This ends up requiring more than twice the number of actual file inodes to represent the full filesystem on the master. One full copy of the entire metadata state, one copy spread across all of the servers for their metadata state replica on the master server, and some fraction of the filesystem in cache for frequent and/or recent file access.

The Master filesystem would be mounted somewhere handy to be filled, like /master:

master$ mkdir /master
master$ /opt/cornfs/current/bin/cornfs /master

Any new files created under /master would be written to the cache until the user closes the file. On file close, the Master needs to:

  1. Lock the file in the metadata state tree so that no two close operations can occur in parallel. Run a "df" on all of the /data/cornfs/import/ filesystems to see which two have the most available space, then fork off a copy to those respective filesystems.
  2. Creates a /data/cornfs/metadata/state/ sparse file
  3. Tag the /data/cornfs/metadata/state/ file with a "mirror1" extended attribute when the copy completes (setxattr). Update the /data/cornfs/metadata/SERVER/ file to mark that the copy was successful.
  4. Tag the /data/cornfs/metadata/state file with a "mirror2" extended attribute when the copy completes (setxattr). update the /data/cornfs/metadata/SERVER/ file to mark that the copy was successful.

When release() is called for a file, if any write() calls were used on the file, it should have been flagged as "dirty" (by an associative array in memory, along with an extended attribute just in case the running daemon is killed). If a file is dirty, it needs to be written out to the mirrors on release(). If a file is clean, don't do anything at all! The file is handily in the cache for the next access.

When reading a file:

  1. Check /data/cornfs/metadata/cache/ for the file. Open if it exists.
  2. If the file does not exist, one of the mirrors would be selected for the file.
  3. Copy the file to the cache. There is nothing wrong with allowing the client to read, as long as it doesn't try to read more data than has been streamed from the mirror server so far (seek or read() past the EOF as the cache file grows). In that case, the read or seek should block until the entire file is in the cache.
  4. If no mirrors are accessible, an error would be returned.

When moving a file/directory:

  1. Move the state/ copy of the file, if it exists. If this fails for any reason, pass the error code up.
  2. Move the cache/ copy of the file, if it exists.
  3. Iterate through the local metadata/SERVER, moving the file, if it exists.
  4. Iterate through the remote import/SERVER, moving the file, if it exists.

When unlinking (removing) a file/directory:

  1. Remove the state/ copy of the file, if it exists. If this fails for any reason, pass the error code up.
  2. Remove any cache/ copy of the file, if it exists.
  3. Iterate through the local metadata/SERVER, removing the file/dir, if it exists.
  4. Iterate through the remote import/SERVER, removing the file/dir, if it eixsts.

Changing permissions, access times, or ownership would really only affect the /data/cornfs/metadata/state/ sparse file.

Most metadata information would use the state sparse file.

A "helper daemon" needs to run periodically to make sure that servers are accessible.

  1. If a server becomes unreachable but has not timed out as "dead", read()s fail over to the other mirror (or fail if both mirrors are unreachable - such operations should probably trigger a mirror copy() as well), and write()s move the unreachable mirror of a file over to another reachable server.
  2. If a server is totally inaccessible for a period of time to mark it as "dead", the helper daemon needs to refer to the /data/cornfs/metadata/SERVER/ tree and create a new mirrored copy for each file across the farm. In the process, the metadata/SERVER tree will be pruned.
  3. A "sanity" script must be periodically run against each metadata/SERVER tree to see if a copy of a file exists on the server that does NOT exist in the metadata/SERVER tree. If so, that's an orphaned mirror, and should be deleted. Orphans would happen when the master's metadata state for a server says something shouldn't be there, but the server has been down during the time when the mirror would have been removed

As metadata state is updated, locking must be used to ensure atomic operations on the metadata tree. We would not want multiple updates to a file to occur out of order due to a delay in a copy operation to a server in the field.

Speed and availability should be consistently monitored to select faster responding mirrors (if possible) and/or noting that nodes are unreachable for file operations to trigger a mirror for a file with a broken mirror.

Symlinks, block/character devices, and other non-files are stored in the metadata state/ tree alongside the sparse files that represent the actual files that are being distributed.

There is no "inode" construct per se, outside of the metadata state/ tree. That is the "master metadata" that most filesystem operations use. Only when reading/writing, opening/closing, moving, or unlinking, do the mounted server filesystems under import/ get involved to hold the data.

Making this a single instance store (ideal for backups) would require just a bit more logic to include an SHA1/MD5 hash encoded as a directory tree (broken up by octet to a path tree structure); something like:

/data/cornfs/metadata/state/SHA1/MD5/object

Another neat extension would be to build a "revision history" of documents in the filesystem by:

  1. On close(), if a file has changed, it should be archived.
  2. Move original version of files into a revision/ metadata tree by hash ID.
  3. Copy in the new version of file from the cache to the mirrors.
  4. Tag the state/ tree of the new file with an extended attribute as to the "previous revision"'s SHA1/MD5 HASH in the revision/ metadata tree.

This would address files that change, but would not save us from directory trees that are removed. For this, we would want an archive/ metadata tree by datestamp:

  1. On unlink(), create an archive/TIMESTAMP/ metadata tree and move the file there.

Moving files and/or directory trees around in state/ would maintain the extended attributes, effectively retaining the revisionist history FOR FREE! When files are moved, the mirrors must be moved as well.

Reconstructing things from the revision/ and archive/ trees would be interesting, but well beyond the initial scope of this endeavor.

The quickest way to throw this together would be with the Fuse.pm perl module. I'm actively writing code now.

The eventual goal would be to write a thread aware C version based on the above prototype, primarily for speed reasons.

More to come.. SOON..

Google
 
Web ian.blenke.com