What do Google Reader engineers read?
What Would Depression 2009 Look Like?
Smart car turned into a monster
The computer science behind EMC’s cloud storage
Let's get "Teh" in the dictionary! Think of the Scrabble® players!!!
Expanding the Cloud: Amazon CloudFront
Rasputine, An XMPP Telnet Client
● High quality YouTube video hack
Holding a conversation with Google
Your Search Activity Predicts Flu Outbreaks
Exclusive: First Look at Blue Spruce, IBM's Next Generation Browser Platform
kenai.com - xVM Server Project site
58% Spam Drop from one colo shutdown
Xenomips - a Xen friendly domU version of Dynamips - Emulate a Cisco 7200
Debian and Android dual-boot on the G1
Sipper (SIPr) - a SIP testing framework in ruby
DBslayer - a SQL abstraction layer using JSON
Fingerworks keyboard in a MacBookPro
The Phoenix BIOS hypervisor is Xen
Do you live in a Constitution-Free zone?
Puppet presentation at NYCOSUG this month
XenSmartIO - Infiniband IO for Xen
Starting with b100, OpenSolaris has virtual consoles
OpenSolaris testfarm build server interface now available
Firefox M9 Fenric - Maemo alpha
SystemZ - aka Sirius - a port of OpenSolaris to IBM System Z mainframe OS running in z/VM mode
Solaris and ZFS on a Dell 2950, tweaking notes
Early Access Windows PV drivers for xVM
Economics: The Theory of Interstellar Trade
The Financial Crisis: What Happened and What's Next?
Cisco to run Windows 2008 on their appliance virtually for services
Packetfence: an OpenSource Network Access Control system
persist.js - an alternative to gears
Chinese building "impossible" EM drive
COMSTAR SMTF - solaris FC, SAS, and iSCSI targets
Flexiscale - yet another control panel?
RightScale - cloud control panels?
Criticial ESXi remote vulnerability in openwsman
Microsoft FUD on VMWare: vmwarecostswaytoomuch.com
nmap builds zenmap topology maps
Don't forget about BarCampTampaBay
The LHC accelerates, and that's what it's all about.
Sun's launch of xVM, live webinar
Microsoft to give away Hyper-V for free, live migration by 2010
Ubuntu's Intrepid Ibex will be followed by Jaunty Jackalope
Why Xen traps negative segment offsets
Rails 2.1.1 more REXML bug fixes
Indiana OS2008.03 RN3 released - based on nv_b96
Skype Mobile Phone (Not in the US)
Youtube gets closed captioning support
Getting xVM to work on OpenSolaris 2008.05
How a VoIP E911 call is handled
MonetDB - a column based RDBMS, ideal for time series data
VMfaq's comparison of virtual storage IO
Xen and Solaris, a log of experience.
OpenSolaris CR#6654713 - 32G limit bug stemmed from bad USB hardware? Perhaps fixed?
OpenSolaris CommonArrayManager
Sharity-Light - smbfs derived samba clone
Drizzle, a thin mysql, generating buzz
VMWare to offer ESX hypervisor for free
Fan, the programming language.
Blackberry Thunder with Haptics keyboard
iPhone App Store Live Walkthrough now available
Overclocking tool for the Mac Pro
ADO.NET Entity Framework (Microsoft's new ORM) given a non-confidence vote by beta testers
Ruby interpreter flaws make the case for JRuby
AdvFS - Tru64 filesystem ported to Linux
OpenSolaris 2005.05 repository update to b91 - follow these instructions carefully
SXCE can ZFS install as of b90
Vertebra: EngineYard's Next Generation Cloud Computing Platform
Skype 4.0 beta overhauls video chat
Mozilla org receives traditional IE cake
Toyota Prius to go entirely Electric
Bill Gates steps down permanently for philanthropic activities
Men write code from Mars, Women write more helpful code from Venus
DRBD LVM Xen = Bug. A rather nasty one at that.
Intel unveils Ct as an extension for C/C to encourage threaded programming for multiple cores
VMWare ThinApp - Run any Windows app on any version of Windows
JRuby-Rack <-- a JRuby port of Rack
Rack <-- a lighter cousin to Merb, fully threaded and no Mutex.
Solaris Cluster Express (SCX) 6/08 released.
Changing solaris' default password hashing
Texas based service provider explosion affects 9,000 servers and 7,500 customers.
Jruby on Rails on Tomcat deployed as as WAR file
42 more of the best Linux games
Use Google's cached ajax libraries
Arduino microcontroller with OS/X
The metasploit page describing the full impact of the poor RNG.
Holger Bert's blog post on the openssl RNG fiasco
Cayac - Cherokee MySQL PHP5 phpMyAdmin
ZFS very slow under an xVM kernel
Dynamically editing libvirt xml configs while a VM is running to redefine reboot flags.
Chronoton - the time travelling robot who's best friend is a talking pie game
Rietveld - Google's code review tool
Opensource multitouch displays
Ono - an efficient way to locate nearby peers
Solaris CIFS integrated AD with ZFS acls
Samba Winbind and ZFS acl working together
Why's unholy Ruby to Python .pyc compiler
OpenSolaris 2008.05 final ISO image
Twitter abandoning Ruby on Rails
HP makes memory from a once-theoretical circuit
Setting Up an OpenSolaris NAS Box: Father-Son Bonding - The Video
Linux kernel Xen self-ballooning patch
Coolstack - Yet another group of solaris packages
SFE - Spec Files Extra - or, solaris's ports system
ksplice - live linux kernel patching
ZFS-102-A.pkg - binary package build of newer ZFS for Mac
Changing boot flags for a solaris domU guest
callflow - SIP callflow diagram generator
sdedit - quick sequence diagram editor
Milax - The OpenSolaris Small Live CD
Big Nerd Ranch on Windows/Linux/Leopard single signon
Sun touts big plans for OpenSolars as first release nears
Heroku - EC2 based Rails hosting.
Meadowcourt's compiled WindowsXenPV driver, v0.8.8, as built from win-pvdrivers.hg repo
Network Solutions hijacks all customer's unused subdomains
ZFS speed bump: set zfs_nocacheflush = 1
We Don't Use Software That Costs Money Here
Hubble - a PlanetLab realtime Internet "blackhole" monitor
Citrix price jumps on rumors of potential IBM/Cisco bidding ware
TechCrunch labs on their AppEngine deployment
pash - because powershell was too cool to let microsoft keep to itself
Brazil migrates 430 thousand boting machines to Linux
The Machine Emulator - TME can emulate a sparc4 with OBP
Google releases new GCC linker
Automatic generation of peephole superoptimizers
Xen.org Trademark Policy for Review
SXCE b85 has problems booting under Xen 3.2
VNRP == opensolaris quagga rbridges crossbow xVM
problems reprobing iscsi devices with solaris 10
LSI MegaRAID SAS/Dell PERC5 driver for Solaris
dm-band block IO bandwidth controller
Dojo.storage - Google Gears workalike?
ooma.com - free phone service after you buy their device
Hacking defibrilators shockingly easy
Microsoft working with Eclipse.
Pentagon attack last June stole an "amazing amount" of data
Solaris and Solaris Cluster on HP ProLiant Servers
Apple Introduces new MacBook and MacBook Pro models
Sun leaks 6-core Xeon, Nehalem details
Xen and Solaris - a journal of sorts
How to save the world with ZFS and 12 USB sticks
Xvm: a summary of creation of various Xen domU
OpenSolaris b82 comes with CoolStack
Dilber PHB on Virtualization Consultants
Sun xVM Ops Center GA v1.0 tomorrow
KernelTrap on the 2.6.23 Xen merge
IETF XMPP/SIMPLE Interworking Draft
PSYCed - IRC/XMPP server that gateways transparently between both
OTR - Off The Record, Homepage. IM Encryption.
SIPE - Pidgin plugin for SIP/SIMPLE with Microsoft LCS compatibility hacks
Price Waterhouse Cooper's Global Cable Map
Solaris Windows iSCSI speedup disabling NAGLE
OpenSolaris Storage Developer Wish List
Nexenta Builder - build your own Nexenta based distribution
Microsoft to acquire SideKick maker Danger
Linux Kernel 2.6.23-2.6.24 vmsplice local root exploit
The evolution of Tech Company logos
Mindstorms NXT Rubiks Cube Solver
Cut four undersea cables, shame on you, cut a fifth, also shame on you
Koha - OpenSource Integrated Library System
SIPE - SIP Exchange protocol - or, how to get Pidgin to talk to Microsoft Live Communication Server
Amazon SimpleDB written in Erlang
Xen DR7 and CR4 Registers Multiple Local DoS vulnerabilities
XMLPulse - parse xen dom0/domu stats
The rist of the FOSS spinmeister
Smartphones patented - lawsuits immediately filed
H-Sphere cross-platform hosting control-panel
Mystery infestation strikes Linux/Apache web sites
GNU/Solaris - When the fun begins
KDE goes cross platform with Windows and Mac/OSX support.
Microsoft prints get-out-of-jail card for Vista Home
Tsung - an erlang based multi-protocol distributed load testing tool
Microsoft relents, ban on vista virtualization is lifted
Hyperic podcast talking smack with Luke KAnies of Puppet
The Mysql storage engines, and when they are appropriate.
MADOCA - Message And Database Oriented Control Architecture
SMP Xen HVM Windows guests need timer_mode=1
James Randi is coming to Tampa
Information Of Those Who Appealed Watch List Compromised
Tata Nano - $2500 world's cheapest car
Air Travel with Spare Batteries? Check the changes to what is permitted starting tomorrow.
Open Configuration and Management Layer
FiveRuns RM-Manage - rails project monitoring
VLDB - Very Large Data Base Endowment Inc - nonprofit
Elastix - a more friendly Trixbox fork
A Glimpse and a Hook - a take on resumes
Xirrus - LISA used 7 arrays to provide WiFi
dopd - an easier way to keep drbd primary/secondaries in sync
OpenSIM - run your own SecondLife grid.
$4million in hardware lost in London data center heist
iscsi block device script for /etc/xen/scripts
Quaqua - Aqua look and feel widgets for jvm
Chimps beat humans in memory tests.
Level 3 needs technicians with FIREBALLS
10 steps to close down an open society
Longer flights to avoid air traffic control charges
News release from Six Apart about LJ sale to SUP
Optimus keyboard is finally available
pkgGen and logGen and Packagemaker - repackage os/x packages to deploy
Jumpbox.com - virtual appliances
TelegraphCQ - barkeley database research - adaptive dataflow capture, combine, analyze
UK loses CD of private info on 25million citizens
Solaris Automatic Migration opensourced
AVS ZFS Demo <-- replicated ZFS pool
Xen Virtualization book not yet published for sell on Amazon
Phoenix BIOS releasing its own hypervisor
Andrew Warfield's other publications
Parallax - managing storage for a million virtual machines, from the Xen guys at Cambridge
Kepler project - GRID scientific workflow engine
Google Code Map/Reduce mini lectures
What 24 would have been like in 1994.
WaterRoof - Mac OS/X Firewall Manager
10 reasons why Oracle databases run best on VMWare
Google Caja - allow scripts in a 3rd party context
Xen Windows PV drivers - opensource mercurial repository
QuickSilver - opensourced 11/06/07
vmcasting.org - someone else "gets it"
ASUS EEEPC701 starts to appear
Perian - Opensource quicktime codecs
RSnapshot - an rsync based dirvish like tool
Flyback - a google code project equivalent to Apple's Time Machine, for Linux
Apple tablet PC is real, says Asus.
producten.hema.nl - wait for this one to load
Google rolls out the Open Handset Alliance
Cost analysis of Windows Vista Content Protection
Git - a Google Talk by Randal Schwartz
indeed.com - MIT search engine for jobs crawled from monster, dice, etc.
Tomshardware's RAID Migration Adventure
Theo de Raadt on Virtualization, and the sate of OpenBSD Xen
Bitlbee - IRC gateway all of your other IM traffic
Off The Record - encrypted IM overlay
Please excuse this brain dump. As ideas come up, I continue to edit this node. Eventually, some structure will be enforced.
Inspired by SSHFS and SHFS, what would it take to make a filesystem that spans a cluster of servers and exposes aggregate diskspace while still mirroring data?
Exposing a filesystem with FUSE on a master node would be ideal, with some form of WebDAV network access (using something as simple as Apache mod_dav) for client access.
Most distributed filesystems have the idea of a "master" for metadata:
There are others, but these are the "big boys" that I can think of.
There are a couple of distributed filesystems that run without a master server. This isn't trivial to implement:
Storage servers in the cluster might each have some space set aside to this purpose. The easiest way would be to create and mount a loopback file filesystem with the space to be shared:
storage-node$ mkdir -p /data/cornfs/spool/ /data/cornfs/export/
storage-node$ dd if=/dev/zero of=/data/cornfs/spool/storage_fs bs=1M count=5k
storage-node$ mke2fs -f /data/cornfs/spool/storage_fs
storage-node$ mount -o loop /data/cornfs/spool/storage_fs /data/cornfs/export/storage
On the Master, each storage server's remote filesystem would be mounted based on the master's config (which is modeled likewise in a filesystem tree):
master-node$ mkdir -p /data/cornfs/cfgs/nodes
master-node$ cd /data/cornfs/cfgs/nodes
master-node$ echo /data/cornfs/export/storage > storage-node1
master-node$ echo /data/cornfs/export/storage > storage-node2
master-node$ mkdir -p /data/cornfs/import
master-node$ for node in * ; do mkdir -p /data/cornfs/import/$node ; shfsmount $node:`cat $node` /data/cornfs/import/$node ; done
The beauty of this is that shfs caches files and works with pretty much any host you can ssh into (including Windows via Cygwin). There are some shortcomings to shfs: "df -i" doesn't work, extended attributes aren't maintained, and it only works from linux kernels (were there only a Mac port ;)
Each file in the master tree will have a FILE pathname, including the filename.
Ideally, each file would have at least two copies. For our purposes, I'll suggest that this filesystem should endeavor to track two mirrors for every file, and clean up any "extra" copies.
The Master itself should have a few trees for the metadata. This leaves us with a few directory trees:
/data/cornfs/metadata/state/FILE
- the FILE has the same owner, group, permissions, ctime/atime/mtime, and size as the actual FILE (as a sparse file).
- Extended attributes make a great storage for things like the primary and secondary mirror server names (setxattr/getxattr).
/data/cornfs/import/SERVER/FILE
- contains the actual file, if SERVER is one of the FILE mirrors.
/data/cornfs/metadata/SERVER/FILE
- this is a sparse version of the above file, used as a sanity check and for regenerating a SERVER from scratch.
- This local metadata replica of a remote server is the masters opinion of what the server actually holds.
- If something does not exist in this copy, but exists on the server, it should be removed from that server.
- If something exists in this copy but not on the server, corruption has occurred.
/data/cornfs/metadata/cache/FILE
- a directory tree containing the past N days worth of accessed FILEs (pruned via cron)
This ends up requiring more than twice the number of actual file inodes to represent the full filesystem on the master. One full copy of the entire metadata state, one copy spread across all of the servers for their metadata state replica on the master server, and some fraction of the filesystem in cache for frequent and/or recent file access.
The Master filesystem would be mounted somewhere handy to be filled, like /master:
master$ mkdir /master
master$ /opt/cornfs/current/bin/cornfs /master
Any new files created under /master would be written to the cache until the user closes the file. On file close, the Master needs to:
When release() is called for a file, if any write() calls were used on the file, it should have been flagged as "dirty" (by an associative array in memory, along with an extended attribute just in case the running daemon is killed). If a file is dirty, it needs to be written out to the mirrors on release(). If a file is clean, don't do anything at all! The file is handily in the cache for the next access.
When reading a file:
When moving a file/directory:
When unlinking (removing) a file/directory:
Changing permissions, access times, or ownership would really only affect the /data/cornfs/metadata/state/ sparse file.
Most metadata information would use the state sparse file.
A "helper daemon" needs to run periodically to make sure that servers are accessible.
As metadata state is updated, locking must be used to ensure atomic operations on the metadata tree. We would not want multiple updates to a file to occur out of order due to a delay in a copy operation to a server in the field.
Speed and availability should be consistently monitored to select faster responding mirrors (if possible) and/or noting that nodes are unreachable for file operations to trigger a mirror for a file with a broken mirror.
Symlinks, block/character devices, and other non-files are stored in the metadata state/ tree alongside the sparse files that represent the actual files that are being distributed.
There is no "inode" construct per se, outside of the metadata state/ tree. That is the "master metadata" that most filesystem operations use. Only when reading/writing, opening/closing, moving, or unlinking, do the mounted server filesystems under import/ get involved to hold the data.
Making this a single instance store (ideal for backups) would require just a bit more logic to include an SHA1/MD5 hash encoded as a directory tree (broken up by octet to a path tree structure); something like:
/data/cornfs/metadata/state/SHA1/MD5/object
Another neat extension would be to build a "revision history" of documents in the filesystem by:
This would address files that change, but would not save us from directory trees that are removed. For this, we would want an archive/ metadata tree by datestamp:
Moving files and/or directory trees around in state/ would maintain the extended attributes, effectively retaining the revisionist history FOR FREE! When files are moved, the mirrors must be moved as well.
Reconstructing things from the revision/ and archive/ trees would be interesting, but well beyond the initial scope of this endeavor.
The quickest way to throw this together would be with the Fuse.pm perl module. I'm actively writing code now.
The eventual goal would be to write a thread aware C version based on the above prototype, primarily for speed reasons.
More to come.. SOON..