Fenxi - Performance analysis made easy
Changing libgnomecups For Multiple Evolution Users
Happy National Sys Admin Appreciation Day!
ESX iSCSI Basic Configuration from the CLI
Tape Rants and Raves: LTO4 Rules
apparently you aren't dead until you start to stink
Charlie Goes to Candy Mountain
Seattle Scalability Conference, Pt II
Overclocking tool for the Mac Pro
ADO.NET Entity Framework (Microsoft's new ORM) given a non-confidence vote by beta testers
Ruby interpreter flaws make the case for JRuby
AdvFS - Tru64 filesystem ported to Linux
OpenSolaris 2005.05 repository update to b91 - follow these instructions carefully
SXCE can ZFS install as of b90
Vertebra: EngineYard's Next Generation Cloud Computing Platform
Skype 4.0 beta overhauls video chat
Mozilla org receives traditional IE cake
Toyota Prius to go entirely Electric
Bill Gates steps down permanently for philanthropic activities
Men write code from Mars, Women write more helpful code from Venus
DRBD LVM Xen = Bug. A rather nasty one at that.
Intel unveils Ct as an extension for C/C to encourage threaded programming for multiple cores
VMWare ThinApp - Run any Windows app on any version of Windows
JRuby-Rack <-- a JRuby port of Rack
Rack <-- a lighter cousin to Merb, fully threaded and no Mutex.
Solaris Cluster Express (SCX) 6/08 released.
Changing solaris' default password hashing
Texas based service provider explosion affects 9,000 servers and 7,500 customers.
Jruby on Rails on Tomcat deployed as as WAR file
42 more of the best Linux games
Use Google's cached ajax libraries
Arduino microcontroller with OS/X
The metasploit page describing the full impact of the poor RNG.
Holger Bert's blog post on the openssl RNG fiasco
Cayac - Cherokee MySQL PHP5 phpMyAdmin
ZFS very slow under an xVM kernel
Dynamically editing libvirt xml configs while a VM is running to redefine reboot flags.
Chronoton - the time travelling robot who's best friend is a talking pie game
Rietveld - Google's code review tool
Opensource multitouch displays
Ono - an efficient way to locate nearby peers
Solaris CIFS integrated AD with ZFS acls
Samba Winbind and ZFS acl working together
Why's unholy Ruby to Python .pyc compiler
OpenSolaris 2008.05 final ISO image
Twitter abandoning Ruby on Rails
HP makes memory from a once-theoretical circuit
Setting Up an OpenSolaris NAS Box: Father-Son Bonding - The Video
Linux kernel Xen self-ballooning patch
Coolstack - Yet another group of solaris packages
SFE - Spec Files Extra - or, solaris's ports system
ksplice - live linux kernel patching
ZFS-102-A.pkg - binary package build of newer ZFS for Mac
Changing boot flags for a solaris domU guest
callflow - SIP callflow diagram generator
sdedit - quick sequence diagram editor
Milax - The OpenSolaris Small Live CD
Big Nerd Ranch on Windows/Linux/Leopard single signon
Sun touts big plans for OpenSolars as first release nears
Heroku - EC2 based Rails hosting.
Meadowcourt's compiled WindowsXenPV driver, v0.8.8, as built from win-pvdrivers.hg repo
Network Solutions hijacks all customer's unused subdomains
ZFS speed bump: set zfs_nocacheflush = 1
We Don't Use Software That Costs Money Here
Hubble - a PlanetLab realtime Internet "blackhole" monitor
Citrix price jumps on rumors of potential IBM/Cisco bidding ware
TechCrunch labs on their AppEngine deployment
pash - because powershell was too cool to let microsoft keep to itself
Brazil migrates 430 thousand boting machines to Linux
The Machine Emulator - TME can emulate a sparc4 with OBP
Google releases new GCC linker
Automatic generation of peephole superoptimizers
Xen.org Trademark Policy for Review
SXCE b85 has problems booting under Xen 3.2
VNRP == opensolaris quagga rbridges crossbow xVM
problems reprobing iscsi devices with solaris 10
LSI MegaRAID SAS/Dell PERC5 driver for Solaris
dm-band block IO bandwidth controller
Dojo.storage - Google Gears workalike?
ooma.com - free phone service after you buy their device
Hacking defibrilators shockingly easy
Microsoft working with Eclipse.
Pentagon attack last June stole an "amazing amount" of data
Solaris and Solaris Cluster on HP ProLiant Servers
Apple Introduces new MacBook and MacBook Pro models
Sun leaks 6-core Xeon, Nehalem details
Xen and Solaris - a journal of sorts
How to save the world with ZFS and 12 USB sticks
Xvm: a summary of creation of various Xen domU
OpenSolaris b82 comes with CoolStack
Dilber PHB on Virtualization Consultants
Sun xVM Ops Center GA v1.0 tomorrow
KernelTrap on the 2.6.23 Xen merge
IETF XMPP/SIMPLE Interworking Draft
PSYCed - IRC/XMPP server that gateways transparently between both
OTR - Off The Record, Homepage. IM Encryption.
SIPE - Pidgin plugin for SIP/SIMPLE with Microsoft LCS compatibility hacks
Price Waterhouse Cooper's Global Cable Map
Solaris Windows iSCSI speedup disabling NAGLE
OpenSolaris Storage Developer Wish List
Nexenta Builder - build your own Nexenta based distribution
Microsoft to acquire SideKick maker Danger
Linux Kernel 2.6.23-2.6.24 vmsplice local root exploit
The evolution of Tech Company logos
Mindstorms NXT Rubiks Cube Solver
Cut four undersea cables, shame on you, cut a fifth, also shame on you
Koha - OpenSource Integrated Library System
SIPE - SIP Exchange protocol - or, how to get Pidgin to talk to Microsoft Live Communication Server
Amazon SimpleDB written in Erlang
Xen DR7 and CR4 Registers Multiple Local DoS vulnerabilities
XMLPulse - parse xen dom0/domu stats
The rist of the FOSS spinmeister
Smartphones patented - lawsuits immediately filed
H-Sphere cross-platform hosting control-panel
Mystery infestation strikes Linux/Apache web sites
GNU/Solaris - When the fun begins
KDE goes cross platform with Windows and Mac/OSX support.
Microsoft prints get-out-of-jail card for Vista Home
Tsung - an erlang based multi-protocol distributed load testing tool
Microsoft relents, ban on vista virtualization is lifted
Hyperic podcast talking smack with Luke KAnies of Puppet
The Mysql storage engines, and when they are appropriate.
MADOCA - Message And Database Oriented Control Architecture
SMP Xen HVM Windows guests need timer_mode=1
James Randi is coming to Tampa
Information Of Those Who Appealed Watch List Compromised
Tata Nano - $2500 world's cheapest car
Air Travel with Spare Batteries? Check the changes to what is permitted starting tomorrow.
Open Configuration and Management Layer
FiveRuns RM-Manage - rails project monitoring
VLDB - Very Large Data Base Endowment Inc - nonprofit
Elastix - a more friendly Trixbox fork
A Glimpse and a Hook - a take on resumes
Xirrus - LISA used 7 arrays to provide WiFi
dopd - an easier way to keep drbd primary/secondaries in sync
OpenSIM - run your own SecondLife grid.
$4million in hardware lost in London data center heist
iscsi block device script for /etc/xen/scripts
Quaqua - Aqua look and feel widgets for jvm
Chimps beat humans in memory tests.
Level 3 needs technicians with FIREBALLS
10 steps to close down an open society
Longer flights to avoid air traffic control charges
News release from Six Apart about LJ sale to SUP
Optimus keyboard is finally available
pkgGen and logGen and Packagemaker - repackage os/x packages to deploy
Jumpbox.com - virtual appliances
TelegraphCQ - barkeley database research - adaptive dataflow capture, combine, analyze
UK loses CD of private info on 25million citizens
Solaris Automatic Migration opensourced
AVS ZFS Demo <-- replicated ZFS pool
Xen Virtualization book not yet published for sell on Amazon
Phoenix BIOS releasing its own hypervisor
Andrew Warfield's other publications
Parallax - managing storage for a million virtual machines, from the Xen guys at Cambridge
Kepler project - GRID scientific workflow engine
Google Code Map/Reduce mini lectures
What 24 would have been like in 1994.
WaterRoof - Mac OS/X Firewall Manager
10 reasons why Oracle databases run best on VMWare
Google Caja - allow scripts in a 3rd party context
Xen Windows PV drivers - opensource mercurial repository
QuickSilver - opensourced 11/06/07
vmcasting.org - someone else "gets it"
ASUS EEEPC701 starts to appear
Perian - Opensource quicktime codecs
RSnapshot - an rsync based dirvish like tool
Flyback - a google code project equivalent to Apple's Time Machine, for Linux
Apple tablet PC is real, says Asus.
producten.hema.nl - wait for this one to load
Google rolls out the Open Handset Alliance
Cost analysis of Windows Vista Content Protection
Git - a Google Talk by Randal Schwartz
indeed.com - MIT search engine for jobs crawled from monster, dice, etc.
Tomshardware's RAID Migration Adventure
Theo de Raadt on Virtualization, and the sate of OpenBSD Xen
Bitlbee - IRC gateway all of your other IM traffic
Off The Record - encrypted IM overlay
SATA drive -> NES cartridge style
Amazon's one-click patents struck down
Morgan Stanley sells entire New York Times stake
Massive installation management tools
GULP: a unified logging architecture for authentication data
EC2 outage loses customer data
FutureOfWebApps conference underway
Microsoft releasing the Source Code for the .NET libraries
Windows 2003 Server Emergency Management Services (EMS) - Special Administration Console (SAC)
Catalyst - the Perl web framework analog to Rails
Fusion io - the power of 1000 harddrives in the palm of your hand
Proggyfonts.com - fixed width font downloads
BarCamp Orlando is this weekend
How to us CHDK to give your Canon digial camera RAW support
Cygnal - When Red5 just won't cut it for an RTMP server
IBM's CoScripter - automating web-based processes
AjaxWindows.com - Another Michael Robertson company
p0f passive fingerprinting IDS
Talking storage systems with Sun's ZFS team
SproutCore - a MVC scaffolding for actual Application development
Skype protocol obfuscation layer
Microsoft Silverlight and the Mono team at Novell join up to create the Moonlight project
Bitlbee - bridge IM client networks to an IRC channel.
EJBCA - The J2EE Certificate Authority
Mcell 3.5" drive has 1GB of DDR RAM 2.5" drive == 110MB/s transfer rates
OpenSolaris Xen domU with a linux dom0
Tentakel: distributd command execution
Ganeti: Opensource virtual server management software for Xen
Seemless dynamic image resizing
Mono and XPCOM scripting VirtualBox
podbrix young woz and jobs playset
Woz gets a speeding ticket for 104mph in a Prius
Google Starts Shared Storage Service
Storm Worm DDoSes scanning machines
Defendant wins access to the Intoxilyzer 5000EN Breathalyzer source code
How to replace graffiti 2 with the original graffiti on a Palm
customizegoogle.com - a firefox plugin for customizing google
Matt Pelletier queried the Mongrel mailing list asking for any personal experiences or tips regarding mongrel and rails hosting in general for some book currently underway.
This was my quick response:
At the moment, the largest site we host is Kapture (www.kapture.info), a Web 2.0 startup. I would be happy to cover our infrastructure in detail if you're interested.
To avoid missing class errors for your models, be sure to define all your models with "model" at the beginning of your application.rb. (NOTE: Edge rails deprecates the use of "model" for loading models, and should load models as it needs them). Alternatively, add something like the following to your application.rb:
# Pre-load every .rb file in the models dir
Dir.foreach(File.dirname(__FILE__) + "/../models") {|file|
model $1.to_sym if file =~ /(\w+).rb$/
}
If you build your mongrel by hand and have a http11.so, do not copy it willy nilly wherever you would like. I had copied it into my gems path (/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/http11.so) by building my own gem with a pre-compiled http11.so as the production boxes don't have a compiler. After installing the gem, I spent countless hours debugging what was causing bizarre SEGFAULT errors. Normally, mongrel installs http11.so to your local siteruby directory (/usr/local/lib/siteruby/1.8/i486-linux/http11.so). Apparently if you have http11.so in your gems lib dir, it causes some deep magic problem with ruby that you'd have to float by Zed or someone to fully explain.
Use a proxy that can quickly serve static content. We started with Pound but had problems with various satellite provider's transparent proxies spitting up 500 server errors for some weird reason. I then moved to Apache 2.2 with modproxybalancer, and static content noticably improved. Mongrel is "ok" at serving static content, but it doesn't hurt to have something in front of it to speed things up.
Mongrel will spin out of control. Use a tool like "monit" to monitor the individual mongrel daemons and kill them off and restart them if they stop responding.
Running mongrel with -B will spit out all kinds of fun debugging stuff to the mongrel_debug/ directory.
Sending a SIGUSR1 will cause a mongrel daemon to die off, and spit out a backtrace of the running state of threads when it recevied a signal. This is a good way to trace why mongrel is spinning.
Sending a SIGUSR2 turns on debugging on a running mongrel server. According to Zed, the debug impact is minimal, and thus fairly safe to use in production.
Use apache's mod_deflate (or turn on gzip content encoding in general). The speed improvement is noticable.
Get your application running. Worry about scaling later. Rails will scale, with only minor tweaking.
Use the exception_notifier plugin on your production boxes to send you email whenever a Rails error occurs.
Run memcached and memcache-client (or memcache-client-stats) instead of the default local file based session store.
Learn to use fragment caching and run memcache_fragments. It is well worth the optimization. Use page caching sparingly, particularly on your main page or pages that are being actively slashdotted.
If you run memcached, consider using memcache-client-stats to get some visibility into the activity to your memcache servers.
Keep complex objects out of your session store as much as possible. Avoid storing complex objects in pstore. Don't be afraid to use memcache directly, particularly with alternate namespaces.
If you're running a cluster, run a centralized syslogng and log to syslog. Consider turning off rails logging altogether in your production cluster.
While svn:externals is a wonderful thing for development (/vendors/plugins), freezing plugins, gems, and rails to your project makes it far happier in a production environment; also, much easier to deploy.
You can use Sequoia / C-JDBC with Carob's libmysql replacement with Rails to both mirror and stripe data across a large number of servers.
The Postgres pgcluster project is flakey, and the sole maintainer doesn't consider anything beyond 1.0.11 "stable" enough for anyone to actually use in a production environment. Unfortunately, that is tied to Postgres 7.3, which destroys any hope for ALTERing tables after they are created (among other things). Avoid it for now. If you must replicate, use something like Slony instead (I haven't found anything in ActiveRecord about shunting writes to one database connection and reads to a cluster of replicated slaves).
There are many more things to add to this list, these are just the items that came immediately to mind.
We've taken to writing rules on our whiteboards.
Today I was driven to write the following on my whiteboard:
Ian's Operational Rules
I'm sure this list will grow over time.
Venting this way feels much more constructive for some reason.
While playing with mod_auth_remote, it became apparent that libapr1's changes have slightly broken it.
There were two errors during compilation:
$ /usr/bin/apxs2 -i -a -c mod_auth_remote.c
mod_auth_remote.c: In function` get_remote_auth':
mod_auth_remote.c:109: warning: passing arg 4 of `apr_socket_create' makes integer from pointer without a cast
mod_auth_remote.c:109: error: too few arguments to function `apr_socket_create'
mod_auth_remote.c:117: warning: implicit declaration of function `apr_setsocketopt'
mod_auth_remote.c:117: error: `APR_SO_TIMEOUT' undeclared (first use in this function)
mod_auth_remote.c:117: error: (Each undeclared identifier is reported only once
mod_auth_remote.c:117: error: for each function it appears in.)
mod_auth_remote.c:126: warning: implicit declaration of function `apr_connect'
mod_auth_remote.c:143: warning: implicit declaration of function `apr_base64_encode'
mod_auth_remote.c:147: warning: implicit declaration of function `apr_send'
mod_auth_remote.c:158: warning: implicit declaration of function `apr_recv'
apxs:Error: Command failed with rc=65536
A quick couple of googles later, I worked up this patch to fix both of the above errors:
--- mod_auth_remote.c.orig 2006-09-14 13:33:51.000000000 -0400
+++ mod_auth_remote.c 2006-09-14 13:33:33.000000000 -0400
@@ -17,6 +17,8 @@
#include "http_protocol.h"
#include "http_request.h"
+#include "apr_version.h"
+
typedef struct {
int port;
@@ -106,7 +108,12 @@
apr_sockaddr_t *addr;
- if((val =apr_socket_create(&socket, APR_INET, SOCK_STREAM, r->pool))
+ if((val =apr_socket_create(&socket, APR_INET,
+ SOCK_STREAM,
+#if (APR_MAJOR_VERSION > 0)
+ APR_PROTO_TCP,
+#endif
+ r->pool))
!= APR_SUCCESS)
{
ap_log_rerror(APLOG_MARK, APLOG_ERR, val, r,
@@ -114,7 +121,12 @@
return HTTP_INTERNAL_SERVER_ERROR;
}
+/* APR_SO_TIMEOUT is deprecated in favor of apr_socket_timeout_{set|get} */
+#if (APR_MAJOR_VERSION > 0)
+ apr_socket_timeout_set(socket, (int)r->server->timeout);
+#else
apr_setsocketopt(socket, APR_SO_TIMEOUT, (int)r->server->timeout);
+#endif
if((val = apr_sockaddr_info_get(&addr, conf->remote_server, APR_INET,
conf->port,0,r->pool)) != APR_SUCCESS)
{
And now everything builds just fine.
Again, the patch is here: mod_auth_remote-apache2.2.patch
The Problem
We have been getting a "undefined class/module" error on 2 of our 6 app servers in one testbed.
Same sourcecode, same installs (systemimager and an isconf style configuration management harness), same binaries. Identical everything. Pound + mongrel + memcached + postgres (pgcluster).
The odd bit here is that our "dev" and "prod" pairs had one node reporting the error, but the other node did not. The "qa" pair worked just fine with both app servers, without incident.
Jason Hall reported the same thing on IRC, and we compared notes.
Basically, the problem looks identical to this unanswered RubyForge error.
Here is a snippet from the logs where I was seeing the problem:
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/command.rb:199:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/bin/mongrel_rails:235
/usr/bin/mongrel_rails:18
undefined class/module Project
/usr/lib/ruby/gems/1.8/gems/memcache-client-1.0.3/lib/memcache.rb:128:in `get'
/usr/lib/ruby/1.8/thread.rb:135:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/memcache-client-1.0.3/lib/memcache.rb:98:in `get'
All of them identical. An error restoring an object from a memcached session.
I caught Zed online and asked about it, he suggested removing any "complex datatypes from the session".
I was going this direction when Jason Hall found a "solution":
The Solution
Jason Hall found a workaround: merely add a "model" line to the app/controller/application.rb, and those
model :user, :group, :project
From what I've been told, "model" is deprecated in Edge Rails, so this isn't a permanent solution. But this does make for an interim workaround.
I guess my question is: why would a class be undefined? Is this some kind of a loadpath issue?
After patching up mod_auth_pgsql2, I ran into another painful wall.
In order to use ActiveRBAC's static_permissions schema to authenticate wether a user has permission to a specific repository path, I would need to parse the APR request_rec.uri path to pull out a repostory name and pass that to a rather complex SQL query:
SELECT users.id FROM users
LEFT JOIN groups_users ON groups_users.user_id = users.id
LEFT JOIN groups ON groups.id = groups_users.user_id
LEFT JOIN groups_roles ON groups_roles.group_id = groups.id
LEFT JOIN roles_users ON roles_users.user_id = users.id
LEFT JOIN roles ON roles.id = roles_users.role_id OR roles.id = groups_roles.role_id
LEFT JOIN roles_static_permissions ON roles_static_permissions.role_id = roles.id
LEFT JOIN static_permissions ON static_permissions.id = roles_static_permissions.static_permission_id
WHERE users.login = '%s' AND
( roles.title = 'Administrator' OR
static_permissions.title = ( 'View Project ' ||
( SELECT id FROM projects WHERE projects.name = '%s' )
))
Ouch.
So I started patching up mod_auth_pgsql2 even more, ending up with some mighty painful logic and configuration options. Argh. Not the Rails way.
Rather than make keep and maintain such a monstrosity, I googled one last time for an alternative.
And I found it: mod_auth_remote.
With mod_auth_remote, apache now makes HTTP requests to a configured Rails method that returns a 2xx result code if access is permitted.
Here's my mod_auth_remote apache config:
AuthRemotePort 3000
AuthRemoteServer localhost
AuthRemoteURL /repo/auth
And my RepoController method auth:
def auth
user = User.find_with_credentials(params[:login], params[:password])
if user.nil?
render :nothing => true, :status => "401 Unauthorized"
else
if user.has_role?("Administrator") or user.has_permission?("View Repo #{params[:path].split('/')[1]}")
render :nothing => true, :status => "200 OK"
else
render :nothing => true, :status => "403 Forbidden"
end
end
end
Now I did need to modify mod_auth_remote slightly to pass the user/password and path as parameters. Thankfully, this patch is a one liner:
req_b = apr_pstrcat(r->pool, "HEAD ", conf->remote_uri, "?login=", user, "&password=", passwd, "&path=", r->uri," HTTP/1.0",
CRLF,"Authorization: Basic ",encoded,CRLF,CRLF,NULL);
Simple! Subversion now uses ActiveRBAC to authenticate read requests.
What's more, I've written a generic ruby "hook" script for Subversion that makes SOAP calls to a Rails web services API, but that's for a future post.