Matt Pelletier queried the Mongrel mailing list asking for any personal experiences or tips regarding mongrel and rails hosting in general for some book currently underway.

This was my quick response:

We use mongrel for hosting our rails projects

At the moment, the largest site we host is Kapture (www.kapture.info), a Web 2.0 startup. I would be happy to cover our infrastructure in detail if you're interested.

Complex Rails things we have stumbled on:

To avoid missing class errors for your models, be sure to define all your models with "model" at the beginning of your application.rb. (NOTE: Edge rails deprecates the use of "model" for loading models, and should load models as it needs them). Alternatively, add something like the following to your application.rb:

# Pre-load every .rb file in the models dir
Dir.foreach(File.dirname(__FILE__) + "/../models") {|file|
  model $1.to_sym if file =~ /(\w+).rb$/
}

Mongrel things NOT to do:

If you build your mongrel by hand and have a http11.so, do not copy it willy nilly wherever you would like. I had copied it into my gems path (/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/http11.so) by building my own gem with a pre-compiled http11.so as the production boxes don't have a compiler. After installing the gem, I spent countless hours debugging what was causing bizarre SEGFAULT errors. Normally, mongrel installs http11.so to your local siteruby directory (/usr/local/lib/siteruby/1.8/i486-linux/http11.so). Apparently if you have http11.so in your gems lib dir, it causes some deep magic problem with ruby that you'd have to float by Zed or someone to fully explain.

Use a proxy that can quickly serve static content. We started with Pound but had problems with various satellite provider's transparent proxies spitting up 500 server errors for some weird reason. I then moved to Apache 2.2 with modproxybalancer, and static content noticably improved. Mongrel is "ok" at serving static content, but it doesn't hurt to have something in front of it to speed things up.

Mongrel will spin out of control. Use a tool like "monit" to monitor the individual mongrel daemons and kill them off and restart them if they stop responding.

Mongrel debugging:

Running mongrel with -B will spit out all kinds of fun debugging stuff to the mongrel_debug/ directory.

Sending a SIGUSR1 will cause a mongrel daemon to die off, and spit out a backtrace of the running state of threads when it recevied a signal. This is a good way to trace why mongrel is spinning.

Sending a SIGUSR2 turns on debugging on a running mongrel server. According to Zed, the debug impact is minimal, and thus fairly safe to use in production.

Web tier things to do:

Use apache's mod_deflate (or turn on gzip content encoding in general). The speed improvement is noticable.

Rails things to do:

Get your application running. Worry about scaling later. Rails will scale, with only minor tweaking.

Use the exception_notifier plugin on your production boxes to send you email whenever a Rails error occurs.

Run memcached and memcache-client (or memcache-client-stats) instead of the default local file based session store.

Learn to use fragment caching and run memcache_fragments. It is well worth the optimization. Use page caching sparingly, particularly on your main page or pages that are being actively slashdotted.

If you run memcached, consider using memcache-client-stats to get some visibility into the activity to your memcache servers.

Keep complex objects out of your session store as much as possible. Avoid storing complex objects in pstore. Don't be afraid to use memcache directly, particularly with alternate namespaces.

If you're running a cluster, run a centralized syslogng and log to syslog. Consider turning off rails logging altogether in your production cluster.

While svn:externals is a wonderful thing for development (/vendors/plugins), freezing plugins, gems, and rails to your project makes it far happier in a production environment; also, much easier to deploy.

Database layer things to consider:

You can use Sequoia / C-JDBC with Carob's libmysql replacement with Rails to both mirror and stripe data across a large number of servers.

The Postgres pgcluster project is flakey, and the sole maintainer doesn't consider anything beyond 1.0.11 "stable" enough for anyone to actually use in a production environment. Unfortunately, that is tied to Postgres 7.3, which destroys any hope for ALTERing tables after they are created (among other things). Avoid it for now. If you must replicate, use something like Slony instead (I haven't found anything in ActiveRecord about shunting writes to one database connection and reads to a cluster of replicated slaves).

There are many more things to add to this list, these are just the items that came immediately to mind.