Matt Pelletier queried the Mongrel mailing list asking for any personal experiences or tips regarding mongrel and rails hosting in general for some book currently underway.

This was my quick response:

We use mongrel for hosting our rails projects

At the moment, the largest site we host is Kapture (www.kapture.info), a Web 2.0 startup. I would be happy to cover our infrastructure in detail if you're interested.

Complex Rails things we have stumbled on:

To avoid missing class errors for your models, be sure to define all your models with "model" at the beginning of your application.rb. (NOTE: Edge rails deprecates the use of "model" for loading models, and should load models as it needs them). Alternatively, add something like the following to your application.rb:

# Pre-load every .rb file in the models dir
Dir.foreach(File.dirname(__FILE__) + "/../models") {|file|
  model $1.to_sym if file =~ /(\w+).rb$/
}

Mongrel things NOT to do:

If you build your mongrel by hand and have a http11.so, do not copy it willy nilly wherever you would like. I had copied it into my gems path (/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/http11.so) by building my own gem with a pre-compiled http11.so as the production boxes don't have a compiler. After installing the gem, I spent countless hours debugging what was causing bizarre SEGFAULT errors. Normally, mongrel installs http11.so to your local siteruby directory (/usr/local/lib/siteruby/1.8/i486-linux/http11.so). Apparently if you have http11.so in your gems lib dir, it causes some deep magic problem with ruby that you'd have to float by Zed or someone to fully explain.

Use a proxy that can quickly serve static content. We started with Pound but had problems with various satellite provider's transparent proxies spitting up 500 server errors for some weird reason. I then moved to Apache 2.2 with modproxybalancer, and static content noticably improved. Mongrel is "ok" at serving static content, but it doesn't hurt to have something in front of it to speed things up.

Mongrel will spin out of control. Use a tool like "monit" to monitor the individual mongrel daemons and kill them off and restart them if they stop responding.

Mongrel debugging:

Running mongrel with -B will spit out all kinds of fun debugging stuff to the mongrel_debug/ directory.

Sending a SIGUSR1 will cause a mongrel daemon to die off, and spit out a backtrace of the running state of threads when it recevied a signal. This is a good way to trace why mongrel is spinning.

Sending a SIGUSR2 turns on debugging on a running mongrel server. According to Zed, the debug impact is minimal, and thus fairly safe to use in production.

Web tier things to do:

Use apache's mod_deflate (or turn on gzip content encoding in general). The speed improvement is noticable.

Rails things to do:

Get your application running. Worry about scaling later. Rails will scale, with only minor tweaking.

Use the exception_notifier plugin on your production boxes to send you email whenever a Rails error occurs.

Run memcached and memcache-client (or memcache-client-stats) instead of the default local file based session store.

Learn to use fragment caching and run memcache_fragments. It is well worth the optimization. Use page caching sparingly, particularly on your main page or pages that are being actively slashdotted.

If you run memcached, consider using memcache-client-stats to get some visibility into the activity to your memcache servers.

Keep complex objects out of your session store as much as possible. Avoid storing complex objects in pstore. Don't be afraid to use memcache directly, particularly with alternate namespaces.

If you're running a cluster, run a centralized syslogng and log to syslog. Consider turning off rails logging altogether in your production cluster.

While svn:externals is a wonderful thing for development (/vendors/plugins), freezing plugins, gems, and rails to your project makes it far happier in a production environment; also, much easier to deploy.

Database layer things to consider:

You can use Sequoia / C-JDBC with Carob's libmysql replacement with Rails to both mirror and stripe data across a large number of servers.

The Postgres pgcluster project is flakey, and the sole maintainer doesn't consider anything beyond 1.0.11 "stable" enough for anyone to actually use in a production environment. Unfortunately, that is tied to Postgres 7.3, which destroys any hope for ALTERing tables after they are created (among other things). Avoid it for now. If you must replicate, use something like Slony instead (I haven't found anything in ActiveRecord about shunting writes to one database connection and reads to a cluster of replicated slaves).

There are many more things to add to this list, these are just the items that came immediately to mind.

We've taken to writing rules on our whiteboards.

Today I was driven to write the following on my whiteboard:

Ian's Operational Rules

1. No whining
2. Own the problem
3. Understand the problem
4. Fix the problem
5. Ask for help
6. Document the help
7. Own up to mistakes

I'm sure this list will grow over time.

Venting this way feels much more constructive for some reason.

While playing with mod_auth_remote, it became apparent that libapr1's changes have slightly broken it.

There were two errors during compilation:

$ /usr/bin/apxs2  -i -a -c mod_auth_remote.c
mod_auth_remote.c: In function` get_remote_auth':
mod_auth_remote.c:109: warning: passing arg 4 of `apr_socket_create' makes integer from pointer without a cast
mod_auth_remote.c:109: error: too few arguments to function `apr_socket_create'
mod_auth_remote.c:117: warning: implicit declaration of function `apr_setsocketopt'
mod_auth_remote.c:117: error: `APR_SO_TIMEOUT' undeclared (first use in this function)
mod_auth_remote.c:117: error: (Each undeclared identifier is reported only once
mod_auth_remote.c:117: error: for each function it appears in.)
mod_auth_remote.c:126: warning: implicit declaration of function `apr_connect'
mod_auth_remote.c:143: warning: implicit declaration of function `apr_base64_encode'
mod_auth_remote.c:147: warning: implicit declaration of function `apr_send'
mod_auth_remote.c:158: warning: implicit declaration of function `apr_recv'

apxs:Error: Command failed with rc=65536

A quick couple of googles later, I worked up this patch to fix both of the above errors:

--- mod_auth_remote.c.orig      2006-09-14 13:33:51.000000000 -0400
+++ mod_auth_remote.c   2006-09-14 13:33:33.000000000 -0400
@@ -17,6 +17,8 @@
 #include "http_protocol.h"
 #include "http_request.h"

+#include "apr_version.h"
+

 typedef struct {
   int port;
@@ -106,7 +108,12 @@
   apr_sockaddr_t *addr;


-  if((val =apr_socket_create(&socket, APR_INET, SOCK_STREAM, r->pool))
+  if((val =apr_socket_create(&socket, APR_INET,
+                                 SOCK_STREAM,
+#if (APR_MAJOR_VERSION > 0)
+                                                APR_PROTO_TCP,
+#endif
+                                 r->pool))
      != APR_SUCCESS)
     {
       ap_log_rerror(APLOG_MARK, APLOG_ERR, val, r,
@@ -114,7 +121,12 @@
       return HTTP_INTERNAL_SERVER_ERROR;
     }

+/* APR_SO_TIMEOUT is deprecated in favor of apr_socket_timeout_{set|get} */
+#if (APR_MAJOR_VERSION > 0)
+   apr_socket_timeout_set(socket, (int)r->server->timeout);
+#else
    apr_setsocketopt(socket,  APR_SO_TIMEOUT, (int)r->server->timeout);
+#endif
    if((val = apr_sockaddr_info_get(&addr, conf->remote_server, APR_INET,
                                    conf->port,0,r->pool)) != APR_SUCCESS)
      {

And now everything builds just fine.

Again, the patch is here: mod_auth_remote-apache2.2.patch

The Problem

We have been getting a "undefined class/module" error on 2 of our 6 app servers in one testbed.

Same sourcecode, same installs (systemimager and an isconf style configuration management harness), same binaries. Identical everything. Pound + mongrel + memcached + postgres (pgcluster).

The odd bit here is that our "dev" and "prod" pairs had one node reporting the error, but the other node did not. The "qa" pair worked just fine with both app servers, without incident.

Jason Hall reported the same thing on IRC, and we compared notes.

Basically, the problem looks identical to this unanswered RubyForge error.

Here is a snippet from the logs where I was seeing the problem:

/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/command.rb:199:in `run'
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/bin/mongrel_rails:235
/usr/bin/mongrel_rails:18
undefined class/module Project
/usr/lib/ruby/gems/1.8/gems/memcache-client-1.0.3/lib/memcache.rb:128:in `get'
/usr/lib/ruby/1.8/thread.rb:135:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/memcache-client-1.0.3/lib/memcache.rb:98:in `get'

All of them identical. An error restoring an object from a memcached session.

I caught Zed online and asked about it, he suggested removing any "complex datatypes from the session".

I was going this direction when Jason Hall found a "solution":

The Solution

Jason Hall found a workaround: merely add a "model" line to the app/controller/application.rb, and those

model :user, :group, :project

From what I've been told, "model" is deprecated in Edge Rails, so this isn't a permanent solution. But this does make for an interim workaround.

I guess my question is: why would a class be undefined? Is this some kind of a loadpath issue?

After patching up mod_auth_pgsql2, I ran into another painful wall.

In order to use ActiveRBAC's static_permissions schema to authenticate wether a user has permission to a specific repository path, I would need to parse the APR request_rec.uri path to pull out a repostory name and pass that to a rather complex SQL query:

    SELECT users.id FROM users 
    LEFT JOIN groups_users ON groups_users.user_id = users.id 
    LEFT JOIN groups ON groups.id = groups_users.user_id
    LEFT JOIN groups_roles ON groups_roles.group_id = groups.id 
    LEFT JOIN roles_users ON roles_users.user_id = users.id 
    LEFT JOIN roles ON roles.id = roles_users.role_id OR roles.id = groups_roles.role_id
    LEFT JOIN roles_static_permissions ON roles_static_permissions.role_id = roles.id
    LEFT JOIN static_permissions ON static_permissions.id = roles_static_permissions.static_permission_id 
    WHERE users.login = '%s' AND 
    ( roles.title = 'Administrator' OR  
      static_permissions.title = ( 'View Project ' || 
       ( SELECT id FROM projects WHERE projects.name = '%s' )
    ))

Ouch.

So I started patching up mod_auth_pgsql2 even more, ending up with some mighty painful logic and configuration options. Argh. Not the Rails way.

Rather than make keep and maintain such a monstrosity, I googled one last time for an alternative.

And I found it: mod_auth_remote.

With mod_auth_remote, apache now makes HTTP requests to a configured Rails method that returns a 2xx result code if access is permitted.

Here's my mod_auth_remote apache config:

    AuthRemotePort 3000
    AuthRemoteServer localhost
    AuthRemoteURL /repo/auth

And my RepoController method auth:

  def auth
    user = User.find_with_credentials(params[:login], params[:password])
    if user.nil?
      render :nothing => true, :status => "401 Unauthorized"
    else
      if user.has_role?("Administrator") or user.has_permission?("View Repo #{params[:path].split('/')[1]}")
        render :nothing => true, :status => "200 OK"
      else
        render :nothing => true, :status => "403 Forbidden"
      end
    end
  end

Now I did need to modify mod_auth_remote slightly to pass the user/password and path as parameters. Thankfully, this patch is a one liner:

    req_b = apr_pstrcat(r->pool, "HEAD ", conf->remote_uri, "?login=", user, "&password=", passwd, "&path=", r->uri," HTTP/1.0",
                        CRLF,"Authorization: Basic ",encoded,CRLF,CRLF,NULL);

Simple! Subversion now uses ActiveRBAC to authenticate read requests.

What's more, I've written a generic ruby "hook" script for Subversion that makes SOAP calls to a Rails web services API, but that's for a future post.