Emailing Server Errors

I've talked about checking the server's health, but its even nicer if the server lets you know if it has a problem. In particular, you don't want user's experiencing errors and for you to remain in blissful ignorance. A simple fix is to have your server email you whenever a user encounters an error. Here is how I did this in rails.

Catch the Error
The first thing to do is to capture any exceptions that get thrown. This can be done easily by adding the following line to your ApplicationController.

unless Rails.application.config.consider_all_requests_local # don't rescue on local testing
  rescue_from Exception, :with => :handle_error
end

As of this writing, apparently Rail 3 doesn't support 404 errors this way, so a fix for this is to add a catch-all route at the bottom of your config/routes.rb that looks something like:

# this must be the LAST rule in the file
# it allows our custom 404 error to execute
match '*path', :to =>'application#routing'

Handle the Error
Now that I have caught unhandled exceptions and 404 errors, you need to do something with them. These methods in the the ApplicationController are the ones called from the above code:

def routing(exception = nil)
  record_error "404 error", exception
  render :template => "/errors/404.html", :status => 404
end
 
def handle_error(exception)
  msg = exception ? exception.message : "unknown error"
  record_error(msg, exception)
  render :template => "/errors/500.html", :status => 500
end

def record_error(msg, exception)
  return nil if Rails.application.config.consider_all_requests_local # don't email during local testing

  referrer = request.env['HTTP_REFERER']
  url = request.url
  app_name = Rails.application.class.to_s
  user = current_user
  name = user ? user.username : "UNKNOWN"
   
  error_msg = msg
  if exception
    error_msg += " #{exception.message}\n#{exception.backtrace.join '\n'}"
  end
   
  ip = request.remote_ip
  remote_ip = request.env['HTTP_X_FORWARDED_FOR']

  logger.warn "Error #{error_msg} for url [#{url}] coming from url [#{referrer}] from user [#{user}] at ip [#{ip}] forwarded from ip [#{remote_ip}]"
   
  mail = ErrorMailer.log_error(msg, app_name, url, referrer, ip, remote_ip, name, exception)
  mail.deliver
end

You can make the 404 and 500 error pages that these methods render be whatever you like, though it is suggested that you make them useful. The record_error method that these methods calls, collect a bunch of information about the request, log them (in case mailing fails), and then emails them. The emailing is done using Rails mailing capability. Since the actual configuration for talking to the mail server will depend on your situation, you should read the documentation for that.

Emailer
To send the email, I created mailers/error_mailer.rb which is analogous to a controller. All it does is package all the variables into instance variables that the view will have:

class ErrorMailer < ActionMailer::Base
  default :from => "myApplication@myurl.com"
  helper :application

  def log_error(subject, app_name, url, referrer, ip, remote_ip, user_name, error)
    @app_name = app_name
    @url = url
    @referrer = referrer
    @ip = ip
    @remote_ip = remote_ip
    @user_error = user_error
    @user_name = user_name
    @error = error
    mail :to => "myAddress@somewhere.com", :subject=>"RAILS ERROR in #{@app_name}: #{subject}"
  end
end

The view (views/error_mailer/log_error.text.haml) turns this information into an email message:

There was an error in app #{@app_name} at time #{Time.now}.

URL:  #{@url}
Referrer:  #{@referrer}
IP:  #{@ip}
IP via a portal:  #{@remote_ip}
User: #{@user_name}
 
- if @error
  Error class: #{@error.class}
  Error message: #{@error.message}
  Error Stack Trace:
  - @error.backtrace.each do |line|
    = line
- else
  No exception to log

Warning
I do have one piece of advice: Make sure the subject you give your error emails is something very distinct that you can write mail filter rules. After I deployed a similar solution at work, the security teams scanner found my app and proceeded to run through its entire litany of tests against the server. I came in the next morning to more than 4,000 emails from 404 errors. The fact that I had put the entire development team as the recipients of these emails was just the icing on the cake.

Maybe emailing every error isn't great, since you can look in the log files for the same information. However, I find the immediacy and in your face nature of an email is good for making sure that errors get fixed and aren't just silently experienced by your users (and ex users). However, be aware that if something goes wrong, you have the potential to receive a lot of email. (not to mention choice comments from anyone else who is also getting those emails).