Monday, August 15, 2011

Emailing Server Errors

I've talked about checking the server's health, but its even nicer if the server lets you know if it has a problem. In particular, you don't want user's experiencing errors and for you to remain in blissful ignorance. A simple fix is to have your server email you whenever a user encounters an error. Here is how I did this in rails.

Catch the Error
The first thing to do is to capture any exceptions that get thrown. This can be done easily by adding the following line to your ApplicationController.
unless Rails.application.config.consider_all_requests_local # don't rescue on local testing
  rescue_from Exception, :with => :handle_error
As of this writing, apparently Rail 3 doesn't support 404 errors this way, so a fix for this is to add a catch-all route at the bottom of your config/routes.rb that looks something like:
# this must be the LAST rule in the file
# it allows our custom 404 error to execute
match '*path', :to =>'application#routing'
Handle the Error
Now that I have caught unhandled exceptions and 404 errors, you need to do something with them. These methods in the the ApplicationController are the ones called from the above code:
def routing(exception = nil)
  record_error "404 error", exception
  render :template => "/errors/404.html", :status => 404
def handle_error(exception)
  msg = exception ? exception.message : "unknown error"
  record_error(msg, exception)
  render :template => "/errors/500.html", :status => 500

def record_error(msg, exception)
  return nil if Rails.application.config.consider_all_requests_local # don't email during local testing

  referrer = request.env['HTTP_REFERER']
  url = request.url
  app_name = Rails.application.class.to_s
  user = current_user
  name = user ? user.username : "UNKNOWN"
  error_msg = msg
  if exception
    error_msg += " #{exception.message}\n#{exception.backtrace.join '\n'}"
  ip = request.remote_ip
  remote_ip = request.env['HTTP_X_FORWARDED_FOR']

  logger.warn "Error #{error_msg} for url [#{url}] coming from url [#{referrer}] from user [#{user}] at ip [#{ip}] forwarded from ip [#{remote_ip}]"
  mail = ErrorMailer.log_error(msg, app_name, url, referrer, ip, remote_ip, name, exception)
You can make the 404 and 500 error pages that these methods render be whatever you like, though it is suggested that you make them useful. The record_error method that these methods calls, collect a bunch of information about the request, log them (in case mailing fails), and then emails them. The emailing is done using Rails mailing capability. Since the actual configuration for talking to the mail server will depend on your situation, you should read the documentation for that.

To send the email, I created mailers/error_mailer.rb which is analogous to a controller. All it does is package all the variables into instance variables that the view will have:
class ErrorMailer < ActionMailer::Base
  default :from => ""
  helper :application

  def log_error(subject, app_name, url, referrer, ip, remote_ip, user_name, error)
    @app_name = app_name
    @url = url
    @referrer = referrer
    @ip = ip
    @remote_ip = remote_ip
    @user_error = user_error
    @user_name = user_name
    @error = error
    mail :to => "", :subject=>"RAILS ERROR in #{@app_name}: #{subject}"
The view (views/error_mailer/log_error.text.haml) turns this information into an email message:
There was an error in app #{@app_name} at time #{}.

URL:  #{@url}
Referrer:  #{@referrer}
IP:  #{@ip}
IP via a portal:  #{@remote_ip}
User: #{@user_name}
- if @error
  Error class: #{@error.class}
  Error message: #{@error.message}
  Error Stack Trace:
  - @error.backtrace.each do |line|
    = line
- else
  No exception to log
I do have one piece of advice: Make sure the subject you give your error emails is something very distinct that you can write mail filter rules. After I deployed a similar solution at work, the security teams scanner found my app and proceeded to run through its entire litany of tests against the server. I came in the next morning to more than 4,000 emails from 404 errors. The fact that I had put the entire development team as the recipients of these emails was just the icing on the cake.

Maybe emailing every error isn't great, since you can look in the log files for the same information. However, I find the immediacy and in your face nature of an email is good for making sure that errors get fixed and aren't just silently experienced by your users (and ex users). However, be aware that if something goes wrong, you have the potential to receive a lot of email. (not to mention choice comments from anyone else who is also getting those emails).


Anonymous said...

Some other good advice is don't send your error emails to your mobile phone that makes an audible alarm every time an email comes in. If you are going to do that, then make sure your phone only gets email for the really big emergencies. Email filtering should normally take care of that if you made good subject lines. If you aren't careful about keeping your phone from going off 4000 times a night then your wife may pressure you to get another job. This is not a problem you want to be fixing at 3am (I knew a guy like this)

Michael Haddox-Schatz said...

Was his name Alan? :) I had a coworker whose phone sound for emergencies was the Star Trek Alarm. Of course, he had the normal messages and texts make the Star Trek communicator sound. Kind of made for a surreal work environment. Everyone definitely knew when the servers went down. (he was single at the time, so didn't have a wife to pressure him about night time alarms).

Anonymous said...

No, his name was not Alan. I used to work with a guy who had a whole bunch of different star trek sounds play depending on the email he got. If it was email from the company president we would hear, "Sir! message from the captain!"

David Joerg said...

Thank you! This worked for me, I was going nuts.