Tuesday, 11 February 2014

The Java/Tomcat PermGen Problem

Running the project on Tomcat, I find that it gets slower and slower, until it grinds to a halt. Java does not bother to garbage collect class definitions, as they are not expected to change much once a project is running. However, running JRuby, that is not the case, and new classes are being generated on the fly all the time. Consequently, the JVM runs out of memory (what it calls PermGen memory), and everything stops.

The solution appears to be modifying start-up options in the JVM. This can be done by running tomcat6w.exe (on Windows anyway), going to the Java tab, and adding these options:

-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
Whether this is a solution remains to be seen, but I am giving it a go.
See also
http://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmgc.html
http://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do/3334954#3334954


Monday, 27 January 2014

Loading records from YAML to Rails

Occasionally you need to save data from your database, to be loaded back in at a later data, or to be loaded into another database. YAML makes saving the data trivial:

  def self.save
    File.open('data.yml', 'w') do |out|   # To file
       YAML.dump(MyRecord.all.to_a, out)
    end
  end

Loading it into Rails again is not as straighhtforward. Well, getting it into Rails is easy, getting Rails to save the data to the database is not.

There are two issues. The first is that Rails will use the SQL UPDATE method when you try to save the date. That is not going to work unless there is already a record present in the database. I found the simplest way around that s to use raw SQL to create each record with the correct ID, and then use save to update that record with the correct values (obviously you could do all that in one step, which would avoid the second issue, but require more complicated SQL).

The second issue is that ActiveRecord.save will only update attributes that are flagged as dirty, so you need to flag every column.

This is the solution I ended up with:

  def self.load
    ary = YAML.load_file "data.yml"
    ary.each do |data|
      ActiveRecord::Base.connection.execute(
            "INSERT INTO my_records (id) VALUES (#{data.id})"
      )
      MyRecord.column_names.each { |s| eval("data.#{s}_will_change!") }
      data.save
    end


Monday, 19 August 2013

Listing all columns in all models

As I look to upgrade to Rails 4, and I am looking at strong parameters - more on them later - but something I need on the way is a list of columns for each model. Here is some code that will do that.

First, make sure all your models are loaded (this can take some time) (cache_classes must be on, which it is by default in development mode):

Rails.application.eager_load!

Then get an array of all  ActiveRecord::Base sub-classes. It also get sub-classes of sub-classes, by the way.


ary = ActiveRecord::Base.descendants

Then you just need to list them:

puts ary.map {|m| "#{m.to_s} ~ :#{m.column_names.join ', :'}" }.join("\n")

Here is a useful page about preparing the upgrade to Rails 4, by the way:
https://iprog.com/posting/2013/07/preparing-for-an-upgrade-to-rails-4-0

Monday, 1 July 2013

Code Coverage with SimpleCov


Recently I have been using SimpleCov to test my code coverage. When UI first tried to use it, it threw up errors, but having updated the infrastructure, it seems to work fine (though I do get a ton of warnings). Just so you know, I am using JRuby 1.7.4 and Rails 3.2.13 - not sure which of these was causing the problem.

To get coverage statistics, install the SimpleCov gem, and put it in the gemfile.

group :test do
  gem 'simplecov'
end

In your test/test_helper file require it. Here I require it only if COVERAGE is set in the command line:

require 'simplecov' if ENV["COVERAGE"]

Then set some parameters in that file too. Here I exclude some folders and files, and then tell SimpleCov to group results (again, only if that command line flag is set).

SimpleCov.start do
  add_filter 'test/'
  add_filter 'config/'
  add_filter 'vendor/'
  add_filter 'mod_db.rb'  # One use methods for modifying the database

  add_group 'Controllers', 'app/controllers'
  add_group 'Models', 'app/models'
  add_group 'Helpers', 'app/helpers'
  add_group 'Mailers', 'app/mailers'
  add_group 'Views', 'app/views'
  add_group 'Library', 'lib/my_lib'
end if ENV["COVERAGE"]

It is slower than normal testing, so check that everything passes first. I use this on my command line, so err.txt catches all the warnings.

jruby -S rake test COVERAGE=true 2>err.txt >tmp.txt

The results appear in coverage/index.html, and look like this:



 Clicking on a file name will bring up that file, and any untested lines will be highlighted in red. I found a few places when a page should show a record and a set of associated sub-records, but the code for the associated was missed because I had not set the record to have any sub-records in my test.

Thursday, 6 June 2013

Testing Mailers

The first thing to check when testing your project sends e-mails is that your project is not sending them when you are testing. Rails does this by default by a setting in config/environments/test.rb:

  config.action_mailer.delivery_method = :test

Compare to the line in config/environments/development.rb

  config.action_mailer.delivery_method = :smtp


Setting this to test stops e-mails being sent, and instead they are sent to an array, ActionMailer::Base.deliveries. This array is reset before each test, by the way.

Here is a simple mailer to test.


class Notifier < ActionMailer::Base
  default :from => "DB@mysite.com"
  add_template_helper(ApplicationHelper)
  
  # Sets up an e-mail for notifying the user to activate his account.
  def signup_notification user
    @name = user.username
    @login = user.login
    @url  = "http://mysite.com/activate/#{user.activation_code}"
    mail :to => user.email, :subject => 'Activate your account'
  end

  def page_error(err, request)
    @err = err
    @request = request
    mail :to => 'admin@mysite.com', :subject => 'Page error'
  end
end

In your controller, you might invoke the first like this:

  Notifier.signup_notification(@user).deliver

In the test, you will need to break that up, so you can examine the mail object.

class NotifierTest < ActionMailer::TestCase
  test "signup_notification" do
    # Create a mock user
    user = TestUser.new 'tester', 'tester@nowhere.com'
    # Invoke the mailer method
    mail = Notifier.signup_notification user
    # Deliver the mail
    mail.deliver
    # Check the mail got sent
    assert !ActionMailer::Base.deliveries.empty?
    # Check it is the right mail
    assert_equal 'Activate your account', mail.subject
    assert_equal ["tester@nowhere.com"], mail.to
    assert_equal ["DB@mysite.com"], mail.from
    assert_match "Visit this url to activate your account",
                    mail.body.encoded
  end
end

Here is the TestUser definition.

class TestUser
  attr_reader :username, :email, :login

  def initialize username, email
    @username = username
    @email = email
    @login = username.gsub ' ', ''
  end
  
  def activation_code; "abcd"; end
end

The second method in the mailer above is for sending error reports to the administrator. Here is a method that generates an error, and sends that to the mailer:

  test "page_error" do
    request = TestRequest.new
    begin
      raise "A test error"
    rescue Exception => err
      mail = Notifier.page_error(err, request)
      mail.deliver
      assert !ActionMailer::Base.deliveries.empty?
      assert_equal "Page error", mail.subject
      assert_equal ["admin@mysite.com"], mail.to
      assert_equal ["DB@mysite.com"], mail.from
      assert_match "error encountered!", mail.body.encoded
    end
  end

Here is the TestRequest definition; it simply returns the string "good" if the method name is recognised - that is enough for the mailer to wok with, and will still highlight any mistyped or made-up method names.

class TestRequest
  # Returns the string "good" if the method is recognised
  def method_missing method, *args
    return "good" if [:fullpath, :request_method, :query_parameters,
                      :request_parameters, :referer].include? method
    super  
  end
end
end

Tuesday, 4 June 2013

Testing Protected Methods in a Controller, Part 2

See part 1 here.

Unit Testing Controller Methods

You are not obliged to test controller methods in with your functional tests. You can mix them in with your unit tests, though they do need a separate class (and so ideally a separate file). The trick is to sub-class from ActionController::TestCase, and to then nominate a controller you are testing (this is normally derived from the test name, so would not be necessary if you maintain the naming convention):

class DrumsControllerMethodsTest < ActionController::TestCase
  tests DrumLog::DrumsController

  # tests
end

Whether this is a good idea is debatable. I would argue that testing these protected methods is proper unit testing, and is not functional testing, so that suggests they should be with the rest of the unit tests (as your helper tests already are), however, I think I read that in Rails 4 they will be relabelled as model tests and controller tests, which kind of undermines that argument.

One solution is to move them to a library file (create a module, and include it in ApplicationController), then you can unit test them with your other library files. Obviously this works best for more generic methods.

Using A Mock Controller


Sometimes you want test methods that invoke methods like render and redirect_to that are doing all sorts of things behind the scenes. Better (in my opinion) to create a mock controller, and over-write these methods.

Here is an example. It is testing a method check_security_level (a filter in fact) that will send the user to the log-in page if he tries to access the edit page when not logged in, but does nothing if he tries to access the index page.

# Create a mock class to subvert redirect_to,
# inheriting from ApplicationController.
# You could inherit an actual controller class.
class MockSampleController < ApplicationController
  attr_reader :destination
  
  # Redefine direct_to to just note the destination and do nothing more.
  def redirect_to s; @destination = s; end
end



class MyControllerTest < ActionController::TestCase
  tests MockSampleController
  
  # First test, check_security_level should redirect the user to a login page
  def test_check_security_level_1
    # Set the request parameters (must be strings not symbols)
    @controller.params[:action] = 'edit'  
    @controller.send(:check_security_level)
    assert_equal '/user_log/sessions', @controller.destination[:controller]
  end

  # Second test, check_security_level should do nothing
  def test_check_security_level_2
    @controller.params[:action] = 'index'  
    @controller.send(:check_security_level)
    assert_nil @controller.destination
  end
end

Add New Routing For Tests


Sometimes you want to create a mock controller with its own set of actions for testing methods in your controllers. This is not trivial as you need to have a set of routes for your test actions, and it is bad practice to put them into routes.rb, as they will end up in your production environment.

Creating a routes.rb for testing is not ideal, as it will get out of sync with the real routes.rb, and there is no simple way to append new routes (the Application.routes.draw method clears all existing routes).

The solution is to use the with_routing method, which is a standad part of Rails. However, it is somewhat neglected, as the API shows...

This is the example from 2.3.8, when it was in  ActionController::TestProcess:
http://apidock.com/rails/ActionController/TestProcess/with_routing

 with_routing do |set|
    set.draw do |map|
      map.connect ':controller/:action/:id'
        assert_equal(
          ['/content/10/show', {}],
          map.generate(:controller => 'content', :id => 10, :action => 'show')
      end
    end
  end


Spot the missing right bracket to match the one in "assert_equal("?

In Rails 3 it got moved to ActionDispatch::Assertions::RoutingAssertions...

 with_routing do |set|
    set.draw do |map|
      map.connect ':controller/:action/:id'
        assert_equal(
          ['/content/10/show', {}],
          map.generate(:controller => 'content', :id => 10, :action => 'show')
      end
    end
  end

Still missing that right bracket. And still using Rails 2 style routing!

Here is an example that works:

  def test_render_generic_form
    with_routing do |set|
      set.draw do
        resources :mocks do
          collection do
            get :test_action
          end
        end
      end
    
    
      get :test_action
      # assertions
    end
  end


Friday, 31 May 2013

Testing Protected Methods in a Controller, Part 1

Methods in a controller (including applicatin_controller.rb) can be divided between those that are actions (so invoked by HTTP requests) and those that are not. Those that are not should all be set as protected (or private), and it is these that I am posting about.

The first thing to do is to shift as much as you can out of the controllers. If at all possible, put in in a model or a library, and then just unit test them. However, that is not always easy, say because you want to invoke a redirect from the method or you need quick access to the session (I expect you could handle the session in a library file, but I think it would be messy and more trouble than it is worth).

So we have a bunch of protected methods, and they all need testing.

While you could test them in the same file you test your actions, thematically they are very different, and I think it makes more sense to put them in their own file. Here is an example. The only tricky part is that you need to say what controller it should use with the "tests" command. Oh, and you need to use "send" to access the method being tested, as it is protected.

require 'test/test_helper'

class ControllerMethodsTest < ActionController::TestCase
  tests PostsController
  
  def setup
    user_setup
  end

  def test_current_user
    login_as ['trainer', 'it'], request
    assert_equal 'tester', @controller.send(:current_user).login
  end
end

The "user_setup" methods puts some roles into the testing database. The "login_as" method is one that I have used a lot for testing actions relating to web pages that have the user logged in.

  def login_as roles, request
    create_user
    user = UserLog::User.find_by_login 'tester'
    roles = [roles] unless roles.is_a? Array
    roles.each { |role| assign_role user, role }
    request.session[:user_id] = user.id
    #p "set request.session[:user_id] to #{request.session[:user_id]}"
  end

  def create_user name = "tester"
    user = UserLog::User.new
    user.login = name
    user.email = "#{name}@domain.com"
    user.username = name.titlecase
    user.password = "12345678"
    user.password_confirmation = "12345678"
    user.save(:validate => false)
    user.send(:activate!)
    UserLog::User.find_by_login name
  end

  def assign_role(user, rolename)
    role = UserLog::Role.find_by_rolename(rolename)
    raise "Failed to find role #{rolename}" if role.nil?
    permission = UserLog::Permission.new
    permission.role = role
    permission.user = user
    permission.save(:validate => false)
  end

See part 2 here.