Thursday, 23 April 2015

Installing gems with JRuby gets "marshal data too short" error

Took me many hours to track this down. The error seems to be pretty generic, and various solutions have worked over the years:

http://stackoverflow.com/questions/5409675/marshal-data-too-short-error-message-while-installing-watir-webdriver-on-windo
http://stackoverflow.com/questions/19070698/how-to-modify-the-gemfile-marshal-data-too-short

Setting the verbose flag showed there was no network problem:

>gem install refile -V
io/console not supported; tty will not be manipulated
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=refile
200 OK
ERROR:  While executing gem ... (ArgumentError)
  marshal data too short

In the end it turned out to be JRuby in Ruby 2.0 mode.

>jruby --1.9 -S gem install carrierwave
io/console not supported; tty will not be manipulated
Fetching: carrierwave-0.10.0.gem (100%)
Successfully installed carrierwave-0.10.0
1 gem installed

I was trying to install refile to replace paperclip, which seems to be dodgy with Windows (requiring DevKit). Unfortunately, refile requires Ruby 2.1, so the gem cannot be downloaded with JRuby in 1.9 mode, and gem install does not work unless JRuby is in 1.9 mode. Great.

So I am looking at CarrierWave...

Thursday, 12 March 2015

Java (and Tomcat) Memory Settings

I am having a problem with my Rails project on Tomcat slowing down and grinding to a halt. As part of the investigation into that, I have been looking at the memory settings in Java (which also apply to Tomcat). I upgraded to Tomcat 8 and Java 8 first, so this applies to Java 8 (which no longer has PermGen, something that featured in the error logs).

Java stores stuff on the heap. This is divided into two sections, the New Generation and the Old Generation, and the former is further divided into two again.

New generation/Eden Space: When a new object is created, it will go here. Garbage Collection (GC) is initiated when this is full, and that involves culling unused objects and promoting used objects.

New generation/Survivor Space: Objects promoted from Eden Space go here. In fact there are two Survivor Spaces, and at GC objects are either culled or moved to the other survivor space (to avoid fragmentation). Objects that have survived a certain number of GCs are further promoted.

Old generation: Long-lived objects eventually turn up here, when promoted from the survivor space..

Permanent generation, or PermGen, was non-heap memory, used for class definitions. It is not used in Java 8, though there are other non-heap memory sections.

Settings


 The -Xms and -Xmx parameters define the minimum and maximum heap sizes, respectively.


-Xmn defines both the minimum and maximum size for the new generation, Eden and Survivor combined.

-XX:NewRatio defines the ratio of young to old. It defaults to 2. Clearly you have to have the old generation at least the size of he new (thogh using -Xmn is a way around that; no point setting both this and -Xmn).

-XX:SurvivorRatio defines the ratio of Survivor to Eden space. Again, it follows that the Survivor Space must be greater than the Eden Space, and that it defaults to 8 indicates it should be much bigger. By default, Java uses an adaptive policy, so -XX:SurvivorRatio will be ignored (but you can set an initial value with -XX:InitialSurvivorRatio). You can turn off the adaptive policy with -XX:-UseAdaptiveSizePolicy.

Here is an example (these are just for illustration of how to use the parameters).

-Xmn1024m
-Xms3092m
-Xmx3092m

-XX:NewRatio=2

-XX:-UseAdaptiveSizePolicy
-XX:SurvivorRatio=8

Thread size

Each thread gets allocated its own chunk of memory, and the size is dicated by -Xss (or -XX:ThreadStackSize). This seems to usually default to 512k, but depends of the JVM, the OS, etc. Some web pages suggest this can be reduced to 128k, whilst others say set it tio 8m. If you have a lot of threads, that will have a serious impact on your memory usage! On the other hand, if the value is too small, you will see StackOverflowErrors.

Tomcat

In Tomcat on Windows you can set these by running Tomcat8w, and putting the values into the Java Options box on the Java tab. There are also boxes specifically for initial heap, maximim heap and thread size, and these may be all you need to fiddle with.

If you go to your Tomcat server status page, you can see the current usage. The page may well be here:

http://localhost:8080/manager/status/all

The memory section might look like this:




This is using the adaptive policy for the survivor ratio, and the survivor space is very small. The important thing (I think) is that there is plenty of room left in the Old Generation area.

By the way the -Xmn1024m format for parameters is the original. Hotspot introduced a shed load more parameters, and also the -XX:NewRatio=2 format for them. Hotspot converts the former to the latter internally for backwards compatibility.

Invoke Dynamic

In the end, what had the most impact was turning off the "invoke dynamic" feature:

-Djruby.compile.invokedynamic=false

Tuesday, 11 February 2014

The Java/Tomcat PermGen Problem

Running the project on Tomcat, I find that it gets slower and slower, until it grinds to a halt. Java does not bother to garbage collect class definitions, as they are not expected to change much once a project is running. However, running JRuby, that is not the case, and new classes are being generated on the fly all the time. Consequently, the JVM runs out of memory (what it calls PermGen memory), and everything stops.

The solution appears to be modifying start-up options in the JVM. This can be done by running tomcat6w.exe (on Windows anyway), going to the Java tab, and adding these options:

-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
Whether this is a solution remains to be seen, but I am giving it a go.
See also
http://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmgc.html
http://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do/3334954#3334954


Monday, 27 January 2014

Loading records from YAML to Rails

Occasionally you need to save data from your database, to be loaded back in at a later data, or to be loaded into another database. YAML makes saving the data trivial:

  def self.save
    File.open('data.yml', 'w') do |out|   # To file
       YAML.dump(MyRecord.all.to_a, out)
    end
  end

Loading it into Rails again is not as straighhtforward. Well, getting it into Rails is easy, getting Rails to save the data to the database is not.

There are two issues. The first is that Rails will use the SQL UPDATE method when you try to save the date. That is not going to work unless there is already a record present in the database. I found the simplest way around that s to use raw SQL to create each record with the correct ID, and then use save to update that record with the correct values (obviously you could do all that in one step, which would avoid the second issue, but require more complicated SQL).

The second issue is that ActiveRecord.save will only update attributes that are flagged as dirty, so you need to flag every column.

This is the solution I ended up with:

  def self.load
    ary = YAML.load_file "data.yml"
    ary.each do |data|
      ActiveRecord::Base.connection.execute(
            "INSERT INTO my_records (id) VALUES (#{data.id})"
      )
      MyRecord.column_names.each { |s| eval("data.#{s}_will_change!") }
      data.save
    end


Monday, 19 August 2013

Listing all columns in all models

As I look to upgrade to Rails 4, and I am looking at strong parameters - more on them later - but something I need on the way is a list of columns for each model. Here is some code that will do that.

First, make sure all your models are loaded (this can take some time) (cache_classes must be on, which it is by default in development mode):

Rails.application.eager_load!

Then get an array of all  ActiveRecord::Base sub-classes. It also get sub-classes of sub-classes, by the way.


ary = ActiveRecord::Base.descendants

Then you just need to list them:

puts ary.map {|m| "#{m.to_s} ~ :#{m.column_names.join ', :'}" }.join("\n")

Here is a useful page about preparing the upgrade to Rails 4, by the way:
https://iprog.com/posting/2013/07/preparing-for-an-upgrade-to-rails-4-0

Monday, 1 July 2013

Code Coverage with SimpleCov


Recently I have been using SimpleCov to test my code coverage. When UI first tried to use it, it threw up errors, but having updated the infrastructure, it seems to work fine (though I do get a ton of warnings). Just so you know, I am using JRuby 1.7.4 and Rails 3.2.13 - not sure which of these was causing the problem.

To get coverage statistics, install the SimpleCov gem, and put it in the gemfile.

group :test do
  gem 'simplecov'
end

In your test/test_helper file require it. Here I require it only if COVERAGE is set in the command line:

require 'simplecov' if ENV["COVERAGE"]

Then set some parameters in that file too. Here I exclude some folders and files, and then tell SimpleCov to group results (again, only if that command line flag is set).

SimpleCov.start do
  add_filter 'test/'
  add_filter 'config/'
  add_filter 'vendor/'
  add_filter 'mod_db.rb'  # One use methods for modifying the database

  add_group 'Controllers', 'app/controllers'
  add_group 'Models', 'app/models'
  add_group 'Helpers', 'app/helpers'
  add_group 'Mailers', 'app/mailers'
  add_group 'Views', 'app/views'
  add_group 'Library', 'lib/my_lib'
end if ENV["COVERAGE"]

It is slower than normal testing, so check that everything passes first. I use this on my command line, so err.txt catches all the warnings.

jruby -S rake test COVERAGE=true 2>err.txt >tmp.txt

The results appear in coverage/index.html, and look like this:



 Clicking on a file name will bring up that file, and any untested lines will be highlighted in red. I found a few places when a page should show a record and a set of associated sub-records, but the code for the associated was missed because I had not set the record to have any sub-records in my test.

Thursday, 6 June 2013

Testing Mailers

The first thing to check when testing your project sends e-mails is that your project is not sending them when you are testing. Rails does this by default by a setting in config/environments/test.rb:

  config.action_mailer.delivery_method = :test

Compare to the line in config/environments/development.rb

  config.action_mailer.delivery_method = :smtp


Setting this to test stops e-mails being sent, and instead they are sent to an array, ActionMailer::Base.deliveries. This array is reset before each test, by the way.

Here is a simple mailer to test.


class Notifier < ActionMailer::Base
  default :from => "DB@mysite.com"
  add_template_helper(ApplicationHelper)
  
  # Sets up an e-mail for notifying the user to activate his account.
  def signup_notification user
    @name = user.username
    @login = user.login
    @url  = "http://mysite.com/activate/#{user.activation_code}"
    mail :to => user.email, :subject => 'Activate your account'
  end

  def page_error(err, request)
    @err = err
    @request = request
    mail :to => 'admin@mysite.com', :subject => 'Page error'
  end
end

In your controller, you might invoke the first like this:

  Notifier.signup_notification(@user).deliver

In the test, you will need to break that up, so you can examine the mail object.

class NotifierTest < ActionMailer::TestCase
  test "signup_notification" do
    # Create a mock user
    user = TestUser.new 'tester', 'tester@nowhere.com'
    # Invoke the mailer method
    mail = Notifier.signup_notification user
    # Deliver the mail
    mail.deliver
    # Check the mail got sent
    assert !ActionMailer::Base.deliveries.empty?
    # Check it is the right mail
    assert_equal 'Activate your account', mail.subject
    assert_equal ["tester@nowhere.com"], mail.to
    assert_equal ["DB@mysite.com"], mail.from
    assert_match "Visit this url to activate your account",
                    mail.body.encoded
  end
end

Here is the TestUser definition.

class TestUser
  attr_reader :username, :email, :login

  def initialize username, email
    @username = username
    @email = email
    @login = username.gsub ' ', ''
  end
  
  def activation_code; "abcd"; end
end

The second method in the mailer above is for sending error reports to the administrator. Here is a method that generates an error, and sends that to the mailer:

  test "page_error" do
    request = TestRequest.new
    begin
      raise "A test error"
    rescue Exception => err
      mail = Notifier.page_error(err, request)
      mail.deliver
      assert !ActionMailer::Base.deliveries.empty?
      assert_equal "Page error", mail.subject
      assert_equal ["admin@mysite.com"], mail.to
      assert_equal ["DB@mysite.com"], mail.from
      assert_match "error encountered!", mail.body.encoded
    end
  end

Here is the TestRequest definition; it simply returns the string "good" if the method name is recognised - that is enough for the mailer to wok with, and will still highlight any mistyped or made-up method names.

class TestRequest
  # Returns the string "good" if the method is recognised
  def method_missing method, *args
    return "good" if [:fullpath, :request_method, :query_parameters,
                      :request_parameters, :referer].include? method
    super  
  end
end
end