Thursday 23 April 2015

Installing gems with JRuby gets "marshal data too short" error

[Note to Blogger: Why was this unpublished? I see nothing that could cause offense and the notice gives no clue at all.]

Took me many hours to track this down. The error seems to be pretty generic, and various solutions have worked over the years:

http://stackoverflow.com/questions/5409675/marshal-data-too-short-error-message-while-installing-watir-webdriver-on-windo
http://stackoverflow.com/questions/19070698/how-to-modify-the-gemfile-marshal-data-too-short

Setting the verbose flag showed there was no network problem:

>gem install refile -V
io/console not supported; tty will not be manipulated
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=refile
200 OK
ERROR:  While executing gem ... (ArgumentError)
  marshal data too short

In the end it turned out to be JRuby in Ruby 2.0 mode.

>jruby --1.9 -S gem install carrierwave
io/console not supported; tty will not be manipulated
Fetching: carrierwave-0.10.0.gem (100%)
Successfully installed carrierwave-0.10.0
1 gem installed

I was trying to install refile to replace paperclip, which seems to be dodgy with Windows (requiring DevKit). Unfortunately, refile requires Ruby 2.1, so the gem cannot be downloaded with JRuby in 1.9 mode, and gem install does not work unless JRuby is in 1.9 mode. Great.

So I am looking at CarrierWave...

Thursday 12 March 2015

Java (and Tomcat) Memory Settings

I am having a problem with my Rails project on Tomcat slowing down and grinding to a halt. As part of the investigation into that, I have been looking at the memory settings in Java (which also apply to Tomcat). I upgraded to Tomcat 8 and Java 8 first, so this applies to Java 8 (which no longer has PermGen, something that featured in the error logs).

Java stores stuff on the heap. This is divided into two sections, the New Generation and the Old Generation, and the former is further divided into two again.

New generation/Eden Space: When a new object is created, it will go here. Garbage Collection (GC) is initiated when this is full, and that involves culling unused objects and promoting used objects.

New generation/Survivor Space: Objects promoted from Eden Space go here. In fact there are two Survivor Spaces, and at GC objects are either culled or moved to the other survivor space (to avoid fragmentation). Objects that have survived a certain number of GCs are further promoted.

Old generation: Long-lived objects eventually turn up here, when promoted from the survivor space..

Permanent generation, or PermGen, was non-heap memory, used for class definitions. It is not used in Java 8, though there are other non-heap memory sections.

Settings


 The -Xms and -Xmx parameters define the minimum and maximum heap sizes, respectively.


-Xmn defines both the minimum and maximum size for the new generation, Eden and Survivor combined.

-XX:NewRatio defines the ratio of young to old. It defaults to 2. Clearly you have to have the old generation at least the size of he new (thogh using -Xmn is a way around that; no point setting both this and -Xmn).

-XX:SurvivorRatio defines the ratio of Survivor to Eden space. Again, it follows that the Survivor Space must be greater than the Eden Space, and that it defaults to 8 indicates it should be much bigger. By default, Java uses an adaptive policy, so -XX:SurvivorRatio will be ignored (but you can set an initial value with -XX:InitialSurvivorRatio). You can turn off the adaptive policy with -XX:-UseAdaptiveSizePolicy.

Here is an example (these are just for illustration of how to use the parameters).

-Xmn1024m
-Xms3092m
-Xmx3092m

-XX:NewRatio=2

-XX:-UseAdaptiveSizePolicy
-XX:SurvivorRatio=8

Thread size

Each thread gets allocated its own chunk of memory, and the size is dicated by -Xss (or -XX:ThreadStackSize). This seems to usually default to 512k, but depends of the JVM, the OS, etc. Some web pages suggest this can be reduced to 128k, whilst others say set it tio 8m. If you have a lot of threads, that will have a serious impact on your memory usage! On the other hand, if the value is too small, you will see StackOverflowErrors.

Tomcat

In Tomcat on Windows you can set these by running Tomcat8w, and putting the values into the Java Options box on the Java tab. There are also boxes specifically for initial heap, maximim heap and thread size, and these may be all you need to fiddle with.

If you go to your Tomcat server status page, you can see the current usage. The page may well be here:

http://localhost:8080/manager/status/all

The memory section might look like this:




This is using the adaptive policy for the survivor ratio, and the survivor space is very small. The important thing (I think) is that there is plenty of room left in the Old Generation area.

By the way the -Xmn1024m format for parameters is the original. Hotspot introduced a shed load more parameters, and also the -XX:NewRatio=2 format for them. Hotspot converts the former to the latter internally for backwards compatibility.

Invoke Dynamic

In the end, what had the most impact was turning off the "invoke dynamic" feature:

-Djruby.compile.invokedynamic=false