Is Rails broken?

I am increasing worried that Ruby on Rails is not a stable environment. I was trying to update my project, and hit error after error in the infrastructure.

Warbler does not work with JRuby 9.2.7

See here:
https://github.com/jruby/warbler/issues/451

Using an earlier version resolves it. Is this a Warbler issue or JRuby? Who knows, but it starts to shake my confidence.

It would help if Warbler gave some clue it was doing something. It takes around 30 minutes for me, during which it reports that is is deleting the old version, and later that it is creating the new one. Most of that time.... Nothing. Has it crashed? Who knows.

Cannot create new projects with Rails: 5.2.3

I was trying to do this to investigate other issue, and found I could not even create a new project! This is basic!

I started a thread on Stack Overflow, without much response.
https://stackoverflow.com/questions/56787268/jruby-and-bindex-error-failed-to-build-gem-native-extension?noredirect=1

I tried three different versions of Java, two versions of Windows, two versions of JRuby:

Java: Zulu 11 (64 bit), Zulu 8 (32 bit), 1.8.0 (Win7)
JRuby: 9.2.7, 9.1.17

Turns out I can use Rails 5.2.2. It produces a Rails 5.2.3 project, but does work. But seriously, why would I choose to use something that is obviously broken?

Rake version mismatching

How often do I see this error?

Gem::LoadError: You have already activated rake 12.3.0, but your Gemfile requires rake 12.3.2

I have no reference to Rake in Gemfile, and I can delete Gemfile.lock, but still there is a mismatch between versions when I do bundle. This is exactly what Bundler should be preventing. The solution is to uninstall rake, then install the version the Gemfile requires, 12.3.2 in this case. Why does Bundler not do that?!?

Tomcat will not run my project.

I just see this error on the screen (and no errors in any of the log!):

Trying to register Bundler::GemfileError for status code 4 but Bundler::GemfileError is already registered

This is with both JRuby: 9.2.7 and 9.1.17, and Rails 5.2.3. I tried Bundler 2.0.2, 2.0.1 and 1.16.6, as I suspected it was the culprit, but none worked.

Google cannot find ANY pages with that exact error... It could be because I have Rails 5.2.2 installed, but much of the other rails gems are 5.2.3


My production server is running Tomcat 9.0.14, and 32 bit Java jre 1.8.0 191, so I started from scratch, with Tomcat and Java as per the server, and the last .war that works on the sever, and got that working on my PC.

Then I copied gemfile and gemfil.lock from the working version to the new, and ran bundler successfully, but warbler then said:

Gem::LoadError: You have already activated rake 12.3.0, but your Gemfile requires rake 12.3.2.

Why had that reverted to 12.3.0? Who knows. I had been messing around with Rails versions, so could have done it by accident.



I still do not know the underlying reason for the error, but at least it is running now.

Is Ruby on Rails Broken?

I have to wonder. It feels less stable at version 5 than it did at version 2 or 3 when I adopted it 10 years ago. I would have hoped it was getting more stable as it gets more mature, not less.

I want a framework I have confidence in, and Rails is not that any more.

You must use Bundler 2 or greater with this lockfile.

I have been getting this error recently when doing some things, but not others... These worked:

rails c

rails test

rails test:models

These failed:

jruby -S rake test:models

jruby -S rails test:models

jruby test/models/db_section_test.rb

The error message suggests it is using an earlier version of Bundler, so I spent ages hunting down any rogue version, deleting old versions of JRuby, searching folders and gemfiles. Version 2.0.1 was definitely the only one, so how come it was complaining?

I was also suspicious of JRuby.

Eventually I found a Github discussion for bundler, with the solution buried pretty deep. This seemed to fix it.

gem update --system

bundle (not sure if this was needed)

gem install bundler

Once gem was updated, a new bundler version was available, 2.0.2, and all seemed well at first, but I layter hit upon this:

Gem::GemNotFoundException: can't find gem bundler (>= 0.a) with executable bundle

This seems to occur because JRuby 9.2.7 comes with a slightly older version of Gem, one that cannot run Bundler 2.0.2 (is there really so big a difference between 2.0.1 and 2.0.2?).

I never got this to work, and the only solution was to start from scratch, re-installing JRuby, and install Bundler version 1.16.6.

Updated gem is a bad idea too, as it them tries to run ruby.exe, rather than jruby.exe.

I guess you need to use a modified version of gem for JRuby, which currently is still on 2.0.1 or equivalent, and that means if you are using JRuby, stick with Bundler 1.16.6 for now.

Functional testing with Rails 5

I have not blogged here for a couple of years, mostly because my blogging was part of the learning process, and I got to a point where I knew enough to get by. I upgraded to Rails 5 a little while ago, somewhat late because I use JRuby, and I waited for them to be compatible. I am not adding two new models/controllers to my project, and found out about functional testing in Rails 5.

There is none.

It has been replaced by integration testing.

Okay, I can see the point. You want to test if a request gives the right response. If the user has to log in first, you do the request first, then do the one you want to test. However...

Updating tests

I have over 800 functional tests in my project. If I do a full upgrade to Rails 5, then I have a huge amount of work modifying each and every test.

And I do mean every test, as the way we make requests has changed fundamentally, so:

get :new

... has become:

get root_path

The parameters are different too, so a simple find/replace is not going to work here.

At the moment, that is not necessary, but functional tests are to be deprecated, so it is either update all 800 or so tests or add another gem to the project.

No assigns

Integration testing is "black box", it gives no way to look inside to see what variables are getting set. This seems like a really bad idea in practice.

How can it possibly be a good thing to stop checking something? Because that is what I will have to do. A fair proportion of those 800 tests check the value of a variable or three. If they stop doing that, then I cannot be as confident that passing the test means the code is actually okay.

Another big problem is that I use assigns as a way to see what is going on when a test fails. With integration testing, I will know it fails, but have no idea why. Again, it feels like Rails is deliberately withholding useful information, just because it is philosophically better to do it this way.

Backwards compatibility

If only! Changes like this make me regret choosing Rails. I doubt I would choose it again, to be honest.

Installing gems with JRuby gets "marshal data too short" error

[Note to Blogger: Why was this unpublished? I see nothing that could cause offense and the notice gives no clue at all.]

Took me many hours to track this down. The error seems to be pretty generic, and various solutions have worked over the years:

http://stackoverflow.com/questions/5409675/marshal-data-too-short-error-message-while-installing-watir-webdriver-on-windo
http://stackoverflow.com/questions/19070698/how-to-modify-the-gemfile-marshal-data-too-short

Setting the verbose flag showed there was no network problem:

>gem install refile -V
io/console not supported; tty will not be manipulated
HEAD https://api.rubygems.org/api/v1/dependencies
200 OK
GET https://api.rubygems.org/api/v1/dependencies?gems=refile
200 OK
ERROR:  While executing gem ... (ArgumentError)
  marshal data too short

In the end it turned out to be JRuby in Ruby 2.0 mode.

>jruby --1.9 -S gem install carrierwave
io/console not supported; tty will not be manipulated
Fetching: carrierwave-0.10.0.gem (100%)
Successfully installed carrierwave-0.10.0
1 gem installed

I was trying to install refile to replace paperclip, which seems to be dodgy with Windows (requiring DevKit). Unfortunately, refile requires Ruby 2.1, so the gem cannot be downloaded with JRuby in 1.9 mode, and gem install does not work unless JRuby is in 1.9 mode. Great.

So I am looking at CarrierWave...

Java (and Tomcat) Memory Settings

I am having a problem with my Rails project on Tomcat slowing down and grinding to a halt. As part of the investigation into that, I have been looking at the memory settings in Java (which also apply to Tomcat). I upgraded to Tomcat 8 and Java 8 first, so this applies to Java 8 (which no longer has PermGen, something that featured in the error logs).

Java stores stuff on the heap. This is divided into two sections, the New Generation and the Old Generation, and the former is further divided into two again.

New generation/Eden Space: When a new object is created, it will go here. Garbage Collection (GC) is initiated when this is full, and that involves culling unused objects and promoting used objects.

New generation/Survivor Space: Objects promoted from Eden Space go here. In fact there are two Survivor Spaces, and at GC objects are either culled or moved to the other survivor space (to avoid fragmentation). Objects that have survived a certain number of GCs are further promoted.

Old generation: Long-lived objects eventually turn up here, when promoted from the survivor space..

Permanent generation, or PermGen, was non-heap memory, used for class definitions. It is not used in Java 8, though there are other non-heap memory sections.

Settings

 The -Xms and -Xmx parameters define the minimum and maximum heap sizes, respectively.


-Xmn defines both the minimum and maximum size for the new generation, Eden and Survivor combined.

-XX:NewRatio defines the ratio of young to old. It defaults to 2. Clearly you have to have the old generation at least the size of he new (thogh using -Xmn is a way around that; no point setting both this and -Xmn).

-XX:SurvivorRatio defines the ratio of Survivor to Eden space. Again, it follows that the Survivor Space must be greater than the Eden Space, and that it defaults to 8 indicates it should be much bigger. By default, Java uses an adaptive policy, so -XX:SurvivorRatio will be ignored (but you can set an initial value with -XX:InitialSurvivorRatio). You can turn off the adaptive policy with -XX:-UseAdaptiveSizePolicy.

Here is an example (these are just for illustration of how to use the parameters).

-Xmn1024m
-Xms3092m
-Xmx3092m

-XX:NewRatio=2

-XX:-UseAdaptiveSizePolicy
-XX:SurvivorRatio=8

Thread size

Each thread gets allocated its own chunk of memory, and the size is dicated by -Xss (or -XX:ThreadStackSize). This seems to usually default to 512k, but depends of the JVM, the OS, etc. Some web pages suggest this can be reduced to 128k, whilst others say set it tio 8m. If you have a lot of threads, that will have a serious impact on your memory usage! On the other hand, if the value is too small, you will see StackOverflowErrors.

Tomcat

In Tomcat on Windows you can set these by running Tomcat8w, and putting the values into the Java Options box on the Java tab. There are also boxes specifically for initial heap, maximim heap and thread size, and these may be all you need to fiddle with.

If you go to your Tomcat server status page, you can see the current usage. The page may well be here:

http://localhost:8080/manager/status/all

The memory section might look like this:

This is using the adaptive policy for the survivor ratio, and the survivor space is very small. The important thing (I think) is that there is plenty of room left in the Old Generation area.

By the way the -Xmn1024m format for parameters is the original. Hotspot introduced a shed load more parameters, and also the -XX:NewRatio=2 format for them. Hotspot converts the former to the latter internally for backwards compatibility.

Invoke Dynamic

In the end, what had the most impact was turning off the "invoke dynamic" feature:

-Djruby.compile.invokedynamic=false

The Java/Tomcat PermGen Problem

Running the project on Tomcat, I find that it gets slower and slower, until it grinds to a halt. Java does not bother to garbage collect class definitions, as they are not expected to change much once a project is running. However, running JRuby, that is not the case, and new classes are being generated on the fly all the time. Consequently, the JVM runs out of memory (what it calls PermGen memory), and everything stops.

The solution appears to be modifying start-up options in the JVM. This can be done by running tomcat6w.exe (on Windows anyway), going to the Java tab, and adding these options:

-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled

Whether this is a solution remains to be seen, but I am giving it a go.
See also
http://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmgc.html
http://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do/3334954#3334954

Loading records from YAML to Rails

Occasionally you need to save data from your database, to be loaded back in at a later data, or to be loaded into another database. YAML makes saving the data trivial:

  def self.save
    File.open('data.yml', 'w') do |out|   # To file
       YAML.dump(MyRecord.all.to_a, out)
    end
  end

Loading it into Rails again is not as straighhtforward. Well, getting it into Rails is easy, getting Rails to save the data to the database is not.

There are two issues. The first is that Rails will use the SQL UPDATE method when you try to save the date. That is not going to work unless there is already a record present in the database. I found the simplest way around that s to use raw SQL to create each record with the correct ID, and then use save to update that record with the correct values (obviously you could do all that in one step, which would avoid the second issue, but require more complicated SQL).

The second issue is that ActiveRecord.save will only update attributes that are flagged as dirty, so you need to flag every column.

This is the solution I ended up with:

  def self.load
    ary = YAML.load_file "data.yml"
    ary.each do |data|
      ActiveRecord::Base.connection.execute(

            "INSERT INTO my_records (id) VALUES (#{data.id})"

      )
      MyRecord.column_names.each { |s| eval("data.#{s}_will_change!") }
      data.save
    end