Two Days with Solr, or 25x Speed Up of Reindexing

by Eugene MelnikovMarch 5, 2013

This blog post shares best practices and lessons learned on how to increase the speed of reindexing with Solr in a Ruby-on-Rails app.

Table of Contents

Increasing the speed of reindexing with Solr

I have chosen Solr for my Ruby-on-Rails application, because the platform allows for using flexible fields for indexing. However, when deploying the app to production, I faced the negative argument error. This was due to progress_bar. It was easy to fix. I moved it to the development section in Gemfile.

However, the database was already quite big (300,000 records for indexing), and reindexing took approximately an hour. I decided to investigate the problem.

After reading this article, I uncommented mergeScheduler in solr/conf/solrconfig.xml, as well as set ramBufferSizeMB to 960, mergeFactor to 40, and termIndexInterval to 1024. It didn’t increase speed at all, though. I checked how busy my virtual machine was. There was about 50% of free memory, and a central processing unit (CPU) was loaded with 100%. After adding two more cores to CPU, it used 33% of each core on average.

After that, I started to investigate possible options for indexing. The first option is batch_size. There are a lot of suggestions to increase indexing to 1,000 records (no more because of the Java garbage collector). However, the rake sunspot:solr:reindex[1000] task still worked slowly. I tried to run indexing from the Model.solr_reindex(:batch_size => 1000) console, and it took less time than the rake task!

I have found two more interesting options: include and batch_commit. Include allows to select the same batch of rows from a database, as you defined to avoid the n+1 problem. batch_size indexes all rows at once and, after that, commits it. Using include and batch_size, I got much better results. I created the rake task that reindexes all my models using the discovered options and got speed of approximately 750 records per second.

It was still strange for me why the rake task worked so slow. After looking into the :sunspot namespace, it became obvious. sunspot:solr:reindex is a deprecated task, and it just runs sunspot:reindex without any options. Ok. I have found out how to pass batch_size. How to pass include and batch_commit then?

Fortunately, sunspot_solr and sunspot_rails were recently updated, and they were able to pass include from the searchable method in the model and hardcode batch_commit to false. I started from 80 records per second, and, by this moment, I got around 1,100 records per second.

New, the error I got was undefined method `closed?' for nil:NilClass. In the source code of rsolr, I have found that this is a confirmed Ruby bug. I updated Ruby to Ruby 1.9.3-p362 and got around 2,000 records per second!

There is just one small issue with speed decrease. You can use your own rake task until it’s fixed.

Conclusion

These are the major lessons learned:

uncomment mergeScheduler in solr/conf/solrconfig.xml, as well as set ramBufferSizeMB to 960, mergeFactor to 40, and termIndexInterval to 1024
use the latest version of Ruby and gems
use the rake task correctly: rake sunspot:reindex[1000]
define related tables in your models using :include
remember that the searchable methods—if, unless, ignore_attribute_changes_of, and only_reindex_attribute_changes_of—accept a lot of interesting options that can speed up you application