Solr update / sunspot commit way too slow

We have recently updated one of our applications to ruby 2.1.2 and rails 3.2 and, as part of the process, updated most of the gems used. One of the gems updated was sunspot, from 1.3.0 to 2.1.0.

After this update we started experiencing terrible performance in solr update when running a sunspot commit. And by terrible I mean awful. 45s average per commit.

If you are experiencing something like this too, here are some tips (and, at the end, the solution to our problem):

• Check if your system has enough free memory. If you are swapping, that could be your problem.

• If you use the sunspot-solr gem to start solr, run it in the foreground to get the full log on your screen and try to identify which step is taking long (in our case it was SpellChecker):

sunspot-solr -s /my_app/shared/solr/ -d /my_app/shared/solr/data --pid-dir=/my_app/shared/pids/ run 

will start solr in the foreground.

• In the configuration file, solrconfig.xml, check if the values of autoCommit.maxTime and autoSoftCommit.maxTime aren't to low:

         <autoCommit>

           <maxTime>60000</maxTime>

         </autoCommit>

         <autoSoftCommit>

           <maxTime>15000</maxTime>

         </autoSoftCommit>

To be honest, I don't know what are the ideal values for them, but 60s between each autoCommit and 15s between each softCommit seemed fine for our purposes.

• Check if the value of the autowarmCount property for the caches is not too high. In our case, it was 0 and so it remained.

autowarmCount="0"

• Check if the automatic building of the spellcheck dictionary on every commit is not enabled. In our case, that was the culprit! This is a very expensive operation and it was being invoked after every single commit. We simply changed the property buildOnCommit of the spellchecker to false and the average commit time dropped to 19ms. Excellent!

Performance killer:

<str name="buildOnCommit">true</str>

Way to go:

<str name="buildOnCommit">false</str>

Just as an observation, buildOnCommit = true was the default value on the configuration file generated by sunspot-solr.