More reasons to avoid REXML (like the plague)

I’ve been having a great time playing with ruby — but one of the things that I find myself constantly running up against is the “ruby way”.  The place where I notice this most is when dealing with XML.  One of the things I’ve noticed in working with XML in ruby is that the ruby crowd doesn’t seem to know what to do with it.  In most documentation that I’ve read — it appears that most ruby folks find little value in XML and prefer to work in YAML.  However, in my world — XML is king and working around it really isn’t an option.  So with XML — what are the ruby options?  There are a few — but by default — ruby pushes a component known as REXML.  In general, I’ve found this to be a nifty little library with lots of convenience functions.  However, it comes a very high cost.  First, I’ve found that there’s a limit to the size of file that can be loaded (~5 MB, then the component starts breaking) — and its slow.  Oh, how is it slow.  But how slow?

Well, I’ve been spending a lot of time reworking the API for LibraryFind, our soon to be open source hybrid Federated Search system.  The big project of course has been moving the data into ruby and out of PHP so we can place the project within a web framework (in our case, Rails).  Well, part of the project deals with harvesting data for local indexing — and in many cases, the data being harvested is in OAI.  So, I’ve been playing with Ed Summer’s ruby-oai (very cool) and its based around REXML.  After installing the module and testing the harvesting of a small collection, I was pretty dismayed with the speed of the application.  I suspected that it was REXML that was causing the slow down. 

So after chatting with Ed a little bit, he’d said that he’d be open to hacking up a very that supported libxml provided the changes didn’t:

  1. Turn the code into an unholy mess
  2. Didn’t require custom XPath statements (and they almost do)

Given that I was going to make these changes anyway for our own local instance — I thought this sounded like a good idea (and I’ve never had a chance to work with Ed, with I couldn’t pass up either πŸ™‚ )

So I spent sometime today modifying the ruby-oai module and finished integrating libxml into the module (I guess I’ll wait now to see if I violated number 1 though πŸ™‚ ) — and after running a small benchmarking application — all is well with the world again. 

So how did the benchmarking go?  Funny you should ask. πŸ™‚  I made an OAI request using the REXML codebase which returned 394 records parsing just the identifier from the header.  Total time:

Time to run: 21.685583
Records returned: 394

Same codebase but just changing the parser — what’s the difference.  Let’s see:

Time to run: 0.75901
Records returned: 394

This kindof a difference really brings a smile to my face.  And considering how many oai sites I have to harvest from (~300), these extra seconds really start to add up.  Test code below…

–TR

Test Code:


require 'oai'

buffer = ""
start_time = Time.now()

client = OAI::Client.new 'http://digitalcollections.library.oregonstate.edu/cgi-bin/oai.exe', :parser =>'libxml'

last_check = Date.new(2006,1,1)
records = client.list_records :set => 'archives', :metadata_prefix => 'oai_dc', :from => last_check
x = 0
records.each do |record|
  #fields = record.serialize_metadata(record.metadata, "oai_dc", "Oai_Dc")
  #puts "Primary Title: " + fields.title[0] + "\n"
  buffer << record.header.identifier + "\n"
  x += 1
end

end_time = Time.now()

puts buffer
puts "Time to run: " + (end_time - start_time).to_s + "\n"
puts "Records returned: " + x.to_s



Posted

in

,

by

Tags:

Comments

6 responses to “More reasons to avoid REXML (like the plague)”

  1. Ed Summers Avatar

    Hey those times are quite an improvement!

  2. Will Groppe Avatar

    Hey thanks Terry! I’m working on a project in Rails that requires both a OAI harvester, and and OAI provider. You and Ed have solved half the problem for me.

  3. Administrator Avatar
    Administrator

    Glad to have helped.

    –TR

  4. Thomas Fee Avatar
    Thomas Fee

    I used REXML in a program that detects “XML diffs”. The REXML programming model was very pleasant, but it was _Ex_Treme_Ly _sLow_. I was glad I realized this before handing out the program for co-workers to use.

    Hate to say it, but I was forced to relegate this library to “junk” status, since operations that shouldn’t take long takes practically forever. It seems to be an exponential function of the XML data size.

  5. Jaime King…

    I Googled for something completely different, but found your page…and have to say thanks. nice read….

  6. […] the swivel gem again and needing to greatly simplify it. at first, it was REXML, but REXML is dog slow. we considered switching to libxml, but that was a totally new interface, plus we use the entire […]