LibraryFind 0.9.2 update

 LibraryFind  Comments Off on LibraryFind 0.9.2 update
Jun 272009

I posted this update Thursday.  It fixes some problems found in 0.9.1.  The main issue with 0.9.1 was that the svn got foobared a little bit and some files needed for install were left out of the package.  Files that were left out and errors that they caused:

1) migrations 031 & 032 – would cause a method not found error of is_private when adding/editing collections

2) QueryAPI note found – the services directory was left out

3) classic_pagination not found – the classic_pagination plugin was left out


The errors were a bit of suprise because all the files were in the svn, but they had become weirdly locked.  I had to run svn cleanup a few times and then switch branches between the dev and trunk to finally get everything synced up.  But, I’ve had outside confirmation that everything is good now.  So if you were trying to install 0.9.1 and had trouble, pick up 0.9.2.  It should solve your problem. 

You can find it at:



 Posted by at 10:34 am

LibraryFind 0.9.2 update coming

 LibraryFind  Comments Off on LibraryFind 0.9.2 update coming
Jun 192009

I’ve been getting some questions from a number of people trying to upgrade to LF 0.9.1 and it appears that our SVN has gone a little wonky.  When I checked in the last set of changes, appearly, SVN marked a number of the files and keep the changes from moving from my development branch to the trunk.  The files in question are:

  1. app/services (the wsdl api element which will result in a queryapi missing error)
  2. migrations 031 and 032 missing (resulting in an is_private undefined error on collection)
  3. vendor/plugins/classic_pagination (resulting in a missing pagination reference).

I’m sorting through the svn right now and will post an update to 0.9.2 by Monday that includes a few changes and the missing files.



 Posted by at 1:20 pm

LF WorldCat Connector

 LibraryFind  Comments Off on LF WorldCat Connector
Mar 022009

I’m going to spend some time getting the WorldCat API gem that I created before code4lib uploaded to rubyforge – but in the mean time, I’ve created a worldcat connector using this gem that is now in the LF 0.9 branch. 

For LF 0.9.5 – the big change will be something that I learned about at c4lib.  Bess Sadler talked about how they were able to turn a rails product that they create into a plugin making installation easier using a rails engine.  I’ve been looking over the documentation and this seems like something that will be very, very easy to do – so it will get done.  That should make updating the program much, much easier going forward.


 Posted by at 1:09 am

I think I like mod_rails

 LibraryFind  Comments Off on I think I like mod_rails
Jan 052009

I’ve been doing a little performance and memory benchmarking, but let me say that I think I like mod_rails.  I’ll write up something a little bit later, but for distribution, mod_rails = sysadmin love.  Just install a gem, run an installer and add 5  lines to your httpd.conf file and your running.  As a bonus, it’s even slightly faster than mongrel (though you pay for it with a slightly higher footprint upfront).  Anyway, you got to love that.

Anyway, LibraryFind 0.9 is “finished”, having been migrated to Rails 2.2.2 on a server running mod_rails for testing.  I’m going to be working to wrap this up this week (I hope) and move it from R&D testing to public testing here in the library.  And as an fyi — I have a few more cool changes to note — though those will be subject of a different post.



 Posted by at 4:05 pm

LibraryFind 0.9: SOAP and REST APIs

 LibraryFind  Comments Off on LibraryFind 0.9: SOAP and REST APIs
Dec 242008

While LibraryFind has always supported SOAP, LibraryFind 0.9 will finish the process of adding json apis for all the SOAP based apis.  This process started in LibraryFind and continued in release — but will be finished with 0.9.  I’m working up documentation for the Json calls (which basically emulate the SOAP calls for simplicity), so I’ll post the link once its finished.



 Posted by at 2:09 am

LibraryFind 0.9: Using Custom Connectors to bridge non-standard collections

 LibraryFind  Comments Off on LibraryFind 0.9: Using Custom Connectors to bridge non-standard collections
Dec 242008

One of the tenants behind LibraryFind has always been that LibraryFind would only query materials that provide some kind of standard search protocol.  However, there are many sites that provide API access, but it’s no a standard api access like OpenSearch for example.  For example, a user wanting to query Yahoo or Flickr (where many libraries are starting to build collections) would have previously been unable to use LibraryFind to query these resources.  However, that will change with LibraryFind 0.9.   LibraryFind 0.9 introduces a custom connectors framework, that will allow users (including OSU) to develop custom connectors to resources that utilize stable, formalized APIs within LibraryFind. 

Configuring these new resources is easy.  In the collection administration screen (note, this might change slightly), a user would simply note that the connection type is connector, and then name the connector in the Host area.  From there, the user doesn’t need to define any other elements (though you can). 

Admin Interface Example:


Once set, the application will utilize the connector as any other standard search class.  So far example, I created a test group and queried my name using our IR, Flickr and Yahoo.  Using these elements, I retrieve the following:


Here you can see an integration of Internet resources (from yahoo), images (from flickr) and Articles (our IR).  Bringing Internet resources into the results complicates relevancy ranking (in part because there is so little metadata about the items being retrieved), but that’s something that I’ll worry about as we start to work with these items within the results set.

So how will this work.  Well, I thought about going the plugin route (since Rails already provides a good model), but instead decided that I wanted to keep these custom search classes near the predefined search classes. So, in the environment.rb file, I defined an additional load_path under models (custom_connectors).  Within this directory, users can drop their home made custom connectors for use by the application. 

The connectors themselves must use the same format as the general search connector.  Within the directory, I’ll include an example connector, but in a nutshell, the code generally looks like the following:

   1:  # LibraryFind - Quality find done better.
   2:  # Copyright (C) 2007 Oregon State University
   3:  #         
   4:  # This program is free software; you can redistribute it and/or modify it under 
   5:  # the terms of the GNU General Public License as published by the Free Software 
   6:  # Foundation; either version 2 of the License, or (at your option) any later 
   7:  # version.
   8:  #       
   9:  # This program is distributed in the hope that it will be useful, but WITHOUT 
  10:  # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS 
  11:  # FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
  12:  # this program; if not, write to the Free Software Foundation, Inc., 59 Temple 
  13:  # Place, Suite 330, Boston, MA 02111-1307 USA
  14:  # 
  15:  # Questions or comments on this program may be addressed to:
  16:  #   
  17:  # LibraryFind
  18:  # 121 The Valley Library
  19:  # Corvallis OR 97331-4501
  20:  #
  21:  #
  23:  require 'rubygems'
  25:  class ExampleSearchClass < ActionController::Base
  26:    @cObject = nil
  27:    @pkeyword = ""
  28:    @feed_id = 0
  29:    @search_id = 0
  31:      logger.debug("collection entered")
  32:      @cObject = _collect
  33:      @pkeyword = _qstring.join(" ")
  34:      @feed_id =
  35:      @search_id = _last_id
  36:      begin
  37:        #perform the search
  38:        results = your_search(@pkeyword, _max.to_i)
  39:      rescue Exception => bang
  40:        if _action_type != nil
  41:           _lxml = ""
  42:           logger.debug("ID: " + _last_id.to_s)
  43:           return my_id, 0
  44:        else
  45:           return nil
  46:        end
  47:      end
  49:      if results != nil
  50:        begin
  51:           _lrecord = parse_yahoo(results)
  52:        rescue Exception => bang
  53:          if _action_type != nil
  54:             _lxml = ""
  55:             return my_id, 0
  56:          else
  57:        end
  58:      end
  60:          _lxml = CachedSearch.build_cache_xml(_lrecord)
  62:          if _lxml != nil: _lprint = true end
  63:          if _lxml == nil: _lxml = "" end
  65:          #============================================
  66:          # Add this info into the cache database
  67:          #============================================
  68:          if _last_id.nil?
  69:                  # FIXME:  Raise an error
  70:                  logger.debug("Error: _last_id should not be nil")
  71:          else
  72:                  status = LIBRARYFIND_CACHE_OK
  73:                  if _lprint != true
  74:                          status = LIBRARYFIND_CACHE_EMPTY
  75:                  end
  76:          end
  77:       else
  78:          _lxml = ""
  79:       end
  81:       if _action_type != nil
  82:          if _lrecord != nil
  83:            return my_id, _lrecord.length
  84:          else
  85:            return my_id, 0
  86:          end
  87:       else
  88:          return _lrecord
  89:       end
  90:    end
  92:    def self.strip_escaped_html(str, allow = [''])
  93:          str = str.gsub("&#38;lt;", "<")
  94:          str = str.gsub("&#38;gt;", ">")
  95:          str = str.gsub("&lt;", "<")
  96:          str = str.gsub("&gt;", ">")
  97:          str.strip || ''
  98:          allow_arr = allow.join('|') << '|\/'
  99:          str = str.gsub(/<(\/|\s)*[^(#{allow_arr})][^>]*>/, ' ')
 100:          str = str.gsub("<", "&lt;")
 101:          str = str.gsub(">", "&gt;")
 102:          return str
 104:    def self.your_search(query, max)
 105:      xml = yourquery(query, max)
 106:      _objRec =
 107:      _title = ""
 108:      _authors = ""
 109:      _description = ""
 110:      _subjects = ""
 111:      _publisher = ""
 112:      _link = ""
 114:      #Parse your data
 115:      _start_time =
 117:      #loop through your results and populate Record.
 118:      nodes.each  { |item|
 119:         begin
 120:            record =
 121:            record.vendor_name = @cObject.alt_name
 122:            record.ptitle = normalize(_yourtitle)
 123:            record.title =  normalize(_yourtitle)
 124:            record.atitle =  ""
 125:            record.issn =  ""
 126:            record.isbn = ""
 127:            record.abstract = normalize(_yourdescription)
 128:   = ""
 129:   = normalize(_yourauthors)
 130:   = ""
 131:            record.doi = ""
 132:            record.openurl = ""
 133:            record.direct_url = normalize(_yourlink)
 134:            record.static_url = ""
 135:            record.subject = normalize(_yoursubjects)
 136:            record.publisher = ""
 137:            record.callnum = ""
 138:            record.vendor_url = normalize(@cObject.vendor_url)
 139:            record.material_type = normalize(@cObject.mat_type)
 140:            record.volume = ""
 141:            record.issue = ""
 142:   = ""
 143:            record.number = ""
 144:            record.start = _start_time.to_f
 145:            record.end =
 146:            record.hits = _hit_count
 147:            _record[_x] = record
 148:            _x = _x + 1
 149:         rescue Exception => bang
 150:          logger.debug(bang)
 151:          next
 152:         end
 153:      }
 154:      return _record
 156:    end
 158:    def self.normalize(_string)
 159:      return _string.gsub(/\W+$/,"") if _string != nil
 160:      return ""
 161:      #_string = _string.gsub(/\W+$/,"")
 162:      #return _string
 163:    end
 165:  end


However, within the custom_connectors directory, there will at least be the yahoo_search_class.rb and the flickr_search_class.rb which will provide sample code sets for users wanting to see how a custom_connector may be created.

Anyway, as I continue marching towards the release of the 0.9 code-base, I’ll continue to post some of the new functionality that folks should expect to see in the new version. 



 Posted by at 2:00 am

LibraryFind 0.9: Advanced Search Changes

 LibraryFind  Comments Off on LibraryFind 0.9: Advanced Search Changes
Dec 182008

A number of changes that will be present in LibraryFind 0.9 will be UI related.  One of those changes will be seen in the Advanced Search page.  The layout of the page has been changed significantly to make it easier for users to specify specific (or combined) queries, as well as make it easier for users to select the materials that they want to query.




 Posted by at 12:39 am

LibraryFind 0.9: Embedding RSS query feeds

 LibraryFind  Comments Off on LibraryFind 0.9: Embedding RSS query feeds
Dec 162008

During the past development cycle, one of the most often requested items by our students and faculty has been the ability to watch a query for an extended period of time.  In 0.9, this will be possible.  At any point, a user can create an RSS feed of a specific search and add it to their feed reader.  At this point, that output looks like the following:



 Posted by at 11:06 pm
Dec 082008

As I’ve been working on 0.9, I’ve been trying to migrate few odds and ends into the current 0.8 branch so that I can move them into production faster on our end.  To that end, I’ll be posting an updated to LF by the beginning of next week.  These updates will include:

  1. Update to the Harvester (this will make it a bit more fault tolerant) as well as allowing harvesting of sets (currently) and root level oai providers (not provided currently).  This change required significant changes to the search component as well that deals with the harvested materials.
  2. Auto-detection of namespaces (for oai and sru — needed for libxml)
  3. Removed rexml dependencies for opensearch component
  4. Frozen gems for oai, sru and opensearch into vendor directories (and have added that to the environment.rb file)
  5. Prep code for solr/ferret decision.  I’ll be adding support to use either ferret or solr as your backend indexer for harvesting for 0.9, but some changes are being made to make this easier.  Ferret provides an integrated rails solutions, while solr would provide a hosted index option.
  6. In addition to this, some changes to the libxml module have deprecated a call being used in the oai gem (maybe the sru gem).  I’ll take a look at both this week and update appropriately [as well as keep backworks compatibility if possible]

Something else, we are starting to work with mod_rails.  This allows apache to manage the rails environment — eliminating the need to run rails through packs of mongrels (or other specialized serving mechanism).  I’ll write up something on our experiences for others that might be interested in this approach.


 Posted by at 9:11 am