2009 March » Terry's Worklog

MarcEdit 5.1 update

MarcEdit Comments Off

Mar 222009

Couple of quick changes.

Delete Fields – you can delete multiple fields using “x�? syntax, where the ‘x’ is no longer case sensitive. For example:
a) 900 – deletes the 900 field
b) 90x – deletes fields 900, 901, 902, 903, etc.
c) 9xX – deletes fields 900, 901, 902, 903…910, 911…920, 921, etc.
Originally, x’s had be be lower case. I’ve modified the code so that this is no longer case sensitive.
Edit Subfield Function: When working with Control fields (00x fields) – MarcEdit use to require data in the Position element. If the position element is empty, the program will do a usual find/replace – including prepending and concating data.
MarcEdit Preview Function (Bug Fix): If a user loads a MARC file directly into the MarcEditor – the program would lose the temp file – so if you clicked on the link to load the entire file into the MarcEdit, it would get lost. This only occurred when users opened a .mrc (MARC) file directly into the MarcEditor. In all other contexts, the preview link works as documented.

If either of these affect your workflow, you can download the update here: MarcEdit_Setup.msi

–TR

An OA mandate for the OSU library faculty

Digital Libraries, Open Access 2 Responses »

Mar 162009

The OSU Library faculty recently adopted an OA mandate, which is pretty cool. You can read about it here: http://ir.library.oregonstate.edu/dspace/handle/1957/10850 and what a few others are saying about it:

I think that this is important on a number of levels.

Symbolically, it’s important. It’s very difficult for the library to go to faculty on campus and ask them to contribute content to the IR, when in fact, the Library faculty itself is not regularly submitting to the IR. This changes that – and hopefully – will act as a catalysis for other departments on campus to follow the Library faculty’s lead.
As tenured faculty, the research (both papers and presentations) our librarians generate represent an important contribution to the scholarly community. As researchers and scholars, preserving our content and making it freely accessible to future researchers is indeed one of our primary responsibilities as faculty.
This was really a faculty initiated endeavor, that has a great back story, but I won’t include it here right now. But suffice it to say, a good number of people at OSU deserve a lot of credit for making this happen, chief among those being Michael Boock and Janet Webster – who have worked tirelessly from the beginning to advertise, grow and advocate for the IR in the library. And for the faculty as well, for stepping up and making this a reality.
Finally, it’s just one more example of that Beaver ingenuity and can do’edness.

–TR

MarcEdit Update

MarcEdit Comments Off

Mar 162009

I just uploaded a new version of MarcEdit 5.1. Most of the updates are related to changes in the MarcEditor. So here’s the list of changes:

MarcEdit Find (specifically when the regular expression option is selected) – Previously, when searches were done, items were located but the window didn’t scroll to the located item. That’s been corrected.
MarcEdit Replace All (Regular Expressions): One of the changes made during the last update of MarcEdit was to change MarcEdit’s MarcEditor’s replace all function (when using regular expressions) from a single line evaluation to evaluating whole records. This allows for the ability to perform replacement actions by evaluating multiple fields – but I had neglected to consider how this might break current workflows that relied on the previous functionality. So, I’ve returned the functionality to evaluating single lines and added a switch to allow users that want to process data across multiple lines. So for example:
(=6.*)([^.]$) – this would evaluate field by field (line by line)
(=6.*)([^.]$)/m – the /m tells marcedit to evaluate multiple lines – so this would run this expression against the entire record.
MarcEdit Replace All (Regular Expressions): The expression evaluator was too greedy – causing matches to blank records. This should never happen any longer.
MarcEditor – when openning a blank .mrk file using the Open button – the window would hang. That’s been corrected.
MarcEditor – when opening a .mrk file by double clicking on it, then opening a new window – closing either window would close both windows. This has been corrected.
XSLT file updates – I’ve added Creative Commons Zero license headers to the FGDC stylesheets distributed in MarcEdit.
Help File has been updated to cover some of the noted changes.

Couple of notes – I’m currently writing up some new notes on using MarcEdit on Linux. Mono 2.0+ essentially has added all the functionality necessary to run MarcEdit on Linux. I’ll be creating a handful of Youtube videos for folks interested in giving this a try. As for running on a Mac – well, I’ll look at that next.

You can download the new version of MarcEdit from here: MarcEdit_Setup.msi

–TR

WorldCat API gem moved to rubyforge

OCLC, rails, ruby Comments Off

Mar 132009

I had mentioned that I’d quickly developed a helper gem for simplifying using the WorldCat API in ruby (at least, it greatly simplifies using the API for my needs). This was created in part for C4L and partly because I’m moving access from Z39.50 to the API when working with LF and I basically wanted a be able to do the search and interact with the results as a set of objects (rather and XML).

Anyway, the first version of the gem (which is slightly different from the code first posted prior to C4L) can be found at the project page, here: http://rubyforge.org/projects/wcapi/

–TR

Translating Project Gutenberg records

Uncategorized Comments Off

Mar 032009

I wrote this up some time ago, but I still occasionally get questions about it (in fact, I got one today, hence this note). The Project Gutenberg (PG) provides its metadata for download in RDF format on it’s website at: http://www.gutenberg.org/feeds/catalog.rdf.zip. I wrote an XSLT transformation for this data (fairly basic) when I was visiting the Internet Archive last year, and posted it here: http://oregonstate.edu/~reeset/marcedit/html/downloads.html (or directly, at: Project Gutenberg RDF = MARC).

Running the RDF records through MarcEdit using this stylesheet produce the following MARC21 recordset: http://oregonstate.edu/~reeset/marcedit/anonymous/catalog.zip. The process is really a straightforward one. You download the above XSLT stylesheet, register it with MarcEdit and then you can be off on your merry way translating data to your heart’s content. Of course, occasionally, folks ask about translating into other metadata formats, and that’s cool too. If you can work with the API, you can do this in one step. However, if you plan on using the MarcEdit UI, you need to do it in two.

Setup a PG => MARCXML translation (the xslt stylesheet above will do that)
Create or use one of MarcEdit’s provided MARCXML => [format] stylesheets to complete the translation.

So, it’s really a two step process. While many of the YouTube videos that I’ve uploaded in the past few days cover parts of this process, I decided to upload on final video on the topic that demonstrates this process (processing the PG data into both MARC and MODS3) as a reference case for future users looking to do something similar. Hopefully this will help. You can find the video here:

http://www.youtube.com/watch?v=1zHKIJ6D_dA

Cheers,

–TR

EAD => MARC tutorial

MarcEdit 1 Response »

Mar 032009

Just an FYI, since some people ask how we go about generating MARC records from our EAD records using MarcEdit…I’ve posted a short video tutorial. What I didn’t include was the EAD translation (it’s somewhat specific to OSU), but I’m happy to add a link to it if anyone is interested.

Anyway, you can find the video here:

http://www.youtube.com/watch?v=9LP4TTHx7e0

And just for reference, I think going forward, as I create tutorials or clarify information in the documentation – I’ll likely upload a video to youtube and will use MarcEdit as a common tag.

–TR

LF WorldCat Connector

LibraryFind Comments Off

Mar 022009

I’m going to spend some time getting the WorldCat API gem that I created before code4lib uploaded to rubyforge – but in the mean time, I’ve created a worldcat connector using this gem that is now in the LF 0.9 branch.

For LF 0.9.5 – the big change will be something that I learned about at c4lib. Bess Sadler talked about how they were able to turn a rails product that they create into a plugin making installation easier using a rails engine. I’ve been looking over the documentation and this seems like something that will be very, very easy to do – so it will get done. That should make updating the program much, much easier going forward.

–TR

MarcEdit 5.1 Update

LibLime, MarcEdit 3 Responses »

Mar 022009

Getting this last update out has taken a little more time than I would have liked, but I really wanted to think some of the issues that this update raised through so that the update process would be seamless. Realistically, were I versioning MarcEdit in any realistic versioning process, this would likely be at the very least, a new point release. However, I’ve already planned out my 5.5 release, and this request, while major, fell into this gray area – so I decided to keep this into the 5.1 branch. Anyway, March 2nd is a good day to officially make this release. March 2nd is my 32nd birthday, and this version of MarcEdit will be present to the MarcEdit user community. Cheers.

Major Changes:

In the continued work towards helping enterprise users, I’ve finally finished the installer update process that began with the last update. In the previous update, MarcEdit’s installer was modified (as was MarcEdit itself) to make it more user aware. What do I mean here – the program was changed so that users running MarcEdit no longer needed to be administrators when running the program, but rather, all configuration and mutable files were moved into the users Application Data directories. This had a number of unintended benefits like supporting multiple MarcEdit users on the same machines (using custom user profiles) and making it easier to copy configuration settings from one computer to another.

This update takes this one step further. As Libraries continue to move to more sophisticated application management, I’ve been running into more users that have their software managed through a central IT source. The IT groups manage the software by automating a process to do distributed installation. In the past, MarcEdit’s installer really didn’t do this well. Well, about 2 months ago, I was contacted by a large IT group on the west coast wanting to know if this would be possible. This required finishing the migration from MarcEdit’s custom installer to the Microsoft Windows Installer – while at the same time, making sure that the program cleans up the previous install while still keeping the users previous settings throughout the upgrade. After a lot of testing (over the last month, spanning multiple institutions and users in different development environments) – I feel like this is ready to go.

When you install the new msi installer, what will happen.
1. MarcEdit will evaluate your current installation – if you have never installed MarcEdit, it simply installs the application
2. If a previous version of MarcEdit is present, the program will copy, in order: config. data in the User Application directory, data in the MarcEdit Program directory – and then silently uninstall the previous version of MarcEdit. Once the previous version has been removed, the installer when then install the new version of the Application.
3. Part of the clean up process of the new install is to move the copied user data back into scope of the application.

So what does this mean to you? Well, if you are an individual user, who manages MarcEdit on your own machine, very little. The one benefit that you will likely see from this migration is the eventual development of an automated updater. The msi installer provides a number of very powerful and integrated functions that I should be able to leverage to potentially create an unmediated upgrade process for users. For enterprise users however, this change will for the first time give your IT administrators the ability to install MarcEdit on multiple machines simply by using their enterprise software management system. For colleges and universities that manage thousands of users, this should be a really big win.
Z39.50 Changes: This will be an incremental process, but for the first time, MarcEdit will allow users to query multiple user databases during a Z39.50 lookup. This will allow users to query multiple Z39.50 targets to return data about a search. This initial implementation allows multiple searching to be done in the Single Search mode. In a planned future update, this will be extended to the batch search update, with rules regarding how to disambiguous duplicate records (for example, the ability to accept records from one target over another, etc.). So how does this work. Essentially, when you enter the single search, you select Select database and then select multiple items. Up front, this is limited to 3 databases, but that limit will eventually be removed (especially as I get UI feedback). When you select multiple database, the Single search screen changes to look like the below…
Do you see the data in the red box? This is how you can see what resources MarcEdit will be querying. Also, see the green box. You can see here that MarcEdit’s Z39.50 results list has changed slightly to let users see what institution each record is from.
Help File is now local again: Sadly, some topics are already out of date slightly (the Z39.50 info for example doesn’t represent the multiple querying functionality) – but this makes the help available both online and offline. The online help will always be more up todate, but will be updated on each build.
Youtube tutorials. If you go to Youtube and look for marcedit, you will find a series of tutorials related to MarcEdit topics. At present, you will find topics for:
1. Breaking your file

http://www.youtube.com/watch?v=MOLJlQQ5RL8
2. Making your file

http://www.youtube.com/watch?v=8UU7gjv_eo8
3. Editing a MARC file

http://www.youtube.com/watch?v=kNMExHdki9k
4. Converting a files characterset

http://www.youtube.com/watch?v=gHxMALV-C4k
5. Adding a new XML Function

http://www.youtube.com/watch?v=3x5Ke81AoEU
6. Updating a current XML Function
7. Using the Delimited Text Wizard

http://www.youtube.com/watch?v=Kp_N3ncjS7Q
8. Extracting a subset of records from a larger set

http://www.youtube.com/watch?v=2F8DgEO4zSg
9. Using the Z39.50 Client

http://www.youtube.com/watch?v=y0YibTP1dIs
10. Harvesting OAI data into MARC

http://www.youtube.com/watch?v=gvBrMVH6j7U
11. Managing Plug-ins through the plug-in editor.

http://www.youtube.com/watch?v=ZTxgL1BAmew
OCLC Connexion Client Plugin has been updated (was needed do to updates in a few other components). If you use the Connexion Client plugin, you will need to update this plugin once you update.
MarcEditor Editor update to better support UTF-8 data loading. Essentially, in layman’s terms, here’s what has changed. In previous versions of MarcEdit, loading UTF8 data into the MarcEditor would sometimes cause the process to load slowly. The why this occurred had to do with the way that the specific windows component that I was using handled text. I’ve updated MarcEdit so that this process has been changed (as has the component) making it so that data loaded into the MarcEditor now uses a new editing component and one that natively handles UTF8 data. The lag time that users previously experienced should no longer be applicable. In addition to this fix, MarcEdit’s MarcEditor’s memory footprint has been reduced. Not drastically, but a bit. One thing to remember when loading data into the MarcEditor. There is roughly a 4-1 memory usage when loading bytes into a visual interface in Windows. So, for example, load a 20 MB file, and Windows will allocate ~80 MB of memory to view the file. Open a 120 MB file, and Windows will need to allocate ~480 MBs to render the file. The new Editor is able to reclaim some of this memory on the high end, but this is in part, how visual interfaced work.
Also, note the 1/2 GB limit of data loading into the Editor, but that data of any size can continue to be edited in the editor if one makes use of the Preview mode.
Yaz Update – Previous version of MarcEdit used Yaz 1.+ because it was small and fast. I had need for some of the enhanced functionality, so I’ve updated the version of used Yaz to 3.+.
‡biblios.net Proof of concept Plug-in: As noted in a recent post, while attending code4lib this week, the folks at Liblime demonstrated their new ‡biblios.net platform. For those that haven’t heard, the ‡biblios.net platform is an attempt to create a large, shared, Open Data repository of bibliographic metadata. What I find most interesting about LibLime’s effort has been the development of an open API to provide push/pull functionality into the database. In theory, this allows library developers the ability to develop tools around the ‡biblios.net platform. The plug-in demonstrates how this works, as well as providing folks that want to work with the ‡biblios.net platform a way to integrate their workflow with MarcEdit. You can see the Youtube video talking about how it works, here:

http://www.youtube.com/watch?v=Y8tldqWhFjs

Minor Changes:

UI changes to the Z39.50 (to accommodate the changes in functionality)
Extended regular expression support in the Replace function so that regular expressions can be run over multiple lines.
Updated workflow – when converting data from UTF-8 to MARC8 using the MarcBreaker, the 9th byte in the leader isn’t set correctly. This is partly because the previous workflow assumed moving the other direction. Since this cause some problems with some loaders, its been corrected.
Updated the Marc21XML xslt function to accommodate the following:
a. Up to 9 indicators (per UniMARC)
b. Ability for indicators to be mixed. The current version assumes indicator order, the update allows indicators to appear in whatever order.
Other minor changes

You can pick up the update at: MarcEdit_Setup.msi.

If you run into any problems, please give me a holler.

–TR