Aug 012014
 

After spending the last 3 months of work – I’ve posted the latest and greatest version of MarcEdit; MarcEdit 6.0.  This represents a significant change on a number of fronts – first and foremost, a lot of code has been reformatted to provide better support for non-Windows systems.  For the first time, a MarcEdit packaged App file is being provided to give Mac users one click installation of the application.  This means that at this point, the installation process for a Mac is as simple as – Install Mono, then download the MarcEdit app.  Unzip, and run.  For Linux systems, a zip file can be downloaded, which includes the 3 steps needed to run the application. 

No while I have spent a significant amount of time and code working on MarcEdit to make it run better on non-Windows systems, I will say up front that this is still a work in progress.  You will occasionally run into UI issues and very likely run into times when the program crashes.  Additionally, at this time, Z39.50 functionality will not work on the Mac.  This should be temporary as I work on implementing a webservice that will simplify how MarcEdit provides these services and removes a dependency.  At this point, I have all the code for the service, I just need to find a place to host the project.  If you are using MarcEdit on a Mac – I’d love to hear how it’s going.  I’ll likely be spending the next year working to translate the over 150 GUI elements to MonoMac, a native Cocoa implementation of the UI elements – this should solve many of the Mac UI problems and allow MarcEdit to look and feel like a regular Mac application.  However, this is a significant time commitment so before I consider doing this work – I’ll want to get an idea of how widely used the Mac version of the application is.  So, if you are a Mac user and want to continue to see this improve – send me feedback.  Oh, and I’d be remiss if I didn’t thank the folks at the Ohio State University Libraries for making a Mac system available for me to do testing.  Without it, the Mac app bundle simply wouldn’t exist.

Now the changes – this version saw the deletion of over 10,000 lines of code, and the creation of close to 28,000 lines of new code.  There were a handful of new tools, lots of new functions, and a number of bug and UI fixes.  In the end, I closed 47 tickets.  For the full list of changes – please see the below. 

So how do you get the program.  If you are current MarcEdit user, the program will prompt you to update the next time you run the program.  The application will automatically try to migrate your data from MarcEdit 5.x to the new MarcEdit 6.0 directory – and will prompt you if successful.  If not, you can manually transfer the data by selecting the Help menu, About, and Migrate data.  If you are a Mac or Linux user, go to the download page and select the appropriate download. 

For direct links:

MarcEdit 6 is an in place replacement of MarcEdit 5.  You cannot run these two versions on the same system.  For that reason, I’m leaving the last version of MarcEdit 5.9 available just in case users run into issues with 6.0 as everything shakes out.  To use this version, you will have to uninstall MarcEdit 6, and then go to the download page at: http://marcedit.reeset.net/downloads.

–TR

**** Update List ****

Summary Priority Category Project Opened Date OpenedBy Due Date Resolved As
MARC Tools window refresh (3) Minor Performance MarcEdit 11-Jun-14 reeset   Fixed
Hard to find EULA after install (4) Trivial Bug MarcEdit 31-Jul-14 reeset   Fixed
Extract by Material type (3) Major Enhancement MarcEdit 14-Jun-14 reeset   Fixed
Allow tasks to have comments (3) Major Enhancement MarcEdit 14-Jun-14 reeset   Fixed
Task List — Swap Field not saving Process one per field (2) Major Bug MarcEdit 12-Jun-14 reeset   Fixed
UI Issue — Validation (3) Minor Bug MarcEdit 12-Jun-14 reeset   Fixed
Adding multiple fields in the Add/Delete Function (3) Major Enhancement MarcEdit 12-Jun-14 reeset   Fixed
Script Wizard — add/delete field with modifier (3) Major Bug MarcEdit 12-Jun-14 reeset   Fixed
RobertCompare/MARCCompare (3) Major Enhancement MarcEdit 12-Jun-14 reeset   Fixed
RDA Helper (3) Major Bug MarcEdit 11-Jun-14 reeset   Fixed
Task Manager & Task List tool (3) Major Enhancement MarcEdit 14-Jun-14 reeset   Fixed
Create a Mac Installer (3) Major Bug MarcEdit 11-Jun-14 reeset   Fixed
Remove empty field — missing fields that should be removed (3) Major Bug MarcEdit 16-Jul-14 reeset   Fixed
Refresh the Delimited Text Translator (3) Minor Enhancement MarcEdit 11-Jun-14 reeset   Fixed
Add/Delete Field — Find What Field (3) Minor Bug MarcEdit 11-Jun-14 reeset 01-Jul-14 Fixed
Sharing Tasks which refer to other tasks (3) Minor Bug MarcEdit 11-Jun-14 reeset 01-Jul-14 Fixed
RDA Helper — generating extra blank lines (4) Trivial Bug MarcEdit 22-Jul-14 reeset   Fixed
Preview File — show file name being previewed (3) Minor Bug MarcEdit 22-Jul-14 reeset   Fixed
Edit Field Utility — field drop down list (3) Major Bug MarcEdit 30-Jul-14 reeset   Fixed
Edit Field Utility — can’t edit the ldr (3) Major Bug MarcEdit 30-Jul-14 reeset   Fixed
OCLC Integration — validation errors (3) Major Bug MarcEdit 23-Jul-14 reeset   Fixed
On first load, Mac systems not auto detecting font (3) Major Bug MarcEdit ######## reeset   Fixed
Changing Capital Letters with Diacritics to lower case (3) Major Bug MarcEdit 11-Jun-14 reeset   Fixed
Allow the RDA Helper to be part of the Task List (3) Major Bug MarcEdit 11-Jul-14 reeset   Fixed
merge records tool — customizing match points causes merges to not occur (3) Major Bug MarcEdit 19-Jun-14 reeset   Fixed
Koha Integration – unable to authenticate when using self-signed certificates (3) Major Enhancement MarcEdit 19-Jun-14 reeset   Fixed
Removing duplicate subject headings based on indicators (4) Trivial Enhancement MarcEdit 17-Jun-14 reeset   Fixed
Prompt for confirmation of task deletion (4) Trivial Enhancement MarcEdit 17-Jun-14 reeset   Fixed
Mac Fonts (3) Major Bug MarcEdit 17-Jun-14 reeset   Fixed
Ability to import RIS format (3) Major Enhancement MarcEdit 17-Jun-14 reeset   Fixed
Task List: Syncing lists between multiple computers (3) Major Enhancement MarcEdit 14-Jun-14 reeset   Fixed
Merged Records: Error thrown when dealing with records with more than 10 duplicate control numbers (1) Critical Bug MarcEdit 15-Jul-14 reeset   Fixed
Edit Subfield and Swap Field Edits (3) Major Enhancement MarcEdit 14-Jun-14 reeset   Fixed
Find All — Usability Error; won’t jump to first record. (3) Major Bug MarcEdit 11-Jul-14 reeset   Fixed
Validate Temp File Path Before Generating files (3) Minor Bug MarcEdit 20-Jun-14 reeset   Fixed
Generate Control Number — invalid lines (3) Major Bug MarcEdit 10-Jul-14 reeset   Fixed
Plugin Updates — make it easier to track plugin updates (3) Minor Enhancement MarcEdit 20-Jun-14 reeset   Fixed
Task List: Allowing Description to show in Task Manager and Task Editor (3) Minor Enhancement MarcEdit 20-Jun-14 reeset   Fixed
Automated update of plugins (3) Major Bug MarcEdit 20-Jun-14 reeset   Fixed
Add Export to Excel on the OCLC Plugin (3) Major Enhancement MarcEdit 17-Jul-14 reeset   Fixed
Edit field status reporting fields edited not actual modifications (3) Major Bug MarcEdit 17-Jul-14 reeset   Fixed
Ability to Undo a Task (2) Major Enhancement MarcEdit 17-Jul-14 reeset   Fixed
Initial case bug (3) Major Bug MarcEdit 17-Jul-14 reeset   Fixed
 Posted by at 11:27 pm
Jul 222014
 

What is MARCCompare/RobertCompare?

Very rarely do I create programs for individuals to meet very specific user needs.  I’ve always taken the approach with MarcEdit that tools should be generalizable, and not tied to a specific individual or project.  RobertCompare was different.  The tool was created to support Mr.Robert (Bob) Ellett’s (ALA Tribute, Link to Dissertation Record in WorldCat) research for his Ph.D. dissertation, and only after completion, was the tool generalized for wider use. 

When I moved MarcEdit from the 4.x to the 5.x codebase, I dropped this utility because it had seemed to have run its course.  This was something Bob would periodically give me a hard time about — I think that he liked the idea of RobertCompare kicking around.  Of course, the program was terribly complicated, and without folks asking for it, converting the code from assembly to C# just wasn’t a high priority. 

Well, that changed last year when Bob suddenly passed away.  I liked Bob a lot — he was immeasurably kind and easy to get along with.  After his passing, I decided I wanted to bring RobertCompare back…I wanted to do something to remember my friend.  It’s taken a lot more time than I’d hoped, in part due to a move, a job change, and the complexity of the code.  However, after an extended absence, RobertCompare is being reintroduced back into MarcEdiit with MarcEdit 6.0. 

MARCCompare/RobertCompare 2.0

The original version of RobertCompare was designed to answer a very specific set of questions.  The program didn’t just look for differences between records, but rather, utilizing a probability engine, made determinations regarding the types of changes that had been made in the records.  Bob’s research centered around the use of PCC records at non-PCC libraries, and he was particularly interested in the types of changes these libraries were making to the records when downloading them for use.  The original version of RobertCompare was very good at analyzing record sets and generating a change history based on the current state of the records.  But the program was incredibly complicated and slow…really, really slow. 

In order to make this tool more multi-use, I’ve removed much of the code centered around probability matrix, and instead created a tool that utilizes a differential equation to generate an output file that graphically represents the changes between MARC files.  The output of the file is in HTML and at this point, pretty simple – but has been created in a way that I should be able to add additional functionality if this tool proves to have utility within the community.

So what does it look like?  The program is pretty straightforward.  There is a home menu where in identify the two files that you want to compare, and then a place to designate a save file path. 

image
Figure 1: MARCCompare/RobertCompare main window

The program can take MARC files and mnemonic files and compare them to determine what changes have been made between each record.  At this point, the files to be compared need to be in the same order.  This has been done primarily for performance reasons, as it allows the program to very quickly chew through very large files which was what I was looking for as part of this re-release. 

As noted above, the output of the files has changed.  Rather than breaking down changes into categories in an attempt to determine if changes were updated fields, new fields or deleted field data – the program now just notes additions/changes and deletions to the record and outputs this as an HTML record.  Figure 2 shows a sample of what the report might look like (format is slightly fluid and still changing). 

image 
Figure 2: MARCCompare/RobertCompare output

Final thoughts

While I’m not sure that RobertCompare was ever widely used by the MarcEdit community, I do know that it had its champions.  Over the past year, I’ve heard from a handful of users asking about the tool, and letting me know that they still have MarcEdit 4.0 on their systems specifically to utilize this program.  Hopefully by adding this tool back into MarcEdit, they will finally be able to send MarcEdit 4.x into retirement and jump to the current version of the application.  For me personally, working on this tool again was a chance to remember a very good man, and finish something that I know probably would have given him a good laugh.  

–TR

Jun 212014
 

One of the changes that will be present in MarcEdit 6 is an expansion of the Material Types Report.  Currently, the program provides an option to allow users to generate a report detailing the type of record as designated by the LDR and 008.  This allows users to get a quick snapshot of how the records are coded within the file.  One of the limitations of this process has been that these reports are read only.  Once a user ran a report, if they wanted to see, for example, all the Computer Files – they would need to construct that query themselves. 

In MarcEdit 6, this will change.  The program has reorganized the Materials Report menu, creating a new entry for Generating the Report, an a new set of subcommands that allow users to designate a material type and find the records that correspond to that type. 

image
Figure 1: Finding the Material Types Report

image
Figure 2: Material Type Report Results.  One new enhancement has been the ability to copy data to the clipboard from this window.  This impacts all functions that utilize this generic results window.

image
Figure 3: Menu to Find Records of a specific type

image
Figure 4: Material Results query and results

This is a request that comes up periodically – hopefully, these changes will make querying and examining record types easier for users.

–tr

 Posted by at 10:00 pm
Jun 212014
 

Thought I’d give the folks that are interested an idea of where I am with the MarcEdit update.  In terms of new features, that is pretty much set at this point.  I have one feature request regarding conditional field editing by field position that I’m still mulling over, but otherwise, I’m pretty much finished.  The last things todo right now is to finish the installers for Linux and Mac systems, and then make sure that the installs work exactly as I want them to.  One thing that I have found with the Linux and Mac systems – the Z39.50 dependencies are one of the biggest dependency hurdles (I can’t really automate their install well) – so I’m working on a method of creating a Z39.50 web service at marcedit.reeset.net that I can then call via non-Windows systems.  This will be functionally equivalent with the locally loaded Z39.50 libraries, though there will likely be a performance hit (small one, you shouldn’t notice too much) – but will remove a major dependency making the program much easier to install.  Also, I’m just not sure how often the Z39.50 client actually gets used at this point (when can we finally phase that out) – so I’m hoping this will be a good enough compromise. 

MarcEdit on Linux:

At this point, I’m testing MarcEdit on Suse and it looks to be going well.  The installer is now a single .sh file, that will expand creating the directory, run the bootloader, and create a shortcut on the desktop.  What it doesn’t do yet is make sure Mono is installed – not certain if that will be part of the first installer, but this should get you to the point that if you have mono installed, you will just run the installer and an icon will be on the desktop ready to be activated. 

Couple of the current screenshots:

image
Figure 1: Start Screen on Linux

image
Figure 2: MarcEditor

image
Figure 3: Delimited Text Translator

image
Figure 4: RDA Helper

image

Figure 5: MARC Tools Window

At this point, on Linux, the version looks and feels pretty much like you’d expect the Windows version to work.  I’m testing performance, especially with larger files in the Editor (for rendering) and it looks like all UI issues should be dealt with in MarcEdit 6.  I’m starting the process of doing the same type of testing using the current version of OS X with Mono 3.4 – hopefully I’ll see the same UI smoothness.

Some Stats on the Update

Some stats – refactoring, I’ve removed close to 15,000 lines of code streamlining older processes (these are things you likely won’t see, outside of maybe a few things being faster).  New code – I’ve written close to 27,000 lines of code developing new features or enhancements for this update.  Much of this has been around multiple platform support, new functions in the editing tools, etc.  Once MARCCompare has been completed, that number will jump to around 37,000 lines – so this will be a significant update in which a number of tools and functions have had significant revisions made too improve code maintenance, performance, add new features or correct bugs, or all three. 

Below you will find a list of topics the represent some of the areas where changes are being made.  This is a subset of a report from the internal issue tracker I use for tracking issues/enhancement requests.  At this point, it still looks like this update will be scheduled for Mid-July.

–tr

 

Tracking Report

Summary

Category

Sharing Tasks which refer to other tasks

Bug

Add/Delete Field — Find What Field

Bug

Refresh the Delimited Text Translator

Enhancement

MARC Tools window refresh

Performance

Changing Capital Letters with Diacritics to lower case

Bug

RDA Helper

Bug

Script Wizard — add/delete field with modifier

Bug

Adding multiple fields in the Add/Delete Function

Enhancement

UI Issue — Validation

Bug

Task List — Swap Field not saving Process one per field

Bug

Allow tasks to have comments

Enhancement

Edit Subfield and Swap Field Edits

Enhancement

Task Manager & Task List tool

Enhancement

Task List: Allowing Description to show in Task Manager and Task Editor

Enhancement

Plugin Updates — make it easier to track plugin updates

Enhancement

Ability to import RIS format

Enhancement

Mac Fonts

Bug

Prompt for confirmation of task deletion

Enhancement

Koha Integration – unable to authenticate when using self-signed certificates

Enhancement

merge records tool — customizing match points causes merges to not occur

Bug

Validate Temp File Path Before Generating files

Bug

 Posted by at 8:53 am
Jun 122014
 

This summer, I have been spending a lot of my free development preparing MarcEdit to make the transition from 5.9 to 6.0.  This will be a big update, and at the same time, and update that I’m not sure folks will necessarily see many of the changes.  But maybe that will be a good thing, I don’t know.  So, a couple of notes about the update.

User Interface Changes

So, MarcEdit has been around for a very long time – 1999 to be exact.  Because of that, there are some very old tools in the application…tools who’s interface haven’t been updated in 6-7 years – even if the underlying functionality has changed.  Well, I’m taking the time to refresh a number of the forms found within the application.  The places where most users will see these changes are in the Delimited Text Translator (the Windows 2000 Wizard interface is going away) and the MARC Tools interface.  There are a few others that will see updates; but these are the two that you’ll see for sure.

Delimited Text Translator (screen 1)

image

Delimited Text Translator (screen 2)

image

MARC Tools Revisions (proposed)

image

These are proposed changes – but are being done to make the look more consistent as well as make some screens easier to work with (the MARC Tools window for instance was getting really long).

Bug fixes

For two months, I’ve been going through my internal ticketing system, and closing every open issue.  When I post this update, every bug report that has been saved, will be closed.  This has required rewriting a lot of code within the application – which for many older functions – is definitely a good thing.  At present – this means that 13 bug tickets have been closed.  All are minor, with general workarounds – but this should have a practical impact for folks.

New features

At present, 7 enhancements/new features have been added to the program.  The remaining change to be completed (if it’s completed) will be the re-inclusion of MARCCompare (or RobertCompare as I fondly referred to it).  This is a specialized diff tool for comparing MARC records that is smarter than just a plain diff because it takes into account MARC structured data.

Cross Platform Work

One of the pseudo supported options in MarcEdit has been the ability to run the tool on non-windows platforms.  Being written in C#, the program runs fine under MONO on both Linux and Mac systems.  However, this has traditionally worked much better for Linux users because: MONO was created by a bunch of people on Linux, and I work a lot on Linux so I would test there.  One of the big efforts this time around has been to try and identify where UI issues or other issues exist and put solid support for folks interested in working with MarcEdit within non-Windows environments.  The tool will likely always run best there – but I’d like to provide a version with installers, for both Linux and Mac systems and 6.0 will represent the first attempt at that.  I’ll admit, I have some trepidation around doing this and providing more formalized support because I don’t own a Mac so I won’t get to test and do the kind of dedicated development as I might like – but I have regular enough access to one that I’m willing to give it a go.  At the very least – users will get a much easier process for installing and that at least is a win.

Go live date

This is still in the air a little bit.  I’d hoped for early July but my wife and I are buying a house, I’m doing a little travelling for work – so this is definitely pushing things back.  But ideally, these changes will start being rolled out in July 2014.

If you have questions – feel free to give me a holler.

-tr

 Posted by at 1:15 am
May 142014
 

Since last December, I’ve had the opportunity to spend a good deal of time working with the OCLC WorldCat Metadata API.  The focus was primarily around kicking the tires, and then eventually developing some integration components with MarcEdit, as well as a C# library (https://github.com/reeset/oclc_api) for those that may have use of such things.

However, most folks in libraries tend to not use C# – they tend to focus on lighter-weight languages like ruby, python, or PHP.  Well, I work in ruby as well, so I decided to take the C# library and port it to Ruby.  The resulting code can be found here: https://github.com/reeset/wc_metadata_api

It’s pretty straightforward – and provides wrappers for all the current functionality found in the API, with the caveat being that the defined transfer format between OCLC and the client is in MARCXML, and response formats are in application/atom+xml.  The API supports a handful of other formats, but I’ve standardized on MARCXML since it’s well understood and there are lots of tools that can work with it.

The code isn’t well commented at this point.  I’ll take some time over the next few days to add some commenting, improve exception handling, and possibly add a few helper functions to make message processing easier – but this should help anyone interested in knowing how the API works get a better understanding.

–tr

Code Example:

** Get Local Bib Record

require ‘rubygems’
require ‘wc_metadata_api’

key = ‘[your key]‘
secret = ‘[your secret]‘
principalid = ‘[your principal_id]‘
principaldns = ‘[your principal_idns]‘
schema = ‘LibraryOfCongress’
holdingLibraryCode=’[your holding code]‘
instSymbol = ‘[your oclc symbol]‘

client = WC_METADATA_API::Client.new(:wskey => key, :secret => secret, :principalID => principalid, :principalDNS => principaldns, :debug =>false)

response = client.WorldCatReadLocalBibRecord(:oclcNumber =>’338544583′, :schema => schema, :holdingLibraryCode => holdingLibraryCode, :instSymbol => instSymbol)

puts response

 Posted by at 10:31 pm
Apr 042014
 

One of the new functions in MarcEdit is the inclusion of a Field Edit Data function.  For the most part, batch edits on field data has been handled primarily via regular expressions using the traditional replace function.  For example, if I had the following field:

=999  \\$zdata1$ydata2

And I wanted to swap the subfield order, I’d use a regular expression in the Replace function and construct the following:
Find: (=999.{4})(\$z.*[^$])(\$y.*)
Replace: $1$3$2

This works well – when needing to do advanced edits.  The problem was that any field edits that didn’t fit into the Edit Subfield tool needed to be done as a regular expression.  In an effort to simplify this process, I’ve introduced an Edit Field Data tool.

image

This tool exposes the data, following the indicators, for edit.  So, in the following field:
=999  \\$aTest Data

The Edit Field Data tool could interact with the data: “$aTest Data”.  Ideally, this will potentially simplify the process of doing most field edits.  However, it also opens up the opportunity to do recursive group and replacements.

When harvesting data, often times subjects may be concatenated with a delimiter, so for example, a single 650 field may represent multiple subjects, separated by a delimiter of a semicolon.  The new function will allow users to capture data, and recursively create new fields from the groups.  So for example, if I had the following data:
=650  \7$adata – data; data – data; data – data;

And I wanted the output to look like:
=650  \7$adata – data
=650  \7$adata – data
=650  \7$adata – data

I could now use this function to achieve that result.  Using a simple regular expression, I can create a recursively matching group, and then generate new fields using the “/r” parameter.  So, to do this, I would use the following arguments:
Field: 650
Find: (data – data[; ]?)+
Replace: $$a$+/r
Check Use Regular Expressions.

 image

The important part of the above expression is in the Replacement syntax.  To tell MarcEdit that the recursion should result in a new line, the mnemonic /r is used at the end of the string to tell the tool that the recursion should result in a new line.

This new function will be available for use as of 4/7/2014. 

–tr

 Posted by at 8:44 pm
Mar 192014
 

After posting the update last night, a user noted that the Delimited Text Translators 008 generation utilized some older defaults that were causing some validation issues.  So, I’ve posted a limited update to correct this specific problem.

If you have automatic updating turned on, MarcEdit will prompt you automatically.  Otherwise, you can download from:

–tr

 Posted by at 9:38 pm