reeset

Jul 112016
 

The past 3 weeks, I’ve been doing a lot of work on MarcEdit.  These initial changes impact just the windows and linux version of MarcEdit.  I’ll be taking some time tomorrow and Wed. to update the Mac version.  The current changes are as follows:

6.2.190
* Enhancement: Language files have been updated
* Enhancement: Command-line tool: -task option added to support tasks being run via the command-line.
* Enhancement: Command-line tool: -clean and -validate options updated to support structure validation.
* Enhancement: Alma integration: Updating version numbers and cleaned up some windowing in the initial release.
* Enhancement: Small update to the validation rules file.
* Enhancement: Update to the linked data rules file around music headings processing.
* Enhancement: Linked Data Platform: collections information has been moved into the configuration file.  This will allow local indexes to be added so long as they support a json return.
* Enhancement: Merge Records — 001 matching now looks at the 035 and included oclc numbers by default.
* Enhancement: MarcEngine: Updated the engine to accommodate invalid data in the ldr.
* Enhancement: MARC SQL Explorer — added an option to allow mysql database to be created as UTF8.
* Enhancement: Handful of odd UI changes.

You can get the update from the downloads page (http://marcedit.reeset.net/downloads) or via the automated update tools.

–tr

 Posted by at 8:39 pm
Jul 112016
 

MarcEdit’s command-line function has always had the ability to run validation tasks against the MarcEdit rules file.  However, the program hasn’t included access to the cleaning functions of the validator.  As of the last update, this has changed.  If the –validate command is invoked without a rules file defined, the program will validate the structure of the data.  If the –clean option is passed, the program will remove invalid structural data from the file. 

Here’s an example of the command:

>> cmarcedit.exe -s “C:\Users\rees\Desktop\CLA_UCB 2016\Data File\sample data\bad_sample_records.mrc” –validate

–tr

 Posted by at 8:19 pm
Jul 112016
 

MarcEdit’s task list functionality has made doing repetitive tasks in MarcEdit a fairly simple process.  But one limitation has always been that the tasks must be run from within MarcEdit.  Well, that limitation has been lifted.  As of the last update, a new option has been added to the command-line tool: –task.  When run with a path to a task to run, MarcEdit will preform the task from the command-line. 

Here’s an example of a command:

cmarcedit.exe -s “C:\Users\rees\Desktop\withcallnumbers.mrk” -d “C:\Users\rees\Desktop\remote_task.mrk” -task “C:\Users\rees\AppData\Roaming\marcedit\macros\tasksfile-2016_06_17_190213223.txt”

This functionality is only available in the Windows and Linux version of the application.

–tr

 Posted by at 8:15 pm
Jul 072016
 

Over the past year and half, I’ve been working with MarcEdit to provide the ability for users interested in doing URI entification against various known vocabularies, as well as adding these vocabularies into an authority processing workflow.  The processes work well for the handful (15-20) defined indexes that MarcEdit currently queries.  But long-term, libraries may see a proliferation of indexing services, and the ability to add custom linking services into MarcEdit will become more important. 

So, with the next version of MarcEdit – I’ll be making a change.  While MarcEdit will still have custom code that handles some of the larger national indexes (because this makes sense) – all the information about the collections, their internal labels, and JSON object paths (for values) will be moved into the rules file.  The upside to this is that users will be able to customize the queries MarcEdit is making against a linked data service (if they want) – but more important, they can add their own linked data services.  Initially, custom services will need to return well-formed JSONLD, where the value for the specified URI can be referenced as a JSON object path (i.e., this the path used with the AAT index: results.bindings[0].subj.value), but if it matches that criteria, you’ll be able to add your own services into MarcEdit, and if MarcEdit encounters your defined index label within your metadata, it will ping the service. 

Within the rules file, you’ll see this new structure represented as:

<collection>
      <name>US Library of Congress Subject Heading</name>
      <label>lcsh</label>
      <uri>http://id.loc.gov/authorities/subjects/label/{search_terms}</uri>
      <path />
    </collection>

Since many of the indexes (national indexes) MarcEdit queries now do not use JSON in their return (or MarcEdit utilizes a special, optimized process) – these will not have paths.  But for other sources like MESH headings, Getty headings, you’ll be able to see how this structure is being utilized.

I’m still testing these changes – as the changes to the linked data framework affect a number of parts of the application – but these will be part of the next update.

–tr

 Posted by at 11:17 am
Jun 152016
 

Last night, I posted an update squashing a couple bugs and adding some new features.  Here’s the change log:

* Bug Fix: Merge Records Tool: If the user defined field is a title, the merge doesn’t process correctly.
* Bug Fix: Z39.50 Batch Processing: If the source server provides data in UTF8, characters from multi-byte languages may be flattened.
* Bug Fix: ILS Integration..Local:  In the previous version, one of the libraries versions didn’t get updated and early beta testers had some trouble.
* Enhancement: Join Records — option added to process subdirectories.
* Enhancement: Batch Processing Tool — option added to process subdirectories
* Enhancement: Extract Selected Records — Allowing regular expressions as an option when processing file data.
* Enhancement: Alma Integration UI Improvements

Downloads can be picked up via the automated updating tool or via the downloads (http://marcedit.reeset.net/downloads) page.

 

–tr

 Posted by at 2:36 pm
Jun 052016
 

This update has been a little while coming and represents a significant number of updates, bug fixes and enhancements.  On the Mac side, the two largest updates where the implementation of the Delimited Text Translator and the OAI Harvester, on all platforms – Alma Integration.  You can find notes about the Alma integration here: http://blog.reeset.net/archives/1950

Please see the full list of updates below.  Downloads can be picked up through the automatic update mechanism in MarcEdit or via the downloads page at: http://marcedit.reeset.net/downloads

–tr

MarcEdit Windows/Linux Updates:

* Bug Fix: ILS Integration: Local Integration — corrected display rendering and search for keyword
* Bug Fix: Add/Delete Records — Corrected problem when using the Add field only if not a duplicate option
* Enhancement: Validate Headings — added dynamic caching
* Enhancement: Build Links — added dynamic caching
* Enhancement: ILS Integration — First version of Alma integration
* Bug Fix: Math conversion — Degree/minute/seconds to Degrees correction
* Settings Change: Updated the RDA Field conversion to limit abbreviation checking in the 245 field to the 245$c
* Enhancement: RDA Abbreviations — new abbreviations added
* Enhancement: Select/Delete MARC Records — added option to expose specialized search options like Field # searching, Range Searching, File Searching and Size searching.
* Bug Fix: OAI Harvester — Debug URL wasn’t correct when adding date values.
* Bug Fix: RDA Helper — Added Data validation to ensure that invalid 008 data doesn’t cause a data crash.
* Enhancement: Delimited Text Translator — Added more preview options
* Enhancement: Delimited Text Translator — Added Holdings LDR/008 values
* Enhancement: UI Improvements — a large number of textboxes that accept file paths now support drag and drop.

MarcEdit Mac Updates:

* Bug Fix: ILS Integration: Local Integration — corrected display rendering and search for keyword
* Bug Fix: Add/Delete Records — Corrected problem when using the Add field only if not a duplicate option
* Enhancement: Validate Headings — added dynamic caching
* Enhancement: Build Links — added dynamic caching
* Enhancement: ILS Integration — First version of Alma integration
* Bug Fix: Math conversion — Degree/minute/seconds to Degrees correction
* Settings Change: Updated the RDA Field conversion to limit abbreviation checking in the 245 field to the 245$c
* Enhancement: RDA Abbreviations — new abbreviations added
* Enhancement: Select/Delete MARC Records — added option to expose specialized search options like Field # searching, Range Searching, File Searching and Size searching.
* Bug Fix: RDA Helper — Added Data validation to ensure that invalid 008 data doesn’t cause a data crash.
* Enhancement: UI Improvements — a large number of textboxes that accept file paths now support drag and drop.
* Enhancement: OAI Harvester Implemented
* Enhancement: Delimited Text Translator implemented

 Posted by at 10:27 pm
Jun 052016
 

Over the past month, I’ve been working with ExLibris (thank you to Ori Miller at ExLibris) and Boston College (thanks to Margaret Wolfe) to provide direct integration between MarcEdit and Alma via the Alma Apis.  Presently, the integration allows users to search, create, and update records.  Setup is pretty easy (I think) and once you have your API access setup correctly – you should be off and running.  But, it will be interesting to see if that’s the case as more people play around with this in their sandboxes.

Setting up integration

MarcEdit Alma integration requires that you configure an API key with Alma that supports the bib api and the user api.  The bib api represents the endpoints where the record editing and retrieval happen, while the user api is used to provide a thin layer of authentication before MarcEdit attempts to run an operation (since Alma doesn’t have it’s own authentication process separate from having a key). 

I’d recommend testing this first in your Sandbox.  To do this, you’ll need to know your sandbox domain, and be able to configure the API accordingly.  If you don’t know how to do this, you’ll want to contact ExLibris. 

Once you have your API key, open MarcEdit’s main window and click the Preferences icon.

image

This will open the Preference’s window.  Select the ILS Integration Link, and then check the Enable ILS Integration Checkbox, select Alma from the listbox and then enter the domain for your sandbox.  Alma’s API doesn’t require a username, so leave that blank, but enter your API key into the Password Textbox.  Finally, you’ll need to have setup a Z39.50 connection to your instance.  This is how MarcEdit searches Alma for record retrieval.  If you haven’t setup a Z39.50 Connection, you can do that here, or you can open the Z39.50 Client, Select Modify Databases, Add a new Z39.50 Server, and enter the information for your Alma Instance.  Here’s an example configuration (minus the username and password) for Boston College’s Sandbox:

image

With your Z39.50 Server configured and selected – the ILS Integration Preference’s window will look something like this:

image

Save these settings.  Now, when you open the MarcEditor, you’ll see a new menu item:

image

This menu item will allow you to search and update/create records.  To find items, click on the menu and select Search.  You’ll get the following window:

image

If I run a search for Boston, I’ll retrieve 5 results based on the limit set in the Limit textbox:

image

You can either download all the items by clicking the Download All Items, or you can select the items individually that you want to download, and right click on the Results.  This will give you a menu allowing you to download the records. 

When downloaded, the record will be opened into MarcEdit like the below:

image

Couple notes about the download.  If the download includes an 852 (and they can) – you’ll want to delete that field, otherwise the field will get duplicated.  Right now, I’m trying to figure out if MarcEdit should just remove the value, or if there is an applicable use case for keeping it. 

Download the record, make the edits that you want to make to the record, and then click the Update/Create option from the Alma window.

image

When you click the Update/Create – the tool will upload your data to your Alma server.  If there is an error, you’ll receive the returned error message.  If the process was successful, you’ll get an message telling you that the data had been processed. If you are interesting in seeing the resulting XML output – MarcEdit automatically copies the data to the clipboard. 

Couple of notes about the process – in my testing, I found that updating Serials records was spotty.  I’m thinking this might have something to do with permissions – but I’m not positive about that.  I’m hoping to do a bit more investigation – but I wanted to get this out for folks to start playing with it and maybe providing some feedback.

Secondly, there is a holdings API – it would be possible to allow users to modify holdings data via MarcEdit, but I’d need use-cases in order to see how it fits into this process.

I’m sure this will be a process that I’ll be refining over the next few weeks – but in the mean time, I’d welcome any and all comments. 

–tr

* I’ll be posting a short youtube video and will update the url here.

MarcEdit Update

 MarcEdit  Comments Off on MarcEdit Update
May 232016
 

Yesterday, I posted a significant update to the Windows/Linux builds and a maintenance update to the Mac build that includes a lot of prep work to get it ready to roll in a number of changes that I’ll hopefully complete this week.  Unfortunately, I’ve been doing a lot of travelling, which means that my access to my mac setup has been pretty limited and I didn’t want to take another week getting everything synched together. 

So what are the specific changes:

ILS Integrations
I’ve been spending a lot of time over the past three works head down working on ILS integrations.  Right now, I’m managing two ILS integration scenarios – one is with Alma and their API.  I’m probably 80% finished with that work.  Right now, all the code is written, I’m just not getting back expected responses from their bibliographic update API.  Once I sort out that issue – I’ll be integrating this change into MarcEdit and will provide a youtube video demonstrating the functionality. 

The other ILS integration that I’ve been accommodating is working with MarcEdit’s MARC SQL Explorer and the internal database structure.  This work builds on some work being done with the Validate Headings tool to close the authority control loop.  I’ll likely be posting more about that later this week as I’m currently have a couple libraries test this functionality to make sure I’ve not missed anything.  Once they give me the thumbs up, this will make its way into the MarcEditor as well. 

But as part of this work, I needed to create a way for users to edit and search the local database structure in a more friendly way.  So, leveraging the ILS platform, I’ve included the ability for users to work with the local database format directly within the MarcEditor.  You can see how this works here (https://www.youtube.com/watch?v=dMJ_pUxyoFc&feature=youtu.be): Integrating the MarcEditor with a local SQL store.  I’m not sure what the ideal use case is for this functionality – but over the past couple of weeks, it had been requested by a couple of power users currently using the MARC SQL Explorer for some data edits, but hoping for an easier to user interface.  This work will be integrated into the Mac MarcEdit version at the end of this week.  All the prep work (window/control development) has been completed.  At this point, its just migrating the code so that it works within the Mac’s object-C codebase.

Edit Shortcuts
I created two new edit shortcuts in the MarcEditor.  The first, Find Records With Duplicate Tags, was created to help users look for records that may have multiple tags or a tag/subfield combination with a set of records.  This is work that can be done in the Extract Selected Records tool, but it requires a bit a trickery and knowledge of how MarcEdit formats data. 

image

How does this work – say you wanted to know which records had multiple call numbers (050) fields in a record.  You would select this option, enter 050 in the prompt, and then the tool would create for you a jump list showing all the records that met your criteria. 

Convert To Decimal Degrees
The second Edit ShortCut function is the first Math function (I’ll be adding two more, specifically around finding records with dates greater than or less than a specific value) targeting the conversion of Degree/Minutes/Seconds to decimal degrees.  The process has been created to be MARC agnostic, so users can specify the field, and subfields to process.  To run this function, select it from the Edit Shortcuts as demonstrated in the screenshot below:

image

When selected, you will get the following prompt:

image

This documents the format for defining the field/subfields to be processed.  Please note, it is important to define the all four potential values for conversion – even if they are not used within the record set. 

Using this function, you can now convert a value like:
=034  1\$aa$b1450000$dW1250000$eW1163500$fN0461500$gN0420000
To:
=034  1\$aa$b1450000$d+125.0000$e+116.5800$f+046.2500$g+042.0000

This function should allow users to transition their cartographic data to a format that is much more friendly to geographic interpretation if desired.

Bug Fixes:
This update also addressed a bug in the Build New field parser.  If you have multiple arguments, side-by-side, within the same field grouping (i.e., {100$a}{100$b}{100$c} – the parser can become confused.  This has been corrected.

Updates:
Included and update to the linked data rules file, updating the 7xx fields to include the $t in the processing.  Also updated the UNIMARC translation to include a 1:1 translation for 9xx data.

Over the next week, I hope to complete the Alma integration, but will focusing the development work in my free time on getting the Mac version synched with these changes.

–tr

 Posted by at 6:39 pm

MarcEdit and Windows XP

 MarcEdit  Comments Off on MarcEdit and Windows XP
May 052016
 

I’ve been supporting XP now for close to 15 years in MarcEdit, and I’m finding the number of areas in the code where I have to work around XP limitations is continually growing. The tipping point for me occurred about a month ago, when I had to write a new URI parser because the version found in current version of .NET and the one found on XP are worlds different and what XP provides wasn’t robust enough and has a number of problematic bugs.

So, you can probably guess where I’m going with this. I’m starting to think about plans for essentially dropping XP support and freezing a version of MarcEdit (that wouldn’t be updated) for those libraries still using XP. Ideally, I’d like to not provide a frozen version at all because this version will become out of date very quickly – but I’m also unsure of how many users still run XP and how long XP will continue to kick around within the library environment. I haven’t picked a date yet, but I definitely want to have this conversation. Does XP support continue to be important to this community, and more importantly, if we look out say 1-1.5 years, will that still be true.

One last thing; I plan on doing a little bit of log analysis to understand more about the current MarcEdit XP user community. If this community is largely international, I may just suck it up and continue finding a way to make it work. I want to be sensitive to the fact I work in an academic bubble, and I know that many libraries have to struggle simply to be open for their patrons. For anyone in that position, XP probably works good enough. But I think that it’s time to start asking this question and evaluating what the tipping points might be within the MarcEdit community around XP and it’s continued use. 

At some point, XP support will need to end.  It’s just so long in the tooth, that continuing to support it will eventually limit some of the work I might to do with MarcEdit.  The question at this point is when that might happen…1 year from now, 2 years?   I just don’t know.

Thanks,

–tr

 Posted by at 8:07 am
May 012016
 

This weekend, I posted a new MarcEdit update.  This is one of the biggest changes that I’ve made in a while.  While the actual changelog is brief – these changes represented ~17k lines of code Windows (~10K not related to UI work) and ~15.5k lines of code on the OSX side (~9K not related to UI work).

Specific changes added to MarcEdit:

Windows/Linux:

  • Enhancement: UNIMARC Tools: Provides a lite-weight tool to convert data to MARC21 from UNIMARC and to UNIMARC from MARC21.
  • Enhancement: Replace Function: Option to support External search/replace criteria.
  • Enhancement: MARCEngine COM Object Updates

MacOSX

  • Enhancement: UNIMARC Tools: Provides a lite-weight tool to convert data to MARC21 from UNIMARC and to UNIMARC from MARC21.
  • Enhancement: Replace Function: Option to support External search/replace criteria.
  • Update: Installation has been changed to better support keeping configuration information sync’d between updates.
  • Bug Fix: Add/Delete Function — Add field if not a duplicate:  Option wasn’t always working.  This has been corrected.

I’m created some videos to demonstrate how these two elements work, and then a third video showing how to use the Add Field if not a duplicate (added in the previous update).  You can find these videos here:

Add Field If Not a Duplicate
URL: https://youtu.be/ObRKgAD9ye8

MarcEdit’s UNIMARC Tools:
URL: https://youtu.be/4rdzOCAwhSU

MarcEdit: Batch Replacement using External Criteria
URL: https://youtu.be/uJB9Uqg6bJs

You can get the changes from the downloads page or through MarcEdit’s automated update tool.

–tr

 Posted by at 8:10 pm