May 232016
 

Yesterday, I posted a significant update to the Windows/Linux builds and a maintenance update to the Mac build that includes a lot of prep work to get it ready to roll in a number of changes that I’ll hopefully complete this week.  Unfortunately, I’ve been doing a lot of travelling, which means that my access to my mac setup has been pretty limited and I didn’t want to take another week getting everything synched together. 

So what are the specific changes:

ILS Integrations
I’ve been spending a lot of time over the past three works head down working on ILS integrations.  Right now, I’m managing two ILS integration scenarios – one is with Alma and their API.  I’m probably 80% finished with that work.  Right now, all the code is written, I’m just not getting back expected responses from their bibliographic update API.  Once I sort out that issue – I’ll be integrating this change into MarcEdit and will provide a youtube video demonstrating the functionality. 

The other ILS integration that I’ve been accommodating is working with MarcEdit’s MARC SQL Explorer and the internal database structure.  This work builds on some work being done with the Validate Headings tool to close the authority control loop.  I’ll likely be posting more about that later this week as I’m currently have a couple libraries test this functionality to make sure I’ve not missed anything.  Once they give me the thumbs up, this will make its way into the MarcEditor as well. 

But as part of this work, I needed to create a way for users to edit and search the local database structure in a more friendly way.  So, leveraging the ILS platform, I’ve included the ability for users to work with the local database format directly within the MarcEditor.  You can see how this works here (https://www.youtube.com/watch?v=dMJ_pUxyoFc&feature=youtu.be): Integrating the MarcEditor with a local SQL store.  I’m not sure what the ideal use case is for this functionality – but over the past couple of weeks, it had been requested by a couple of power users currently using the MARC SQL Explorer for some data edits, but hoping for an easier to user interface.  This work will be integrated into the Mac MarcEdit version at the end of this week.  All the prep work (window/control development) has been completed.  At this point, its just migrating the code so that it works within the Mac’s object-C codebase.

Edit Shortcuts
I created two new edit shortcuts in the MarcEditor.  The first, Find Records With Duplicate Tags, was created to help users look for records that may have multiple tags or a tag/subfield combination with a set of records.  This is work that can be done in the Extract Selected Records tool, but it requires a bit a trickery and knowledge of how MarcEdit formats data. 

image

How does this work – say you wanted to know which records had multiple call numbers (050) fields in a record.  You would select this option, enter 050 in the prompt, and then the tool would create for you a jump list showing all the records that met your criteria. 

Convert To Decimal Degrees
The second Edit ShortCut function is the first Math function (I’ll be adding two more, specifically around finding records with dates greater than or less than a specific value) targeting the conversion of Degree/Minutes/Seconds to decimal degrees.  The process has been created to be MARC agnostic, so users can specify the field, and subfields to process.  To run this function, select it from the Edit Shortcuts as demonstrated in the screenshot below:

image

When selected, you will get the following prompt:

image

This documents the format for defining the field/subfields to be processed.  Please note, it is important to define the all four potential values for conversion – even if they are not used within the record set. 

Using this function, you can now convert a value like:
=034  1\$aa$b1450000$dW1250000$eW1163500$fN0461500$gN0420000
To:
=034  1\$aa$b1450000$d+125.0000$e+116.5800$f+046.2500$g+042.0000

This function should allow users to transition their cartographic data to a format that is much more friendly to geographic interpretation if desired.

Bug Fixes:
This update also addressed a bug in the Build New field parser.  If you have multiple arguments, side-by-side, within the same field grouping (i.e., {100$a}{100$b}{100$c} – the parser can become confused.  This has been corrected.

Updates:
Included and update to the linked data rules file, updating the 7xx fields to include the $t in the processing.  Also updated the UNIMARC translation to include a 1:1 translation for 9xx data.

Over the next week, I hope to complete the Alma integration, but will focusing the development work in my free time on getting the Mac version synched with these changes.

–tr

 Posted by at 6:39 pm
May 052016
 

I’ve been supporting XP now for close to 15 years in MarcEdit, and I’m finding the number of areas in the code where I have to work around XP limitations is continually growing. The tipping point for me occurred about a month ago, when I had to write a new URI parser because the version found in current version of .NET and the one found on XP are worlds different and what XP provides wasn’t robust enough and has a number of problematic bugs.

So, you can probably guess where I’m going with this. I’m starting to think about plans for essentially dropping XP support and freezing a version of MarcEdit (that wouldn’t be updated) for those libraries still using XP. Ideally, I’d like to not provide a frozen version at all because this version will become out of date very quickly – but I’m also unsure of how many users still run XP and how long XP will continue to kick around within the library environment. I haven’t picked a date yet, but I definitely want to have this conversation. Does XP support continue to be important to this community, and more importantly, if we look out say 1-1.5 years, will that still be true.

One last thing; I plan on doing a little bit of log analysis to understand more about the current MarcEdit XP user community. If this community is largely international, I may just suck it up and continue finding a way to make it work. I want to be sensitive to the fact I work in an academic bubble, and I know that many libraries have to struggle simply to be open for their patrons. For anyone in that position, XP probably works good enough. But I think that it’s time to start asking this question and evaluating what the tipping points might be within the MarcEdit community around XP and it’s continued use. 

At some point, XP support will need to end.  It’s just so long in the tooth, that continuing to support it will eventually limit some of the work I might to do with MarcEdit.  The question at this point is when that might happen…1 year from now, 2 years?   I just don’t know.

Thanks,

–tr

 Posted by at 8:07 am
May 012016
 

This weekend, I posted a new MarcEdit update.  This is one of the biggest changes that I’ve made in a while.  While the actual changelog is brief – these changes represented ~17k lines of code Windows (~10K not related to UI work) and ~15.5k lines of code on the OSX side (~9K not related to UI work).

Specific changes added to MarcEdit:

Windows/Linux:

  • Enhancement: UNIMARC Tools: Provides a lite-weight tool to convert data to MARC21 from UNIMARC and to UNIMARC from MARC21.
  • Enhancement: Replace Function: Option to support External search/replace criteria.
  • Enhancement: MARCEngine COM Object Updates

MacOSX

  • Enhancement: UNIMARC Tools: Provides a lite-weight tool to convert data to MARC21 from UNIMARC and to UNIMARC from MARC21.
  • Enhancement: Replace Function: Option to support External search/replace criteria.
  • Update: Installation has been changed to better support keeping configuration information sync’d between updates.
  • Bug Fix: Add/Delete Function — Add field if not a duplicate:  Option wasn’t always working.  This has been corrected.

I’m created some videos to demonstrate how these two elements work, and then a third video showing how to use the Add Field if not a duplicate (added in the previous update).  You can find these videos here:

Add Field If Not a Duplicate
URL: https://youtu.be/ObRKgAD9ye8

MarcEdit’s UNIMARC Tools:
URL: https://youtu.be/4rdzOCAwhSU

MarcEdit: Batch Replacement using External Criteria
URL: https://youtu.be/uJB9Uqg6bJs

You can get the changes from the downloads page or through MarcEdit’s automated update tool.

–tr

 Posted by at 8:10 pm
Mar 282016
 

As part of the last update, I added a new feature that is only available in the Mac Version of MarcEdit at this point.  One of the things that had been missing in the Export Tab Delimited Tool was the ability to save and load one’s export settings.  I added that as part of the most recent update.  At the same time, I though that the ability to batch process multiple files using same criteria may be useful as well.  So this has been added to the Mac interface as well.

Screen Shot 2016-03-29 at 12.14.29 AM

In the image above, you initiate the batch processing mode by checking the batch process checkbox.  This will change the marc file and save file textbox and buttons to directory paths.  You will also be prompted to select a file extension to process.

I’m not sure if this will be useful — but as I’m working through new functionality, I’ll be noting changes being made to the MarcEdit Mac version.  And this is notable, because this is the first time that the Mac version contains functionality that is not in the Windows version.

–tr

 Posted by at 9:15 pm
Mar 282016
 

I’ve posted a new update for all versions of MarcEdit this afternoon.  Last night, when I posted the new update, I introduced a bug into the RDA Helper that rendered it basically unusable.  When adding functionality to the tool to enable support for abbreviations at a subfield level, I introduced a problem that removed the subfield codes from fields where abbreviations would take place.

So what does this bug look like?  When processing data, a set field that would look like this:
=300  \\$a1 vol $b ill.

would be replaced to look like:
=300  \\a1 volume b illustrations

As one case see, the delimiter symbol “$” has been removed.  This occurred in all fields where data abbreviations were occurring.  This has been corrected with this update.

You can get the update from the downloads page: http://marcedit.reeset.net/downloads or via the automated update tool.

–tr

 Posted by at 9:05 pm
Mar 272016
 

I spent some time this week working through a few updates based on some feedback I’ve gotten over the past couple of weeks.  Most of the updates at this point are focused on the Windows/Linux builds, but the Mac build has been updated as well as all new functionality found in the linking libraries and RDA changes apply there as well.  I’ll be spending this week focusing on making Mac MarcEdit UI to continue to work towards functional parity with the Windows version.

Windows/Linux Updates:

6.2.100
* 6.2.100
** Bug Fix: Build Links Tool — when processing a FAST heading without a control number, the search would fail.  This has been corrected.
** Bug Fix: MarcEditor — when using the convenience function that allows you to open mrc files directly into the MarcEditor and saving directly back to the mrc file — when using a task, this function would be disconnected.  This has been corrected.
** Enhancement: ILS Integration — added code to enable the use of profiles.
** Enhancement: ILS Integration — added a new select option so users can select from existing Z39.50 servers.
** Enhancement: OAI Harvesting — Added a debug URL string so that users can see the URL MarcEdit will be using to query the users server.
** UI Change: OAI Harvesting — UI has been changed to have the data always expanded.
** Enhancement: MarcValidator — Rules file has been updated to include some missing fields.
** Enhancement; MarcValidator — Rules file includes a new parameter: subfield, which defines the valid subfields within a field.  If a subfield appears not in this list, it will mark the record as an error.
** Enhancement: Task Menu — Task menu items have been truncated according to Windows convention.  I’ve expanded those values so users can see approximately 45 characters of a task name.
** Cleanup: Validate Headings — did a little work on the validate headings to clean up some old code.  Finishing prep to start allowing indexes beyond LCSH based on the rules file developed for the build links tool.

 

Mac Updates:

1.4.43
**********************************************
** 1.4.43 ChangeLog
**********************************************
* Bug Fix: Build Link Tool: Generating FAST Headings would work when an identifier was in the record, but wasn’t correctly finding the data when looking.
* Enhancement: RDA Helper:  Rules file has been updated and code now exists to allow users to define subfields that are valid.
* Bug Fix: RDA Helper: Updated library to correct a processing error when handling unicode replacement of characters in the 264.
* Enhancement: RDA Helper: Users can now define fields by subfield.  I.E. =245$c and abbreviation expansion will only occur over the defined subfields.

MarcValidator Changes:

One of the significant changes in the program this time around has been a change in how the Validator works.  The Validator currently looks at data present, and determines if that data has been used correctly.  I’ve added a new field in the validator rules file called subfield (Example block):

020    R    INTERNATIONAL STANDARD BOOK NUMBER
# Uncomment these lines and add validation routines like:
#valida    [^0-9x]    Valid Characters
#validz    [^0-9x]    Valid Characters
ind1    blank    Undefined
ind2    blank    Undefined
subfield    acqz68    Valid Subfields
a    NR    International Standard Book Number
c    NR    Terms of availability
q    R    Qualifier
z    R    Canceled/invalid ISBN
6    NR    Linkage
8    R    Field link and sequence number

The new block is the subfield item – here the tool defines all the subfields that are valid for this field.  If this element is defined and a subfield shows up that isn’t defined, you will receive an error message letting you know that the record has a field with an improper subfield in it.

RDA Helper

The other big change came in the RDA Helper.  Here I added the ability for the abbreviation field to be defined at a finer granularity.  Up to this point, abbreviation definitions happened at the field or field group level.  Users can now define down to the subfield level.  For example, if the user wanted to just target the 245$c, for abbreviations but leave all other 245 subfields alone, one would just define =245$c in the abbreviation field definition file.  If you want to define multiple subfields for processing, define each as its own unit…i.e:
=245$c
=245$p

You can get the download from the MarcEdit website (http://marcedit.reeset.net/downloads) or via the MarcEdit automatic download functionality.

Questions, let me know.

–tr

 Posted by at 10:24 pm

MarcEdit Update

 MarcEdit  Comments Off on MarcEdit Update
Mar 142016
 

I posted a small MarcEdit Update for Linux/Windows users that corrects some file path issues on linux and corrects a problem introduced in doing unicode character replacement in the 260/264 process of the RDA Helper.  You can get the update from the downloads page or via the automated updates.

I also wanted to take this as a time to make a quick reminder to something because it was something that came up during my testing.  I test on a wide range of VMs when I push an update.  This doesn’t mean that I catch everything, but it means that I do endeavor to try and minimize the problems that can occur due to the Windows Installer (and there are many).  On one of my Windows 10 VMS, and update that touched the .NET framework somehow invalidated the MarcEdit install.  When this happens, you have a couple options.  The one I recommend —

1) Uninstall MarcEdit completely.  This includes going to the Program Directory and Deleting the MarcEdit program directory.  The Windows Installer does various types of black magic, and the only way to make sure that this goes away is to get rid of the directory.

2) if you cannot uninstall the program (said Windows Installer black magic has gone haywire) – there is a program called the msicleaner on the MarcEdit downloads page.  Download that, run it as an administrator – and then go to the Program Directory and delete the MarcEdit program directory.  Then reinstall.  Again, the msi installer with unstuck the Windows Installer – but removing the contents of the directory will prevent future black magic chicanery. 

Again – this showed up on 1 on the 12 or 15 VMs I test on – but since it showed up after an update, its hard to know if this is something that will affect others.  Given that – I thought this would be a good time to remind users of how to overcome issues with the Windows Installer when/if they occur.

 

–tr

 Posted by at 2:03 am
Mar 032016
 

Yesterday, I had someone indicate that there was a problem with the Add/Delete Field function.  An update in the last version to allow for deduplication deletions based on subfields tripped other deletions.  This was definitely problematic.  This has been corrected, in addition to a couple other changes.

Change log:

6.2.88

  • Bug Fix: Add/Delete Field: I introduced an element into the Delete function to allow dedup deletions to happen at the subfield level. This tripped non-dedup deletions. This has been corrected.
  • Update: Build New Links: FAST headings in the 600,611,630 weren’t being processed. I’ve updated the rules file appropriately.
  • Update: RDA Helper Abbrevs File: Add S.L. abbreviation.
  • Bug Fix: Validate Headings: The Check A only when subject checking wasn’t honoring that option. This is corrected.

Changes can be found on the downloads page: http://marcedit.reeset.net/downloads

 

tr

 Posted by at 11:01 pm

MarcEdit Update

 MarcEdit  Comments Off on MarcEdit Update
Feb 282016
 

Update was posted Feb. 27 to all versions.  Update Contains the following changes:

6.2.85

  • Enhancement: Characterset Detection: MarcEdit is including a tool that will provide a heuristical analysis of a file to provide best guess characterset detection. (http://blog.reeset.net/archives/1897)
  • Enhancement: Build New Tool Function: Adding a find macro to the function so that users can now identify specific fields when building new fields from data in a MARC record. (http://blog.reeset.net/archives/1902)
  • Update: Build Links — improved handling of MESH data ** Update: Build Links — improved handling of AAT data
  • Update: Build Links — improved handling of ULAN data
  • Update: Build Links — added work around to character escaping issues found in .NET 4.0. Issue impacts URIs with trailing periods and slashes (/). Apparently, the URI encoding tool doesn’t escape them properly because of how Windows handles file paths.
  • Update: Build Links — Rules file updated to include refined definitions for the 6xx fields.
  • Update: MarcEdit Command-Line: program updated to include new build links functional updates
  • Update: COM object: Updated character encoding switching to simplify streaming functions.
  • Update: Validate Headings: Integrated rules file into checking.
  • Bug Fix: Validate Headings: headings validation was being tripped by the URI escaping issue in .NET 4.0. This has been corrected.
  • Update: RDA Helper: Finished code refinements
  • Update: Build Links — tool is now asynchronous
  • Enhancement: Build Links — Users can now select and build their own rules files
  • Enhancement: Build Links — Tool now includes a function that will track resolution speed from linked services and attempt to provide notification when services are performing poorly. First version won’t identify particular services — just that data isn’t being processed in a timely manner.
  • Bug Fix: Character Conversion — UTF-8 to MARC-8, the {dollar} literal isn’t being converted back to a literal dollar sign. This is related to removing the fall back entity checking in the last update. This has been corrected.

Updates can be picked up through the automated update tools in MarcEdit or via the downloads page: http://marcedit.reeset.net/downloads

 

–tr

 Posted by at 6:49 am

MarcEdit: Build New Field Enhancement

 MarcEdit  Comments Off on MarcEdit: Build New Field Enhancement
Feb 232016
 

I’m wrapping up a few odds and ends prior to releasing the next MarcEdit update – mostly around the linked data work and how the tool works with specific linked data services – but one of the specific changes that should make folks using the Build New Field tool happy is the addition of a new macro that can be used to select specific data elements when building a new field. 

So, for those that might not be aware, the build new field tool is a pattern based tool that allows users to select information from various MARC fields in a record and create a new field.  You can read about the initial description at: http://blog.reeset.net/archives/1782 and the enhancements that added a kind of macro language to the tool here: http://blog.reeset.net/archives/1853

When the tool runs, one of the assumptions that is made is that the tool pulls the data for the pattern from the first field/field/subfield combination that meets the pattern criteria.  This works well if your record has only a single field for the data that you need to capture.  But what if you have multiple fields.  Say for example, the user needs to create a call number, and one of those elements will be the ISBN – however, the record has multiple ISBN fields like:
=020  \\$a123456 (ebook)
=020  \\$a654321 (hardcopy)

Say I need to specifically get the ISBN from the hardcopy.  In the current build new tool function, this wouldn’t be possible without first changing the first 020 to something else (like an 021) – then changing it back when the operation was completed.  This is because if I used say:
=099  \\$aMyCall Number {020$a}

I would get the first 020$a value.  There hasn’t been a way to ask for the tool to find specific field data in this function.  But that has changed – I’ve introduced: find. 

Function: .find
Arguments: needle
Example: {020$a.find(“hardcopy”)}

Find will allow you to selectively find data in a field.  So, in the example above, I can now select the correct 020.
=020  \\$aMyCall Number {020$a.find(“hardcopy”).replace(“(hardcopy)”,””)}

This will output:
=020  \\$aMyCall Number 654321

A couple notes about usage.  Find must always be the first option in a chain of macros.  This is because the tool actually does the other operations like substitutions – so the criteria being queried must reflect the data in the record at read – not after it has been processed.  If you place find in any other position, you may invalidate your pattern. 

This will be part of the next upcoming MarcEdit update.

–tr

 Posted by at 8:20 am