Aug 022015
 

MarcEdit Mac users, a new preview update has been made available.  This is getting pretty close to the first “official” version of the Mac version.  And for those that may have forgotten, the preview designation will be removed on Sept. 1, 2015.

So what’s been done since the last update?  Well, I’ve pretty much completed the last of the work that was scheduled for the first official release.  At this point, I’ve completed all the planned work on the MARC Tools and the MarcEditor functions.  For this release, I’ve completed the following:

****************************
** 1.0.9 ChangeLog
****************************

  • Bug Fix: Opening Files — you cannot select any files but a .mrc extension. I’ve changed this so the open dialog can open multiple file types.
  • Bug Fix: MarcEditor — when resizing the form, the filename in the status can disappear.
  • Bug Fix: MarcEditor — when resizing, the # of records per page moves off the screen.
  • Enhancement: Linked Data Records — Tool provides the ability to embed URI endpoints to the end of 1xx, 6xx, and 7xx fields.
  • Enhancement: Linked Data Records — Tool has been added to the Task Manager.
  • Enhancement: Generate Control Numbers — globally generates control numbers.
  • Enhancement: Generate Call Numbers/Fast Headings – globally generated call numbers/fast headings for selected records.
  • Enhancement: Edit Shortcuts — added back the tool to enabled Record Marking via a comment.

Over the next month, I’ll be working on trying to complete four other components prior to the first “official” release Sept. 1.  This means that I’m anticipating at least 1, maybe 2 more large preview releases before Sept. 1, 2015.  The four items I’ll be targeting for completion will be:

  1. Export Tab Delimited Records Feature — this feature allows users to take MARC data and create delimited files (often for reporting or loading into a tool like Excel).
  2. Delimited Text Translator — this feature allows users to generate MARC records from a delimited file.  The Mac version will not, at least initially, be able to work with Excel or Access data.  The tool will be limited to working with delimited data.
  3. Update Preferences windows to expose MarcEditor preferences
  4. OCLC Metadata Framework integration…specifically, I’d like to re-integrate the holdings work and the batch record download.

How do you get the preview?  If you have the current preview installed, just open the program and as long as you have the notifications turned on – the program will notify that an update is available.  Download the update, and install the new version.  If you don’t have the preview installed, just go to: http://marcedit.reeset.net/downloads and select the Mac app download.

If you have any questions, let me know.

–tr

 Posted by at 4:42 pm
Jul 292015
 

I hadn’t planned on putting together an update for the Windows version of MarcEdit this week, but I’ve been working with someone putting the Linked Data tools through their paces and came across instances where some of the linked data services were not sending back valid XML data – and I wasn’t validating it.  So, I took some time and added some validation.  However, because the users are processing over a million items through the linked data tool, I also wanted to provide a more user friendly option that doesn’t require opening the MarcEditor – so I’ve added the linked data tools to the command line version of MarcEdit as well. 

Linked Data Command Line Options:

The command line tool is probably one of those under-used and unknown parts of MarcEdit.  The tool is a shim over the code libraries – exposing functionality from the command line, and making it easy to integrate with scripts written for automation purposes.  The tool has a wide range of options available to it – and for users unfamiliar with the command line tool – they can get information about the functionality offered by querying help.  For those using the command line tool – you’ll likely want to create an environmental variable pointing to the MarcEdit application directory so that you can call the program without needing to navigate to the directory.  For example, on my computer, I have an environmental variable called: %MARCEDIT_PATH% which points to the MarcEdit app directory.  This means that if I wanted to run the help from my command line for the MarcEdit Command Line tool, I’d run the following and get the following results:

C:\Users\reese.2179>%MARCEDIT_PATH%\cmarcedit -help
***************************************************************
* MarcEdit 6.1 Console Application
* By Terry Reese
* email: reeset@gmail.com
* Modified: 2015/7/29
***************************************************************
Arguments:
        -s:     Path to file to be processed.
                        If calling the join utility, source must be files
                        delimited by the ";" character
        -d:     Path to destination file.
                          If call the split utility, dest should specify a fold
r
                        where split files will be saved.
                        If this folder doesn't exist, one will be created.
        -rules: Rules file for the MARC Validator.
        -mxslt: Path to the MARCXML XSLT file.
        -xslt:  Path to the XML XSLT file.
        -batch: Specifies Batch Processing Mode
        -character:     Specifies character conversion mode.
        -break: Specifies MarcBreaker algorithm
        -make:  Specifies MarcMaker algorithm
        -marcxml:       Specifies MARCXML algorithm
        -xmlmarc:       Specifics the MARCXML to MARC algorithm
        -marctoxml:     Specifies MARC to XML algorithm
        -xmltomarc:     Specifies XML to MARC algorithm
        -xml:   Specifies the XML to XML algorithm
        -validate:      Specifies the MARCValidator algorithm
        -join:  Specifies join MARC File algorithm
        -split: Specifies split MARC File algorithm
        -records:       Specifies number of records per file [used with split c
mmand].
        -raw:   [Optional] Turns of mnemonic processing (returns raw data)
        -utf8:  [Optional] Turns on UTF-8 processing
        -marc8: [Optional] Turns on MARC-8 processing
        -pd:    [Optional] When a Malformed record is encountered, it will modi
y the process from a stop process to one where an error is simply noted and a s
ub note is added to the result file.
        -buildlinks:    Specifies the Semantic Linking algorithm
This function needs to be paired with the -options parameter
        -options        Specifies linking options to use: example: lcid,viaf:lc
oclcworkid,autodetect           lcid: utilizes id.loc.gov to link 1xx/7xx data
                autodetect: autodetects subjects and links to know values
                oclcworkid: inserts link to oclc work id if present
                viaf: linking 1xx/7xx using viaf.  Specify index after colon. I
 no index is provided, lc is assumed.
                        VIAF Index Values:
                        all -- all of viaf
                        nla -- Australia's national index
                        vlacc -- Belgium's Flemish file
                        lac -- Canadian national file
                        bnc -- Catalunya
                        nsk -- Croatia
                        nkc -- Czech.
                        dbc -- Denmark (dbc)
                        egaxa -- Egypt
                        bnf -- France (BNF)
                        sudoc -- France (SUDOC)
                        dnb -- Germany
                        jpg -- Getty (ULAN)
                        bnc+bne -- Hispanica
                        nszl -- Hungary
                        isni -- ISNI
                        ndl -- Japan (NDL)
                        nli -- Israel
                        iccu -- Italy
                        LNB -- Latvia
                        LNL -- Lebannon
                        lc -- LC (NACO)
                        nta -- Netherlands
                        bibsys -- Norway
                        perseus -- Perseus
                        nlp -- Polish National Library
                        nukat -- Poland (Nukat)
                        ptbnp -- Portugal
                        nlb -- Singapore
                        bne -- Spain
                        selibr -- Sweden
                        swnl -- Swiss National Library
                        srp -- Syriac
                        rero -- Swiss RERO
                        rsl -- Russian
                        bav -- Vatican
                        wkp -- Wikipedia

        -help:  Returns usage information

The linked data option uses the following pattern: cmarcedit.exe –s [sourcefile] –d [destfile] –buildlinks –options [linkoptions]

As noted above in the list, –options is a comma delimited list that includes the values that the linking tool should query.  A user, for example, looking to generate workids and uris on the 1xx and 7xx fields using id.loc.gov – the command would look like:

<< cmarcedit.exe –s [sourcefile] –d [destfile] –buildlinks –options oclcworkid,lcid

Users interesting in building all available linkages (using viaf, autodetecting subjects, etc. would use:

<< cmarcedit.exe –s [sourcefile] –d [destfile] –buildlinks –options oclcworkid,lcid,autodetect,viaf:lc

Notice the last option – viaf. This tells the tool to utilize viaf as a linking option in the 1xx and the 7xx – the data after the colon identifies the index to utilize when building links.  The indexes are found in the help (see above).

Download information:

The update can be found on the downloads page: http://marcedit.reeset.net/downloads or using the automated update tool within MarcEdit.  Direct links:

Mac Port Update:

Part of the reason I hadn’t planned on doing a Windows update of MarcEdit this week is that I’ve been heads down making changes to the Mac Port.  I’ve gotten good feedback from folks letting me know that so far, so good.  Over the past few weeks, I’ve been integrating missing features from the MarcEditor into the Port, as well as working on the Delimited Text Translation.  I’ll now have to go back and make a couple of changes to support some of the update work in the Linked Data tool – but I’m hoping that by Aug. 2nd, I’ll have a new Mac Port Preview that will be pretty close to completing (and expanding) the initial port sprint. 

Questions, let me know.

–tr

 Posted by at 9:39 pm
Jul 242015
 

Whew – it’s be a wonderfully exhausting past few days here in Columbus, OH as the Libraries played host to Code4LibMW.  This has been something that I’ve been looking forward to ever since making the move to The Ohio State University; the C4L community has always been one of my favorites, and while the annual conference continues to be one of the most important meetings on my calendar – it’s within these regional events where I’m always reminded why I enjoy being a part of this community. 

I shared a story with the folks in Columbus this week.  As one of the folks that attended the original C4L meeting in Corvallis back in 2006 (BTW, there were 3 other original attendees in Columbus this week), there are a lot of things that I remember about that event quite fondly.  Pizza at American Dream, my first experience doing a lightening talk, the joy of a conference where people were writing code as they were standing on stage waiting their turn to present, Roy Tennant pulling up the IRC channel while he was on stage, so he could keep an eye on what we were all saying about him.  It was just a lot of fun, and part of what made it fun was that everyone got involved.  During that first event, there were around 80 attendees, and nearly every person made it onto the stage to talk about something that they were doing, something that they were passionate about, or something that they had been inspired to build during the course of the week.  You still get this at times at the annual conference, but with it’s shear size and weight, it’s become much harder to give everyone that opportunity to share the things that interest them, or easily connect with other people that might have those same interests.  And I think that’s the purpose that these regional events can serve. 

By and large, the C4L regional events feel much more like those early days of the C4L annual conference.  They are small, usually free to attend, with a schedule that shifts and changes throughout the day.  They are also the place where we come together, meet local colleagues and learn about all the fantastic work that is being done at institutions of all sizes and all types.  And that’s what the C4LMW meeting was for me this year.  As the host, I wanted to make sure that the event had enough structure to keep things moving, but had a place for everyone to participate.  For me – that was going to be the measure of success…did we not just put on a good program – but did this event help to make connections within our local community.  And I think that in this, the event was successful.  I was doing a little bit of math, and over the course of the two days, I think that we had a participation rate close to 90%, and an opportunity for everyone that wanted to get up and just talk about something that they found interesting.  And to be sure – there is a lot of great work being done out here by my Midwest colleagues (yes, even those up in Michigan Smile).

Over the next few days, I’ll be collecting links and making the slides available via the C4LMW 2015 home page as well as wrapping up a few of the last responsibilities of hosting an event, but I wanted to take a moment and again thank everyone that attended.  These types of events have never been driven by the presentations, the hosts, or the presenters – but have always been about the people that attend and the connections that we make with the people in the room.  And it was a privilege this year to have the opportunity to host you all here in Columbus. 

Best,

–tr

 Posted by at 7:17 pm
Jul 212015
 

With the last update, I made a few significant modifications to the Merge Records tool, and I wanted to provide a bit more information around how these changes may or may not affect users.  The changes can be broken down into two groups:

  1. User Defined Merge Field Support
  2. Multiple Record merge support

Prior to MarcEdit 6.1, the merge records tool utilized 4 different algorithms for doing record merges.  These were broken down by field class, and as such, had specific functionality built around them since the limited scope of the data being evaluated, made it possible.  Two of these specific functions was the ability for users to change the value in a field group class (say, change control numbers from 001 to 907$b) and the ability for the tool to merge multiple records in a merge file, into the source.

When I made the update to 6.1, I tossed out the 3 field specific algorithms, and standardized on a single processing algorithm – what I call the MARC21 option.  This is an algorithm that processes data from a wide range of fields, and provides a high level of data evaluation – but in doing this, I set the fields that could be evaluated, and the function dropped the ability to merge multiple records into a single source file.  The effect of this was that:

  • Users could no longer change the fields/subfields used to evaluate data for merge outside of those fields set as part of the MARC21 option.
  • if a user had a file that looked like the following —
    sourcefile1 – record 1
    mergefile – record1 (matches source1)
    mergefile – record2
    mergefile – record3 (matches source1)

    Only data from the mergefile – record 1 would be merged.  The tool didn’t see the secondary data that might be in the merge file.  This has always been the case when working with the MARC21 merge option, but by making this the only option, I removed this functionality from the program (as the 3 custom field algorithms did make accommodations for merging data from multiple records into a single source).

With the last update, I’ve brought both of these to elements back to the tool.  When a user utilizes the Merge Records tool, they can change the textbox with the field data – and enter a new field/subfield combination for matching (at this point, it must be a field/subfield combination).  Secondly, the tool now handles the merging of multiple records if those data elements are matched via a title or control number.  Since MarcEdit will treat user defined fields as the same class as a standard number (ISBN technically) for matching – users will now see that the tool can merge duplicate data into a single source file.

Questions about this – just let me know.

–tr

 Posted by at 9:06 am
Jul 202015
 

This update will have four significant changes to three specific algorithms that are high use — so I wanted to give folks a heads up.

1) Merge Records — I’ve updated the process in two ways.  

   a) Users can now change the data in the dropdown box to a user-defined field/subfield combination.  At present, you have defined options: 001, 020, 022, 035, marc21.  You will now be able to specify another field/subfield combination (must be the combination) for matching.  So say you exported your data from your ILS, and your bibliographic number is in a 907$b — you could change the textbox from 001 to 907$b and the tool will now utilize that data, in a control number context — to facilitate matching.  

   b) This meant making a secondary change.  When I shifted to using the MARC21 method, I removed the ability for the algorithm to collapse multiple records of the same type with the merge file into the source.  For example, after the change to the marc21 algorithm, in the following scenario, the following would be true:

 source 1 — record 1
merge 1 — matches record 1
merge 2 — matches record 2
merge 3 — matches record 3

 

The data moved into source 1 would be the data from merge1 — merge 3 wouldn’t be seen.  In the previous version prior to utilizing just the Marc21 option, users could collapse records when using the control number index match.  I’ve updated the merge algorithm, so that default is now to assume that all source data could have multiple merge matches.  This has the practical option of essentially allowing users to take a merge file with multiple duplicates, and merge all data into a single corresponding source file.  But this does represent a significant behavior change — so users need to be aware.

 

2) RDA Helper — 

   a) I’ve updated the error processing to ensure that the tool can fail a bit more gracefully

   b) Updating the abbreviation expansion because the expression I was using could miss values on occasion.  This will catch more content — it should also be a bit faster.

 

3) Linked Data tools — I included the ability to link to OCLC works ids — there were problems when the json outputted was too nested.  This has been corrected.

 

4) Bibframe tool — I’ve updated the mapping used to the current LC flavor.

 

Updates can be found on the downloads page (Windows/Linux) or via the automated update tool.

Direct Links:

 

 Posted by at 11:51 pm
Jul 102015
 

A new MarcEdit 6.1 build is available.  This build closes one bug related to the Select Individual Records for Edit function in the MarcEdit, and adds four new enhancements.  You can get the update through the automated download process within MarcEdit or from the downloads page.

Download URL: http://marcedit.reeset.net/downloads
32-bit URL: http://marcedit.reeset.net/software/MarcEdit_Setup.msi
64-bit URL: http://marcedit.reeset.net/software/MarcEdit_Setup64.msi

–TR

****************************************************************************
** Change Log
****************************************************************************

6.1.25
* 6.1.25
** Bug Fix: Selected Records for Edit within the MarcEditor
** Enhancement: New button added to the join window to allow you to jump directly to the MarcEditor and work with the newly joined file.
** RDA Helper enhancement: I'm doing some additional refinements around the 380 and when bibliography is provided (since it's still probably too often to be useful)
** RDA Helper enhancement: Option to parse the 502 so that it becomes delimited.
** Enhancement Linked Data tool -- New option to add a link to OCLC's work entities.  Data is represented as a 787$n$o.  This could change...this is an experimental mapping. 

 

 Posted by at 12:49 am
Jul 082015
 

This build is a continued refinement of the preview build.  It really doesn’t include anything that is significantly new, but addresses a couple of early gaps folks had noticed while working with the tool.  Change log is below.

Download URL: http://marcedit.reeset.net/downloads
Direct URL: http://marcedit.reeset.net/software/MarcEdit.dmg

–tr

1.0.8
****************************
** 1.0.8 ChangeLog
****************************
* Bug Fix: Field Count -- When clicking on a field to retrieve information about specific indicator/subfield
usage, an error would be thrown.  This has been corrected.
* Enhancement: Main Menu -- Added a Windows menu to the MarcEdit OSX main window to make it easier to 
get back to windows that might have been hidden.
* Enhancement: Main Menu/Help/Help -- Linked to the Online Help
* Enhancement: Main Menu/Help/Report Bug/Suggestion -- Linked to the MarcEdit online reporting tool.
* Enhancement: Main Menu/Help/About Author -- Linked to online contact information.
* Enhancement: Join MARC Records -- Added an Edit File button so that users can move directly 
from Joining files together to editing the data in the MarcEditor.
* Enhancement: MarcEditor -- Exposed the mrc extension so that users can now open mrc files 
directly into the MarcEditor.  This isn't quite as smooth as the Windows version yet, but
its getting there.
* Enhancement: MarcEditor/Reports/Validate ISSNs -- Exposed the Validate ISSNs function.
 Posted by at 8:58 pm
Jul 062015
 

Interesting thing about software development — everything can work so great within your own environments, but then be so uneven once they move outside of them.  The variable that changed — real data…and that’s why you make things available for folks to play with.

First, thanks to those that downloaded the preview and gave it a whirl.  I got responses that ranged from — looks great, when will [my favorite missing function] be ported to, I tried click on this button and things crashed.  The crashing was something I didn’t expect — but it was a good lesson in making sure that all user data is validated.  I took for granted that all data passed between the API components would be OK — and it wasn’t, and when it wasn’t, problems ensued, which could not be fixed without resetting the config settings manually (which made me realize I need to ensure this can be done automatically like in the Windows/Linux version).

So, I had a late night ahead of me for some unrelated reasons, and I took a crack and hardening the validation and making the portions of the program that accept user data more fault tolerant.  And, I’m back to the point where I can’t break it…so, I’ll let you all take another crack at it.

If you downloaded the preview yesterday — the first time you open the program, you’ll be notified that a new version is available.  You can click on the download button and follow the link.  Otherwise, you can download the program from the downloads page.

Download Page URL: http://marcedit.reeset.net/downloads
Direct Link: http://marcedit.reeset.net/software/MarcEdit.dmg

Change log is below

–tr

****************************
1.0.7 ChangeLog
****************************
* Bug Fix: Open/Save Dialog Validation — These functions were not validating user data and this was causing problems. These functions now validate data, and if they cannot recover from an error, will simply return a blank value.
* Bug Fix: Run Tasks — Some of the task elements were not running. This has been corrected.
* Bug Fix: Window flashing when running tasks — this still exists a little bit (small flicker), but prior, windows were opening and staying open on each task element.
* Bug Fix: Change File prompt not being run on close – this occurred when an update was made that returned zero results. The value that managed data changes was cleared, and the window was allowed to close without prompt. This has been corrected.
* Bug Fix: The about page wasn’t listing the names that supported this development. This was a regression due to some changes made to how this particular UI component renders. This has been corrected.
* Enhancement: MARC Tools — when select a file to process, the program autofills the save file with the appropriate extension.
* Enhancement: MARC Tools — the Edit File button is now enabled after breaking
* Enhancement: Document Types — I’ve enabled document type support within the program. The application does not yet self register file extensions to the application, but if you associate the .mrc or .mrk files with the application, it will now handle opening these files correctly.

 Posted by at 9:53 pm
Jul 052015
 

It’s with a little trepidation that I’m formally making the first Public Preview of the MarcEdit OSX version available for download and use.  In fact, as of today, this version is now *the* OSX download available on the downloads page.  I will no longer be building the old code-base for use on OSX.

When I first started this project around Mid-April, I began knowing that this process would take some time.  I’ve been working on MarcEdit continuously for a little over 16 years.  It’s gone through one significant rewrite (when the program moved from Assembly to C#) and has had way too many revisions to count.  In agreeing to take on the porting work — I’d hoped that I could port a significant portion of the program over the course of about 8 months and that by the end of August, I could produce a version of MarcEdit that would cover the 80% or so of the commonly used application toolset.  To do this, it meant porting the MARC Tools portion of the application and the MarcEditor.

Well, I’m ahead of schedule.  Since about 2014, I’ve been reworking a good deal of the application to support a smoother porting process sometime in the future — though, honestly, I wasn’t sure that I’d ever actual do the porting work.  Pleasantly, this early work has made a good deal of the porting work easier allowing me to move faster than I’d anticipated.  As of this posting, a significant portion of that 80% has been converted, and I think that for many people — most of what they probably use daily — has been implemented.  And while I’m ahead of schedule and have been happy with how the porting process has gone, make no mistake — it’s been a lot of work, and a lot of code.  Even though this work has primarily been centered around rewriting just the UI portions of MarcEdit, you are still talking, as of today, close to 200,000 lines of code.  This doesn’t include the significant amount of work I’ve done around the general assemblies that have provided improvements to all MarcEdit users.  Because of that — I need to start getting feedback from users.  While the general assemblies go through an automated testing process — I haven’t, as of yet, come up with an automated testing process for the OSX build.  This means that I’m testing things manually, and simply cannot go through the same leveling of testing that I do each time I build the Windows version.  Most folks may not realize it, but it takes about a day to build the Windows version — as the program goes through various unit tests processing close to 25 million records.  I simply don’t have an equivalent of that process yet, so I’m hoping that everyone interested in this work will give it a spin, use it for real work, and let me know if/when things fall down.

In creating the Preview, I’ve tried to make the process for users as easy as possible.  Users interested in running the program simply need to be running at least OSX 10.8 and download the dmg found here: http://marcedit.reeset.net/downloads.  Once downloaded, run the dmg an a new disk will mount called MarcEdit OSX.  Run this file, and you’ll see the following installer:

MarcEdit OSX installer

MarcEdit OSX installer

Drag the MarcEdit icon into the Applications folder and the application will either install, or overwrite an existing version.  That’s it.  No other downloads are necessary.  On first run, the program will generate a marcedit folder under /users/[yourid]/marcedit.  I realize that this isn’t completely normal — but I need the data accessible outside of the normal app sandbox to easily support updates.  I’d also considered the User Documents folder, but the configuration data probably shouldn’t live there either.  So, this is where I ended up putting it.

So what’s been completed — Essentially, all the MARC Tools functions and a significant amount of the MarcEditor has been completed.  There are some conspicuous functions that are absent at this point though.  The Call Number and Fast Heading generation, the Delimited Text Translator and Exporter, the Select and Delete Selected Records, everything Z39.50 related, as well as the Linked Data tools and the Integration work with OCLC and Koha.  All these are not currently available — but will be worked on.  At this point, what users can do is start letting me know what absent components are impacting you the most, and I’ll see how they fit into the current development roadmap.

Anyway — that’s it.  I’m excited to let you all give this a try, and a little nervous as well.  This has been a significant undertaking which has definitely pushed me a bit, requiring me to learn Object-C in a short period of time, as well as quickly assimilate a significant portion of Apples SDK documents relating to UI design.  I’m sure I’ve missed things, but it’s time to let other folks start working with it.

If you have been interested in this work — download the installer, kick the tires, and give feedback.  Just remember to be gentle.  :)

–TR

Download URL: http://marcedit.reeset.net/downloads

 

 Posted by at 8:40 pm
Jul 052015
 

This was something I’d hoped to get into the last update, but didn’t get the time to test it; so I got it done now.  While at the first MarcEdit User Group meeting at ALA, there was a question about supporting 880 fields when exporting data via tab delimited format.  When you use the tool right now, the program will export all the 880 fields, not a specific 880 field.  This update changes that.  After the update, when you select the 880 field in the Export tab delimited tool, the program will ask you for the linking field.  In this case, the program will then match the 880$6[linkingfield], and pull the selected subfield.  I’m not sure how often this comes up — but it certainly made a lot of sense when the problem was described to me.

You can pick up the download at: http://marcedit.reeset.net/downloads

–tr

 Posted by at 8:33 pm