Sep 022015

I’m not sure how I’ve missed creating something like this for so long, but it took a question from a cataloger to help me crystalize the need for a new feature.  Here was the question:

Add an 856 url to all records that uses the OCLC number from the 035 field, the title from 245 $a, and the ISSN, if present. This will populate an ILLiad form with the publication title, ISSN, call number (same for all records), and OCLC number. Although I haven’t worked it in yet, the link in our catalog will also include instructions to “click for document delivery form” or something like that.

In essence, the user was looking to generate a link within some records – however, the link would need to be made up of data pulled from different parts of the MARC record.  It’s a question that comes up all the time, and in many cases, the answer I generally give points users to the Swap Field Function – a tool designed around moving data between fields.  For fields that are to be assembled from data in multiple fields – multiple swap field operations would need to be run.  The difference here was how the data from the various MARC fields listed above, needed to be presented.  The swap field tool moves data from one subfield to another – where as this user was looking to pull data from various fields and reassemble that data using a specific data pattern.  And in thinking about how I would answer this question – it kind of clicked – we need a new tool.

Build New Field:

The build new field tool is the newest global editing tool being added to the MarcEditor tool kit.  The tool will be available in the Tools menu:


And will be supported in the Task Automation list.  The tool is designed around this notion of data patterns – the idea that rather than moving data between fields – some field data needs to be created via a specific set of data patterns.  The example provided by the user asking this question as:

  •[Title from 245$a]&&ISSN=[ISSN from 022$a]&CallNumber=[CallNumber from 099$a]&ESPNumber=[oclc number from the 035]

While the swap field could move all this data around, the tool isn’t designed to do this level of data integration when generating a new field.  In fact, none of MarcEdit’s present global editing tasks are configured for this work.  To address this gap, I’ve introduced the Build New Field tool:


The Build New Field tool utilizes data patterns to construct a new MARC field.  Using the example above, a user could create a new 856 by utilizing the following pattern:

=856  41$u{245$a}&ISSN={022$a}&CallNumber={099$a}&ESPNumber={035$a(OcLC)}

Do you see the pattern?  This tool allows users to construct their field, replacing the variable data to be extracted from their MARC records using the mnemonic structure: {field$subfield}.  Additionally, in the ESPNumber tag, you can see that in addition to the field and subfield, qualifying information was also included.  The tool allows users to provide this information, which is particularly useful when utilizing fields like the 035 to extract control numbers.

Finally, the new tool provides two additional options.  For items like proxy development, data extracted from the MARC record will need to be URL encoded.  By checking the “Escape data from URL” option, all MARC data extracted and utilized within the data pattern will be URL encoded.  Leaving this item unchecked will allow the tool to capture the data as presented within the record. 

The second option, “Replace Existing Field or Add New one if not Present” tells the tool what to do if the field exists.  If left unchecked, the tool will create a new field if this option is not selected (were the field defined in the pattern exists or not).  If you check this option, the the tool will replace any existing field data or create a new field if one doesn’t exist, for the field defined by your pattern.

Does that make sense?  This function will be part of the next MarcEdit release, so if you have questions or comments, let me know.


 Posted by at 3:13 pm

MarcEdit 6.1 Update

 Uncategorized  Comments Off on MarcEdit 6.1 Update
Jul 102015

A new MarcEdit 6.1 build is available.  This build closes one bug related to the Select Individual Records for Edit function in the MarcEdit, and adds four new enhancements.  You can get the update through the automated download process within MarcEdit or from the downloads page.

Download URL:
32-bit URL:
64-bit URL:


** Change Log

* 6.1.25
** Bug Fix: Selected Records for Edit within the MarcEditor
** Enhancement: New button added to the join window to allow you to jump directly to the MarcEditor and work with the newly joined file.
** RDA Helper enhancement: I'm doing some additional refinements around the 380 and when bibliography is provided (since it's still probably too often to be useful)
** RDA Helper enhancement: Option to parse the 502 so that it becomes delimited.
** Enhancement Linked Data tool -- New option to add a link to OCLC's work entities.  Data is represented as a 787$n$o.  This could change...this is an experimental mapping. 


 Posted by at 12:49 am

MarcEdit OSX Preview Build Update

 Uncategorized  Comments Off on MarcEdit OSX Preview Build Update
Jul 082015

This build is a continued refinement of the preview build.  It really doesn’t include anything that is significantly new, but addresses a couple of early gaps folks had noticed while working with the tool.  Change log is below.

Download URL:
Direct URL:


** 1.0.8 ChangeLog
* Bug Fix: Field Count -- When clicking on a field to retrieve information about specific indicator/subfield
usage, an error would be thrown.  This has been corrected.
* Enhancement: Main Menu -- Added a Windows menu to the MarcEdit OSX main window to make it easier to 
get back to windows that might have been hidden.
* Enhancement: Main Menu/Help/Help -- Linked to the Online Help
* Enhancement: Main Menu/Help/Report Bug/Suggestion -- Linked to the MarcEdit online reporting tool.
* Enhancement: Main Menu/Help/About Author -- Linked to online contact information.
* Enhancement: Join MARC Records -- Added an Edit File button so that users can move directly 
from Joining files together to editing the data in the MarcEditor.
* Enhancement: MarcEditor -- Exposed the mrc extension so that users can now open mrc files 
directly into the MarcEditor.  This isn't quite as smooth as the Windows version yet, but
its getting there.
* Enhancement: MarcEditor/Reports/Validate ISSNs -- Exposed the Validate ISSNs function.
 Posted by at 8:58 pm

MarcEdit OSX Public Build #2

 Uncategorized  Comments Off on MarcEdit OSX Public Build #2
Jul 062015

Interesting thing about software development — everything can work so great within your own environments, but then be so uneven once they move outside of them.  The variable that changed — real data…and that’s why you make things available for folks to play with.

First, thanks to those that downloaded the preview and gave it a whirl.  I got responses that ranged from — looks great, when will [my favorite missing function] be ported to, I tried click on this button and things crashed.  The crashing was something I didn’t expect — but it was a good lesson in making sure that all user data is validated.  I took for granted that all data passed between the API components would be OK — and it wasn’t, and when it wasn’t, problems ensued, which could not be fixed without resetting the config settings manually (which made me realize I need to ensure this can be done automatically like in the Windows/Linux version).

So, I had a late night ahead of me for some unrelated reasons, and I took a crack and hardening the validation and making the portions of the program that accept user data more fault tolerant.  And, I’m back to the point where I can’t break it…so, I’ll let you all take another crack at it.

If you downloaded the preview yesterday — the first time you open the program, you’ll be notified that a new version is available.  You can click on the download button and follow the link.  Otherwise, you can download the program from the downloads page.

Download Page URL:
Direct Link:

Change log is below


1.0.7 ChangeLog
* Bug Fix: Open/Save Dialog Validation — These functions were not validating user data and this was causing problems. These functions now validate data, and if they cannot recover from an error, will simply return a blank value.
* Bug Fix: Run Tasks — Some of the task elements were not running. This has been corrected.
* Bug Fix: Window flashing when running tasks — this still exists a little bit (small flicker), but prior, windows were opening and staying open on each task element.
* Bug Fix: Change File prompt not being run on close – this occurred when an update was made that returned zero results. The value that managed data changes was cleared, and the window was allowed to close without prompt. This has been corrected.
* Bug Fix: The about page wasn’t listing the names that supported this development. This was a regression due to some changes made to how this particular UI component renders. This has been corrected.
* Enhancement: MARC Tools — when select a file to process, the program autofills the save file with the appropriate extension.
* Enhancement: MARC Tools — the Edit File button is now enabled after breaking
* Enhancement: Document Types — I’ve enabled document type support within the program. The application does not yet self register file extensions to the application, but if you associate the .mrc or .mrk files with the application, it will now handle opening these files correctly.

 Posted by at 9:53 pm

MarcEdit OSX Public Preview 1

 MarcEdit, Uncategorized  Comments Off on MarcEdit OSX Public Preview 1
Jul 052015

It’s with a little trepidation that I’m formally making the first Public Preview of the MarcEdit OSX version available for download and use.  In fact, as of today, this version is now *the* OSX download available on the downloads page.  I will no longer be building the old code-base for use on OSX.

When I first started this project around Mid-April, I began knowing that this process would take some time.  I’ve been working on MarcEdit continuously for a little over 16 years.  It’s gone through one significant rewrite (when the program moved from Assembly to C#) and has had way too many revisions to count.  In agreeing to take on the porting work — I’d hoped that I could port a significant portion of the program over the course of about 8 months and that by the end of August, I could produce a version of MarcEdit that would cover the 80% or so of the commonly used application toolset.  To do this, it meant porting the MARC Tools portion of the application and the MarcEditor.

Well, I’m ahead of schedule.  Since about 2014, I’ve been reworking a good deal of the application to support a smoother porting process sometime in the future — though, honestly, I wasn’t sure that I’d ever actual do the porting work.  Pleasantly, this early work has made a good deal of the porting work easier allowing me to move faster than I’d anticipated.  As of this posting, a significant portion of that 80% has been converted, and I think that for many people — most of what they probably use daily — has been implemented.  And while I’m ahead of schedule and have been happy with how the porting process has gone, make no mistake — it’s been a lot of work, and a lot of code.  Even though this work has primarily been centered around rewriting just the UI portions of MarcEdit, you are still talking, as of today, close to 200,000 lines of code.  This doesn’t include the significant amount of work I’ve done around the general assemblies that have provided improvements to all MarcEdit users.  Because of that — I need to start getting feedback from users.  While the general assemblies go through an automated testing process — I haven’t, as of yet, come up with an automated testing process for the OSX build.  This means that I’m testing things manually, and simply cannot go through the same leveling of testing that I do each time I build the Windows version.  Most folks may not realize it, but it takes about a day to build the Windows version — as the program goes through various unit tests processing close to 25 million records.  I simply don’t have an equivalent of that process yet, so I’m hoping that everyone interested in this work will give it a spin, use it for real work, and let me know if/when things fall down.

In creating the Preview, I’ve tried to make the process for users as easy as possible.  Users interested in running the program simply need to be running at least OSX 10.8 and download the dmg found here:  Once downloaded, run the dmg an a new disk will mount called MarcEdit OSX.  Run this file, and you’ll see the following installer:

MarcEdit OSX installer

MarcEdit OSX installer

Drag the MarcEdit icon into the Applications folder and the application will either install, or overwrite an existing version.  That’s it.  No other downloads are necessary.  On first run, the program will generate a marcedit folder under /users/[yourid]/marcedit.  I realize that this isn’t completely normal — but I need the data accessible outside of the normal app sandbox to easily support updates.  I’d also considered the User Documents folder, but the configuration data probably shouldn’t live there either.  So, this is where I ended up putting it.

So what’s been completed — Essentially, all the MARC Tools functions and a significant amount of the MarcEditor has been completed.  There are some conspicuous functions that are absent at this point though.  The Call Number and Fast Heading generation, the Delimited Text Translator and Exporter, the Select and Delete Selected Records, everything Z39.50 related, as well as the Linked Data tools and the Integration work with OCLC and Koha.  All these are not currently available — but will be worked on.  At this point, what users can do is start letting me know what absent components are impacting you the most, and I’ll see how they fit into the current development roadmap.

Anyway — that’s it.  I’m excited to let you all give this a try, and a little nervous as well.  This has been a significant undertaking which has definitely pushed me a bit, requiring me to learn Object-C in a short period of time, as well as quickly assimilate a significant portion of Apples SDK documents relating to UI design.  I’m sure I’ve missed things, but it’s time to let other folks start working with it.

If you have been interested in this work — download the installer, kick the tires, and give feedback.  Just remember to be gentle.  :)


Download URL:


 Posted by at 8:40 pm
Jun 302015

Changes in this update:

* 6.1.21
** Bug Fix: Conditional Delete - When selecting regular expressions -- there were times when the process wasn't being recognized.
** Enhancement: Conditional Delete - This function use to only work when using the Regular Expression option.  This now works for all options.
** Bug Fix: ValidateISBNs - Process would only process the first subfield.  If the subfield to be processed wasn't the first one, it wouldn't be validated.
** Enhancement: ValidateISSN: Uses mathematical formula to validate ISSNs.
** Bug Fix: Generate Fast Headings (Stand alone tool) -- LDR fields could be deleted.  
** Enhancement: Working to make the global edit functions a little more fault tolerant around record formatting.
** Enhancement: Generate MARC record from URL -- program generates MARC records from Webpages.  If you pass it an LC URL, it will generate data from the MARCXML.

At this point, only the Windows and Linux downloads were updated.  I'll be replacing the Mac download with the first version of the native OSX build the July 4th weekend.  You can get the updates either via the Automated updated tool or from the website at:

 Posted by at 8:45 pm

Updating resources on the main UI thread: OSX & C#

 Uncategorized  Comments Off on Updating resources on the main UI thread: OSX & C#
Jun 062015

One of the things that has been a learning experience for me has been understanding how OSX handles threading, understanding what objects are thread-safe, and how to capture user feedback from within an thread running outside of the main UI thread.

Conceptually, Windows, OSX, Linux — UIs tend to be run through a single, primary thread.  This is why long-running processes can lock a window from updating.  In Windows, this can be overcome by creating background threads.  While these threads should not interact with the main UI objects — you do have the ability to spawn a new UI thread that can be used to capture interaction from users.  You just need to be careful to ensure that objects stay compartmentalized since these objects are not explicitly thread-safe.

Within the OSX environment, the UI components cannot be run from outside of the main thread.  This means that if you have a background threading process, you cannot interact with the user because all UI objects are marked as not safe for threading.  Not very practical since there are times when user interaction is definitely necessary.  For example, within MarcEdit’s Mac port — I need to be able to tell users when the program has updated — and I’d like this process to occur outside the primary UI thread so not to slow down application startup or user interactions.  To do this, you need to find a way to thread switch — so that you can encapsulate the code that will interact with the UI around the necessary code to switch execution back to the main UI thread.  In C# and on Windows, you can take non-thread-safe components and using a delegate and Invoke, thread switch within your application.  OSX has something similar, but after hours of looking, I wanted to make sure I document how it works here, so I won’t have to look this up again.  The two functions are: InvokeOnMainThread and BeginInvokeOnMainThread.  These two functions have some limitations — they can only be run within the context of a UI thread (so a window controller because you need access to the NSWindow handle) — but from within this block, you have the ability to generate an NSAlert class (though tricky) as well as generate new NSWindow objects to capture user input.

Here’s an example of how this would work below.  This is the code used for MarcEdit’s Mac notification service.  The program starts a thread when the application loads the main window for the first time — that thread initiates a System.Net.HttpWebRequest object to retrieve information about the current MarcEdit Mac Build.  The program then parses the version information, compares it to the local version and build number — and if the remote build number is newer — generates a UI element that tells the user what has changed, and provides buttons to download the new application.  This code lives in a couple of different classes, but these are the relevant parts.


public override void AwakeFromNib ()
	this.Window.BackgroundColor = NSColor.White;
	base.AwakeFromNib ();
	System.Threading.Thread objT = new System.Threading.Thread((System.Threading.ThreadStart)delegate { CheckUpdate();});
	objT.Start ();


void CheckUpdate ()
	helpers objH = new helpers ();
	string data = objH.GetWebContent(clsGlobal.MacNotificationURI);
	string sversion = data.Split (System.Environment.NewLine.ToCharArray ()) [0].Trim();

	if (!string.IsNullOrEmpty (data)) {
		NSObject ver = NSBundle.MainBundle.InfoDictionary ["CFBundleVersion"];
		NSObject ver2 = NSBundle.MainBundle.InfoDictionary ["CFBundleShortVersionString"];

		Version vweb = new Version (sversion);
		Version vlocal = new Version (ver2.ToString() + "." + ver.ToString ());

		if (vweb > vlocal) {
			//To generate a UI element from outside the main UI thread, 
			//use InvokeOnMainThread and create a delegate function block
			InvokeOnMainThread (() => {
				frmUpdateNotificationController objU = new frmUpdateNotificationController();
				objU.LoadMessage("Build #" + sversion + System.Environment.NewLine + System.Environment.NewLine + 

//helpers class

public string GetWebContent(string uri) {

	System.Net.HttpWebRequest myReq =
				(System.Net.HttpWebRequest)System.Net.WebRequest.Create (new Uri (uri));
	System.Net.HttpWebResponse response = (System.Net.HttpWebResponse)myReq.GetResponse ();
	System.IO.Stream responseStream = response.GetResponseStream ();
	System.IO.StreamReader reader = new System.IO.StreamReader (responseStream, System.Text.Encoding.UTF8);
	string val = reader.ReadToEnd ();
	reader.Close ();
	response.Close ();
	return val;

 Posted by at 9:28 am

MarcEdit 6 Update Posted

 Uncategorized  Comments Off on MarcEdit 6 Update Posted
Dec 162014

A new update has been posted.  The changes are noted below:

  • Enhancement: Installation changes – for administrators, the program will now allow for quite installations and an option to prevent users from enabling automated updates.  Some IT admins have been asking for this for a while.  The installation program will take an command-line option: autoupdate=no, to turn this off.  The way this is disabled (since MarcEdit manages individual profiles) is a file will be created into the program directory that if present, will present automatic updates.  This file will be removed (or not recreated) if this command-line isn’t set – so users doing automated installations will need to remember to always set this value if they wish to prevent this option from being enabled.  I’ve also added a not in the Preferences window noting if the administrator has disabled the option.
  • Bug Fix: Swap Field Task List – one of the limiters wasn’t being passed (the process one field per swap limiter)
  • Bug Fix: Edit Field Task List – when editing a control field, the positions text box wasn’t being shown. 
  • Bug Fix: Edit Field Regular Expression options – when editing control fields, the edit field function evaluated the entire field data – not just the items to be edited.  So, if I wanted to use a regular expression to evaluate two specific values, I couldn’t.  This has been corrected.
  • Enhancement: Linked Data Linker – added support for FAST headings. 
  • Bug Fix: Linked Data Linker – when processing data against LC’s, some of the fuzzy matching was causing values to be missed.  I’ve updated the normalization to correct this.
  • Enhancement: Edit Subfield Data – Moving Field data – an error can occur if the field having data moved to is a control field, and the control field is smaller than the position where the data should be moved to.  An error check has been added to ensure this error doesn’t pop up.
  • Bug Fix: Auto Translation Plug-in – updated code because some data was being dropped on translation, meaning that it wouldn’t show up in the records.

Update can be found at: or via the automated updating tool.  The plug-in updates can be downloaded via the Plug-in Manager within the MarcEdit Application.


 Posted by at 10:25 pm

MarcEdit 6 Update

 Uncategorized  Comments Off on MarcEdit 6 Update
Nov 272014

Happy Thanksgiving to those celebrating.  Rather than over indulging in food, my family and I spent our day relaxing and enjoying some down time together.  After everyone went to bed, I had a little free time and decided to wrap up the update I’ve been working on.  This update includes the following changes:

  • Language File changes.
  • Export/Delete Selected Records: UI changes
  • Biblinker — updated the tool to provide support for linking to FAST headings when available in the record
    • Updated the fields processed (targeted to ignore uncontrolled or local items)
  • Z39.50 Client — Single Search, multiple databases selected bug when number of results exceed data limit, blank data would be returned.
  • RDA Helper Bug Fix — Updated an error where under certain conditions, bracketed data would be incorrectly parsed.
  • Miscellaneous UI changes to support language changes


The Language file changes represent a change in how internationalization of the interface works.  Master language files are now hosted on GItHub, with new files added on update.  The language files are automatically generated, so they are not as good as if they were done by an individual – though some individuals are looking at the files and providing updates.  My hope is that through this process of automated language generation, coupled with human intervention, the new system will significantly help non-English speakers.  But I guess time will tell.

The download can be found by using the automated update tool in MarcEdit, or downloading the update from:

 Posted by at 10:27 pm

MarcEdit 6 Update

 Uncategorized  Comments Off on MarcEdit 6 Update
Aug 152014

So I missed one – I made some changes to how MarcEdit loads data into the MarcEditor to improve performance, especially on newer equipment, and introduced a bug.  When making multiple global updates, the program may (probably) will lose track of the last page of records.  This slipped through my unit tests because the program was reporting the correct number of changes, but when the program analyzed the for indexing, it was dropping the last page.  Oops.  This was introduced in the update posted at 1 am on Aug. 15th.  I’ve corrected the problem and updated my unit tests so that this type of regression shouldn’t occur again.

One question that did come up to me privately was why make this change to begin with.  Primarily, it was about performance.  MarcEdit’s paging process requires the program to index the records within a MARC block.  In the previous approach, this resulted in a lot of disk reads.  Generally, this isn’t a problem, but I’ve had occasion where folks with lower disk speeds have had performance issues.  This change will correct that.  For files under 50 MB, the program will now read the entire file into memory, and process the data in memory to generate paging.  This is a more memory intensive task (the previous method utilized a small amount of memory, whereas the new process can require allocations of 100-120 MB of system memory for processing) but removes the disk reads which is the largest bottleneck within the process.  The effect of this change is a large performance gain.  On my development system, which has a solid state drive, the improvement loading a 50 MB file is over a second, going from 3.3 seconds to 1.8 seconds.  That’s a pretty significant improvement – especially on a system where disk reads tend to happen very quickly.  On my secondary systems, the improvements are more noticeable.  On an Intel I-5 with a non-solid state drive and 6 GB of RAM, the old process took between 3.7 to 4.1 seconds, while the new method loaded the file between 1.6-1.8 seconds.  And on a tablet with an older Atom processor and 2 GB of RAM, the old process took approximately 22 seconds, while the new only 9 seconds.  These are big gains that I hope users will be able to see and benefit from.


Testing Results Old Process

Machine Description File Description Time to Load
I-7 Dell XPS Ultrabook, 8 GB RAM, SSD 45,922 records; 50 MB 1st load: 3.4s
2nd load: 3.3s
3rd load: 3.2s
I-5 Dell Workstation, 6 GB RAM, 7200 rpm HD 45,922 records, 50 MB 1st load: 4.1 s
2nd load: 4.0s
3rd load: 3.8s
Atom 1.5 mhz ACER tablet, 2 GB RAM, SSD 45,922 records; 50 MB 1st load: 27s
2nd load: 22s
3rd load: 23s


Testing Results New Process

Machine Description File Description Time to Load Diff
I-7 Dell XPS Ultrabook, 8 GB RAM, SSD 45,922 records; 50 MB 1st load: 1.4s
2nd load: 1.3s
3rd load: 1.3s
I-5 Dell Workstation, 6 GB RAM, 7200 rpm HD 45,922 records, 50 MB 1st load: 1.8 s
2nd load: 1.6s
3rd load: 1.6s
Atom 1.5 mhz ACER tablet, 2 GB RAM, SSD 45,922 records; 50 MB 1st load: 10.1s
2nd load: 9.6s
3rd load: 9.7s


While the new process appears to provide better performance on many different types of systems, I realize that there may be some system variations that not benefit from this new method.  To that end, I’ve added a new configuration option in the MarcEditor Preferences that will allow users to decide to turn off the new paging method.  By default, this option is selected.


If you update the program via MarcEdit, the download will be offered automatically the next time you use the program.  Otherwise, you can get the update at:



 Posted by at 11:20 pm