Terry's Worklog

So, I’ve been having a few more questions asking about how to setup MarcEdit on Linux and a Mac.  In general, the program works very well on both platforms, but takes a bit of know-how to get them running.  This is primarily due to the Z39.50 components that I link to.  On Linux, installation is very straightforward – on a Mac, the easiest process I’ve found is using Mac Ports – but it can take as long as an hour to install all the dependencies.  Given that Z39.50 is becoming less and less important as a protocol – I’ve worked out a process so that the program can essentially turn off these components by default when installed on non-Windows systems.  To that end, I’m reworking the zip download, and then will be looking at how I can make an installer that can be used on non-Windows platforms to make the installation a little bit easier.  On linux, I don’t think this will be all that difficult (I work primarily in a Windows/Linux environment) – on a Mac, it might take me just a little bit longer because I simply don’t use the platform often. 

If you have questions or some interest in this work, let me know.

–tr

One of the hard and fast rules that MarcEdit has consistently enforced is that you are not allowed to mix character set streams.  What this means – if your data is in UTF8 – MarcEdit will not process mnemonic data.  There are some good reasons for this – but best being that mnemonics in MarcEdit map back to MARC8 representations of a character which  is completely incompatible with the UTF8 character set. 

I’ve tried a few times to look at different ways to deal with this – but in most cases, I’ve been thwarted by the way C# handles streams.  In C#, all data is typed as UTF8 streams, unless the data is specifically types as otherwise.  In order to support MARC8 formatted data, MarcEdit reads all data as either UTF8 or as binary.  This allows MarcEdit to move easily between MARC8 and UTF8.  The problem occurs when someone wants to use mnemonics in a string that is already UTF8 encoded.  For example:
=246  13$aal-Mujāhid $bRees{aacute}, T{eacute}rry

The above is problematic to process.  Currently, MarcEdit ignores the mnemonics and treats them simply as strings.  Because these mnemonics convert directly to MARC8 bytes – one of these three diacritics sets would be flattened when processed against the stream.  If the stream was defaulted to UTF8, the {aacute} and the {eacute} encoded data will be flattened and the record generated by MarcEdit will have incorrect lengths.  If the Stream is converted to binary, then reconstituted as a UTF8 stream, any UTF8 data present in the stream is flattened, but the mnemonic data is processed correctly.  A bit of a pickle. 

To make this work – I ended up having to atomize the data that is to be processed, meaning that only the data in the mnemonic is processed – and then inserted back into a UTF8 data stream.  So, it would look something like this:

if (RecognizeUTF8(System.Text.Encoding.GetEncoding(1252).GetBytes(str_Source)) == RET_VAL_UTF_8)
{                               
    if (objChar == null)
    {
       objChar = new marc82utf8.MARCDictionary();
       objChar.UTFNormalize = UTFNormalization;
     }
     
     string tmp_diacritic = (string)lc_mnemonics_patch[tmp_string];
     tmp_diacritic = objChar.MARC8UTF8(tmp_diacritic);
     //need to convert bytes
     byte[] bytes = System.Text.Encoding.UTF8.GetBytes(tmp_diacritic);
     tmp_diacritic = System.Text.Encoding.GetEncoding(1252).GetString(bytes);
     str_Source = str_Source.Replace(tmp_string, tmp_diacritic);                                    
}

In this case, the atomized data is in tmp_diacritic and must be processed as MARC8 data to UTF8 utilizing the MarcEdit UTF8 Normalization library. At this point, the stream is switched to UTF8.  This data must now be converted to bytes, then transcoded to the internal base encoding for staging all character data, so it can then be passed back into the library for proper character handling. 

The upshot of this – MarcEdit will soon allow this type of mixed character editing.  The downside is that we still can’t get away from this type of MARC8 legacy crap.

–tr

I have been working hard over the past week to close a couple of outstanding issues with the application.  The biggest of those issues is related to the Find/Find All function.  While making some changes a few weeks ago, it appears that I introduced a bug.  I’ve been working off-line with a few folks that have been helping me debug the issue and it appears that I’ve been able to isolate and correct the issue.  The following changes have been made:

1. Bug Fix:  Find/Find All – Regular Expressions were resulting in “Text Not Found” and Boundary errors.  This has been corrected.
2. Bug Fix:  Find/Find All – The Find process has become incredibly slow due to some of the enhancements made to help the program jump directly to the found text when using Find All.  The culprit was an inefficient loop, which has been corrected.
3. Bug Fix:  Find/Find All – When searching, and then moving the cursor and searching again, the program wouldn’t reset where the search would begin.  This has been corrected.
4. Enhancement:  Find/Find All – I’ve added a directional component.  You can now search up and down the record while using Find.
5. Enhancement:  RDA Helper – I’ve added support for the automatic generation of multiple 380 fields when the data to create those elements is available.
6. Enhancement:  Console Program – I’ve added an –xml switch to allow for the processing of data from MARCXML to other XML schemas.

You can pick up the new build from:

• 32-bit: http://people.oregonstate.edu/~reeset/marcedit/software/development/MarcEdit_Setup.msi
• 64-bit: http://people.oregonstate.edu/~reeset/marcedit/software/development/MarcEdit_Setup64.msi

–tr

I’ve posted a new update that includes the following changes:

1) Bug Fix:  RDA Helper – Corrects the exception that occurs when trying to insert 33x, 38x fields when the highest field in the record is a lower than 329. 

2) Enhancement:  RDA Helper – I added the ability to embed regular expressions into the Abbreviation expansion, and will include 2 as examples with the next update.  This is necessary to deal with items like “v.”  This could expand to volume or volumes depending on the data that follows.  So, I’ve included a regular expression that will evaluate the data prior to “v.” to try and determine what the expansion should look like.  This should give some added flexibility for anyone wanting to augment the substitution list.

3) Behavior Change: RDA Helper – because I’m modifying the substitutions list due to the item above, I need to provide a mechanism to update the codes.  Essentially, I’ve added a routine that will automatically keep the substitution list synchronized with the master list.  However, if you add your own items to the list – they will get overwritten – so I’ve added a notification and save the previous list as a backup so you can move any custom substitutions to the new list.  I don’t anticipate changing the master list often, so this might be good enough.  But I’ll look at other ways of making this process less intrusive in the future.

4) Enhancement: RDA Helper – when generating the 336, I’ve expanded the data elements consulted to help improve the selection of more granular values in the 336$a – specifically for text items.

5) Enhancement:  RDA Helper – I spent a good deal of time last night optimizing how regular expressions are run in the RDA Helper and have seen a good deal of performance gained.

6) Bug Fix: Delimited Text Translator – When using autogenerate, indicators would occasionally be dropped.  This has been corrected.

7) Enhancement:  RDA Helper – MarcEdit will protect data within quotes from substitution.  This makes the assumption that quoted data should be data that has been transcribed from the object.

8) Bug Fix:  RDA Helper – When generating 264s from 260s, the first indicator wasn’t be retained correctly.  This has been corrected. 

9) Bug Fix: RDA Helper – When generating a 264, fields that ended with a hyphen were occasionally having periods added to the end.  This has been corrected.

10) Bug Fix: Find/Find All – I made a change to the jump list that allowed the program to select the text searched for.  Unfortunately, if you are looking for a regular expression, it can expand beyond the select.  This has been corrected.

You can download the program from:

• 32-bit: http://people.oregonstate.edu/~reeset/marcedit/software/development/MarcEdit_Setup.msi
• 64-bit: http://people.oregonstate.edu/~reeset/marcedit/software/development/MarcEdit_Setup64.msi

–TR