MarcEdit 5.5 Update

By reeset / On / In MarcEdit

I’ve posted the MarcEdit 5.5 update.  This is a major update in that it updates the program from .NET 3 to .NET 4.  Here’s the full list of changes:


Changes for this version:

1) Bug fix to the OCLC Classify service (when a local OCLC number is present, depending on the prefix, it can cause problems)

2) OCLC Classify (Enhancement) – I’m pulling this out as a stand-alone tool so you don’t have to run it directly from the MarcEditor.

3) MARCEngine

a. Update: Added error code when fields are too long. (still working on a marc agnostic method for splitting fields. Easy to do in MARC21 – more difficult when MARC21 is taken out of the picture due to other national rules)

4) Plugins:

a. Updates: Rebuilt to support new 32/64 bit architecture

5) Small updates to the Updater (no longer a message box when update is necessary – but move to a scrollable dialog that can hold more content)

6) Architectural updates that:

a. Better 32/64 bit support

b. Switch to the .NET 4 Framework Client Profile (this is an optimized subset of the full .NET 4 Framework)

c. Installer updates slimming the present installer from 20 MB to about 12.5 MB (thanks largely to the Framework change and Yaz changes)

7) Additional Preference option for Automated Updates

a. Users will be able to store administrative credentials in MarcEdit to allow non-Admin users to use the automated updating tool.  I’d sent a note earlier about this change – admin credentials will be encrypted using a 128bit private key, and then salted for security.   YOU must be a part of a DOMAIN for this to work.

8) Delimiter Text Translator:

a. Update: Added code that allows for multiple joining of like fields

b. Bug-fix: When working with files smaller than 8000 bites, there is a read error where some data gets dropped. This is corrected in this version

c. Update: Command Line Options added

9) Task Automation Addition

a. Update: ability to add Copy Field to task automation

10) Copy Field Function Enhancement

a. Added Find options to choose fields for copy.

11) Script Wizard

a. Update: Added In-String matching for deleted fields

b. Update: Added Conditional matching for deleted fields

c. Update: Added ability to modify control data via the script wizard

12) Extract Selected Records:

a. Added a checkbox to make Regular Expressions easier to utilize

13) MarcEditor Compiler updates

a. Added ability to do character conversions from the MarcEditor on save

14) Jump To…

a. Enhancement: Made it so you can jump to records not on the current page

b. Enhancement: Added a jump to page function

15) Yaz updates

a. Enhancement: I’ve updated MarcEdit to use Yaz 4+ by default.

b. Enhancement: yaz build is directly from the source – allowing me to reduce the file size by turning off a number of options that are simply not used in MarcEdit.

Getting the real %windir%\system32 directory from within a 32 bit process on a 64 bit machine

By reeset / On / In C#

When working with the 64-bit flavor of Windows, there are a couple of quirks that you just need to accept.  First, when Microsoft designed windows for 64 bit processing, they weren’t going to break legacy applications and second, how this gets done makes absolutely no sense unless you simply have faith that the Operating System will take care of it all for you.

So why doesn’t it make sense?  Well, Windows 64-bit systems are essentially operating systems with two minds.  One the one hand, you have the system designed to run 64-bit processes, but lives within an ecosystem where nearly all windows applications are still designed for 32-bit systems.  So what is an OS designer to do?  Well, you have both versions of the OS running, obviously.  Really, it’s not that simple (or complex) – in reality, Microsoft has a 32 bit emulator that allows all 32 bit applications to run on 64-bit versions of Windows.  However, this is where it gets complicated.  Since 64 bit processes cannot access 32 bit processes and vise versa – you run into a scenerio where Windows must mirror a number of systems components for both 32 and 64 bit processes.  They do this two ways:

  1. In the registry – Microsoft has what I like to think of as a shadow registry that exists for WOW64 bit processes (that’s the 32 bit emulator if you can’t tell) – to register COM objects and components accessible to 32-bit applications.
  2. The presence of a system32 (this is ironically where the 64bit libraries live) and a SysWow64 (this is where the 32 bit libraries live) system folders which replicate large portions of the Windows API and provide system components for 32-bit and 64 bit processes.

So how does this all work together?  Well, Microsoft makes it work through redirection.  Quietly, transparently – the Windows operating system will access the applicable part of the system based on the process type (being either 64 or 32 bit).  The problem arises when one is running a 32-bit process, but want to have access to a dedicated 64 bit folder or registry key.  Because redirection happens at the system level, programming tools, api access, etc. all are redirected to the appropriate folder for the process type.

Presently, I’m in the process of rebuilding MarcEdit, and one of the requirements is that it run natively in either 32 or 64 bit mode.  The problem is that there are a large cohort of users that are working with MarcEdit on 64 bit systems, but have it installed as a 32 bit app.  So, I’ve been tweaking an Installer class that will evaluate the user’s environment (if 32 bit process running in a 64 bit OS) and move the appropriate 64 bit files into their correct locations so that when they run the program for the first time, the C# code compiled to run on Any CPU (so, if it’s a 64 bit OS, it will run natively as a 64 bit app) – the necessary components will be available.

So how do we do this?  Actually, it’s not all that hard.  Microsoft provides two important API for the task: Wow64DisableWow64FsRedirection and Wow64RevertWow64FsRedirection.  Using these two functions, you can temporarily disable redirection within your application.  However, you want to make sure you re-enable when your required access of these components is over (or apparently, the world as we know it will come to and end).   So, here’s my simple example of how this might work:

#region API_To_Disable_File_Redirection_On_64bit_Machines
public static string gslash = System.IO.Path.DirectorySeparatorChar.ToString();
[DllImport("kernel32.dll", SetLastError = true)]
private static extern bool Wow64DisableWow64FsRedirection(ref IntPtr ptr);
[DllImport("kernel32.dll", SetLastError = true)]
private static extern bool Wow64RevertWow64FsRedirection(IntPtr ptr);
#region Fix 64bit 
if (System.Environment.Is64BitProcess == false && System.Environment.Is64BitOperatingSystem == true)
//This needs to have some data elements moved from the 64bit folder to the System32 directory
string[] files = System.IO.Directory.GetFiles(AppPath() + "64bit" + gslash);
string windir = Environment.ExpandEnvironmentVariables("%windir%");
string system32dir = Path.Combine(windir, "System32");
if (system32dir.EndsWith(gslash) == false) {
system32dir += gslash;

//We need to run off the redirection that happens on 64 bit systems.
IntPtr ptr = new IntPtr();
bool isWow64FsRedirectionDisabled = Wow64DisableWow64FsRedirection(ref ptr);
foreach (string f in files)
string filename = System.IO.Path.GetFileName(f);
if (System.IO.File.Exists(system32dir + filename) == false)
    System.IO.File.Copy(AppPath() + "64bit" + gslash + filename, system32dir + filename);
catch (System.Exception yyy){//System.Windows.Forms.MessageBox.Show(yyy.ToString()); 
//We need to run the redirection back on
if (isWow64FsRedirectionDisabled)
bool isWow64FsRedirectionReverted = Wow64RevertWow64FsRedirection(ptr);



And that’s it.  Pretty straightforward.  As noted above, the big thing to remember is to Reenable the Redirection.  If you don’t, lord knows what will happen.



Installing MarcEdit on Linux–a How To

By reeset / On / In MarcEdit

I posted a YouTube video at:

Installing MarcEdit on Linux

That shows the process for installing MarcEdit on Linux using Mono.  While this demo shows how to install MarcEdit 5.5 (which will drop around April) – the general install process is the same.

I’ll be doing the same thing for a Mac install demo sometime later this week.


Elevating a process to running an MSI from a standard user’s account

By reeset / On / In C#

One of the questions that consistently comes up with the advent of Windows Vista and Windows 7’s use of the UAC is how to run applications or processes without being prompted for a username/password.  There are a number of places online that talk about how to use the C# classes + LogUser API to impersonate a user.  In fact, there is a very nice class written here that makes it quite easy.   However, these options don’t work for everything – and one specific use case is during installation.  So, for example, you’ve written a program and you want to provide automatic updating through the application.  To do the update, you’d like to simply shell to an MSI.  Well, there is a simple way to do this.  Using the System.Diagnostics assembly you can initiate a new process.  So, for example:

System.Security.SecureString password = 
new System.Security.SecureString(); string unsecured_pass = "your-password"; for (int x = 0; x < unsecured_pass.Length; x++) { password.AppendChar(unsecured_pass[x]); } password.MakeReadOnly(); StringBuilder shortPath = new StringBuilder(255); GetShortPathName(@?c:\your file name.msi?, shortPath,
shortPath.Capacity); System.Diagnostics.ProcessStartInfo myProcess =
new System.Diagnostics.ProcessStartInfo(); myProcess.Domain = "my-domain"; myProcess.UserName = "my-admin"; myProcess.Password = password; myProcess.FileName = "msiexec"; myProcess.Arguments = "/i " + shortPath.ToString(); myProcess.UseShellExecute = false; System.Diagnostics.Process.Start(myProcess);

So, using the above code, a program running as a standard user, could initiate a call to an MSI file and run it as though you were an administrator.  So, why does this work?  First, in order for this type of elevation to work, you need to make sure that UseShellExecute is set to fall.  However, if you set it to fall, you cannot execute any process that isn’t an executable (and a .msi isn’t an executable).  So we solve this by calling the msiexec program instead.  When you run an msi installer, this is the parent application that gets called.  So, we call it directly.  Now, since we are calling an executable, we can attach a username/password/domain to the process we are about to spawn.  This allows use to start the program as if we were another user.  And finally, when setting the path to the MSI file we want to run, we need to remember that we have to call that file using the Windows ShortFile Naming convention.  There is an API that will allow you to accomplish this. 

If you have this setup correctly – when the user runs an MSI file, they will not be prompted for a username or password – however, they will still see a UAC prompt asking to continue with the installation.  It looks like this (sorry, had to take a picture):


So, while the user no longer has to enter credentials to run the installer, they do have to still acknowledge that they wish to run the process.  But that seems like a livable trade off.


Changes to the automatic updater

By reeset / On / In MarcEdit

So, one of the things that I’ve been trying to do for a while is make it easier for users to update MarcEdit.  To that end, I introduced an automatic updater that allowed users to be prompted and automatically download new updates for their machines.  And with those changes, things were good…well, for me anyway, and for those users that had access to their administrative username’s and passwords.  However, with a growing number of colleges and campuses moving to a completely managed environment where users have no administrative access to their machines – this is more of a problem.  Since MarcEdit requires administrative access to update itself, users without administrative access are forced to go without or contact their IT to provide the updates – and since I don’t have a formal update schedule, this can be admittedly annoying for IT managers. 

So, I’ve been working on a change to the automated updater.  Essentially, users will be able to have MarcEdit store the administrative username/domain/password within the application, allowing MarcEdit to run the setup program using elevated permissions. 

Ok – so the first question that everyone should ask is – is that really safe.  Well, I think so, and here’s why.  Internally, MarcEdit will store the entered password using the AES encryption standard.  Passwords will be padded and then salted with a 128-bit private key.  Passwords will be encrypted and decrypted within a protected memory space.  It’s not perfect by any means – but I think that it will be more than sufficient for my needs and the MarcEdit user needs to keep this information protected.

So how will it work?  In the MarcEdit preferences, you will see a new option:


If you enter the domain/username/password combination here – the program, on next update, will initiate the installer as part of the administrator group.  If you are on Windows XP, the program will simply run, no prompts.  If you run on Windows Vista or Windows 7 and have the UAC enabled – you will see the following type of prompt (sorry – couldn’t screen shot it so I photographed it on my phone):


So, users of Vista and Windows 7 machines will still be prompted to allow MarcEdit’s setup process to proceed, but, they will no longer be required to provide a username and password because the program will utilize the information stored within the application. 

This change will be made available the next time MarcEdit is updated.