reeset

Dec 052016
 

Posted a MarcEdit Mac update.  This syncs the task management and Edit shortcuts with the Windows version.

************************************************
** 1.9.45
************************************************
* Enhancement: Task Manager: Implemented the ability to include Edit Shortcuts in Tasks
* Enhancement: Task Manager: Updated Task Manager to complete  network task clean up (error messages, file locking)
* Enhancement: Preferences: Updated preferences to include dialogs to find files and folders.

You can get the file from the downloads page: http://marcedit.reeset.net/downloads or via the automated update tool.

-tr

 Posted by at 12:39 am
Nov 302016
 

In what’s become a bit of a tradition, I took some of my time over the Thanksgiving holiday to work through a few things on my list and put together an update (posted last night).  Updates were to all versions of MarcEdit and cover the following topics:

Windows/Linux:

* Enhancement: Dedup Records – addition of a fuzzy match option
* Enhancement: Linked Data tweaks to allow for multiple rules files
* Bug Fix: Clean Smart Characters can now be embedded in a task
* Enhancement: MARC Tools — addition of a MARC=>JSON processing function
* Enhancement: MARC Tools — addition of a JSON=>MARC processing function
* Behavior Change: SPARQL Browser updates — tweaks make it more simple at this point, but this will let me provide better support
* Dependency Updates: Updated Saxon XML Engine
* Enhancement: Command-Line Tool: MARC=>JSON; JSON=>MARC processes added to the command-line tool
* Enhancement: continued updates to the Automatic updater (due to my webhost continuing to make changes)
* removal of some deprecated dependencies

Mac OS

* Enhancement: Dedup Records – addition of a fuzzy match option
* Enhancement: Linked Data tweaks to allow for multiple rules files
* Enhancement: MARC Tools — addition of a MARC=>JSON processing function
* Enhancement: MARC Tools — addition of a JSON=>MARC processing function
* Behavior Change: SPARQL Browser updates — tweaks make it more simple at this point, but this will let me provide better support
* Dependency Updates: Updated Saxon XML Engine
* Enhancement: continued updates to the Automatic updater (due to my webhost continuing to make changes)
* Enhancement: Linked data enhancement — allow selective collection processing
* Enhancement: MarcEditor: Smart Character Cleaner added to the Edit ShortCuts menu
* removal of some deprecated dependencies

Couple notes about the removal of deprecated dependencies.  These were mostly related to a SPARQL library that I’d been using – but having some trouble with due to changes a few institutions have been making.  It mostly was a convenience set of tools for me, but they were big and bulky.  So, I’m rebuilding exactly what I need from core components and shedding the parts that I don’t require.

Couple other notes – I’ll be working this week on adding the Edit Shortcuts functionality into the Mac versions task manager (that will bring the Windows and Mac version back together).  I’ll also be working to do a little video recording on some of the new stuff just to provide some quick documentation on the changes.

You can download from the website: http://marcedit.reeset.net/downloads or assuming my webhost hasn’t broke it, the automatic downloader.  And I should not, the automatic downloader will now work differently – it will attempt to do a download, but if my host causes issues, it will automatically direct your browser to the file for download following this update.

–tr

 Posted by at 9:07 am
Nov 142016
 

I posted an update to the Linux and Windows versions of MarcEdit.  I had hoped to finish work on the Mac as well, but I have, I would guess, about 5 hours of interface work to finish before that is completed and ready to be made available.  I’ll be endeavoring to get that completed over this next week during my spare time in the evenings.  I think, I should be able to have it done by Wed.

The updates really are more refinements to existing functionality.  The full list:

  • Enhancement: Edit Shortcuts; added a new option to clean smart characters.
  • Enhancement: Edit Shortcuts (many) have been enable for use with the Task Manager. 
  • Enhancement: MARCNext; Updated the linking tool to enable individual processing in both the UI and the Console program.
  • Enhancement: Application Error message is now in a window that can be copied.
  • Enhancement: RDA Helper; GMD in linking fields will be deleted if the GMD is selected for deletion.
  • Enhancement: Tab Delimited Records; add multiple subfields (without the delimiter character) to pull multiple subfields into the same column.
  • Bug Fixes (miscellaneous).

While these updates represented enhancements to existing functionality, a few of them required significant work to implement – especially bringing some of the Edit Shortcuts into the task manager.  That required a significant rethinking of how these “shortcuts” get run to fit within the task infrastructure – but I think that I’ve got these set.  I will point out, that not all the edit shortcuts have been enabled.  In the task manager, you will see placeholders for the Clean Smart Characters and the Math Functions.  These were not integrated yet into the tool – I’ll complete these with an update around the same time I update the Mac version of MarcEdit – but I put this out now because folks on the listserv wanted the Clean Smart Characters function sooner than later.   

I’ve create a couple of videos to note the new functionality.  These should give a better idea as to how these refinements have been implemented and give users a good idea of how these changes are made manifest.  Please see the videos below.

If you want to download the update, you should be able to download via MarcEdit’s automated updating mechanism or through manual download at http://marcedit.reeset.net/downloads.

Questions?  Feel free to let me know.

–tr

 Posted by at 9:40 am
Oct 092016
 

This round of MarcEdit updates focused on the Task Manager/Task Editing.  After talking to some folks, I really tried to do some work to make it easier for folks when sharing network tasks.  Change logs:

Windows/Linux:

  • Enhancement: Task List will preserve a back up task list before save, and will restore if the original task list is deleted or zero bytes.
  • Enhancement:Task List: Added .lock files to prevent multiple users from editing files on the network at the same time.
  • Enhancement: Task List: Updated Task Manager/process to remove all file paths.  Please see: http://blog.reeset.net/archives/2004
  • Enhancement: COM Object additions.

MacOS:

  • Bug Fix: Task Manager: when cloning a task that had been edited, a leak occurred which could also corrupt the task list
  • Bug Fix: MarcEditor: Error occurred if autosave wasn’t enabled.  This has been corrected.
  • Enhancement: Task List will preserve a back up task list before save, and will restore if the original task list is deleted or zero bytes.
  • Enhancement:Task List: Added .lock files to prevent multiple users from editing files on the network at the same time.
  • Enhancement: Task List: Updated Task Manager/process to remove all file paths.  Please see: http://blog.reeset.net/archives/2004

 

Downloads can be found at: http://marcedit.reeset.net/downloads.

–tr

 Posted by at 11:02 pm
Oct 042016
 

Below are two messages from a conversation on the MarcEdit list around the Task Manager/Task Functionality in MarcEdit.  I’ve been discussing some changes to the way that this works – particularly related to how the networked task management works.  Since this is a widely used function of the application, I’m pulling these two messages out and making them available here.  If you see something that gives you pause, let me know.

–tr

********************************************************************************************************************************************

Following up – I spent a bit of time working on this and as far as I can tell, everything now works as I’ve described below.  This definitely complicates the management of tasks a bit on my end – but after doing a bit of work – here’s the results.  Please, look over the notes here and below and let me know if you see any potential issues.  If I don’t hear anything, I’ll include this in my update over the weekend.

Simplifying Network Sharing:

When you setup network sharing, the tool now recognizes if your path is a networked path (or something else) and will automatically do a couple things if it’s a networked path:

Adds an import option to the Preferences (option only shows if your path is a network path)

clip_image001

If you click the copy option, it will create a .network option in the local tasks folder and then move a copy of all your items into the network space and into the _tasks.txt file on the network.

clip_image002

On startup, MarcEdit automatically will update your .network task folder with the current data in the defined network folder.

When off-line, MarcEdit automatically will use the data in the .network folder (a local cached copy of your networked data)

When off-line and using a networked path, if you select the task manager, you will see the following:

clip_image003

When you have a networked folder, MarcEdit creates the local .network cache as read-only.  You’ll see this is enforced in the editor.

Changes to the _tasks.txt file – file paths are no longer embedded.  Just the name of the file.  The assumption is that all files will be found in the defined tasks directory.  And the program will determine if there is an old path or just a filename, and will ignore any old path information, extracting just the filename and using it with the defined task paths now managed by the program.

Within the TASKS themselves, MarcEdit no longer stores paths to task lists.  Like the _tasks.txt file, just the name of the task file is stored, with the assumption being that the task list will be in the task defined file.  This means imports and exports can be done through the tool, or just by copying and pasting files into the managed task folder.

Finally – in Windows, networked drives can cause long delays when they are offline.  I’ve created a somewhat novel approach to checking this data – but it means that I’ve set a low timeout (~150 mms) – that means if your network has a lot of latency (i.e., not responsive), MarcEdit could think it’s offline.  I can make this value configurable if necessary, but the timeout here really has an impact on parts of the program because the tasks data is read into the UI of the Editor.  In my testing, the timeouts appear to be appropriate. 

These changes have just been implemented on the windows/linux side right now – I’ll move them to the mac version tonight. 

If you’d like to try these changes, let me know – I won’t release these till the weekend, but would be willing to build a test build for folks interested in testing how this works. 

Finally, I know how much the tasks function is used, so I’m giving time for feedback before going forward.  If you see something described that you think may be problematic, let me know.

Thanks,

–tr

From: Terry Reese
To: MARCEDIT-L@listserv.gmu.edu
Cc:
Subject: Proposed TASK changes…Feedback requested. [Re: Saving tasks lists on the network….]

Following up on this – I’m looking at a behavior change and want to make certain that this won’t cause anyone problems.  Let me explain how the tasks works currently (for access) and how I’ve reworked the code.  Please let me know if anyone is using a workflow where this might be problematic.

Current:

When a task is created or cloned, the task and its full path is saved to the _tasks.txt file (in your local preferences).  Example:

Main task with a whole lot of text is right here               C:\Users\reese.2179\AppData\Roaming\marcedit\macros\tasksfile-2016_03_18_052022564.txt         SHIFT+F1
AAR Library         C:\Users\reese.2179\AppData\Roaming\marcedit\macros\tasksfile-2016_03_18_052317664.txt        
TLA        C:\Users\reese.2179\AppData\Roaming\marcedit\macros\tasksfile-2016_04_19_113756870.txt        

This is a tab delimited list – with the first value being the name, the second value the path to the task, the first being a shortcut if assigned.  The task naming convention has changed a number of times through the years.  Currently, I use GUIDs to prevent any possibility of collision between users.  This is important when saving to the network.

When a user saves to the network, internally, the tool just changes where it looks for the _tasks.txt file.  But data is still saved in that location using the full network path.  So, if users potentially have different drive numbers, or someone uses a UNC path and someone else doesn’t, then you have problems.  Also, when setting up a network folder, if you have existing tasks, you have to import and export them into the folder.  Depending on how deep your task references are, that may require some manual intervention.

If you are off the network and you save to a network path, your tasks are simply unavailable unless you keep a copy in your local store (which most wouldn’t).

Proposed:

Here’s what I propose to do.  This will require changing the import and export processes (trivial), making a change to the preferences (trivial), and updating the run code for the macros (little less trivial). 

First, the _tasks.txt file will have all filepaths removed.  Items will only be referenced by filenames.  Example:

Main task with a whole lot of text is right here    tasksfile-2016_03_18_052022564.txt               SHIFT+F1
AAR Library         tasksfile-2016_03_18_052317664.txt      
TLA        tasksfile-2016_04_19_113756870.txt      

The program already sets the taskfile path so that it can run tasks that are on the network, this will continue that process of normalization.  Now, no file paths will be part of the string, and access to task locations will be completely tied to the information set in the preferences. 

Pros:

  • This will make tasks more portable as I no longer have to fix paths.
  • Makes it easier to provide automatic import of tasks when first setting up a network location
  • Allows me, for the first time, to provide local caching of networked caches so that if a user is configured to use a network location, is offline, their tasks will continue to be available.  At this point, I’m thinking caches would be updated when the user opens MarcEdit, or makes a change to a network task.  The assumption, for simplicities sake, is that if you are using a network drive, you cannot edit local networked tasks – and I’ll likely find a way to enforce that to avoid problems with caching and updates.

Cons:

  • If you use a networked task, you won’t be able to edit the offline cache when you are offline.  You’ll still have access to your tasks (which isn’t true today), but you won’t be allowed to change them.
  • This isn’t possible with the task manager, but in theory, someone could code tasks to live in locations other than the places MarcEdit manages.  This would go away.  All tasks would need to be managed within the defined network path, or the local tasks folder.
  • Using GUIDs for filenames, it shouldn’t happen – but there is always the theoretical chance of duplicate files being created.  I’ll need to ensure I keep checks to make sure that doesn’t happen, which complicates management slightly.

Second – Tasks themselves….

Presently, when a task references a task list, it uses the full path.  Again, the path will be removed to just the filename.  The same pros and cons apply here as above.

For users using tasks, you honestly shouldn’t see a difference.  The first time you’d run the program, MarcEdit would likely modify the tasks that you have (network or otherwise) and things would just go as is.  If you are creating a new networked folder, you’d see a new option in the locations preference that would allow you to automatically copy your current tasks to the network when setting that up.  And, if you are offline and a networked user, you’ll find that you now have access to your networked tasks.  Though, this part represents one more behavior change – presently, if you have networked tasks, you can take yourself offline and create and run tasks local to you.  In the above format, that goes away since MarcEdit will be caching your network tasks locally in a .network location.  If the current functionality is still desired (i.e., folks have workflows where they have are connected to the network for shared tasks, but disconnect to run local only to them tasks), I may be able to setup something so that the task runner checks both the .network and local task directories.  My preference would be to not to, but I understand that the current behavior has been around for years now, and I really would like to minimize the impact of making these changes.

–tr

 Posted by at 5:08 pm

MarcEdit Updates

 MarcEdit  Comments Off on MarcEdit Updates
Oct 032016
 

The following updates have been made to all versions of MarcEdit:

MarcEdit MacOS:

1.9.20
************************************************
** 1.9.20
************************************************
* Update: Linked Data Rules File: Rules file updated to add databases for the Japanese Diet library, 880 field processing, and the German National Library.
* Enhancement: Task Manager: Added a new macro/delimiter.  {current_file} will print the current filename if set.
* Enhancement: Task Manager: Added a new macro/delimiter.  {current_filename} will print the current filename if set.
* Bug Fix: RDA Helper: Abbreviation expansion is failing to process specific fields when config file is changed.
* Bug Fix: MSXML Engine: In an effort to allow the xsl:strip-whitespace element, I broke this process.   The work around has been to use the saxon.net engine.  However, I’ll correct this.  Information on how you emulate the xsl:strip-whitespace element will be here: http://marcedit.reeset.net/xslt-processing-xslstrip-whitespace-issues
* Bug Fix/Enhancement: Open Refine Import: OpenRefine’s release candidate changes the tab delimited output slightly.  I’ve added some code to accommodate the changes.
* Enhancement: MarcEdit Linked Data Platform: adding enhancements to make it easier to add collections and update the rules file
* Enhancement: MarcEdit Linked Data Platform: updating the rules file to include a number of new endpoints
* Enhancement: MarcEdit Linked Data Platform: adding new functionality to the rules file to support the recoding of the rules file for UNIMARC.
* Enhancement: Edit Shortcut: Adding a new edit short cut to find fields missing words
* Enhancement: OCLC API Integration: added code to integrate with the validation.  Not sure this makes its way into the interface yet, but code will be there.
* Enhancement: Saxon.NET version bump
* Enhancement: Autosave option when working in the MarcEditor.  Saves every 5 minutes.  Will protect against crashes.
* Enhancement: MarcEditor: Remove Blank Lines function

MarcEdit Windows/Linux:

6.2.315
* Enhancement: MarcEditor: Remove Blank Lines
* Bug Fix: COM Object — updating array signature
* Update: Automatic Updater: Due to a change in my webhost, automatic updating stopped working.  That should be corrected after this update is applied.

The Windows update includes fixes to the Automated Update mechanism caused due to my webhost making some unannounced changes.  User will likely need to manually download the update from: http://marcedit.reeset.net/downloads.  Following this update, this shouldn’t be necessary any longer.

–tr

 Posted by at 1:00 am

Note on Automatic Updates

 MarcEdit  Comments Off on Note on Automatic Updates
Sep 272016
 

Please note, MarcEdit’s Automated update tool will notify you of the update, but you will need to manually download the file from: http://marcedit.reeset.net/downloads.  My web host, Bluehost, has made a change to their server configuration that makes no sense, but ultimately, dumps traffic sent from non-web browsers (connections without a user-agent).  Right now, users will get this message when they attempt to download using the automatic update:

update_bluehost_error

I can accommodate the requirements that they have setup now, but it will mean that users will need to do manual downloads for the current update posted 9/27/2016 and the subsequent update — which I’ll try to get out tonight or tomorrow.

I apologize for the inconvenience, but after spending 8 hours yesterday and today wrangling with them and trying to explain what this breaks (because I have some personal tools that this change affects), I’m just not getting anywhere.  Maybe something will magically change, maybe not — but for now I’ll be rewriting the update process to try and protect from these kinds of unannounced changes in the future.

So again, you’ll want to download MarcEdit from http://marcedit.reeset.net/downloads since the automatic update download connection is currently being dumped by my web host.

 Posted by at 1:45 pm
Sep 272016
 

I’ve posted a new set of updates.  The initial set is for Windows and Linux.  I’ll be posting Mac updates later this week.  Here’s the list of changes:

  • Behavior Change — Windows/Linux: Intellisense turned off by default (this is the box that shows up when you start to type a diacritic) for new installs. As more folks use UTF8, this option makes less sense. Will likely make plans to remove it within the next year.
  • Enhancement: Select Extracted Records: UI Updates to the import process.
  • Enhancement: Select Extracted Records: Updates to the batch file query.
  • Behavior Change: Z39.50 Client: Override added to the Z39.50 client to enable the Z39.50 client to override search limits. Beware, overriding this option is potentially problematic.
  • Update: Linked Data Rules File: Rules file updated to add databases for the Japanese Diet library, 880 field processing, and the German National Library.
  • Enhancement: Task Manager: Added a new macro/delimiter. {current_file} will print the current filename if set.
  • Bug Fix: RDA Helper – Abbreviation expansion is failing to process specific fields when config file is changed.
  • Bug Fix: MSXML Engine – In an effort to allow the xsl:strip-whitespace element, I broke this process. The work around has been to use the saxon.net engine. However, I’ll correct this. Information on how you emulate the xsl:strip-whitespace element will be here: http://marcedit.reeset.net/xslt-processing-xslstrip-whitespace-issues
  • Bug Fix: Task Manager Editing – when adding the RDA Helper to a new task, it asks for file paths. This was due to some enhanced validation around files. This didn’t impact any existing tasks.
  • Bug Fix: UI changes – I’m setting default sizes for a number of forms for usability
  • Bug Fix/Enhancement: Open Refine Import – OpenRefine’s release candidate changes the tab delimited output slightly. I’ve added some code to accommodate the changes.
  • Enhancement: MarcEdit Linked Data Platform – adding enhancements to make it easier to add collections and update the rules file
  • Enhancement: MarcEdit Linked Data Platform – updating the rules file to include a number of new endpoints
  • Enhancement: MarcEdit Linked Data Platform – adding new functionality to the rules file to support the recoding of the rules file for UNIMARC.
  • Enhancement: Edit Shortcut – Adding a new edit short cut to find fields missing words
  • Enhancement: XML Platform – making it clearer that you can use either XQuery or XSLT for transformations into MARCXML
  • Enhancement: OAI Harvester – code underneath to update user agent and accommodate content-type requirements on some servers.
  • Enhancement: OCLC API Integration – added code to integrate with the validation. Not sure this makes its way into the interface yet, but code will be there.
  • Enhancement: Saxon.NET version bump
  • Enhancement: SPARQL Explorer – Updating the sparql engine to give me more access to low level data manipulation
  • Enhancement: Autosave option when working in the MarcEditor. Saves every 5 minutes. Will protect against crashes. data

Downloads are available from the downloads page (http://marcedit.reeset.net/downloads):

–tr

 Posted by at 12:10 am

MarcEdit PreUpdate Notes

 MarcEdit  Comments Off on MarcEdit PreUpdate Notes
Sep 102016
 

Three significant changes will be coming as part of MarcEdit’s Sunday update.  These impact the MARCEngine, Regular Expression Processing, and the Linked Data Platform.

MARCEngine Changes:

In May 2016, I had to make some changes to the MARCEngine to remove some of the convenience functions that allowed mnemonic processing to occur on UTF8 data or HTML entitles on MARC8 data.  Neither of these should happen, and there were issues that came up related to titles that actually included HTML entitles in the titles.  So, this kind of processing had to be removed.

Over the past few months, I’ve been re-evaluating how these functions use to work, and have been bringing them back.  Sunday will mark the reintroduction of many of these functions (though, not the HTML entity translation when it’s not specifically appropriate).  The upside, is that coupled with the new encoding work, the tool will be able to support more mixed encoding use-cases.

Regular Expression Processing

The most significant changes to the Regular Expression engine processing is how the multi-line processing works in the Replace Function.  To protect users, I’d set up the match any character “.” match all characters but a new line character.  This was done to keep users from accidently deleting data.  However, this meant that when using the multi-line processing option, it really only worked when fields were side by side.  By removing this limitation, users working with the multi-line option will now have full access to the entire record with the Regular Expression processing.  And with that, a word of warning…be careful.  The Multi-line processing is the easiest way to accidently delete bibliographic data through greedy matches.

Additionally, I’ve added an option to the Replace Function dialog that makes it easier to know that MarcEdit has this option.  Right now, users have to know that you need to add a /m to the end of your expression to initiate the multi-line mode.  You can still do that – but for users that don’t know that this is the case, a new option has been added to turn on the Multiline option (see below).

image

The MultiLine Evaluation will be enabled and selectable when the Use regular expressions option is checked.

Linked Data Platform Changes

The Linked Data Platform will be seeing two significant changes.  The first change is occurring in the code to support linking fields like the 880.  This has meant adding a new special processing instruction “linking”, which can now be used to perform reconciliation against data in these fields.  This is particularly important for Asian languages, Arabic languages, Hebrew, etc.

The second change is in the rules file itself.  I’ve profiled the 880 field, as well as a wide range of other collections.

Finally, I’ve added a note to the Main Window that helps users find their rules file for edit, as well as points to the knowledge-base articles and videos explaining the process. (see below)

image

 

Other Changes:

Other changes that will be made in this update:

  1. ISSN Report – tweaked the process due to a bug.
  2. MARCCompare – added a new output type; in addition to the HTML output, there will be a diff file output
  3. Preferences Window/Language – Added a help icon that points to the knowledge-base articles related to fonts and font recommendations.

These changes will be part of the Sunday update, and at this point, look like they will be applicable to all versions of MarcEdit (though, I do have a few UI tweaks that I need to complete on the mac side).

If you have questions, let me know.  Otherwise, these changes will be made available on 9/11/2016 9/12/2016 [evening].

–tr

 Posted by at 9:33 pm