MarcEdit 7: Continued Task Improvement; Part 2

By reeset / On / In MarcEdit

Last week, I discussed some of the work I was doing to continue to evaluate how Task processing will work in MarcEdit 7.  To do some of this work, I’ve been working with a set of outlier data who’s performance in MarcEdit 6.3 left much to be desired.  You can read about the testing and the file set here: MarcEdit 7: Continued Task Refinements

Over the week, I’ve continued to work on how this data is processed, hoping to continue to move the processing time of this data from almost 7 hours in MarcEdit 6.3 to around 1 1/2 hours, and I’ve been able to do that and more.  My guess was that by adding targeted pre-processing statements into the task processing queue, I could improve processing by only running the task processes that absolutely had to be run.  In this case, I had 962 task actions, but on any given record, maybe 20-30 needed to be run.  By adding a preprocessing step, I was able to move the processing time from 2+hours to 25 minutes.  My guess is that I’ve reached the ceiling in terms of optimizations, but I can live with this.  Of course, over the next few days, what I’ll need to do is validate that these new changes don’t cause the program to miss processing a step that should be run.  Generally, I’ve setup the preprocessing steps so that it will fall back to running the task when in doubt.

–tr

Leave a Reply

Your email address will not be published. Required fields are marked *