While continuing to add the finishing touches to the Watch Folders — I realized I needed to provide a couple of additional contextual notes.
Watch Folder Behavior
C# provides a couple of different methods to make this work. Probably the most straight forward is what is called a FileSystemWatcher event that can be tagged to a specific path. This is how I originally considered implementing this functionality — but it caused some very specific problems.
First — I would like users to keep access to their original source data. In the process, the tool will move data from the watched folder to: [watched_folder]\_originals\[filename]. The moving of the items into the _originals folder caused the FileSystemWatcher event to fire again. This was causing a lot of confusion and extra event processing.
Second — This made it very difficult to thread the processing. Within this event — once the event was thrown, it would kick off the evaluation process that, being tied to an event on the GUI, was a little more difficult to segregate into its own space (you can do it, but it makes tracking the process more difficult)
Third — these events stay active for the file of the service. I’m assuming that once running, users may add new watchers or edit existing ones. In this model, those changes wouldn’t occur until the service was restarted. This seems undesirable.
To address these issues, I’ve implemented a timer. The tool will re-evaluate all Watcher Criteria and then re-evaluate folders/files every 15 minutes or schedule all watchers to be run at a specific time (this seems like the best option in most cases). Once the timer fires, if a watcher is run, the timer will pause and a flag will be set in the program that will notify Windows that an active process is occurring. If you try to restart windows while this is happening, you’ll get the window that shows a program is in process and will ask you to wait for the completion. The watcher will then push all watched processes into a thread pool. This way, 3 watchers can be run at any time. Threads will be given low priority — this way, Windows and the CPU will know speed isn’t important and that other operations can be prioritized. The idea is that this will keep the service from impacting general system performance (this works in testing — interested to see when it gets to a wider install base and system types). After the watchers complete, the system is notified that the program is no longer busy (so if you are restarting, it will be released to continue, if not — it will ensure you never see the message) and will then re-enable the timer to run either at a specific interval or time (depending on your settings).
How will I know things have happened
If you are running on Windows 10 — the Watcher service has been tied into the notification/toast system. So you’ll see messages show up in the notifications area:
If you have an older version of Windows, these will show up as ballon tips that will display when a watcher has completed a task. As I implement this on the Mac side — I’m hoping to push results into the notification/toast area.
Additionally, if you right click on the MarcEdit Service Icon, there will be an option to load the log file. This will provide a log of each file processed and at what time. Watcher logs will rotate every 24 hours, and will be purged every 5 days. If there is an interest having these archive (zip) rather than delete, I can look at adding that functionality.
Updating MarcEdit
This is the tricky part. MarcEdit cannot update while the Watcher is running. So, one of the things that will needed to be added to the installer (and this is proving to be tricky) is a messaging tool that can tell the Watcher when it needs to shutdown, and a listener that allows the Watcher to tell the installer to wait until a process has completed. This is the last piece that needs to be completed before it can be made available.
If you have questions, let me know.
–tr