Automating Legacy Applications

Environment: Visual C++ 6.0, Windows 2000

Introduction

Applications that are completely unaware of COM stuff may be automated. The term "Automation" in this context means the ability to control an application from outside of its process. This is useful particularly in the case of old legacy applications. Their functionality and even user interface may be upgraded with COM object(s) injected to the process. The DLL injection technique was proposed by Jeffrey Richter [1] and has already become almost conventional. Recently, John Peloquin's paper [2] reminded me about the issue and stimulated me to move a step further: to embed a COM object into a target application and, using this component, control the application from outside of its process.

Brief Technique Description

Let's assume for simplicity that our target application is running in just one thread. This application has one main frame window having one client (view) window. To automate such an application, the following steps should be taken.

• A Special Loader application injects a Plugin DLL into the target process using a remote thread. This is a worker thread, and its function is DllMain() of the Plugin DLL. After DllMain() returns, the thread will be over.
• The Plugin DLL contains also callback window procedures to subclass the frame and view windows of a target application. These window procedures include message handlers with new functionality for the target application. DllMain() of the Plugin DLL actually performs the subclassings.
• The New window procedure for, say, a frame window has a handler for an additional Plugin-specific Windows message, WM_CREATE_OBJECT. DllMain() performs the target application window's subclassing, and after that posts the WM_CREATE_OBJECT message to a frame window. Then DllMain() returns.
• Upon receiving the WM_CREATE_OBJECT message, a new frame window procedure creates a COM object and calls its method for registration with Running Object Table (ROT).
• Now, Client applications may obtain a proxy of the injected COM object via ROT, thus gaining control over the target application.

To accomplish automation task information about the target application, the window class name is required. It may be revealed by using the Spy utility, which is part of the Visual Studio installation. Another useful tool is ROT Viewer. It is also a Visual Studio utility. ROT viewer allows the developer to inspect the ROT table and therefore check the proper registration of a COM object embedded into the target process.

Test Sample

The well-known Notepad text editor is taken as the target application for automation.

The Loader.exe application is responsible for the injection of the NotepadPlugin.dll in Notepad.exe. Loader finds a running instance of Notepad (or starts a new instance in case no running instances are available), obtains its process handle, and actually performs the injection of NotepadPlugin.dll using a remote thread. (Please refer to [1] and code for details.)

A NotepadPlugin.dll is to be injected into Notepad.exe. Its DllMain() method first finds the frame and view windows of Notepad.exe and then subclasses both of them with the appropriate custom windows procedures. As it was stated above, DllMain() is running not in the Notepad.exe main thread, but in an additional thread created remotely by the Loader application. This additional thread will vanish when NotepadPlugin.dll is completely loaded. A custom WM_CREATE_OBJECT Windows message is posted to the frame window to initiate creation of the COM object for automation and its registration with ROT. The message WM_CREATE_OBJECT handler of a new frame window procedure initializes COM, creates NotepadHandler COM object, and calls its appropriate method for its ROT registration.

A COM in-process server component, NotepadHandler.dll, implements the dual-interface IHandler, specially tailored for our custom Notepad automation. Among others, IHandler has methods to register/unregister the object with the ROT.

The AutomClient.exe application is a sample of an automation client. It creates a proxy for the NotepadHandler COM object by using the component's ROT registration and actually controls the automated instance of Notepad through the IHandler interface implemented by NotepadHandler.

Running the Test

An already compiled demo is available for the test sample. Please note that before you run it, the NotepadHandler.dll COM component has to be registered with the regsvr32.exe utility. For the registration, you have to run the file Register NotepadHandler.bat, located in the demo directory. After registration, you may run the AutomationClient.exe application and press its NOTEPAD AUTOMATE button. It causes the automation of an already running instance of Notepad. If there are no running instances of Notepad at the moment, a new instance is started and automated. A message box containing "I've been Automated!" with the "Notepad" caption appears.

Now you may type some text in the automated Notepad instance and press the Copy Text button of the client application. The last text fragment you typed will appear in the edit box of the client application. To simplify things, only actually typed characters and symbols are copied, not copied-and-pasted text. Pressing the Find and Append Menu buttons of the client causes a corresponding Notepad response. On exiting from Notepad, a "Bye-Bye..." message box generated by NotepadPlugin.dll on Notepad's behalf amuses the user.

To compile projects of the NotepadAuto workspace, load NotepadAuto.dsw to VC++ Studio. Please, make sure that files psapi.h, psapi.lib, and psapi.dll are installed on your machine, and set the proper path to them in your test projects. If there are no such files, they may be loaded from here (11 Kb).

The psapi library contains the EnumWindows() and EnumProcesses() functions required for a systemwide search for particular windows and processes. It is also assumes that Notepad.exe may be found in your [Windows]\System32 directory. After the above arrangements have been made, you may build the _Build_All_Projects project (the rest are its dependants).

The AutomationClient.exe application should be run to test the sample. But, before you start it under Visual Studio, please open the Project Settings dialog (by activating the Project/Settings... menu item) and on its Debug tab, set the Working directory containing all output EXE and DLL files. Then, run the AutomationClient.exe application. By pressing the NOTEPAD AUTOMATE button, you internally start the Loader.exe application. The latter starts Notepad and automates it. A message box containing "I've been Automated!" is thrown by NotepadPlugin.dll on behalf of Notepad. Alternatively, you may run Notepad manually before the AutomationClient.exe or Loader.exe. In this case, Loader automates the already running instance of Notepad.

Further Development

Conclusion

Usage of COM objects in a DLL injected into a target application allows the developer to automate legacy software, thus extending the duration of its active service.

References

[1] Jeffrey Richter. Advanced Windows. Third edition. Microsoft Press, 1997.
[2] John Peloquin. Remote Library Loading.

Downloads

Download demo project - 19 Kb
Download source - 32 Kb
Psapi library files - 11 Kb

Note

This paper presents just a technique to automate an application from outside of its process. Anysoft Inc. has developed the Digital Cortex, a product addressing similar problems, among many others.