Final Report of DMP Research Project for Sarah E. Smith - Summer 2003

Sarah E. Smith
College of Engineering
North Carolina State University
Raleigh, NC 27695
sesmith5@unity.ncsu.edu

1.0 Introduction

This summer I worked with Dr. Laurie Williams at North Carolina State University on several different projects and attended the Agile Development Conference in Salt Lake City, Utah. These projects are divided up into two different categories: academic and industry activities. My main academic activity included working on tutorials that are going to be used in CSC 326, NCSU's Software Engineering class, in the fall of 2003. These tutorials are especially important to me because I am a CSC 326 TA for the fall. On the industry side, I worked two days a week at Nortel, with the NCS Team. I helped implement a system to insert test results into their database, and then evaluated different automated GUI testing tools for later use of GUI testing for the team.

2.0 Tutorials

This year in Software Engineering the students are going to be using Eclipse and Rational XDE as their IDE tools. Eclipse is an open source IDE for Java development. At its lowest level, Eclipse allows users to create and run Java Projects and use JUnit. The functionality of Eclipse can be expanded with the use of plug-ins that can be written by members of the Eclipse community.

Rational XDE is a IDE tool developed by IBM. Rational XDE incorporates the look and feel of Eclipse, but has many more options. XDE allows users to create UML diagrams to model their code, which is an important aspect of what is taught in Software Engineering.

The eight tutorials that were written included: Introduction to Rational XDE, Class Diagrams, Sequence Diagrams, State Diagrams, Use Case Diagrams, Software Metrics, JUnit, and FIT. The tutorials were written to demonstrate how to use the tools that Rational XDE and Eclipse provide. The FIT tutorial was written to show the testing framework at its most basic level.

Two main examples were used to write the tutorials. For the Introduction to Rational XDE and the Class Diagrams tutorial the Greeter example was used. Greeter is a variation of the HelloWorld program and a very simple introduction to the intricacies of Rational XDE. The rest of the tutorials used the GasPump example to demonstrate how to use the different tools described. The GasPump concept was large enough so that there was an example of each aspect we wanted to teach in the tutorial, but small enough so that the tutorial could be completed in a certain time period (around 30 minutes).

3.0 Nortel Activities

At Nortel I worked with the NCS Team which develops software for the BCM and Contivity devices. I worked very closely with Jason Pyles and Julie Gupton, the Verification Engineers, on the different testing aspect of the project. My first assignment at Nortel was to create a system to store automated test results in their database. It was initially estimated that this project would take two weeks, but due to some unforeseen complications it took five weeks.

The next part of my project at Nortel was to evaluate automated GUI testing tools and their ability to handle 7specific activities in their software. I evaluated several different tools including Abbot and WinRunner.

3.1 Difficulties with GUI Testing

GUI testing itself, at a user-testing level, is not a difficult concept, but with larger and more complex GUI programs being written it becomes harder to test these GUI's [1,2]. Writing and maintaining hand-written GUI tests is very time consuming [1]. Automated testing of GUI's is even more complex. Some problems with automated GUI testing is the size and complexity of the GUI itself. There are many different states in a GUI and different arrangements of GUI actions can lead to different states [2,4].

White states that any automated GUI testing tool should include:

1. record and playback of physical events in the GUI,
2. screen image capture and comparison, and
3. shell scripts to control and execute test runs of the GUI [4].

The above description of an automated testing tool allows a user to interact with the GUI to write testing scripts that can be reused later. However, there are problems in the GUI testing tools themselves that need to be solved. The most important is the map of the GUI components and how the object are named and selected by the GUI testing tool. If a GUI testing tool relies on the location of a mouse to perform a certain event, any resizing or movement of the GUI window will cause errors when the test is replayed.

3.2 Analysis of GUI Testing Tools

There where two main tools that were analyzed: Abbot and WinRunner. Abbot is an open source tool, created by Timothy Wall, which has a capture and playback set up. It records the scripts based on the location and name of the components in the GUI. Abbot runs the tests on a specific program that you load up through the Abbot program. Abbot is a very good tool, but there were problems with NCS Team's software. A major problem was that Abbot did not recognize some components.

WinRunner is a commercial tool created by Mercury Interactive. WinRunner can record scripts on anything, including events on the desktop. WinRunner is a more robust system and was able to recognize all components in the software. WinRunner was also able to pass the 7 baseline tests that needed to be passed. There are some particularities associated with WinRunner, as with all software, but they are easy to work around and the support is better.

We decided to use WinRunner for future automated GUI testing.

4.0 Future Work

The projects that I worked on this summer were mostly terminal projects. There is the possibility of future research in automated GUI testing tools. However, I am moving on to work with Hema on a Traceability paper. I am also considering writing my thesis on Traceability.

5.0 Conclusion

This summer with the DMP has been a wonderful opportunity. I was especially lucky to gain experience in both academics and industry. I had already been accepted to Graduate School before I applied for this program, but it helped to reinforce my interest in higher education and research. I am now seriously considering getting a Doctorate in Computer Science in the future. I have already switched from a MCS (Master of Computer Science, a terminal degree) to a MS in Computer Science.

Sources

[1] Memon, Atif, Martha E. Pollack, and Mary Lou Soffa. Hierarchical GUI Test Case Generation Using Automated Planning. in IEEE Transactions on Software Engineering. Vol. 27, No. 2, Feb 2001.

[2] Memon, Atif, Martha E. Pollack, and Mary Lou Soffa. Using a Goal-Driven Approach to Generate Test Cases for GUI's. in Proc. 21st International Conference Software Engineering. pp. 257-266. May 1999.

[3] White, Lee and Husain Almezen. Generating TEst Cases for GUI Responsibilities Using Complete Interaction Sequences. IEEE. 2000

[4] White, Lee. Regression Testing of GUI Event Interactions. In Proceedings of the International Conference of Software Maintenance. pp. 350-358. Washington, Nov. 4-8 1996. IEEE Computer Society Press.