The MultiMahjong Project
K-Team
Team Members:
- Joanna Araminta
- Victor Leung
- Joel Brakey
- Michael Hart
- Dean Cortinovis
- Long Tang
|
(Project Manager)
(Technical Researcher)
(Client Liaison Officer, Web Site Manager)
(Secretary, Web Site Manager, Backup Manager)
(Technical Researcher)
(Technical Researcher and Risk Manager) |
Abstract:
MultiMahjong is a single/multi player Mahjong computer game. This document formally specifies the scope, techniques and resources used for testing the MultiMahjong system.
Maintainer:
Version Control:
Table Of Contents
- Introduction
- Overview of Software Testing Strategy
- Unit Testing
3.1 General Strategy
3.2 Black-Box Test Cast Selection
3.2.1 Equivalence Partitioning
3.2.2 Boundary Value Analysis
3.2.3 Random Selection
3.3 Conduct of Unit Tests
3.4 Unit Test Report
- Integration Testing
4.1 General Strategy
4.2 Order of Integration
4.2.1 UserGUI Module Integration
4.2.2 Game Module Integration
4.2.3 CO Module Integration
4.2.4 MultiMahjongServer Module Integration
4.2.5 UserGUI/Game Module Integration
4.2.6 UserGUI/MultiMahjongServer Module Integration
4.2.7 Game/MultiMahjongServer Module Integration
4.3 Test Case Selection
4.4 Conduct of Integration Tests
4.5 Integration Test Report
- System Testing
5.1 General Strategy
5.1.1 Validation Testing
5.1.2 Performance Testing
5.1.3 Acceptance Testing
5.1.4 Installation Testing
5.2 Failure of System Tests
1. Introduction
This document formally specifies the scope, techniques and resources used by K-Team to ensure the correctness of the MultiMahjong product with respect to the Software Requirements Specification document (http://www.perceptek.com.au/kteam/docs/srs.html).
MultiMahjong is a product consisting of two programs - a MultiMahjongServer and a MultiMahjongClient. This Server/Client architecture will allow up to 4 players to play Mahjong against each other over a TCP/IP network. The MultiMahjongClient program will also allow 1 player to play in a stand-alone mode.
As any game of Mahjong requires 4 players to play, and there may not be 4 people available for a network game, the game will allocate enough computer opponents to make up the required 4 players. In a single player game, the user will play against 3 computer opponents.
To play the game, users will use the MultiMahjongClient. The MultiMahjongServer is to reside on a TCP/IP server and will communicate with MultiMahjongClients.
The client requires the product for commercial purposes. The MultiMahjongClient program is to be sold to potential users and the MultiMahjongServer is to initially reside on a server owned or operated by the client.
The client for this project is:
Our team for the project is called K-Team and consists of:
Joanna Araminta (jiar)
Victor Leung (vhle)
Joel Brakey (jebr)
Michael Hart (mwhart)
Dean Cortinovis (dcort)
Long Tang (lqkt) |
Ph: 9889 4423 (Project Manager)
Ph: 9706 1560
Ph: 9859 6038
Ph: 9859 5419
Ph: 9798 2684
Ph: 9540 8992 |
The supervisor for the project is:
| Anthony Senyard (anthls) |
Ph (W): 9344 1940
Ph (H): 9417 2839 |
2. Overview of Software Testing Strategy
Testing begins at the lowest level of abstraction and moves progressively to higher levels of abstraction. As the coding for each class progresses, informal testing is carried out by the author in order to satisfy themselves that the class methods so far defined are exhibiting the expected behaviour. This occurs in lieu of the actual coding, and may be considered as a part of the coding process. As such, no guidelines have been set for the conduct of informal testing.
Once the coding for a class has been completed, it undergoes unit testing, which ensures that the class functions properly as a discrete, independent unit (Section 3). Unit tested classes are then assembled together incrementally to eventually form the complete software product. Integration testing is conducted throughout the entire assembly process to uncover errors associated with interfacing (Section 4). Finally, system testing is carried out to demonstrate that the software product conforms with the requirements as set out in the Software Requirements Specifications. This is described in Section 5 of this document.
3. Unit Testing
Unit testing focuses on the smallest compilable program unit. In an object oriented context, the smallest unit is a class, which encapsulates a number of class attributes and methods which operate on the data. Each class must pass all of its unit tests before it can be integrated into the rest of the system. This helps to ensure that errors in the code are more readily uncovered and isolated.
3.1 General Strategy
Once the author of a unit has completed coding, they will be required to conduct black-box unit tests on the class. As part of the Booch method, specific roles and responsibilites have been assigned to each class. Unit testing will be aimed to ensure that the class fulfills all of its identified responsibilities and that these are carried out correctly.
It is important to note here that unit testing in the object oriented context differs from ordinary testing procedures of a single module. This is due primarily to the fact that we can no longer test a single method in isolation as the behaviour of the method varies with the class it is encapsulated within. Multiple methods will be contained within an individual class, and therefore each will need to be tested for correctness. The context in which a method is used by other classes will also vary and therefore it will be necessary to test for each. The success of a test will therefore depend on both the method's output and on the state of the class' attributes when it is completed.
3.2 Black-Box Test Case Selection
3.2.1 Equivalence Partitioning
Being a game of chance, there will be instances in Mahjong where the input space is essentially infinite. Since exhaustive testing of every possible combination is clearly not a viable option, we need some method of reducing the number of test cases required.
Equivalence partitioning is expected to achieve this. The general procedure will be to place a certain range of input into classes and test individual input from these classes. Once one member of a class is found to be correct, it is assumed that this must be true for the remaining values. This eliminates the need to duplicate tests. Since this technique covers a wider area than other data selection techniques, there will be a high probability of detecting bugs. Equivalence testing will cover classes of valid inputs and some common invalid inputs. This will ensure that fatal errors are not caused by invalid inputs.
3.2.2 Boundary Value Analysis
Boundary value anaylsis will be applied in conjunction with equivalence partitioning. This form of testing is based on exercising the boundary of input conditions and see how well the tested unit can handle these. Where a range is specified as the input condition, both bounded values of the range, and values above and below these values are tested. All data structures that have a boundary will be tested at the lower and upper limit of the structure.
3.2.3 Random Selection
Random selection occurs when test data is randomly selected from the input domain of the test unit. While random selection would not be as effective as specific input conducted with the above strategies, this form of test case selection may still be useful in developing multiple and large amounts test data.
3.3 Conduct of Unit Tests
The author of the class will begin testing by selecting test cases based on the equivalence partitioning strategy. It is expected that this would lead to a rapid reduction in the number of test cases required to achieve complete coverage of the class' methods, ensuring that the class is fulfilling each of its assigned roles and responsibilities correctly.
In addition equivalence partitioning, boundary value analysis will be performed whenever we encounter a method where the input condition is restricted to some arbitrary range. Any assumptions that justifies the existence of fixed-size data structures, such as arrays, must also be closely examined.
It is recognised that there are certain classes that may require large amounts of test data that are not well-suited to equivalence partitioning. This is exemplified by the mahjong wall, which contains 136 tiles arranged in random order. Each combination of tiles may be considered different for the purposes of test case selection. For these classes, random selection may be an effective test case selection strategy.
Although full-coverage white-box testing will not be conducted, authors are encouraged to examine class internals and ensure that the integrity of the data is maintained as it flows through the class.
The author will produce an Unit Test Report following the completion (satisfactorily or otherwise) of relevant test cases.
3.4 Unit Test Report
The Unit Test Report will be stored in the repository at:
A sample Unit Test Report appears below. This is also available as a template from:
| Unit Test Summary |
| Unit: class Tile extends Component implements Comparable |
| TestID |
Passed/Failed |
| Tile01 |
Passed |
| Tile02 |
FAILED |
| Tile03 |
Passed |
| ... |
... |
| ... |
... |
| 15 test cases: passed 14, failed 1 |
| TestID: Tile01 |
Last completed: Wed Sep 22 09:51:03 EST 1999 by vhle |
| Test Description |
| This test case verifies firstly that the filename corresponding to the tile's face image is formed correctly, and secondly that the image is fetched and returned correctly. |
| Inputs or Test Data |
Using the loop below, we displayed the face image of the all the tiles from the character, circle and bamboo suits, and from the wind and dragon honour tiles. A couple of invalid cases are also included like Tile(5, 6) (there are only 3 kinds of tiles in suit 5 - the dragon honour tiles).
for (int i = 1; i <= 5; i++) {
for (int j = 1; j <= 9; j++) {
try {
Thread.sleep(500);
} catch (InterruptedException e) { }
Tile tile = new Tile(j, i);
ImageIcon icon = new ImageIcon(tile.faceImage());
JLabel label = new JLabel(icon);
frame.getContentPane().add(label);
frame.getContentPane().validate();
}
}
|
| Expected Output |
| The correct face images for all valid tiles should be displayed properly onscreen. The retrieval of the face images of invalid tiles should not cause catastrophic failure. |
| Actual Output |
| Visual inspection of the display shows that the face images for all valid tiles are displayed properly. Specifically, the insertion of System.out.println statements within the faceImage method verifies that the filename for the image is formed correctly. When face tile is called on an invalid tile, the image returned is an empty (1 pixel) image. This is desirable as fetching the image of an invalid tile does not cause the program to fail catastrophically. |
| Additional Comments |
| None |
4. Integration Testing
Various classes must be combined together with the aim of eventually building the complete system. Such integration must occur incrementally, with integration testing performed at each step to uncover errors due to interfacing the classes. To qualify as a candidate for integration, each of the classes involved must have satisfactorily completed unit testing.
Side-effects arising from the addition of new classes to the system are detected with a series of regression tests. This section describes the approach to be taken with respect to the design and conduct of integration testing.
4.1 General Strategy
As an integral part of Stages 2 and 3 of the Booch method's macro process, detailed use cases have been developed to cover the various interaction between the user and the system, as well as the interactions between the objects in the system. Scenario diagrams are subsequently derived from these use cases, representing much the same information in a more concise, diagrammatic form.
Scenario diagrams are invaluable for integration testing as it shows clearly the downstream effects of one input or event to the system. A viable strategy for integration testing is therefore to define and integrate all the classes required to respond to this one input or event. This set of classes forms what is known as a thread, and the integration testing approach chosen is unsurprisingly referred to as thread-based testing. This approach is particularly relevant to our choice of Java as the language in which to implement the system, seeing that both are inherently event-driven.
Please note that in the context of integration testing, a "thread" is defined as the minimal set of classes required to respond to a single input or event to the system. This is completely unrelated to the concept of multi-threaded programming.
4.2 Order of Integration
The order of integration must be defined prior to coding so that we can best determine the order in which the classes are to be implemented. Since we have chosen thread-based testing as our integration testing approach, the order of integration is determined by examination of available scenario diagrams (please refer to the Software Design Document).
Since the software system is comprised of a number of discrete modules, a two-step integration strategy will be adopted. The classes which make up each module are first incrementally assembled together. Stubs and drivers will be used so that each module can be tested as an autonomous entity. Intermodular connections are then formed as the stubs and drivers are replaced by "real" classes from external modules.
4.2.1 UserGUI Module Integration
- Displaying the main application window: UserGUI
- Bringing up a dummy game type selector: (1) + GameTypeSelector
- Reacting to input from game type selector (single player mode): (2) + Player (fields and access methods only)
- Displaying player information: (3) + TabbedPane
- Displaying game information: (4) + StatusPane
- Displaying online help: (5) + OnlineHelp
- Displaying a blank playing screen: (6) + PlayArea
- Displaying individual tiles on the playing screen: (7) + Tile
- Displaying sets of tiles on the playing screen: (8) + Set
- Displaying action buttons for player interaction: (9) + ActionPane
4.2.2 Game Module Integration
- Underlying data structure: Tile + Set
- Reacting to text-based user input: (1) + Game
- Shuffling and dealing out tiles: (2) + Wall + Hand
- Checking and setting fishing and mahjong flags: (3) + CombList + Comb
- Permitting game actions and determining players' turns: (4) + Observer + LocalReferee
- Determining payout and scores: (5) + PayOut + PayOutInfo
4.2.3 Game and CO Module Integration
- Analyzing concealed hand to get optimal discard tile: ComputerOpponent overrides method int getDiscard(bollean, Moves) in its base class Game.
- Determining whether computer opponent can call Mahjong, KONG, PUNG, CHOW or draw wall' tile: ComputerOpponent overrides method int getPlayerSelection(Tile, Next) in its base class Game.
4.2.4 MultiMahjongServer Module Integration
- Creating RMI object: Registrar + MultiMahjongServer
- Adding a list of mini games: (1) + GamesList + MiniGame
- Instantiating RemoteReferees: (2) + RemoteReferee
4.2.5 UserGUI/Game Integration
- Starting a single player game: UserGUI + Game + LocalReferee
- Obtaining game actions from users graphically: (1) + Human + HumanDiscard + HumanSelection
- Reacting to game actions: (2) + RetrieveMove + RetrieveRobKong + RetrieveSelection
- Displaying details of payout: UserGUI + PayOutGUI
4.2.6 UserGUI/MultiMahjongServer Integration
- Obtaining information for starting a multi player game: UserGUI + GameTypeSelector + MultiGameStarter
- Starting a multi player game: (1) + Registrar + RemoteReferee + PlayersObserver
- Obtaining information for joining a multi player game: UserGUI + GameTypeSelector + MultiGameJoiner
- Joining a multi player game: (3) + Registrar + RemoteReferee + GamesObserver
4.2.7 Game/MultiMahjongServer Integration
- Using a RemoteReferee as a Referee: Game + RemoteReferee
- Creating triangle relationship for references: (1) + Observer
4.3 Test Case Selection
To qualify as a candidate for integration, each of the classes involved must have satisfactorily completed unit testing. Nevertheless, regression tests must be run after each integration step to ensure that no adverse side-effects have been introduced into the existing code. It is neither necessary nor desirable to re-run every single unit test. Rather, the tester should concentrate on software components that have been changed, or those that are likely to be affected by the change.
Integration testing will be achieved through a series of black-box tests which concentrate on the data flowing across the interfaces. The test cases will consist of both valid and invalid test data. However, it is recognised that some methods may make certain assumptions about the data being passed across an interface. Such assumptions must be clearly documented in the method description, and all caller objects must ensure that these assumptions hold true under all circumstances.
4.4 Conduct of Integration Tests
Once all the classes in the thread to be tested have been identified, the authors responsible for their respective classes will cooperate to integrate the classes and to create initial test cases based upon the relevant scenario diagram(s). The authors should also indicate the regression tests that should be run, based upon their knowledge of the code that has been modified during the integration process.
A tester for the thread will be allocated during regular team meetings. The tester is responsible for reviewing and enhancing the initial test cases, and for ultimately conducting integration testing on this thread. The tester will also be responsible for running the regression test suites as indicated by the authors. The tester will produce an Integration Test Report following the completion (satisfactorily or otherwise) of relevant test cases.
4.5 Integration Test Report
The Integration Test Report will be stored in the repository at:
A sample Integration Test Report appears below. This is also available as a template from:
| Integration Test Summary |
| Existing classes: UserGUI, GameTypeSelector, Player, TabbedPane |
| Added: class StatusPane extends JPanel |
| TestID |
Passed/Failed |
| StatusPane01-i |
Passed |
| StatusPane02-i |
FAILED |
| StatusPane03-i |
Passed |
| ... |
... |
| ... |
... |
| 15 test cases: passed 14, failed 1 |
| TestID: StatusPane01-i |
Last completed: Wed Sep 22 09:51:03 EST 1999 by vhle |
| Test Description |
| This test case verifies the preferred sizes of the play area and the status pane are correct. |
| Inputs or Test Data |
A new JFrame is created and the PlayArea, StatusPane, a dummy TabbedPane and a dummy ActionPane are added to the frame with the following code:
JFrame frame = new JFrame();
frame.setLayout(new BorderLayout());
JPanel leftPane = new JPanel(new BorderLayout());
leftPane.add(playArea, BorderLayout.CENTER);
leftPane.add(statusPane, BorderLayout.SOUTH);
frame.getContentPane().setLayout(new BorderLayout());
frame.getContentPane().add(leftPane, BorderLayout.CENTER);
frame.getContentPane().add(dummyPane, BorderLayout.EAST);
Where dummyPane is a pane containing a rigid area to simulate the width of the tabbed pane / action pane. |
| Expected Output |
| The status pane should appear directly below the play area, and the width of the status pane should be exactly the same as that of the play area. The play area should look approximately square under the default screen size (800 x 600). |
| Actual Output |
| Visual inspection of the display shows that the actual output matches the expected output. |
| Additional Comments |
| None |
5. System Testing
During the process of unit and integration testing, errors specific to individual classes or groups of classes have the ultimate consideration. This fails to observe key elements that the system must have in order to be considered successful. The system, once it has been fully integrated, must meet the original objectives that were intended to be met, and the performance as a whole must meet the original expectations of the client.
All the criteria relating to the original objective and the performance are set out in sections 4 and 5 of the Software Requirements Specification (SRS). The aim of system testing is to fully demonstrate that the entire software runs according to the required specifications and help uncover any final problems. The tests will be conducted when all parts of the software have been completed and integrated into one.
5.1 General Strategy
Classes identified in the design stage are assigned to responsibilities that will help carry out some component of the original objectives. All requirements specified in the SRS are allocated to the classes. The Booch method then requires that scenarios be constructed to demonstrate how these classes will relate to each other. These will show how the assigned responsibilities, and hence the objectives, will be fulfilled. If this functionality is found in the system then it will demonstrate that the requirements have been met. The scenarios will therefore form the basis of the system test cases.
The system test cases will draw on the scenarios to help uncover bugs and to ensure that all areas interact correctly. Of particular importance will be the areas of:
- User Interaction.
- Communication between Client and Server.
- Game Play.
It is expected that the scenarios will cover both the common and complex aspects of these areas. This will ensure that the system has been fully tested.
All scenarios will be able to be demonstrated with the use of black-box test cases. Input will be developed to match the situations specified at the beginning of the scenario and the sequences to each stage of the diagram will be carried out. This will demonstrate if the system operates as intended. A system test will only considered successful if every aspect of the scenario has been followed and all MultiMahjongClients(MMC) that participate in the test contain the equivalent information.
5.1.1 Validation Testing
Validation testing will be conducted to determine whether the functional requirements, as set out in section 4 of the SRS, have been satisfied. A checklist that matches the requirements, scenarios, and test cases will be used to ensure that every requirement of the project has been tested for. As each test is successfully completed, the associated requirement will be marked off in this checklist. Any requirement that has not been satisfied will be marked as incomplete.
5.1.2 Performance Testing
Performance testing is conducted to examine how well the system performs in all extreme circumstances. These tests are designed to help observe the system's limitations by placing a high demand on its resources, and placing it in abnormal situations. The performance criteria that will be used to determine whether the performance of the system conforms to the stated acceptable level is set out in section 5 of the SRS.
To ensure the success of the system in extreme circumstances, performance testing will be conducted at all levels of the testing process. Tests will be conducted on individual units, the modules and the entire system as a whole. The majority of these tests will be conducted at the unit testing level. This will ensure that the problems can be more easily isolated.
Communications between the MultiMahjongClient(MMC) and the MultiMahjongServer(MMS) will form the majority of performance testing. High and low levels of demand will be placed on the MMS and tests conducted on communications to ensure that the system can deal with these loads. However this form of testing will not be able to be conducted until the entire system is fully integrated.
5.1.3 Acceptance Testing
Acceptance testing is conducted to demonstrate to the client that the system performs to an acceptable level and has met their objectives. This will be conducted in the preliminary and final demonstrations to the client. The preliminary demonstration will be of particular importance, as it will ensure that the functionality conforms to their expectations at an earlier stage. It is also intended to ensure that the graphical user interface is correct.
As mahjong is a game that is not intended to have fixed input, the first test case to be demonstrated will be entirely random. All other test cases to follow will be based on the scenarios, to demonstrate the full functionality of the system. It is intended that these will help convince the client that the requirements have been met, as set out in section 6 of the SRS.
A walkthrough of the user documentation will be the final stage of the acceptance testing.
5.1.4 Installation Testing
Installation testing will be carried out to ensure that the software has been installed correctly and that it will run on the all the systems it was intended to work on. These systems are specified in section 5 of the SRS.
As the software is intended to work on a number of platforms, each will need to be tested individually. Installation testing will need to check that all directory packages have been created and that all binary files contained in the software exist. The final check will involve running the software in a multi-player game.This will ensure that all sections have been installed and function correctly. Installation testing will also form part of the final demonstration to the client.
5.2 Failure of System Tests
Once a system test has been complete, and the actual functionality has found to have deviated from the original objective, then this will be marked off in a deficiency list. All functions that are recorded in the deficiency list are to be investigated and any suggested changes discussed at a team meeting. If the problem occurs during the final demonstration to the client, then any changes will be discussed with the client.
|