Version: 1.0
CS 4500, University of Utah, Spring 2008
This document details the Verification and Validation Results for the first staged release (SR1) of Sonic Boxmo.
Sonic Boxmo will be a puzzle game which will consist of novel and innovative features including a multi-cursor multi-player user interface and fast-paced music based puzzles, which will immerse the players into a unique musical environment. Each player in the game is going to have their own cursor and speaker, and will play in a shared 3D hardware-accelerated environment.
MSI - Microsoft Installer (file extension)
SDS - Software Development Specification
SR1 - Staged Release 1
SR2 - Staged Release 2
SRS - Software Requirements Specification
VVP - Verification adn Validation Plan
VVR - Verification and Validation Results
Sonic Boxmo VVP
Tests were performed as outlined in the VVP where applicable. Tests performed for SR1/VVR1 were repeated to ensure continued functionality.
The final SR3 release code was reviewed in its entirety after testing (only modules modified from SR2 were reviewed). Issues were raised concerning lack of commenting in some modules.
4.1. Component test Results
4.1.1 Component Test Result Layout
Each of the following sections include the original testing strategies for each component as well as the result of tests and what yet needs to be done.
4.1.2 GraphicsMain
SR1: Use very large SceneLists and empty scene lists.
Results: Scene rendered correctly, slowly, but properly. Performance of the engine begins to degrade above 500,000 single-polygonal objects (on test system)
Performed by: Seth
Work yet to do: noneSR2 retest: pass (seth)
SR3 retest: pass (seth)SR1: Set width and height to values beyond screen resolution.
Results: glut returned an error and the program crashed
Performed by: Seth
Work yet to do: The display resolution will be set via in-game settings which will poll the OS for valid settings, need to add a fallback to the minimum settings in the case of a corrupted settings file.SR2 retest: Resolution method re-worked to simply use the current desktop resolution to preven incompatability. Test no longer applicable.
SR3 retest: pass4.1.3 DrawableObject
SR1: load a drawable object with a VERY large vertex list
Results: objects drew correctly for all geometry types
Performed by: seth
Work yet to do: noneSR2 retest: same result (seth)
SR3 retest: pass (seth)SR1: create lots of drawable objects and randomly delete from the middle of the list to test list cohesion
Results: scene list mended itself correctly in all situations
Performed by: seth
Work yet to do: noneSR2 retest: same resut (seth)
SR3 retest: pass (seth)SR1: specify invalid geometry (i.e. triangles with vertex count not a multiple of 3)
Results: excess vertices were ignored by OpenGL no errors, performs as expected
Performed by: seth
Work yet to do: noneSR2 retest: same resut (seth)
SR3retest: pass (seth)verify that obj files can be loaded by importing obj files from Maya
Results: obj files were loaded correctly
Performed by: Justin
Work yet to do: none
SR3 retest: pass (Jared)4.1.4 Material
attempt to load very large texture/bump map files
Results: No errors were found. OBJ files were loaded and textured with large size texture and bump maps
Performed by: Justin
Work yet to do: none
SR3 retest: pass (Justin)specify invalid texture/bump map path
Results: Program recovers properly.
Performed by: Jared
Work yet to do: none
SR3 retest: texture and bump mapping has not and will not be used and support has been removed from the engine for efficiency4.1.5 AudioSample
SR1: test all possible sample rates and bit depths (even those that excede hardware capabilities)
Results: everything that the hardware supported played, unsupported formats were automatically re-mixed while in software mode but generated errors in hardware mode
Performed by: Jared
Work yet to do: noneSR2 retest: same resut (justin)
SR3 retest: pass (Justin)SR1: incrementally increase the number of simultaneous samples being played back until the system fails to find hard limit of simultaneous samples
Results: Hardware mode varried by sound card, but the system never failed, the FMOD api simply clipped excess channels (CPU usage about 7%). In software mode there was no practical limit to simultaneous sample playback; tests were done with 1024 simultaneous voices with no errors or performance loss (CPU usage was arouns 35%).
Performed by: Jared
Work yet to do: noneSR2 retest: same resut (Justin)
SR3 retest: pass (Justin)4.1.6 BoxmoMain
Play test. See section 4.2
4.1.7 PuzzleField
load variable sized fields and test element placement and identification
Result: This test passed the sizing aspect. The identification failures came when we tried to rotate pieces around the centroid of triangles. The idea was formed, and code was changed, to make the puzzle pieces move more simply.
Performed by: Justin
Work to do: None
SR3 retest: pass (Jared)test every possible rotation of elements
Result: Rotating elements has changed due to previous test case.
Performed by: Justin
Work to do: None
SR3 retest: pass (seth)attempt to index element out of bounds
Result: Using vector datatypes, we have dodged this problem by using vector.at(index) function. This does the element bounds checking for us. Testing of out of bounds element is controlled and passes the test.
Performed by: Seth
Work to do: None
SR3 retest: pass (seth)attempt to rotate an invalid element group
Result: Finding the element triangle of drawable objects that the pieces are rotating about is accomplished by clicking triangles. Clicking outside the element triangle does not rotate any triangle of elements. The puzzle field is thus complete.
Performed by: Jared
Work to do: None
SR3 retest: pass (seth)4.1.8 PuzzleElement
while testing PuzzleFied, attempt all possible valid and invalid calls to update and explode
Result: All calls to update and explode make sense.
Performed by: Seth
Work to do: None
SR3 retest: pass (seth)observe for proper interface behavior in PuzzleField testing
Result: PuzzleField has child objects that are the PuzzleElements. PuzzleElements respond properly to the user's clicks. PuzzleField is updated accordingly
Performed by: Seth
Work to do: None
SR3 retest: pass (Justin)4.1.9 Player
manually test input mapping
Result: Input from the player's InputDevice works as designed. Wii remotes work properly to use as input devices. Two buttons on the Wii are designed to map to the two buttons on the mouse. No other input mapping is required.
Performed by: Jared
Work to do: None
SR3 retest: pass (Jared)4.1.10 MusicBar
load track with lots of notes and attempt to display them
Result: Many hundreds of notes can be loaded
Performed by: Jared
Work to do: None
SR3 retest: pass (Justin)specify a very large active section and 0 active section
Result: large active sections are handled correctly. The handling of positions in the puzzlefield are in float increments and appear correctly no matter the division.
Performed by: Seth
Work to do: None
SR3 retest: pass (Justin)4.1.11 MusicTrack
SR1: create invalid song file
Results: application crashed if the song fie was poorly formatted or is the specified sample files were not found
Performed by: Justin
Work yet to do: generate a useful error message and exit the program more gracefully when ill-formatted track files are detectedSR2 retest: ill-formatted tracks are now can be detected (switched to use readline and split on whitespace to ensure correct format) (Jared)
SR3 retest: pass (Jared)SR1: play back fast-tempo tracks with lots of notes
Results: initially there was a lot of variablility in note timiing (beats were very our of sync) but the timing mechanism was re-written to use Windows' high-resolution timing API and time delta variables were changed from single-precision to double-precision variables and now they play in time properly. The only issue is that at the beginning of playback there is a bit of a lag while the samples load and the library initializes.
Performed by: Justin
Work yet to do: None... while the load lag is still present, it will not be an issue in the finished product because the audio will load prior to the other resources and be ready before the game begins.SR2 retest: Player selection takes place while audio is loading and lag is no longer an issue (jared)
SR3 retest: pass ()4.2 Summary of Integration Test Results
Puzzes can be played multiple times to completion without errors or other problems. UI options work in game.
5.1 Evaluation of test cases
The original test cases were still all applicable and useful in spite of many changes to the back-end design since they were written.
5.2 Results from defect tracker
Direct copy of SVN logs here. (Please note: the author names only show who commited changes and we used pair-programming through most of the coding for this release)5.3 Lessons learned
As with SR1, the testing strategy worked quite well with a few minor exceptions. We did much more of the coding using pair programming which actually resulted in much fewer bugs discovered during testing.
- any number of puzzles can be played without any of the following events occuring:
- any operating system errors (pass)
- game crashes (pass)
- game logic errors (pass)
- sound glitches/errors (pass)
- video glitches or errors (pass)
- all game options have been tested and found to produce the desied effect (pass, need to validate input on options)
- sand-box mode produces correct files which can be read and played properly in game-mod (N/A, not implemented yet)
This release will not be packaged in an MSI installer but will be made available in source and binary archives.
This list will include open issues only with features which have been implemented. Features planned for future releases will be ommited.
- sand box mode needs to be implemented
- front-end UI needs to be cleaned up
N/A