Version: 1.0
CS 4500, University of Utah, Spring 2008
This document details the Verification and Validation Results for the first staged release (SR1) of Sonic Boxmo.
Sonic Boxmo will be a puzzle game which will consist of novel and innovative features including a multi-cursor multi-player user interface and fast-paced music based puzzles, which will immerse the players into a unique musical environment. Each player in the game is going to have their own cursor and speaker, and will play in a shared 3D hardware-accelerated environment.
MSI - Microsoft Installer (file extension)
SDS - Software Development Specification
SR1 - Staged Release 1
SR2 - Staged Release 2
SRS - Software Requirements Specification
VVP - Verification adn Validation Plan
VVR - Verification and Validation Results
Sonic Boxmo VVP
Tests were performed as outlined in the VVP where applicable. Tests performed for SR1/VVR1 were repeated to ensure continued functionality.
The final SR2 release code was reviewed in its entirety after testing and some issues were discovered which were addressed (as well as some of the unsatifactory tests) in a second build which is available on the Releases page.
4.1. Component test Results
4.1.1 Component Test Result Layout
Each of the following sections include the original testing strategies for each component as well as the result of tests and what yet needs to be done.
4.1.2 GraphicsMain
SR1: Use very large SceneLists and empty scene lists.
Results: Scene rendered correctly, slowly, but properly. Performance of the engine begins to degrade above 500,000 single-polygonal objects (on test system)
Performed by: Seth
Work yet to do: noneSR2 retest: pass (seth)
SR1: Set width and height to values beyond screen resolution.
Results: glut returned an error and the program crashed
Performed by: Seth
Work yet to do: The display resolution will be set via in-game settings which will poll the OS for valid settings, need to add a fallback to the minimum settings in the case of a corrupted settings file.SR2 retest: Resolution method re-worked to simply use the current desktop resolution to preven incompatability. Test no longer applicable.
4.1.3 DrawableObject
SR1: load a drawable object with a VERY large vertex list
Results: objects drew correctly for all geometry types
Performed by: seth
Work yet to do: noneSR2 retest: same resut (seth)
SR1: create lots of drawable objects and randomly delete from the middle of the list to test list cohesion
Results: scene list mended itself correctly in all situations
Performed by: seth
Work yet to do: noneSR2 retest: same resut (seth)
SR1: specify invalid geometry (i.e. triangles with vertex count not a multiple of 3)
Results: excess vertices were ignored by OpenGL no errors, performs as expected
Performed by: seth
Work yet to do: noneSR2 retest: same resut (seth)
verify that obj files can be loaded by importing obj files from Maya
Results: obj files were loaded correctly
Performed by: Justin
Work yet to do: none4.1.4 Material
attempt to load very large texture/bump map files
Results: No errors were found. OBJ files were loaded and textured with large size texture and bump maps
Performed by: Justin
Work yet to do: nonespecify invalid texture/bump map path
Results: Program recovers properly.
Performed by: Jared
Work yet to do: none4.1.5 AudioSample
SR1: test all possible sample rates and bit depths (even those that excede hardware capabilities)
Results: everything that the hardware supported played, unsupported formats were automatically re-mixed while in software mode but generated errors in hardware mode
Performed by: Jared
Work yet to do: noneSR2 retest: same resut (justin)
SR1: incrementally increase the number of simultaneous samples being played back until the system fails to find hard limit of simultaneous samples
Results: Hardware mode varried by sound card, but the system never failed, the FMOD api simply clipped excess channels (CPU usage about 7%). In software mode there was no practical limit to simultaneous sample playback; tests were done with 1024 simultaneous voices with no errors or performance loss (CPU usage was arouns 35%).
Performed by: Jared
Work yet to do: noneSR2 retest: same resut (Justin)
4.1.6 BoxmoMain
Play test. See section 4.2
4.1.7 PuzzleField
load variable sized fields and test element placement and identification
Result: This test passed the sizing aspect. The identification failures came when we tried to rotate pieces around the centroid of triangles. The idea was formed, and code was changed, to make the puzzle pieces move more simply.
Performed by: Justin
Work to do: Nonetest every possible rotation of elements
Result: Rotating elements has changed due to previous test case.
Performed by: Justin
Work to do: Noneattempt to index element out of bounds
Result: Using vector datatypes, we have dodged this problem by using vector.at(index) function. This does the element bounds checking for us. Testing of out of bounds element is controlled and passes the test.
Performed by: Seth
Work to do: Noneattempt to rotate an invalid element group
Result: Finding the element triangle of drawable objects that the pieces are rotating about is accomplished by clicking triangles. Clicking outside the element triangle does not rotate any triangle of elements. The puzzle field is thus complete.
Performed by: Jared
Work to do: None4.1.8 PuzzleElement
while testing PuzzleFied, attempt all possible valid and invalid calls to update and explode
Result: All calls to update and explode make sense.
Performed by: Seth
Work to do: Noneobserve for proper interface behavior in PuzzleField testing
Result: PuzzleField has child objects that are the PuzzleElements. PuzzleElements respond properly to the user's clicks. PuzzleField is updated accordingly
Performed by: Seth
Work to do: None4.1.9 Player
manually test input mapping
Result: Input from the player's InputDevice works as designed. Wii remotes work properly to use as input devices. Two buttons on the Wii are designed to map to the two buttons on the mouse. No other input mapping is required.
Performed by: Jared
Work to do: None4.1.10 MusicBar
load track with lots of notes and attempt to display them
Result: Many hundreds of notes can be loaded
Performed by: Jared
Work to do: Nonespecify a very large active section and 0 active section
Result: large active sections are handled correctly. The handling of positions in the puzzlefield are in float increments and appear correctly no matter the division.
Performed by: Seth
Work to do: None4.1.11 MusicTrack
SR1: create invalid song file
Results: application crashed if the song fie was poorly formatted or is the specified sample files were not found
Performed by: Justin
Work yet to do: generate a useful error message and exit the program more gracefully when ill-formatted track files are detectedSR2 retest: ill-formatted tracks are now can be detected (switched to use readline and split on whitespace to ensure correct format) (Jared)
SR1: play back fast-tempo tracks with lots of notes
Results: initially there was a lot of variablility in note timiing (beats were very our of sync) but the timing mechanism was re-written to use Windows' high-resolution timing API and time delta variables were changed from single-precision to double-precision variables and now they play in time properly. The only issue is that at the beginning of playback there is a bit of a lag while the samples load and the library initializes.
Performed by: Justin
Work yet to do: None... while the load lag is still present, it will not be an issue in the finished product because the audio will load prior to the other resources and be ready before the game begins.SR2 retest: Player selection takes place while audio is loading and lag is no longer an issue (jared)
4.2 Summary of Integration Test Results
This release incorporates the initial game-play components in a not-yet-playable demo. The components were tested individually but have yet to be fully integrated. Gameplay objects were use-tested to ensure proper behavior and no issues were encountered which required more than a couple minutes to debug.
5.1 Evaluation of test cases
The original test cases were still all applicable and useful in spite of many changes to the back-end design since they were written.
5.2 Results from defect tracker
Direct copy of SVN logs here. (Please note: the author names only show who commited changes and we used pair-programming through most of the coding for this release)5.3 Lessons learned
As with SR1, the testing strategy worked quite well with a few minor exceptions. We did much more of the coding using pair programming which actually resulted in much fewer bugs discovered during testing.
Acceptance testing was limited due to the fact that this release is not a fully functioning game... Our initial acceptance conditions were evaluated anyway for a status-check.
- any number of puzzles can be played without any of the following events occuring:
- any operating system errors (demo app passes)
- game crashes (demo app does not crash under these conditions)
- game logic errors (notes are graded properly, puzzle-field behaves as expected)
- sound glitches/errors (sounds work properly through unit testing and demo-app testing)
- video glitches or errors (graphics engine works properly as per unit testing and demo-app testing)
- all game options have been tested and found to produce the desied effect (N/A, not implemented yet)
- sand-box mode produces correct files which can be read and played properly in game-mod (N/A, not implemented yet)
This release will not be packaged in an MSI installer but will be made available in source and binary archives.
This list will include open issues only with features which have been implemented. Features planned for future releases will be ommited.
- Integration of game-play components not complete
N/A