Version: 1.1
CS 4500, University of Utah, Spring 2008
This document details the Verification and Validation Results for the first staged release (SR1) of Sonic Boxmo.
Sonic Boxmo will be a puzzle game which will consist of novel and innovative features including a multi-cursor multi-player user interface and fast-paced music based puzzles, which will immerse the players into a unique musical environment. Each player in the game is going to have their own cursor and speaker, and will play in a shared 3D hardware-accelerated environment.
MSI - Microsoft Installer (file extension)
SDS - Software Development Specification
SR1 - Staged Release 1
SR2 - Staged Release 2
SRS - Software Requirements Specification
VVP - Verification adn Validation Plan
VVR - Verification and Validation Results
Sonic Boxmo VVP
Tests were performed as outlined in the VVP where applicable. Some components are scheduled to be implemented for SR2 and thus were not tested. Of the tests, 3 results were considered unsatisfactory, 2 of which will require additional work to satisfy the test while the third (4.11.1, test 2) was determined to be a system limitation beyond the scope of the project and a workaround will be implemented.
The final SR1 release code was reviewed in its entirety after testing and some issues were discovered which were addressed (as well as some of the unsatifactory tests) in a second build which is available on the Releases page.
4.1. Component test Results
4.1.1 Component Test Result Layout
Each of the following sections include the original testing strategies for each component as well as the result of tests and what yet needs to be done.
4.1.2 GraphicsMain
Use very large SceneLists and empty scene lists.
Results: Scene rendered correctly, slowly, but properly. Performance of the engine begins to degrade above 500,000 single-polygonal objects (on test system)
Performed by: Seth
Work yet to do: noneSet width and height to values beyond screen resolution.
Results: glut returned an error and the program crashed
Performed by: Seth
Work yet to do: The display resolution will be set via in-game settings which will poll the OS for valid settings, need to add a fallback to the minimum settings in the case of a corrupted settings file.4.1.3 DrawableObject
load a drawable object with a VERY large vertex list
Results: objects drew correctly for all geometry types
Performed by: seth
Work yet to do: nonecreate lots of drawable objects and randomly delete from the middle of the list to test list cohesion
Results: scene list mended itself correctly in all situations
Performed by: seth
Work yet to do: nonespecify invalid geometry (i.e. triangles with vertex count not a multiple of 3)
Results: excess vertices were ignored by OpenGL no errors, performs as expected
Performed by: seth
Work yet to do: none4.1.4 Material
This component is not scheduled to be coded until SR2
4.1.5 AudioSample
test all possible sample rates and bit depths (even those that excede hardware capabilities)
Results: everything that the hardware supported played, unsupported formats were automatically re-mixed while in software mode but generated errors in hardware mode
Performed by: Jared
Work yet to do: noneincrementally increase the number of simultaneous samples being played back until the system fails to find hard limit of simultaneous samples
Results: Hardware mode varried by sound card, but the system never failed, the FMOD api simply clipped excess channels (CPU usage about 7%). In software mode there was no practical limit to simultaneous sample playback; tests were done with 1024 simultaneous voices with no errors or performance loss (CPU usage was arouns 35%).
Performed by: Jared
Work yet to do: none4.1.6 BoxmoMain
Play test. See section 4.2
4.1.7 PuzzleField
Component will not be implemented until SR2
4.1.8 PuzzleElement
Component will not be implemented until SR2
4.1.9 Player
Component will not be implemented until SR2
4.1.10 MusicBar
Component will not be implemented until SR2
4.1.11 MusicTrack
create invalid song file
Results: application crashed if the song fie was poorly formatted or is the specified sample files were not found
Performed by: Justin
Work yet to do: generate a useful error message and exit the program more gracefully when ill-formatted track files are detectedplay back fast-tempo tracks with lots of notes
Results: initially there was a lot of variablility in note timiing (beats were very our of sync) but the timing mechanism was re-written to use Windows' high-resolution timing API and time delta variables were changed from single-precision to double-precision variables and now they play in time properly. The only issue is that at the beginning of playback there is a bit of a lag while the samples load and the library initializes.
Performed by: Justin
Work yet to do: None... while the load lag is still an present, it will not be an issue in the finished product because the audio will load prior to the other resources and be ready before the game begins.4.2 Summary of Integration Test Results
Because this release aimed only at completing the back-end framework integration testing was limited to the simple demo application built on this framework.
The demo application creates a graphical cursor object for each mouse detected on the system while playing a music track in the background and also responds to click events.
The only major integration issue encountered was an incompatibility between the glut window management api and with RawMouse api. They both reuired exclusive control over the applications message handling to work properly. This was remedied by re-writing the mouse input handing with the ManyMouse interface which uses global device polling as opposed to system messages.Tests were performed on various systems with between 1 and 4 mice and no issues were encountered.
5.1 Evaluation of test cases
Most of the testing issues we encountered were due to either outside libraries behaving in an unexected mannor or changing libraries after the test cases were written. The audio tests, for example, were written with OpenAL in mind but under FMOD all of the forseen issues with simultaneous sample playback were handled automatically with no extra effort.
5.2 Results from defect tracker
Direct copy of SVN logs here. (Please ignore author names, there was an issue with SVN accounts which has since been resolved, but most of the author names are wrong)5.3 Lessons learned
The testing strategy worked quite well with a few minor exceptions. Earlier integration testing would have revealed the RawInput/GLUT conflict much earlier and a complete rewrite of the input code would not have been needed. Components were tested by member other than their authors which worked very well.
Acceptance testing was limited due to the fact that this release was not planned to be a completed product but rather a completed framework for the product.
The criteria for acceptance of the produce (as outlined in the VVP) are as follows (with comments on the current state of the project).
- any number of puzzles can be played without any of the following events occuring:
- any operating system errors (demo app passes)
- game crashes (demo app does not crash under these conditions)
- game logic errors (N/A, game logic yet to be implemented)
- sound glitches/errors (sounds work properly through unit testing and demo-app testing)
- video glitches or errors (graphics engine works properly as per unit testing and demo-app testing)
- all game options have been tested and found to produce the desied effect (N/A, not implemented yet)
- sand-box mode produces correct files which can be read and played properly in game-mod (N/A, not implemented yet
This release will not be packaged in an MSI installer but will be made available in source and binary archives.
This list will include open issues only with features which have been implemented. Features planned for future releases will be ommited.
- graphics mode fallback (fallback to minimum graphics settings if there is a graphics initialization error)
- detect invalid music track and generate an appropriate error message.
- fix load-lag for samples (not sure if this is even possible or if a workarround is needed
N/A