12.0 Test Phase

  • The organization should have test standards and the project should have a test plan that are well documented. Requirements for performing specific and various test types should be included in the standards and plan. Test types include unit testing, load testing, integration testing, and acceptance testing.
  • Testing should include help functions built into the application along with manuals for both users, operators, and maintenance personnel.
  • Responsibility for documentation, review, and approval of test results should be assigned in advance of testing.
  • A code review must be performed to be sure the program adheres to set conventions including design specifications, naming conventions, and code standards.
  • The system components should be tested in environments that simulate the production environment and various workloads including the peak workload as close as possible.
  • Whether simulated data or live data are used during testing is based on the circumstances. If live data are used and any data is sensitive, it must be protected with adequate controls to guarantee data confidentiality but sensitive data should not be used for testing if it can be avoided. The test data should simulate live data in format and volume.
  • The testers must be independent of the developers.
  • Test code should be issued through a central code release authority.
  • The testing results must be included in a test report that shows that the system did or did not meet with the specifications and requirements. The test report must be approved by all groups involved with testing before it is finalized and sent to management. Test results must be tracked until all test issues are resolved.
  • The test report must be stored so it can be reviewed later. Any problems detected during testing must be documented in the report and the problems must be prioritized and corrected. The impact of any discrepancies must be evaluated and the discrepancies must be resolved or signed off by the system owner and upper management before the project is implemented.
  • Controls must be in place to ensure that the integrity and confidentiality of test data and results are maintained.
  • Prior to production, the system owners and end users must agree and sign indicating that appropriate testing has been conducted and the test results are satisfactory. The agreement must indicate that all system requirements have been adequately and successfully tested. Controls must be in place to prevent the system from going into production without approval.
  • If known errors exist in the system or programs, the senior management must sign to agree to allow the system to go into production mode with the errors.
  • A test environment should be created to perform testing prior to production. The test environment should be compared to the production environment and emulate actual loads, inputs, and circumstances as close as possible. Security controls and workloads must be considered to be sure they are compared between the environments.
  • When the system is tested for operational functionality, the testing should test all system functions and components. Operational testing conditions should be as much like the actual production environment as possible.
  • Procedures must be in place defining what to do if the system is unacceptable and the cost to fix it is not acceptable.
  • Testers must be trained for planning tests, conducting tests, and documenting test results. Testors must understand and be aware of testing conditions such as buffer overflow, null, negative, and inconsistent values which may cause system errors.

Test Plans

  • All new systems or systems that are having major changes must have a test plan.
  • The test plan must be based on system specifications and ensure that system requirements and functionality are tested.
  • The test plan must include specific tests which are designed to detect unauthorized changes or access.
  • The test plan must apply both valid and invalid conditions considering possible security attack scenarios.
  • There should be a walk through of the test plan before the testing begins. This will help be sure the test plan will be effective. A review of the test plan with organizational standards should be conducted. Acceptance criteria, responsibilities, and test scenarios should be reviewed.
  • There must be test guidelines and standards in the test plan or referred to by the test plan for all types and levels of testing.
  • The test plan must define the roles and responsibilities of the persons involved with the effort.
  • The test plan must provide for all functional requirements to be tested.
  • The test plan must provide for test simulations with transactions or simulated system functionality tests of the system.
  • The test plan must provide a plan for getting or creating data to be used in the tests.
  • The test plan must require that all testing is documented and is satisfactory before the system goes into production.
  • Testing and test results analysis must comply with the methods specified in the test plan.
  • When changes to the application or system are made, a roll back or back out process must exist so an efficient and effective rollback to the previous version can be implemented. The back out process would be used if an adverse impact on the system were experienced due to the change.