Extnded Random Regression Testing Finding valuable bugs by 扩展的随机回归测试发现有价值的错误.docVIP

Extended Random Regression Testing: Finding valuable bugs by running long pseudo-random sequences of already-passed complex tests. Pat McGee Florida Institute of Technology jpmcgee@ Cem Kaner Florida Institute of Technology kaner@ Abstract This paper reports a case study of the use of Extended Random Regression (ERR) testing, a type of high volume automated software testing. High volume automation involves running very long sequences of tests. In ERR, the tests are all regression tests that the program has already passed, run in pseudo-random order. Reuse of regression tests does not increase testing coverage (by most coverage measures), but ERR testing exposes serious (crash) bugs that are otherwise hard to find, such as memory leaks, memory corruption, and timing problems. ERR seems particularly applicable to embedded software that will run for long periods of time between resets. In the case study, ERR was applied to embedded firmware in a line of mass-market products. It would be equally applicable to life-critical products.. Introduction This paper presents a case study of the use of a high volume computer-assisted testing technique that we call Extended Random Regression (ERR) testing. ERR testing strings together a set of existing functional regression tests that the program has already passed, running them in a long random sequence. This method was developed to test embedded software that must run for long periods between resets. It is one of the simplest kinds of long-sequence testing that we can imagine, because it requires no simulator, no modeling, no knowledge of the underlying state machine, and no development of new tests. You just re-use what you have already used, but in a long sequence. To our surprise, even though the software has passed these tests individually, it fails the sequences quite often. The sequences expose problems that are difficult to detect with individual functional tests, such as memory leaks, memory corruption, and timing pro