When Code Coverage Betrayed Me
I’m a fan of code protection as a method to guarantee that there are covering tests. One location that I tend to rely greatly on Code Protection for is to capture any tests that are no longer working properly due to modifications in the production code. That frequently exercises well, however today I got betrayed by the code protection engine.
The code that I dealt with included an if declaration with a multi-step &&& & expression.
bool IsAllWrong ( int importantValue,(* )bool b) { bool a = importantValue= = GetAnswer () &&; bool &&c &&= incorrect ; bool d = incorrect ; if ( ! a & &! b (* )& &! c & &! d (* )) { return(* ) real &&; &&} return incorrect ; } . bool IsAllWrong( int importantValue, bool b) . { . bool a= importantValue= =GetAnswer(); . bool c= incorrect; . bool d= incorrect; . . if (! a & &! b & &! c & &! d) . { . return real; .} . return incorrect; .} Naturally I had tests that made the examination stop working both due to the fact that of importantValue and |
So what happend later on was that GetAnswer( ) was upgraded, without the test for when importantValue being upgraded. Naturally (my bad )that test had actually set b to real , triggering the examination to stop working on b , triggering real to be returned. So the test passed, however not due to the important things I wished to evaluate. In an intricate application, this is bound to take place from time to time. However typically, the code protection ratings will expose that there is an execution course not covered. However not this time! The reliable code protection analysis betrayed me! To comprehend why I was betrayed by code protection, we'll need to take a look at the IL code produced. It appears like Visual Studios' code protection does not examine branch protection, however rather simply IL guideline protection. For the most part that's the exact same, however not this time.// Initialize a,
b
, c & & d. IL_0000: nop IL_0001: ldarg 1. IL_0002: ldarg 0. IL_0003: call UserQuery GetAnswer. IL_0008 : ceq. IL_000A: stloc 0 // a. IL_000B: ldc i4 0. IL_000C: stloc 1 // c. IL_000D: ldc i4 0. IL_000E: stloc 2 // d. // Examine conditional expression. IL_000F: ldloc 0 // a. IL_0010: brtrue s IL_001E. IL_0012: ldarg 2 // b. IL_0013: brtrue s IL_001E. IL_0015: ldloc 1 // c. IL_0016: brtrue s IL_001E. IL_0018: ldloc 2 // d. IL_0019: ldc i4 0. IL_001A: ceq. IL_001C: br s IL_001F. IL_001E: ldc i4 0 // Run if expression was brief circuited by a, b or c holding true IL_001F: stloc 3. IL_0020: ldloc 3. IL_0021: brfalse s IL_0029. IL_0023: nop IL_0024: ldc i4 1. IL_0025: stloc s 04 IL_0027: br s IL_002E. IL_0029: ldc i4 0. IL_002A: stloc s 04 IL_002C: br s IL_002E. IL_002E: ldloc s 04 IL_0030: ret// Initialize a, b, c & & d . IL_0000: nop . IL_0001: ldarg.1 . IL_0002: ldarg.0 . IL_0003: call UserQuery.GetAnswer . IL_0008: ceq . IL_000A: stloc.0// a . IL_000B: ldc.i4.0 . IL_000C: stloc.1// c . IL_000D: ldc.i4.0 . IL_000E: stloc.2// d .// Examine conditional expression . IL_000F: ldloc.0// a . IL_0010: brtrue.s IL_001E . IL_0012: ldarg.2/ / b . IL_0013: brtrue.s IL_001E . IL_0015: ldloc.1 / / c . IL_0016: brtrue.s IL_001E . IL_0018: ldloc .2/ / d . IL_0019: ldc.i4.0 . IL_001A: ceq . IL_001C: br.s IL_001F . IL_001E: ldc.i4.0// Run if expression was brief circuited by a, b or c holding true . . IL_001F: stloc.3 . IL_0020: ldloc.3 . IL_0021: brfalse.s IL_0029 . IL_0023: nop . IL_0024: ldc.i4.1 . IL_0025: stloc.s 04 . IL_0027: br.s IL_002E . IL_0029: ldc.i4.0 . IL_002A: stloc.s 04 . IL_002C: br.s IL_002E . IL_002E: ldloc.s 04 . IL_0030: ret(* )We can see that the examination of the conditional expression takes the kind of filling each of a, b |
c and after that short-circuiting the examination if the variable is real The issue is that if either of a, b or c is real the branch guideline indicate the exact same target: IL_001E So to get all IL guidelines run it suffices to evaluate for 2 cases: a= b= c =d= incorrect a= real; b= c =d= incorrect I still had both those cases covered. And due to the fact that I was careless and passed in real for
bwhen screening
importantValue the system test still passed. So what I have actually discovered today is that: Do not count on code protection to properly determine branch protection. When composing tests for a complex if declaration, it is dangerous to be careless with the criteria checked after the appropriate one. Despite the fact that they are not appropriate due to the fact that of expression brief circuiting when whatever works, bad worths can make the test hand down the incorrect properties.
.
Published in
. C# on 2017-04-06|Tagged Code Protection, Test Driven Advancement, System Tests
.
.
.
.
.
