CS375: Software Engineering - Software Testing and Code Coverage

Activity Goals

The goals of this activity are:
  1. To differentiate between white box and black box testing
  2. To indicate when one testing strategy is appropriate over another
  3. To explain that although testing all input possibilities and configurations is impossible, achieving good code coverage with heterogeneous inputs is a best practice
  4. To write unit tests with 100% code and control flow coverage
  5. To explain that 100% code coverage alone is insufficient because different inputs may exercise those branches differently

Supplemental Reading

Feel free to visit these resources for supplemental background reading material.

The Activity

Directions

Consider the activity models and answer the questions provided. First reflect on these questions on your own briefly, before discussing and comparing your thoughts with your group. Appoint one member of your group to discuss your findings with the class, and the rest of the group should help that member prepare their response. Answer each question individually from the activity, and compare with your group to prepare for our whole-class discussion. After class, think about the questions in the reflective prompt and respond to those individually in your notebook. Report out on areas of disagreement or items for which you and your group identified alternative approaches. Write down and report out questions you encountered along the way for group discussion.

Model 1: Choosing Unit Tests

?
1
2
3
4
5
6
7
8
9
10
11
public class Main {           
    public static int triangleArea(int base, int height) {
        // Broken Code!
        return 1 / 2 * base * height;
    }
     
    public static void main(String[] args) {
        System.out.println(triangleArea(5, 2));
        System.out.println(triangleArea(3, -2));
    }
}

Questions

  1. What's wrong with this code? (there is more than one answer!)
  2. How many calls would you make to triangleArea before you decide that it is "passing?" What parameter inputs would you supply to those calls?
  3. Visit this guide and design a unit test for triangleArea. You can just write the code in your notes: there is no need to compile or execute it now (we will do this in lab instead!).
  4. Recall that floating point types cannot always be compared directly for equality, due to rounding and precision limitations. Change this program to use double data types, and re-generate unit tests for it. Where do you think a floating point tolerance can be added with the assertEquals function?

Model 2: Thinking Critically about Code

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
public class Main {           
    public static double calculateIncomeTax(double income) {
        double result = 0;
         
        double base10 = 9950 * 0.1;
        double base12 = base10 + (40525 - 9950) * 0.12;
        double base22 = base12 + (86375 - 40525) * 0.22;
        double base24 = base22 + (164925 - 86375) * 0.24;
        double base32 = base24 + (209425 - 164925) * 0.32;
        double base35 = base32 + (523600 - 209425) * 0.35;
         
        if(income < 9950) {
            result = income * 0.1;
        } else if(income < 40525) {
            result = base10 + (income - 9950) * 0.12;
        } else if(income < 86375) {
            result = base12 + (income - 40525) * 0.22;
        } else if(income < 164925) {
            result = base22 + (income - 86375) * 0.24;
        } else if(income < 209425) {
            result = base24 + (income - 164925) * 0.32;
        } else if(income < 523600) {
            result = base32 + (income - 209425) * 0.35;
        } else {
            result = base35 + (income - 523600) * 0.37;
        }
         
        return result;
    }
     
    public static void main(String[] args) {
        double taxOwed = calculateIncomeTax(25000);
         
        System.out.println("I owe: $" + taxOwed);
    }
}

Questions

  1. What kinds of inputs would make this function fail (or return values that don't make sense)? What can you do about this?
  2. What tests, at a minimum, would you propose to thoroughly exercise this function?

Model 3: Facilitating Unit Tests

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import java.util.Random;
 
public class Main {           
    public static boolean isHeads() {
        Random rng = new Random();
         
        // Use the random number generator (rng)
        // to generate a value >= 0.0 and < 1.0
        double randomValue = rng.nextDouble();
         
        boolean result = false;
        if(randomValue > 0.5) {
            result = true;
        }
         
        return result;
    }
     
    public static void main(String[] args) {
        System.out.println(isHeads());
        System.out.println(isHeads());
    }
}

Questions

  1. What makes this a difficult function to test?
  2. What could we do to better facilitate testing a function like this? For example, how might running the program and evaluating the output be helpful?
  3. Can black-box testing be automated?

Model 4: Unit Testing and Code Coverage

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def get_grade(numgrade):
    """Returns the letter grade for a course given the numerical  
    grade average.
 
    Parameters
    ----------
    numgrade : float, required
        The number grade between 0-100
 
    Returns
    -------
    The letter grade associated with that numeric grade on a
    90/80/70/60 scale
    """
     
    if numgrade >= 90:
    return "A"
    elif numgrade >= 80:
    return "B"
    elif numgrade >= 70:
    return "C"
    elif numgrade >= 60:
    return "D"
    else:
    return "F"

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import unittest
from main import get_grade
 
# run with python -m unittest test_grade
# generate coverage with coverage run -m unittest test_grade
# view coverage report with coverage report or coverage html
class TestGrades(unittest.TestCase):
 
  def test_A(self):
    self.assertEqual(get_grade(90), "A")
 
  def test_C(self):
    self.assertEqual(get_grade(75), "C")
 
if __name__ == '__main__':
  unittest.main()

Questions

  1. Run the unit tests above and generate a code coverage report.
  2. How can you improve code coverage to 100%, by testing all code branches?
  3. Should you continue to write and run additional unit tests beyond 100% code coverage? Give an example of why this might be necessary.
  4. Would you write your code first or your unit tests first? How might it help to write your unit tests before writing the code?
  5. How can unit testing and code coverage be automated via a github workflow? When would unit testing be executed?

Embedded Code Environment

You can try out some code examples in this embedded development environment! To share this with someone else, first have one member of your group make a small change to the file, then click "Open in Repl.it". Log into your Repl.it account (or create one if needed), and click the "Share" button at the top right. Note that some embedded Repl.it projects have multiple source files; you can see those by clicking the file icon on the left navigation bar of the embedded code frame. Share the link that opens up with your group members. Remember only to do this for partner/group activities!

Submission

I encourage you to submit your answers to the questions (and ask your own questions!) using the Class Activity Questions discussion board. You may also respond to questions or comments made by others, or ask follow-up questions there. Answer any reflective prompt questions in the Reflective Journal section of your OneNote Classroom personal section. You can find the link to the class notebook on the syllabus.