Structuring Tests

• How to write tests so that:
  • It's easy to add or change tests
  • It's easy to see what's been tested, and what hasn't
• A test consists of a fixture, an action, and an expected result
  • A fixture is something that a test is run on
  • Can be as simple as a single value, or as complex as a networked database

• Every test should be independent
  • I.e., the outcome of one test shouldn't depend on what happened in another test
  • Otherwise, faults in early tests can distort the results of later ones
• So each test:
  • Creates a fresh instance of the fixture
  • Performs the operation
  • Checks and records the result

A Simple Example

• Test string.startswith
  • Specification: returns True if the string starts with the given prefix, and False otherwise
  • Hm… What if the prefix is the empty string?
• Store the tests in a table
  • Easy to read and add to

  Tests = [
  # String  Prefix  Expected
  ['a',     'a',    True],
  ['a',     'b',    False],
  ['abc',   'a',    True],
  ['abc',   'ab',   True],
  ['abc',   'abc',  True],
  ['abc',   'abcd', False],
  ['abc',   '',     True]
  ]

  • String and prefix are fixture

• Now run them

  passes = 0
  failures = 0
  for (s, p, expected) in Tests:
  actual = s.startswith(p)
  if actual == expected:
      passes += 1
  else:
      failures += 1
  print 'passed', passes, 'out of', passes+failures, 'tests'

  • Hm… Where's the code to handle and report errors in the tests themselves?

Simple Exception Example

• Try dividing by zero and some non-zero values:

  for num in [-1, 0, 1]:
  try:
      inverse = 1/num
  except:
      print 'inverting', num, 'caused error'
  else:
      print 'inverse of', num, 'is', inverse
  

  inverse of -1 is -1
  inverting 0 caused error
  inverse of 1 is 1
  

Exception Objects

• When Python raises an exception, it creates an object to hold information about what went wrong
  • Typically contains an error message

• Can choose which errors to handle by specifying an exception type in the except statement
  • E.g., handle division by zero, but not out-of-bounds list index

  # Note: mix of numeric and non-numeric values.
  values = [0, 1, 'momentum']
  
  # Note: top index will be out of bounds.
  for i in range(4):
  try:
      print 'dividing by value', i
      x = 1.0 / values[i]
      print 'result is', x
  except ZeroDivisionError, e:
      print 'divide by zero:', e
  except IndexError, e:
      print 'index error:', e
  except:
      print 'some other error:', e
  

  dividing by value 0
  divide by zero: float division
  dividing by value 1
  result is 1.0
  dividing by value 2
  some other error: float division
  dividing by value 3
  index error: list index out of range
  

• The except blocks are tested in order—whichever matches first, wins
  • If a “naked” except appears, it must come last (since it catches everything)
  • Generally better to use except Exception, e so that you have the exception object

Exception Hierarchy

• Exceptions are organized in a hierarchy
  • e.g., ZeroDivisionError, OverflowError, and FloatingPointError are all types of ArithmeticError
  • A handler for the general type catches all its specific sub-types

Functions and Exceptions

Figure 11.2: Stacking Exception Handlers

• Each time Python enters a try/except block, it pushes the except handlers on a stack
  • Just like the function call stack

• When an exception is raised, Python searches this stack for the top-most matching handler
  • Often means jumping out of the middle of a function

    def invert(vals, index):
    try:
        vals[index] = 10.0/vals[index]
    except ArithmeticError, e:
        print 'inner exception handler:', e
    
    def each(vals, indices):
    try:
        for i in indices:
    	invert(vals, i)
    except IndexError, e:
        print 'outer exception handler:', e
    
    # Once again, the top index will be out of bounds.
    values = [-1, 0, 1]
    print 'values before:', values
    each(values, range(4))
    print 'values after:', values
    

    values before: [-1, 0, 1]
    inner exception handler: float division
    outer exception handler: list index out of range
    values after: [-10.0, 0, 10.0]
    

Handling Errors in Tests

• Now know how to check for errors in tests
  • Wrap the test in try/except

  Tests = [
      ['a',     'a',    False],    # wrong expected value
      ['a',     1,      False],    # wrong type
      ['abc',   'a',    True]      # everything legal
  ]
  
  passes = failures = errors = 0
  for (s, p, expected) in Tests:
      try:
          actual = s.startswith(p)
          if actual == expected:
              passes += 1
          else:
              failures += 1
      except:
          errors += 1
  
  print 'tests:', passes + failures + errors
  print 'passes:', passes
  print 'failures:', failures
  print 'errors:', errors
  

• tests: 3
  passes: 1
  failures: 1
  errors: 1
  

• Note the deliberate errors in the test cases to exercise the testing code

Test-Driven Design

• Tests are actually specifications
  • “Given these inputs, this code should behave the following way”
• So write the tests first, then the application code
  • Test-driven development (TDD)

• Sounds backward, but:
  • A great way to clarify specifications
    • I write the tests
    • “All” you have to do is write code that passes those tests
  • Gives programmers a definite goal
    • Coding is finished when all tests run
    • Particularly useful when trying to fix bugs in old code, as it forces you to figure out how to re-create the bug
    • Helps prevent the “one more feature” syndrome
  • Ensures that tests actually get written
    • People are often too tired, or too rushed, to test after coding
  • Helps clarify the Application Programming Interface (API) before it is set in stone
    • If something is awkward to test, it can be redesigned before it's written

Design by Contract

• Functions ought to carry their specifications around with them
  • Keeping specification and implementation together makes both easier to understand
  • And improves the odds that programmers will keep them in sync
• A function is defined by:
  • Its pre-conditions: what must be true in order for the function to work correctly
  • Its post-conditions: what the function guarantees will be true if its pre-conditions are met
  • May also have invariants: things that are true throughout the execution of the function

• Leads to a style of programming called design by contract
• Pre- and post-conditions constrain how the function can evolve
  • Can only ever relax pre-conditions (i.e., take a wider range of input)…
  • …or tighten post-conditions (i.e., produce a narrower range of output)
  • Tightening pre-conditions, or relaxing post-conditions, would violate the function's contract with its callers

Assertions

• Normally specify pre- and post-conditions using assertions
  • A statement that something is true at a particular point in a program
  • If the assertion's condition is not met, Python raises an AssertionError exception
• For example:
  • Pre-condition: input argument is a non-empty list
  • Post-condition: two values from the list such that the first is less than the second

  def find_range(values):
  '''Find the non-empty range of values in the input sequence.'''
  assert (type(values) is list) and (len(values) > 0)
  left = min(values)
  right = max(values)
  assert (left in values) and (right in values) and (left <= right)
  return left, right
  

• Note that the post-condition isn't as exacting as it should be
  • Doesn't check that left is less than or equal to all other values, or that right is greater than or equal to
  • The code to check the condition exactly is as likely to contain errors as the function itself
  • Which is one of the reasons design by contract isn't as popular as it might be

Defensive Programming

• You can (and should) use assert liberally
  • Even if you don't practice design by contract
• Defensive programming is like defensive driving
  • Program as if the rest of the world is out to get you
  • “Fail early, fail often”
    • The less distance there is between the error and you detecting it, the easier it will be to find and fix

• Good practice: every time you fix a bug, put in an assertion and a comment
  • Because if you made the error, the right code can't be obvious
  • And you should protect yourself against someone “simplifying” the bug back in

  def can_transmute(element):
    '''Can this element be turned into gold?'''
  
    # Bug #172: make sure the input is actually an element.
    assert is_valid_element(element)
  
    # Gold is trivial.
    if element is Gold:
        return True
  
    # Trans-uranic metals and halogens are impossible.
    if (element.atomic_number > Uranium.atomic_number) or \
       (element in Halogens):
        return False
  
    # Look for a sequence of steps that leads to gold.
    steps = search_transmutations(element, Gold)
    if steps == []:
        return False
    else:
        # Bug #201: must be at least two elements in sequence.
        assert len(steps) >= 2
        return True
      

Rule 0: Get It Right the First Time

• The simplest bugs to fix are the ones that don't exist
• Design, reflect, discuss, then code
  • “A week of hard work can sometimes save you an hour of thought.”
• Design and build your code with testing and debugging in mind
  • Minimize the amount of “spooky action at a distance”
  • Minimize the number of things programmers have to keep track of at any one time
  • Train yourself to do things right, so that you'll code well even when you're tired, stressed, and facing a deadline

• “Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?” (Brian Kernighan)

Rule 4: Divide and Conquer

• The smaller the gap between cause and effect, the easier the relationship is to see
• So once you have a test that makes the system fail, use it isolate the faulty subsystem
  • Examine the input of the code that's failing
  • If that's wrong, look at the preceding code's input, and so on
• Use assert to check things that ought to be right
  • “Fail early, fail often”
  • A good way to stop yourself from introducing new bugs as you fix old ones

• When you do fix the bug, see whether you can add assertions to prevent it reappearing
  • If you made the mistake once, odds are that you, or someone, will make it again
• Another argument against duplicated code
  • Few things are as frustrating as fixing a bug, only to have it crop up again elsewhere

What's Wrong with Print Statements

• Many people still debug by adding print statements to their programs
• It's error-prone
  • Adding print statements is a good way to add typos
  • Particularly when you have to modify the block structure of your program
• And time-consuming
  • All that typing…
  • And (if you're using Java, C++, or Fortran) all that recompiling…

• And can be misleading
  • Moves things around in memory, changes execution timing, etc.
  • Common for bugs to hide when print statements are added, and reappear when they're removed
• But can be extremely effective
  • May be added using the same tools as programming
  • Can collect lots of data in a single run

Symbolic Debuggers

• A debugger is a program that runs another program on your behalf
  • Sometimes called a symbolic debugger because it shows you the source code you wrote, rather than raw machine code
• While the target program (or debuggee) is running, the debugger can:
  • Pause, resume, or restart the target
  • Display or change values
  • Watch for calls to particular functions, changes to particular variables, etc.

• Do not need to modify the source of the target program!
  • Depending on your language, you may need to compile it with different flags
• And yes, the debugger modifies the target's layout in memory, and execution speed…
  • …but a lot less than print statements…
  • …with a lot less effort from you

Speeding Things Up

• Replace hot spots with faster code
• If a collection of data will be searched repeatedly, instead of a linear list, use a sorted list or a dictionary
• There is usually a tradeoff between bookkeeping overhead and search speed when using “faster” data structures, e.g.,
  • Keeping a list sorted
  • Managing a dictionary
• Which data structure is “best” depends on the data

• Restructure the entire program
  • Take a different approach to solving your problem
  • Throw hardware at “embarrassingly parallel” problems
    • If you have access to a computing cluster, and the problem can be partitioned into multiple jobs easily, use one CPU per job to improve performance
    • Embarrassingly parallel tasks include:
      • Processing multiple independent data sets
      • Repeating simulations with different initial conditions
    • Write a shell script to:
      • Partition big job into a bunch of little jobs
      • Run the jobs, either directly or submitting it into a batch job queue
      • Wait for all jobs to complete
      • Collate results