Black box testing

Overview Software testing is the process of verifying that an application works as expected. While developers often reach for basic unit tests, these tests have a fundamental limitation: they can show the presence of bugs but never their absence. Testing a finite number of cases cannot account for every possible side effect or state. This guide explores the theoretical foundations of program correctness through Hoare logic and introduces advanced techniques like mutation and property-based testing to build more robust software. Prerequisites To follow this tutorial, you should have a solid grasp of Python basics, including functions and loops. Familiarity with basic assertions and the concept of a unit test will help you understand the more advanced testing paradigms discussed here. Key Libraries & Tools - **Mutmut**: A Python library for mutation testing that automatically modifies your source code to see if your tests catch the changes. - **Hypothesis**: A powerful property-based testing library that generates random data to find edge cases where your code might fail. - **Jest**: Mentioned as a common tool for snapshot testing in the JavaScript ecosystem. Code Walkthrough The Limits of Basic Unit Testing Consider a simple function meant to add three to an integer. We might write tests that pass for specific inputs, but those tests can be "cheated" by poor implementation. ```python def add_three(x: int) -> int: if x == 1: return 4 elif x == 2: return 5 return 0 # Fails for any other input These assertions pass, but the code is broken assert add_three(1) == 4 assert add_three(2) == 5 ``` This highlights why we need broader testing strategies. Even with infinite tests, we can't prove correctness for all side effects, such as a function that only fails on a specific date. Mutation Testing Mutation testing introduces "mutants"—slight modifications to your code—to see if your test suite is actually effective. If you change a `+` to a `-` and your tests still pass, your tests are weak. ```python def multiply_by_two(x: int) -> int: return x * 2 Test case assert multiply_by_two(2) == 4 Mutation: change 'x * 2' to 'x + 2' The test STILL passes because 2 * 2 == 2 + 2. This mutant survived, meaning we need more varied test cases. ``` Property-Based Testing Property-based testing checks if a general property (an invariant) holds true across a wide range of inputs. Instead of choosing specific numbers, we test the relationship between functions. ```python import random def add_three(x: int) -> int: return x + 3 def remove_three(x: int) -> int: return x - 3 Property: adding 3 then removing 3 should return the original number for _ in range(100): x = random.randint(-1000, 1000) assert remove_three(add_three(x)) == x ``` Syntax Notes In the examples above, we use simple `assert` statements. In a production environment, you would use frameworks like Pytest. Note the use of **type hints** (`x: int -> int`), which provide static testing benefits by helping IDEs catch type mismatches before the code even runs. Practical Examples - **Bilbo Testing**: Also known as "There and Back Again," this is perfect for encoders/decoders. If you encrypt a string and then decrypt it, you must get the original string back. - **Sorting Invariants**: When testing a sorting algorithm, a key property is that the length of the list should never change, regardless of the input data. - **Data Processing**: Ensure that a processing function never returns a dictionary with empty fields when given valid random inputs. Tips & Gotchas Avoid relying solely on manual test cases. Humans are biased toward "happy paths" and often miss edge cases. Use randomized testing to uncover scenarios you didn't anticipate. However, remember that mutation testing is computationally expensive; start by running it on your most critical logic rather than the entire codebase.