>>> list(itertools.permutations('ABC', 3))

[('A', 'B', 'C'), ('A', 'C', 'B'),

广告:个人专属 VPN,独立 IP,无限流量,多机房切换,还可以屏蔽广告和恶意软件,每月最低仅 5 美元

('B', 'A', 'C'), ('B', 'C', 'A'),

('C', 'A', 'B'), ('C', 'B', 'A')]

1. A string is just a sequence of characters. For the purposes of finding permutations, the string 'ABC' is

equivalent to the list ['A', 'B', 'C'].

2. The first permutation of the 3 items ['A', 'B', 'C'], taken 3 at a time, is ('A', 'B', 'C'). There are

five other permutations — the same three characters in every conceivable order.

3. Since the permutations() function always returns an iterator, an easy way to debug permutations is to pass

that iterator to the built-in list() function to see all the permutations immediately.

198

8.7. OTHER FUN STUFF IN THE itertools MODULE

>>> import itertools

>>> list(itertools.product('ABC', '123'))

[('A', '1'), ('A', '2'), ('A', '3'),

('B', '1'), ('B', '2'), ('B', '3'),

('C', '1'), ('C', '2'), ('C', '3')]

>>> list(itertools.combinations('ABC', 2))

[('A', 'B'), ('A', 'C'), ('B', 'C')]

1. The itertools.product() function returns an iterator containing the Cartesian product of two sequences.

2. The itertools.combinations() function returns an iterator containing all the possible combinations of the

given sequence of the given length. This is like the itertools.permutations() function, except

combinations don’t include items that are duplicates of other items in a different order. So

itertools.permutations('ABC', 2) will return both ('A', 'B') and ('B', 'A') (among others), but

itertools.combinations('ABC', 2) will not return ('B', 'A') because it is a duplicate of ('A', 'B') in

a different order.

>>> names = list(open('examples/favorite-people.txt', encoding='utf-8'))

>>> names

['Dora\n', 'Ethan\n', 'Wesley\n', 'John\n', 'Anne\n',

'Mike\n', 'Chris\n', 'Sarah\n', 'Alex\n', 'Lizzie\n']

>>> names = [name.rstrip() for name in names]

>>> names

['Dora', 'Ethan', 'Wesley', 'John', 'Anne',

'Mike', 'Chris', 'Sarah', 'Alex', 'Lizzie']

>>> names = sorted(names)

>>> names

['Alex', 'Anne', 'Chris', 'Dora', 'Ethan',

'John', 'Lizzie', 'Mike', 'Sarah', 'Wesley']

>>> names = sorted(names, key=len)

>>> names

['Alex', 'Anne', 'Dora', 'John', 'Mike',

'Chris', 'Ethan', 'Sarah', 'Lizzie', 'Wesley']

199

1. This idiom returns a list of the lines in a text file.

2. Unfortunately (for this example), the list(open(filename)) idiom also includes the carriage returns at the

end of each line. This list comprehension uses the rstrip() string method to strip trailing whitespace from

each line. (Strings also have an lstrip() method to strip leading whitespace, and a strip() method which

strips both.)

3. The sorted() function takes a list and returns it sorted. By default, it sorts alphabetically.

4. But the sorted() function can also take a function as the key parameter, and it sorts by that key. In this

case, the sort function is len(), so it sorts by len(each item). Shorter names come first, then longer, then

longest.

What does this have to do with the itertools module? I’m glad you asked.

200

…continuing from the previous interactive shell…

>>> import itertools

>>> groups = itertools.groupby(names, len)

>>> groups

<itertools.groupby object at 0x00BB20C0>

>>> list(groups)

[(4, <itertools._grouper object at 0x00BA8BF0>),

(5, <itertools._grouper object at 0x00BB4050>),

(6, <itertools._grouper object at 0x00BB4030>)]

>>> groups = itertools.groupby(names, len)

>>> for name_length, name_iter in groups:

...

print('Names with {0:d} letters:'.format(name_length))

...

for name in name_iter:

...

print(name)

...

Names with 4 letters:

Alex

Anne

Dora

John

Mike

Names with 5 letters:

Chris

Ethan

Sarah

Names with 6 letters:

Lizzie

Wesley

1. The itertools.groupby() function takes a sequence and a key function, and returns an iterator that

generates pairs. Each pair contains the result of key_function(each item) and another iterator containing

all the items that shared that key result.

2. Calling the list() function “exhausted” the iterator, i.e. you’ve already generated every item in the iterator

to make the list. There’s no “reset” button on an iterator; you can’t just start over once you’ve exhausted

201

it. If you want to loop through it again (say, in the upcoming for loop), you need to call

itertools.groupby() again to create a new iterator.

3. In this example, given a list of names already sorted by length, itertools.groupby(names, len) will put all

the 4-letter names in one iterator, all the 5-letter names in another iterator, and so on. The groupby()

function is completely generic; it could group strings by first letter, numbers by their number of factors, or

any other key function you can think of.

☞ The itertools.groupby() function only works if the input sequence is already

sorted by the grouping function. In the example above, you grouped a list of names

by the len() function. That only worked because the input list was already sorted by

length.

Are you watching closely?

>>> list(range(0, 3))

[0, 1, 2]

>>> list(range(10, 13))

[10, 11, 12]

>>> list(itertools.chain(range(0, 3), range(10, 13)))

[0, 1, 2, 10, 11, 12]

>>> list(zip(range(0, 3), range(10, 13)))

[(0, 10), (1, 11), (2, 12)]

>>> list(zip(range(0, 3), range(10, 14)))

[(0, 10), (1, 11), (2, 12)]

>>> list(itertools.zip_longest(range(0, 3), range(10, 14)))

[(0, 10), (1, 11), (2, 12), (None, 13)]

1. The itertools.chain() function takes two iterators and returns an iterator that contains all the items

from the first iterator, followed by all the items from the second iterator. (Actually, it can take any number

of iterators, and it chains them all in the order they were passed to the function.)

202

2. The zip() function does something prosaic that turns out to be extremely useful: it takes any number of

sequences and returns an iterator which returns tuples of the first items of each sequence, then the second

items of each, then the third, and so on.

3. The zip() function stops at the end of the shortest sequence. range(10, 14) has 4 items (10, 11, 12, and

13), but range(0, 3) only has 3, so the zip() function returns an iterator of 3 items.

4. On the other hand, the itertools.zip_longest() function stops at the end of the longest sequence,

inserting None values for items past the end of the shorter sequences.

OK, that was all very interesting, but how does it relate to the alphametics solver? Here’s how:

>>> characters = ('S', 'M', 'E', 'D', 'O', 'N', 'R', 'Y')

>>> guess = ('1', '2', '0', '3', '4', '5', '6', '7')

>>> tuple(zip(characters, guess))

(('S', '1'), ('M', '2'), ('E', '0'), ('D', '3'),

('O', '4'), ('N', '5'), ('R', '6'), ('Y', '7'))

>>> dict(zip(characters, guess))

{'E': '0', 'D': '3', 'M': '2', 'O': '4',

'N': '5', 'S': '1', 'R': '6', 'Y': '7'}

1. Given a list of letters and a list of digits (each represented here as 1-character strings), the zip function will

create a pairing of letters and digits, in order.

2. Why is that cool? Because that data structure happens to be exactly the right structure to pass to the

dict() function to create a dictionary that uses letters as keys and their associated digits as values. (This

isn’t the only way to do it, of course. You could use a dictionary comprehension to create the dictionary directly.) Although the printed representation of the dictionary lists the pairs in a different order (dictionaries

have no “order” per se), you can see that each letter is associated with the digit, based on the ordering of

the original characters and guess sequences.

The alphametics solver uses this technique to create a dictionary that maps letters in the puzzle to digits in

the solution, for each possible solution.

203

characters = tuple(ord(c) for c in sorted_characters)

digits = tuple(ord(c) for c in '0123456789')

...

for guess in itertools.permutations(digits, len(characters)):

...

equation = puzzle.translate(dict(zip(characters, guess)))

But what is this translate() method? Ah, now you’re getting to the really fun part.

8.8. A NEW KIND OF STRING MANIPULATION

Python strings have many methods. You learned about some of those methods in the Strings chapter:

lower(), count(), and format(). Now I want to introduce you to a powerful but little-known string

manipulation technique: the translate() method.

>>> translation_table = {ord('A'): ord('O')}

>>> translation_table

{65: 79}

>>> 'MARK'.translate(translation_table)

'MORK'

1. String translation starts with a translation table, which is just a dictionary that maps one character to

another. Actually, “character” is incorrect — the translation table really maps one byte to another.

2. Remember, bytes in Python 3 are integers. The ord() function returns the ASCII value of a character,

which, in the case of A–Z, is always a byte from 65 to 90.

3. The translate() method on a string takes a translation table and runs the string through it. That is, it

replaces all occurrences of the keys of the translation table with the corresponding values. In this case,

“translating” MARK to MORK.

204

What does this have to do with solving alphametic

puzzles? As it turns out, everything.

Now you’re

getting to

the really

fun part.

>>> characters = tuple(ord(c) for c in 'SMEDONRY')

>>> characters

(83, 77, 69, 68, 79, 78, 82, 89)

>>> guess = tuple(ord(c) for c in '91570682')

>>> guess

(57, 49, 53, 55, 48, 54, 56, 50)

>>> translation_table = dict(zip(characters, guess))

>>> translation_table

{68: 55, 69: 53, 77: 49, 78: 54, 79: 48, 82: 56, 83: 57, 89: 50}

>>> 'SEND + MORE == MONEY'.translate(translation_table)

'9567 + 1085 == 10652'

1. Using a generator expression, we quickly compute the byte values for each character in a string. characters is an example of the value of sorted_characters in the alphametics.solve() function.

2. Using another generator expression, we quickly compute the byte values for each digit in this string. The

result, guess, is of the form returned by the itertools.permutations() function in the

alphametics.solve() function.

205

3. This translation table is generated by zipping characters and guess together and building a dictionary from the resulting sequence of pairs. This is exactly what the alphametics.solve() function does inside the for

loop.

4. Finally, we pass this translation table to the translate() method of the original puzzle string. This converts

each letter in the string to the corresponding digit (based on the letters in characters and the digits in

guess). The result is a valid Python expression, as a string.

That’s pretty impressive. But what can you do with a string that happens to be a valid Python expression?

8.9. EVALUATING ARBITRARY STRINGS AS PYTHON EXPRESSIONS

This is the final piece of the puzzle (or rather, the final piece of the puzzle solver). After all that fancy string

manipulation, we’re left with a string like '9567 + 1085 == 10652'. But that’s a string, and what good is a

string? Enter eval(), the universal Python evaluation tool.

>>> eval('1 + 1 == 2')

True

>>> eval('1 + 1 == 3')

False

>>> eval('9567 + 1085 == 10652')

True

But wait, there’s more! The eval() function isn’t limited to boolean expressions. It can handle any Python

expression and returns any datatype.

206

>>> eval('"A" + "B"')

'AB'

>>> eval('"MARK".translate({65: 79})')

'MORK'

>>> eval('"AAAAA".count("A")')

5

>>> eval('["*"] * 5')

['*', '*', '*', '*', '*']

But wait, that’s not all!

>>> x = 5

>>> eval("x * 5")

25

>>> eval("pow(x, 2)")

25

>>> import math

>>> eval("math.sqrt(x)")

2.2360679774997898

1. The expression that eval() takes can reference global variables defined outside the eval(). If called within a

function, it can reference local variables too.

2. And functions.

3. And modules.

Hey, wait a minute…

>>> import subprocess

>>> eval("subprocess.getoutput('ls ~')")

'Desktop Library Pictures \

Documents Movies Public \

Music Sites'

>>> eval("subprocess.getoutput('rm /some/random/file')")

207

1. The subprocess module allows you to run arbitrary shell commands and get the result as a Python string.

2. Arbitrary shell commands can have permanent consequences.

It’s even worse than that, because there’s a global __import__() function that takes a module name as a

string, imports the module, and returns a reference to it. Combined with the power of eval(), you can

construct a single expression that will wipe out all your files:

>>> eval("__import__('subprocess').getoutput('rm /some/random/file')")

1. Now imagine the output of 'rm -rf ~'. Actually there wouldn’t be any output, but you wouldn’t have any

files left either.

eval() is

EVIL

Well, the evil part is evaluating arbitrary expressions from untrusted sources. You should only use eval()

on trusted input. Of course, the trick is figuring out what’s “trusted.” But here’s something I know for

certain: you should NOT take this alphametics solver and put it on the internet as a fun little web service.

Don’t make the mistake of thinking, “Gosh, the function does a lot of string manipulation before getting a

208

string to evaluate; I can’t imagine how someone could exploit that.” Someone WILL figure out how to sneak

nasty executable code past all that string manipulation (stranger things have happened), and then you can kiss your server goodbye.

But surely there’s some way to evaluate expressions safely? To put eval() in a sandbox where it can’t

access or harm the outside world? Well, yes and no.

>>> x = 5

>>> eval("x * 5", {}, {})

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "<string>", line 1, in <module>

NameError: name 'x' is not defined

>>> eval("x * 5", {"x": x}, {})

>>> import math

>>> eval("math.sqrt(x)", {"x": x}, {})

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "<string>", line 1, in <module>

NameError: name 'math' is not defined

1. The second and third parameters passed to the eval() function act as the global and local namespaces for

evaluating the expression. In this case, they are both empty, which means that when the string "x * 5" is

evaluated, there is no reference to x in either the global or local namespace, so eval() throws an

exception.

2. You can selectively include specific values in the global namespace by listing them individually. Then

those — and only those — variables will be available during evaluation.

3. Even though you just imported the math module, you didn’t include it in the namespace passed to the

eval() function, so the evaluation failed.

Gee, that was easy. Lemme make an alphametics web service now!

209

>>> eval("pow(5, 2)", {}, {})

25

>>> eval("__import__('math').sqrt(5)", {}, {})

2.2360679774997898

1. Even though you’ve passed empty dictionaries for the global and local namespaces, all of Python’s built-in

functions are still available during evaluation. So pow(5, 2) works, because 5 and 2 are literals, and pow() is

a built-in function.

2. Unfortunately (and if you don’t see why it’s unfortunate, read on), the __import__() function is also a built-

in function, so it works too.

Yeah, that means you can still do nasty things, even if you explicitly set the global and local namespaces to

empty dictionaries when calling eval():

>>> eval("__import__('subprocess').getoutput('rm /some/random/file')", {}, {})

Oops. I’m glad I didn’t make that alphametics web service. Is there any way to use eval() safely? Well, yes

and no.

>>> eval("__import__('math').sqrt(5)",

...

{"__builtins__":None}, {})

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "<string>", line 1, in <module>

NameError: name '__import__' is not defined

>>> eval("__import__('subprocess').getoutput('rm -rf /')",

...

{"__builtins__":None}, {})

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "<string>", line 1, in <module>

NameError: name '__import__' is not defined

1. To evaluate untrusted expressions safely, you need to define a global namespace dictionary that maps

"__builtins__" to None, the Python null value. Internally, the “built-in” functions are contained within a

210

pseudo-module called "__builtins__". This pseudo-module ( i.e. the set of built-in functions) is made

available to evaluated expressions unless you explicitly override it.

2. Be sure you’ve overridden __builtins__. Not __builtin__, __built-ins__, or some other variation that

will work just fine but expose you to catastrophic risks.

So eval() is safe now? Well, yes and no.

>>> eval("2 ** 2147483647",

...

{"__builtins__":None}, {})

1. Even without access to __builtins__, you can still launch a denial-of-service attack. For example, trying to

raise 2 to the 2147483647th power will spike your server’s CPU utilization to 100% for quite some time. (If

you’re trying this in the interactive shell, press Ctrl-C a few times to break out of it.) Technically this

expression will return a value eventually, but in the meantime your server will be doing a whole lot of

nothing.

In the end, it is possible to safely evaluate untrusted Python expressions, for some definition of “safe” that

turns out not to be terribly useful in real life. It’s fine if you’re just playing around, and it’s fine if you only

ever pass it trusted input. But anything else is just asking for trouble.

8.10. PUTTING IT ALL TOGETHER

To recap: this program solves alphametic puzzles by brute force, i.e. through an exhaustive search of all

possible solutions. To do this, it…

1. Finds all the letters in the puzzle with the re.findall() function

2. Find all the unique letters in the puzzle with sets and the set() function

3. Checks if there are more than 10 unique letters (meaning the puzzle is definitely unsolvable) with an assert statement

4. Converts the letters to their ASCII equivalents with a generator object

211

5. Calculates all the possible solutions with the itertools.permutations() function

6. Converts each possible solution to a Python expression with the translate() string method

7. Tests each possible solution by evaluating the Python expression with the eval() function

8. Returns the first solution that evaluates to True

…in just 14 lines of code.

8.11. FURTHER READING

itertools module

itertools — Iterator functions for efficient looping

Watch Raymond Hettinger’s “Easy AI with Python” talk at PyCon 2009

Recipe 576615: Alphametics solver, Raymond Hettinger’s original alphametics solver for Python 2

More of Raymond Hettinger’s recipes in the ActiveState Code repository

Alphametics on Wikipedia

Alphametics Index, including lots of puzzles and a generator to make your own

Many thanks to Raymond Hettinger for agreeing to relicense his code so I could port it to Python 3 and use

it as the basis for this chapter.

212

CHAPTER 9. UNIT TESTING

Certitude is not the test of certainty. We have been cocksure of many things that were not so.

Oliver Wendell Holmes, Jr.

9.1. (NOT) DIVING IN

Kidstoday.Sospoiledbythesefastcomputersandfancy“dynamic”languages.Writefirst,shipsecond,

debug third (if ever). In my day, we had discipline. Discipline, I say! We had to write programs by hand, on

paper, and feed them to the computer on punchcards. And we liked it!

In this chapter, you’re going to write and debug a set of utility functions to convert to and from Roman

numerals. You saw the mechanics of constructing and validating Roman numerals in “Case study: roman

numerals”. Now step back and consider what it would take to expand that into a two-way utility.

The rules for Roman numerals lead to a number of interesting observations:

1. There is only one correct way to represent a particular number as a Roman numeral.

2. The converse is also true: if a string of characters is a valid Roman numeral, it represents only one number

(that is, it can only be interpreted one way).

3. There is a limited range of numbers that can be expressed as Roman numerals, specifically 1 through 3999.

The Romans did have several ways of expressing larger numbers, for instance by having a bar over a numeral

to represent that its normal value should be multiplied by 1000. For the purposes of this chapter, let’s

stipulate that Roman numerals go from 1 to 3999.

4. There is no way to represent 0 in Roman numerals.

5. There is no way to represent negative numbers in Roman numerals.

6. There is no way to represent fractions or non-integer numbers in Roman numerals.

213

Let’s start mapping out what a roman.py module should do. It will have two main functions, to_roman() and

from_roman(). The to_roman() function should take an integer from 1 to 3999 and return the Roman

numeral representation as a string…

Stop right there. Now let’s do something a little unexpected: write a test case that checks whether the

to_roman() function does what you want it to. You read that right: you’re going to write code that tests

code that you haven’t written yet.

This is called test-driven development, or TDD. The set of two conversion functions — to_roman(), and later

from_roman() — can be written and tested as a unit, separate from any larger program that imports them.

Python has a framework for unit testing, the appropriately-named unittest module.

Unit testing is an important part of an overall testing-centric development strategy. If you write unit tests, it

is important to write them early and to keep them updated as code and requirements change. Many people

advocate writing tests before they write the code they’re testing, and that’s the style I’m going to

demonstrate in this chapter. But unit tests are beneficial no matter when you write them.

• Before writing code, writing unit tests forces you to detail your requirements in a useful fashion.

• While writing code, unit tests keep you from over-coding. When all the test cases pass, the function is

complete.

• When refactoring code, they can help prove that the new version behaves the same way as the old version.

• When maintaining code, having tests will help you cover your ass when someone comes screaming that your

latest change broke their old code. (“But sir, all the unit tests passed when I checked it in...”)

• When writing code in a team, having a comprehensive test suite dramatically decreases the chances that

your code will break someone else’s code, because you can run their unit tests first. (I’ve seen this sort of

thing in code sprints. A team breaks up the assignment, everybody takes the specs for their task, writes unit

tests for it, then shares their unit tests with the rest of the team. That way, nobody goes off too far into

developing code that doesn’t play well with others.)

214

9.2. A SINGLE QUESTION

A test case answers a single question about the code it

is testing. A test case should be able to...

• ...run completely by itself, without any human input. Unit

testing is about automation.

Every test is

• ...determine by itself whether the function it is testing

has passed or failed, without a human interpreting the

an island.

results.

• ...run in isolation, separate from any other test cases

(even if they test the same functions). Each test case is

an island.

Given that, let’s build a test case for the first requirement:

1. The to_roman() function should return the Roman numeral representation for all integers 1 to 3999.

It is not immediately obvious how this code does… well, anything. It defines a class which has no

__init__() method. The class does have another method, but it is never called. The entire script has a

__main__ block, but it doesn’t reference the class or its method. But it does do something, I promise.

215

import roman1

import unittest

class KnownValues(unittest.TestCase):

known_values = ( (1, 'I'),

(2, 'II'),

(3, 'III'),

(4, 'IV'),

(5, 'V'),

(6, 'VI'),

(7, 'VII'),

(8, 'VIII'),

(9, 'IX'),

(10, 'X'),

(50, 'L'),

(100, 'C'),

(500, 'D'),

(1000, 'M'),

(31, 'XXXI'),

(148, 'CXLVIII'),

(294, 'CCXCIV'),

(312, 'CCCXII'),

(421, 'CDXXI'),

(528, 'DXXVIII'),

(621, 'DCXXI'),

(782, 'DCCLXXXII'),

(870, 'DCCCLXX'),

(941, 'CMXLI'),

(1043, 'MXLIII'),

(1110, 'MCX'),

(1226, 'MCCXXVI'),

(1301, 'MCCCI'),

(1485, 'MCDLXXXV'),

(1509, 'MDIX'),

216

(1607, 'MDCVII'),

(1754, 'MDCCLIV'),

(1832, 'MDCCCXXXII'),

(1993, 'MCMXCIII'),

(2074, 'MMLXXIV'),

(2152, 'MMCLII'),

(2212, 'MMCCXII'),

(2343, 'MMCCCXLIII'),

(2499, 'MMCDXCIX'),

(2574, 'MMDLXXIV'),

(2646, 'MMDCXLVI'),

(2723, 'MMDCCXXIII'),

(2892, 'MMDCCCXCII'),

(2975, 'MMCMLXXV'),

(3051, 'MMMLI'),

(3185, 'MMMCLXXXV'),

(3250, 'MMMCCL'),

(3313, 'MMMCCCXIII'),

(3408, 'MMMCDVIII'),

(3501, 'MMMDI'),

(3610, 'MMMDCX'),

(3743, 'MMMDCCXLIII'),

(3844, 'MMMDCCCXLIV'),

(3888, 'MMMDCCCLXXXVIII'),

(3940, 'MMMCMXL'),

(3999, 'MMMCMXCIX'))

def test_to_roman_known_values(self):

'''to_roman should give known result with known input'''

for integer, numeral in self.known_values:

result = roman1.to_roman(integer)

self.assertEqual(numeral, result)

217

if __name__ == '__main__':

unittest.main()

1. To write a test case, first subclass the TestCase class of the unittest module. This class provides many

useful methods which you can use in your test case to test specific conditions.

2. This is a list of integer/numeral pairs that I verified manually. It includes the lowest ten numbers, the highest

number, every number that translates to a single-character Roman numeral, and a random sampling of other

valid numbers. You don’t need to test every possible input, but you should try to test all the obvious edge

cases.

3. Every individual test is its own method. A test method takes no parameters, returns no value, and must have

a name beginning with the four letters test. If a test method exits normally without raising an exception,

the test is considered passed; if the method raises an exception, the test is considered failed.

4. Here you call the actual to_roman() function. (Well, the function hasn’t been written yet, but once it is, this

is the line that will call it.) Notice that you have now defined the API for the to_roman() function: it must

take an integer (the number to convert) and return a string (the Roman numeral representation). If the API

is different than that, this test is considered failed. Also notice that you are not trapping any exceptions

when you call to_roman(). This is intentional. to_roman() shouldn’t raise an exception when you call it with

valid input, and these input values are all valid. If to_roman() raises an exception, this test is considered

failed.

5. Assuming the to_roman() function was defined correctly, called correctly, completed successfully, and

returned a value, the last step is to check whether it returned the right value. This is a common question,

and the TestCase class provides a method, assertEqual, to check whether two values are equal. If the

result returned from to_roman() (result) does not match the known value you were expecting (numeral),

assertEqual will raise an exception and the test will fail. If the two values are equal, assertEqual will do

nothing. If every value returned from to_roman() matches the known value you expect, assertEqual never

raises an exception, so test_to_roman_known_values eventually exits normally, which means to_roman()

has passed this test.

218

Once you have a test case, you can start coding the

to_roman() function. First, you should stub it out as an

empty function and make sure the tests fail. If the tests

succeed before you’ve written any code, your tests

aren’t testing your code at all! Unit testing is a dance:

Write a test

tests lead, code follows. Write a test that fails, then

code until it passes.

that fails,

# roman1.py

then code

def to_roman(n):

'''convert integer to Roman numeral'''

until it

passes.

pass

1. At this stage, you want to define the API of the to_roman() function, but you don’t want to code it yet.

(Your test needs to fail first.) To stub it out, use the Python reserved word pass, which does precisely

nothing.

Execute romantest1.py on the command line to run the test. If you call it with the -v command-line

option, it will give more verbose output so you can see exactly what’s going on as each test case runs. With

any luck, your output should look like this:

219

you@localhost:~/diveintopython3/examples$ python3 romantest1.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... FAIL

======================================================================

FAIL: to_roman should give known result with known input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest1.py", line 73, in test_to_roman_known_values

self.assertEqual(numeral, result)

AssertionError: 'I' != None

----------------------------------------------------------------------

Ran 1 test in 0.016s

FAILED (failures=1)

1. Running the script runs unittest.main(), which runs each test case. Each test case is a method within a

class in romantest.py. There is no required organization of these test classes; they can each contain a single

test method, or you can have one class that contains multiple test methods. The only requirement is that

each test class must inherit from unittest.TestCase.

2. For each test case, the unittest module will print out the docstring of the method and whether that test

passed or failed. As expected, this test case fails.

3. For each failed test case, unittest displays the trace information showing exactly what happened. In this

case, the call to assertEqual() raised an AssertionError because it was expecting to_roman(1) to return

'I', but it didn’t. (Since there was no explicit return statement, the function returned None, the Python null

value.)

4. After the detail of each test, unittest displays a summary of how many tests were performed and how long

it took.

5. Overall, the test run failed because at least one test case did not pass. When a test case doesn’t pass,

unittest distinguishes between failures and errors. A failure is a call to an assertXYZ method, like

assertEqual or assertRaises, that fails because the asserted condition is not true or the expected

exception was not raised. An error is any other sort of exception raised in the code you’re testing or the

unit test case itself.

220

Now, finally, you can write the to_roman() function.

roman_numeral_map = (('M', 1000),

('CM', 900),

('D', 500),

('CD', 400),

('C', 100),

('XC', 90),

('L', 50),

('XL', 40),

('X', 10),

('IX', 9),

('V', 5),

('IV', 4),

('I', 1))

def to_roman(n):

'''convert integer to Roman numeral'''

result = ''

for numeral, integer in roman_numeral_map:

while n >= integer:

result += numeral

n -= integer

return result

1. roman_numeral_map is a tuple of tuples which defines three things: the character representations of the

most basic Roman numerals; the order of the Roman numerals (in descending value order, from M all the

way down to I); the value of each Roman numeral. Each inner tuple is a pair of (numeral, value). It’s not

just single-character Roman numerals; it also defines two-character pairs like CM (“one hundred less than one

thousand”). This makes the to_roman() function code simpler.

2. Here’s where the rich data structure of roman_numeral_map pays off, because you don’t need any special

logic to handle the subtraction rule. To convert to Roman numerals, simply iterate through

roman_numeral_map looking for the largest integer value less than or equal to the input. Once found, add

221

the Roman numeral representation to the end of the output, subtract the corresponding integer value from

the input, lather, rinse, repeat.

If you’re still not clear how the to_roman() function works, add a print() call to the end of the while

loop:

while n >= integer:

result += numeral

n -= integer

print('subtracting {0} from input, adding {1} to output'.format(integer, numeral))

With the debug print() statements, the output looks like this:

>>> import roman1

>>> roman1.to_roman(1424)

subtracting 1000 from input, adding M to output

subtracting 400 from input, adding CD to output

subtracting 10 from input, adding X to output

subtracting 10 from input, adding X to output

subtracting 4 from input, adding IV to output

'MCDXXIV'

So the to_roman() function appears to work, at least in this manual spot check. But will it pass the test

case you wrote?

you@localhost:~/diveintopython3/examples$ python3 romantest1.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

----------------------------------------------------------------------

Ran 1 test in 0.016s

OK

222

1. Hooray! The to_roman() function passes the “known values” test case. It’s not comprehensive, but it does

put the function through its paces with a variety of inputs, including inputs that produce every single-

character Roman numeral, the largest possible input (3999), and the input that produces the longest possible

Roman numeral (3888). At this point, you can be reasonably confident that the function works for any good

input value you could throw at it.

“Good” input? Hmm. What about bad input?

9.3. “HALT AND CATCH FIRE”

It is not enough to test that functions succeed when

given good input; you must also test that they fail when

given bad input. And not just any sort of failure; they

must fail in the way you expect.

>>> import roman1

The

>>> roman1.to_roman(4000)

'MMMM'

Pythonic

>>> roman1.to_roman(5000)

'MMMMM'

way to halt

>>> roman1.to_roman(9000)

and catch

'MMMMMMMMM'

fire is to

1. That’s definitely not what you wanted — that’s not even

a valid Roman numeral! In fact, each of these numbers is

outside the range of acceptable input, but the function

returns a bogus value anyway. Silently returning bad

values is baaaaaaad; if a program is going to fail, it is far better if it fails quickly and noisily. “Halt and catch

fire,” as the saying goes. The Pythonic way to halt and catch fire is to raise an exception.

223

The question to ask yourself is, “How can I express this

as a testable requirement?” How’s this for starters:

The to_roman() function should raise an

OutOfRangeError when given an integer greater

raise an

than 3999.

exception.

What would that test look like?

class ToRomanBadInput(unittest.TestCase):

def test_too_large(self):

'''to_roman should fail with large input'''

self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 4000)

1. Like the previous test case, you create a class that inherits from unittest.TestCase. You can have more

than one test per class (as you’ll see later in this chapter), but I chose to create a new class here because

this test is something different than the last one. We’ll keep all the good input tests together in one class,

and all the bad input tests together in another.

2. Like the previous test case, the test itself is a method of the class, with a name starting with test.

3. The unittest.TestCase class provides the assertRaises method, which takes the following arguments: the

exception you’re expecting, the function you’re testing, and the arguments you’re passing to that function. (If

the function you’re testing takes more than one argument, pass them all to assertRaises, in order, and it

will pass them right along to the function you’re testing.)

Pay close attention to this last line of code. Instead of calling to_roman() directly and manually checking that

it raises a particular exception (by wrapping it in a try...except block), the assertRaises method has encapsulated all of that for us. All you do is tell it what exception you’re expecting

(roman2.OutOfRangeError), the function (to_roman()), and the function’s arguments (4000). The

assertRaises method takes care of calling to_roman() and checking that it raises

roman2.OutOfRangeError.

224

Also note that you’re passing the to_roman() function itself as an argument; you’re not calling it, and you’re

not passing the name of it as a string. Have I mentioned recently how handy it is that everything in Python is

an object?

So what happens when you run the test suite with this new test?

you@localhost:~/diveintopython3/examples$ python3 romantest2.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

test_too_large (__main__.ToRomanBadInput)

to_roman should fail with large input ... ERROR

======================================================================

ERROR: to_roman should fail with large input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest2.py", line 78, in test_too_large

self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 4000)

AttributeError: 'module' object has no attribute 'OutOfRangeError'

----------------------------------------------------------------------

Ran 2 tests in 0.000s

FAILED (errors=1)

1. You should have expected this to fail (since you haven’t written any code to pass it yet), but... it didn’t

actually “fail,” it had an “error” instead. This is a subtle but important distinction. A unit test actually has

three return values: pass, fail, and error. Pass, of course, means that the test passed — the code did what

you expected. “Fail” is what the previous test case did (until you wrote code to make it pass) — it executed

the code but the result was not what you expected. “Error” means that the code didn’t even execute

properly.

2. Why didn’t the code execute properly? The traceback tells all. The module you’re testing doesn’t have an

exception called OutOfRangeError. Remember, you passed this exception to the assertRaises() method,

because it’s the exception you want the function to raise given an out-of-range input. But the exception

225

doesn’t exist, so the call to the assertRaises() method failed. It never got a chance to test the

to_roman() function; it didn’t get that far.

To solve this problem, you need to define the OutOfRangeError exception in roman2.py.

class OutOfRangeError(ValueError):

pass

1. Exceptions are classes. An “out of range” error is a kind of value error — the argument value is out of its

acceptable range. So this exception inherits from the built-in ValueError exception. This is not strictly

necessary (it could just inherit from the base Exception class), but it feels right.

2. Exceptions don’t actually do anything, but you need at least one line of code to make a class. Calling pass

does precisely nothing, but it’s a line of Python code, so that makes it a class.

Now run the test suite again.

you@localhost:~/diveintopython3/examples$ python3 romantest2.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

test_too_large (__main__.ToRomanBadInput)

to_roman should fail with large input ... FAIL

======================================================================

FAIL: to_roman should fail with large input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest2.py", line 78, in test_too_large

self.assertRaises(roman2.OutOfRangeError, roman2.to_roman, 4000)

AssertionError: OutOfRangeError not raised by to_roman

----------------------------------------------------------------------

Ran 2 tests in 0.016s

FAILED (failures=1)

226

1. The new test is still not passing, but it’s not returning an error either. Instead, the test is failing. That’s

progress! It means the call to the assertRaises() method succeeded this time, and the unit test framework

actually tested the to_roman() function.

2. Of course, the to_roman() function isn’t raising the OutOfRangeError exception you just defined, because

you haven’t told it to do that yet. That’s excellent news! It means this is a valid test case — it fails before

you write the code to make it pass.

Now you can write the code to make this test pass.

def to_roman(n):

'''convert integer to Roman numeral'''

if n > 3999:

raise OutOfRangeError('number out of range (must be less than 4000)')

result = ''

for numeral, integer in roman_numeral_map:

while n >= integer:

result += numeral

n -= integer

return result

1. This is straightforward: if the given input (n) is greater than 3999, raise an OutOfRangeError exception. The

unit test does not check the human-readable string that accompanies the exception, although you could write

another test that did check it (but watch out for internationalization issues for strings that vary by the user’s

language or environment).

Does this make the test pass? Let’s find out.

227

you@localhost:~/diveintopython3/examples$ python3 romantest2.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

test_too_large (__main__.ToRomanBadInput)

to_roman should fail with large input ... ok

----------------------------------------------------------------------

Ran 2 tests in 0.000s

OK

1. Hooray! Both tests pass. Because you worked iteratively, bouncing back and forth between testing and

coding, you can be sure that the two lines of code you just wrote were the cause of that one test going

from “fail” to “pass.” That kind of confidence doesn’t come cheap, but it will pay for itself over the lifetime

of your code.

9.4. MORE HALTING, MORE FIRE

Along with testing numbers that are too large, you need to test numbers that are too small. As we noted in

our functional requirements, Roman numerals cannot express 0 or negative numbers.

>>> import roman2

>>> roman2.to_roman(0)

''

>>> roman2.to_roman(-1)

''

Well that’s not good. Let’s add tests for each of these conditions.

228

class ToRomanBadInput(unittest.TestCase):

def test_too_large(self):

'''to_roman should fail with large input'''

self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, 4000)

def test_zero(self):

'''to_roman should fail with 0 input'''

self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, 0)

def test_negative(self):

'''to_roman should fail with negative input'''

self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, -1)

1. The test_too_large() method has not changed since the previous step. I’m including it here to show

where the new code fits.

2. Here’s a new test: the test_zero() method. Like the test_too_large() method, it tells the

assertRaises() method defined in unittest.TestCase to call our to_roman() function with a parameter

of 0, and check that it raises the appropriate exception, OutOfRangeError.

3. The test_negative() method is almost identical, except it passes -1 to the to_roman() function. If either

of these new tests does not raise an OutOfRangeError (either because the function returns an actual value,

or because it raises some other exception), the test is considered failed.

Now check that the tests fail:

229

you@localhost:~/diveintopython3/examples$ python3 romantest3.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

test_negative (__main__.ToRomanBadInput)

to_roman should fail with negative input ... FAIL

test_too_large (__main__.ToRomanBadInput)

to_roman should fail with large input ... ok

test_zero (__main__.ToRomanBadInput)

to_roman should fail with 0 input ... FAIL

======================================================================

FAIL: to_roman should fail with negative input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest3.py", line 86, in test_negative

self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, -1)

AssertionError: OutOfRangeError not raised by to_roman

======================================================================

FAIL: to_roman should fail with 0 input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest3.py", line 82, in test_zero

self.assertRaises(roman3.OutOfRangeError, roman3.to_roman, 0)

AssertionError: OutOfRangeError not raised by to_roman

----------------------------------------------------------------------

Ran 4 tests in 0.000s

FAILED (failures=2)

Excellent. Both tests failed, as expected. Now let’s switch over to the code and see what we can do to

make them pass.

230

def to_roman(n):

'''convert integer to Roman numeral'''

if not (0 < n < 4000):

raise OutOfRangeError('number out of range (must be 1..3999)')

result = ''

for numeral, integer in roman_numeral_map:

while n >= integer:

result += numeral

n -= integer

return result

1. This is a nice Pythonic shortcut: multiple comparisons at once. This is equivalent to if not ((0 < n) and

(n < 4000)), but it’s much easier to read. This one line of code should catch inputs that are too large,

negative, or zero.

2. If you change your conditions, make sure to update your human-readable error strings to match. The

unittest framework won’t care, but it’ll make it difficult to do manual debugging if your code is throwing

incorrectly-described exceptions.

I could show you a whole series of unrelated examples to show that the multiple-comparisons-at-once

shortcut works, but instead I’ll just run the unit tests and prove it.

231

you@localhost:~/diveintopython3/examples$ python3 romantest3.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

test_negative (__main__.ToRomanBadInput)

to_roman should fail with negative input ... ok

test_too_large (__main__.ToRomanBadInput)

to_roman should fail with large input ... ok

test_zero (__main__.ToRomanBadInput)

to_roman should fail with 0 input ... ok

----------------------------------------------------------------------

Ran 4 tests in 0.016s

OK

9.5. AND ONE MORE THING…

There was one more functional requirement for converting numbers to Roman numerals: dealing with non-integers.

>>> import roman3

>>> roman3.to_roman(0.5)

''

>>> roman3.to_roman(1.0)

'I'

1. Oh, that’s bad.

2. Oh, that’s even worse. Both of these cases should raise an exception. Instead, they give bogus results.

Testing for non-integers is not difficult. First, define a NotIntegerError exception.

232

# roman4.py

class OutOfRangeError(ValueError): pass

class NotIntegerError(ValueError): pass

Next, write a test case that checks for the NotIntegerError exception.

class ToRomanBadInput(unittest.TestCase):

.

.

.

def test_non_integer(self):

'''to_roman should fail with non-integer input'''

self.assertRaises(roman4.NotIntegerError, roman4.to_roman, 0.5)

Now check that the test fails properly.

233

you@localhost:~/diveintopython3/examples$ python3 romantest4.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

test_negative (__main__.ToRomanBadInput)

to_roman should fail with negative input ... ok

test_non_integer (__main__.ToRomanBadInput)

to_roman should fail with non-integer input ... FAIL

test_too_large (__main__.ToRomanBadInput)

to_roman should fail with large input ... ok

test_zero (__main__.ToRomanBadInput)

to_roman should fail with 0 input ... ok

======================================================================

FAIL: to_roman should fail with non-integer input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest4.py", line 90, in test_non_integer

self.assertRaises(roman4.NotIntegerError, roman4.to_roman, 0.5)

AssertionError: NotIntegerError not raised by to_roman

----------------------------------------------------------------------

Ran 5 tests in 0.000s

FAILED (failures=1)

Write the code that makes the test pass.

234

def to_roman(n):

'''convert integer to Roman numeral'''

if not (0 < n < 4000):

raise OutOfRangeError('number out of range (must be 1..3999)')

if not isinstance(n, int):

raise NotIntegerError('non-integers can not be converted')

result = ''

for numeral, integer in roman_numeral_map:

while n >= integer:

result += numeral

n -= integer

return result

1. The built-in isinstance() function tests whether a variable is a particular type (or, technically, any

descendant type).

2. If the argument n is not an int, raise our newly minted NotIntegerError exception.

Finally, check that the code does indeed make the test pass.

235

you@localhost:~/diveintopython3/examples$ python3 romantest4.py -v

test_to_roman_known_values (__main__.KnownValues)

to_roman should give known result with known input ... ok

test_negative (__main__.ToRomanBadInput)

to_roman should fail with negative input ... ok

test_non_integer (__main__.ToRomanBadInput)

to_roman should fail with non-integer input ... ok

test_too_large (__main__.ToRomanBadInput)

to_roman should fail with large input ... ok

test_zero (__main__.ToRomanBadInput)

to_roman should fail with 0 input ... ok

----------------------------------------------------------------------

Ran 5 tests in 0.000s

OK

The to_roman() function passes all of its tests, and I can’t think of any more tests, so it’s time to move on

to from_roman().

9.6. A PLEASING SYMMETRY

Converting a string from a Roman numeral to an integer sounds more difficult than converting an integer to

a Roman numeral. Certainly there is the issue of validation. It’s easy to check if an integer is greater than 0,

but a bit harder to check whether a string is a valid Roman numeral. But we already constructed a regular

expression to check for Roman numerals, so that part is done.

That leaves the problem of converting the string itself. As we’ll see in a minute, thanks to the rich data

structure we defined to map individual Roman numerals to integer values, the nitty-gritty of the

from_roman() function is as straightforward as the to_roman() function.

236

But first, the tests. We’ll need a “known values” test to spot-check for accuracy. Our test suite already

contains a mapping of known values; let’s reuse that.

def test_from_roman_known_values(self):

'''from_roman should give known result with known input'''

for integer, numeral in self.known_values:

result = roman5.from_roman(numeral)

self.assertEqual(integer, result)

There’s a pleasing symmetry here. The to_roman() and from_roman() functions are inverses of each other.

The first converts integers to specially-formatted strings, the second converts specially-formated strings to

integers. In theory, we should be able to “round-trip” a number by passing to the to_roman() function to

get a string, then passing that string to the from_roman() function to get an integer, and end up with the

same number.

n = from_roman(to_roman(n)) for all values of n

In this case, “all values” means any number between 1..3999, since that is the valid range of inputs to the

to_roman() function. We can express this symmetry in a test case that runs through all the values 1..3999,

calls to_roman(), calls from_roman(), and checks that the output is the same as the original input.

class RoundtripCheck(unittest.TestCase):

def test_roundtrip(self):

'''from_roman(to_roman(n))==n for all n'''

for integer in range(1, 4000):

numeral = roman5.to_roman(integer)

result = roman5.from_roman(numeral)

self.assertEqual(integer, result)

These new tests won’t even fail yet. We haven’t defined a from_roman() function at all, so they’ll just raise

errors.

237

you@localhost:~/diveintopython3/examples$ python3 romantest5.py

E.E....

======================================================================

ERROR: test_from_roman_known_values (__main__.KnownValues)

from_roman should give known result with known input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest5.py", line 78, in test_from_roman_known_values

result = roman5.from_roman(numeral)

AttributeError: 'module' object has no attribute 'from_roman'

======================================================================

ERROR: test_roundtrip (__main__.RoundtripCheck)

from_roman(to_roman(n))==n for all n

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest5.py", line 103, in test_roundtrip

result = roman5.from_roman(numeral)

AttributeError: 'module' object has no attribute 'from_roman'

----------------------------------------------------------------------

Ran 7 tests in 0.019s

FAILED (errors=2)

A quick stub function will solve that problem.

# roman5.py

def from_roman(s):

'''convert Roman numeral to integer'''

(Hey, did you notice that? I defined a function with nothing but a docstring. That’s legal Python. In fact, some programmers swear by it. “Don’t stub; document!”)

238

Now the test cases will actually fail.

you@localhost:~/diveintopython3/examples$ python3 romantest5.py

F.F....

======================================================================

FAIL: test_from_roman_known_values (__main__.KnownValues)

from_roman should give known result with known input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest5.py", line 79, in test_from_roman_known_values

self.assertEqual(integer, result)

AssertionError: 1 != None

======================================================================

FAIL: test_roundtrip (__main__.RoundtripCheck)

from_roman(to_roman(n))==n for all n

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest5.py", line 104, in test_roundtrip

self.assertEqual(integer, result)

AssertionError: 1 != None

----------------------------------------------------------------------

Ran 7 tests in 0.002s

FAILED (failures=2)

Now it’s time to write the from_roman() function.

239

def from_roman(s):

"""convert Roman numeral to integer"""

result = 0

index = 0

for numeral, integer in roman_numeral_map:

while s[index:index+len(numeral)] == numeral:

result += integer

index += len(numeral)

return result

1. The pattern here is the same as the to_roman() function. You iterate through your Roman numeral data structure (a tuple of tuples), but instead of matching the highest integer values as often as possible, you

match the “highest” Roman numeral character strings as often as possible.

If you're not clear how from_roman() works, add a print statement to the end of the while loop:

def from_roman(s):

"""convert Roman numeral to integer"""

result = 0

index = 0

for numeral, integer in roman_numeral_map:

while s[index:index+len(numeral)] == numeral:

result += integer

index += len(numeral)

print('found', numeral, 'of length', len(numeral), ', adding', integer)

240

>>> import roman5

>>> roman5.from_roman('MCMLXXII')

found M of length 1, adding 1000

found CM of length 2, adding 900

found L of length 1, adding 50

found X of length 1, adding 10

found X of length 1, adding 10

found I of length 1, adding 1

found I of length 1, adding 1

1972

Time to re-run the tests.

you@localhost:~/diveintopython3/examples$ python3 romantest5.py

.......

----------------------------------------------------------------------

Ran 7 tests in 0.060s

OK

Two pieces of exciting news here. The first is that the from_roman() function works for good input, at least

for all the known values. The second is that the “round trip” test also passed. Combined with the known values tests, you can be reasonably sure that both the to_roman() and from_roman() functions work

properly for all possible good values. (This is not guaranteed; it is theoretically possible that to_roman() has

a bug that produces the wrong Roman numeral for some particular set of inputs, and that from_roman() has

a reciprocal bug that produces the same wrong integer values for exactly that set of Roman numerals that

to_roman() generated incorrectly. Depending on your application and your requirements, this possibility may

bother you; if so, write more comprehensive test cases until it doesn't bother you.)

241

9.7. MORE BAD INPUT

Now that the from_roman() function works properly with good input, it's time to fit in the last piece of the

puzzle: making it work properly with bad input. That means finding a way to look at a string and determine

if it's a valid Roman numeral. This is inherently more difficult than validating numeric input in the to_roman() function, but you have a powerful tool at your disposal: regular expressions. (If you’re not familiar with

regular expressions, now would be a good time to read the regular expressions chapter.)

As you saw in Case Study: Roman Numerals, there are several simple rules for constructing a Roman

numeral, using the letters M, D, C, L, X, V, and I. Let's review the rules:

• Sometimes characters are additive. I is 1, II is 2, and III is 3. VI is 6 (literally, “5 and 1”), VII is 7, and

VIII is 8.

• The tens characters (I, X, C, and M) can be repeated up to three times. At 4, you need to subtract from the

next highest fives character. You can't represent 4 as IIII; instead, it is represented as IV (“1 less than 5”).

40 is written as XL (“10 less than 50”), 41 as XLI, 42 as XLII, 43 as XLIII, and then 44 as XLIV (“10 less

than 50, then 1 less than 5”).

• Sometimes characters are… the opposite of additive. By putting certain characters before others, you

subtract from the final value. For example, at 9, you need to subtract from the next highest tens character: 8

is VIII, but 9 is IX (“1 less than 10”), not VIIII (since the I character can not be repeated four times). 90

is XC, 900 is CM.

• The fives characters can not be repeated. 10 is always represented as X, never as VV. 100 is always C, never

LL.

• Roman numerals are read left to right, so the order of characters matters very much. DC is 600; CD is a

completely different number (400, “100 less than 500”). CI is 101; IC is not even a valid Roman numeral

(because you can't subtract 1 directly from 100; you would need to write it as XCIX, “10 less than 100, then

1 less than 10”).

Thus, one useful test would be to ensure that the from_roman() function should fail when you pass it a

string with too many repeated numerals. How many is “too many” depends on the numeral.

242

class FromRomanBadInput(unittest.TestCase):

def test_too_many_repeated_numerals(self):

'''from_roman should fail with too many repeated numerals'''

for s in ('MMMM', 'DD', 'CCCC', 'LL', 'XXXX', 'VV', 'IIII'):

self.assertRaises(roman6.InvalidRomanNumeralError, roman6.from_roman, s)

Another useful test would be to check that certain patterns aren’t repeated. For example, IX is 9, but IXIX

is never valid.

def test_repeated_pairs(self):

'''from_roman should fail with repeated pairs of numerals'''

for s in ('CMCM', 'CDCD', 'XCXC', 'XLXL', 'IXIX', 'IVIV'):

self.assertRaises(roman6.InvalidRomanNumeralError, roman6.from_roman, s)

A third test could check that numerals appear in the correct order, from highest to lowest value. For

example, CL is 150, but LC is never valid, because the numeral for 50 can never come before the numeral

for 100. This test includes a randomly chosen set of invalid antecedents: I before M, V before X, and so on.

def test_malformed_antecedents(self):

'''from_roman should fail with malformed antecedents'''

for s in ('IIMXCC', 'VX', 'DCM', 'CMM', 'IXIV',

'MCMC', 'XCX', 'IVI', 'LM', 'LD', 'LC'):

self.assertRaises(roman6.InvalidRomanNumeralError, roman6.from_roman, s)

Each of these tests relies the from_roman() function raising a new exception, InvalidRomanNumeralError,

which we haven’t defined yet.

# roman6.py

class InvalidRomanNumeralError(ValueError): pass

All three of these tests should fail, since the from_roman() function doesn’t currently have any validity

checking. (If they don’t fail now, then what the heck are they testing?)

243

you@localhost:~/diveintopython3/examples$ python3 romantest6.py

FFF.......

======================================================================

FAIL: test_malformed_antecedents (__main__.FromRomanBadInput)

from_roman should fail with malformed antecedents

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest6.py", line 113, in test_malformed_antecedents

self.assertRaises(roman6.InvalidRomanNumeralError, roman6.from_roman, s)

AssertionError: InvalidRomanNumeralError not raised by from_roman

======================================================================

FAIL: test_repeated_pairs (__main__.FromRomanBadInput)

from_roman should fail with repeated pairs of numerals

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest6.py", line 107, in test_repeated_pairs

self.assertRaises(roman6.InvalidRomanNumeralError, roman6.from_roman, s)

AssertionError: InvalidRomanNumeralError not raised by from_roman

======================================================================

FAIL: test_too_many_repeated_numerals (__main__.FromRomanBadInput)

from_roman should fail with too many repeated numerals

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest6.py", line 102, in test_too_many_repeated_numerals

self.assertRaises(roman6.InvalidRomanNumeralError, roman6.from_roman, s)

AssertionError: InvalidRomanNumeralError not raised by from_roman

----------------------------------------------------------------------

Ran 10 tests in 0.058s

FAILED (failures=3)

244

Good deal. Now, all we need to do is add the regular expression to test for valid Roman numerals into the from_roman() function.

roman_numeral_pattern = re.compile('''

^ # beginning of string

M{0,3} # thousands - 0 to 3 Ms

(CM|CD|D?C{0,3}) # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 Cs),

# or 500-800 (D, followed by 0 to 3 Cs)

(XC|XL|L?X{0,3}) # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 Xs),

# or 50-80 (L, followed by 0 to 3 Xs)

(IX|IV|V?I{0,3}) # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 Is),

# or 5-8 (V, followed by 0 to 3 Is)

$ # end of string

''', re.VERBOSE)

def from_roman(s):

'''convert Roman numeral to integer'''

if not roman_numeral_pattern.search(s):

raise InvalidRomanNumeralError('Invalid Roman numeral: {0}'.format(s))

result = 0

index = 0

for numeral, integer in roman_numeral_map:

while s[index : index + len(numeral)] == numeral:

result += integer

index += len(numeral)

return result

And re-run the tests…

245

you@localhost:~/diveintopython3/examples$ python3 romantest7.py

..........

----------------------------------------------------------------------

Ran 10 tests in 0.066s

OK

And the anticlimax award of the year goes to… the word “OK”, which is printed by the unittest module

when all the tests pass.

246

CHAPTER 10. REFACTORING

After one has played a vast quantity of notes and more notes, it is simplicity that emerges as the crowning reward

of art.

Frédéric Chopin

10.1. DIVING IN

Likeitornot,bugshappen.Despiteyourbesteffortstowritecomprehensiveunittests, bugshappen.

What do I mean by “bug”? A bug is a test case you haven’t written yet.

>>> import roman7

>>> roman7.from_roman('') ①

0

1. This is a bug. An empty string should raise an InvalidRomanNumeralError exception, just like any other

sequence of characters that don’t represent a valid Roman numeral.

After reproducing the bug, and before fixing it, you should write a test case that fails, thus illustrating the

bug.

class FromRomanBadInput(unittest.TestCase):

.

.

.

def testBlank(self):

'''from_roman should fail with blank string'''

self.assertRaises(roman6.InvalidRomanNumeralError, roman6.from_roman, '') ①

247

1. Pretty simple stuff here. Call from_roman() with an empty string and make sure it raises an

InvalidRomanNumeralError exception. The hard part was finding the bug; now that you know about it,

testing for it is the easy part.

Since your code has a bug, and you now have a test case that tests this bug, the test case will fail:

you@localhost:~/diveintopython3/examples$ python3 romantest8.py -v

from_roman should fail with blank string ... FAIL

from_roman should fail with malformed antecedents ... ok

from_roman should fail with repeated pairs of numerals ... ok

from_roman should fail with too many repeated numerals ... ok

from_roman should give known result with known input ... ok

to_roman should give known result with known input ... ok

from_roman(to_roman(n))==n for all n ... ok

to_roman should fail with negative input ... ok

to_roman should fail with non-integer input ... ok

to_roman should fail with large input ... ok

to_roman should fail with 0 input ... ok

======================================================================

FAIL: from_roman should fail with blank string

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest8.py", line 117, in test_blank

self.assertRaises(roman8.InvalidRomanNumeralError, roman8.from_roman, '')

AssertionError: InvalidRomanNumeralError not raised by from_roman

----------------------------------------------------------------------

Ran 11 tests in 0.171s

FAILED (failures=1)

Now you can fix the bug.

248

def from_roman(s):

'''convert Roman numeral to integer'''

if not s:

raise InvalidRomanNumeralError('Input can not be blank')

if not re.search(romanNumeralPattern, s):

raise InvalidRomanNumeralError('Invalid Roman numeral: {}'.format(s))

result = 0

index = 0

for numeral, integer in romanNumeralMap:

while s[index:index+len(numeral)] == numeral:

result += integer

index += len(numeral)

return result

1. Only two lines of code are required: an explicit check for an empty string, and a raise statement.

2. I don’t think I’ve mentioned this yet anywhere in this book, so let this serve as your final lesson in string

formatting. Starting in Python 3.1, you can skip the numbers when using positional indexes in a format specifier. That is, instead of using the format specifier {0} to refer to the first parameter to the format()

method, you can simply use {} and Python will fill in the proper positional index for you. This works for any

number of arguments; the first {} is {0}, the second {} is {1}, and so forth.

249

you@localhost:~/diveintopython3/examples$ python3 romantest8.py -v

from_roman should fail with blank string ... ok

from_roman should fail with malformed antecedents ... ok

from_roman should fail with repeated pairs of numerals ... ok

from_roman should fail with too many repeated numerals ... ok

from_roman should give known result with known input ... ok

to_roman should give known result with known input ... ok

from_roman(to_roman(n))==n for all n ... ok

to_roman should fail with negative input ... ok

to_roman should fail with non-integer input ... ok

to_roman should fail with large input ... ok

to_roman should fail with 0 input ... ok

----------------------------------------------------------------------

Ran 11 tests in 0.156s

OK

1. The blank string test case now passes, so the bug is fixed.

2. All the other test cases still pass, which means that this bug fix didn’t break anything else. Stop coding.

Coding this way does not make fixing bugs any easier. Simple bugs (like this one) require simple test cases;

complex bugs will require complex test cases. In a testing-centric environment, it may seem like it takes

longer to fix a bug, since you need to articulate in code exactly what the bug is (to write the test case),

then fix the bug itself. Then if the test case doesn’t pass right away, you need to figure out whether the fix

was wrong, or whether the test case itself has a bug in it. However, in the long run, this back-and-forth

between test code and code tested pays for itself, because it makes it more likely that bugs are fixed

correctly the first time. Also, since you can easily re-run all the test cases along with your new one, you are

much less likely to break old code when fixing new code. Today’s unit test is tomorrow’s regression test.

250

10.2. HANDLING CHANGING REQUIREMENTS

Despite your best efforts to pin your customers to the ground and extract exact requirements from them

on pain of horrible nasty things involving scissors and hot wax, requirements will change. Most customers

don’t know what they want until they see it, and even if they do, they aren’t that good at articulating what

they want precisely enough to be useful. And even if they do, they’ll want more in the next release anyway.

So be prepared to update your test cases as requirements change.

Suppose, for instance, that you wanted to expand the range of the Roman numeral conversion functions.

Normally, no character in a Roman numeral can be repeated more than three times in a row. But the

Romans were willing to make an exception to that rule by having 4 M characters in a row to represent 4000.

If you make this change, you’ll be able to expand the range of convertible numbers from 1..3999 to

1..4999. But first, you need to make some changes to your test cases.

251

class KnownValues(unittest.TestCase):

known_values = ( (1, 'I'),

.

.

.

(3999, 'MMMCMXCIX'),

(4000, 'MMMM'),

(4500, 'MMMMD'),

(4888, 'MMMMDCCCLXXXVIII'),

(4999, 'MMMMCMXCIX') )

class ToRomanBadInput(unittest.TestCase):

def test_too_large(self):

'''to_roman should fail with large input'''

self.assertRaises(roman8.OutOfRangeError, roman8.to_roman, 5000)

.

.

.

class FromRomanBadInput(unittest.TestCase):

def test_too_many_repeated_numerals(self):

'''from_roman should fail with too many repeated numerals'''

for s in ('MMMMM', 'DD', 'CCCC', 'LL', 'XXXX', 'VV', 'IIII'):

self.assertRaises(roman8.InvalidRomanNumeralError, roman8.from_roman, s)

.

.

.

class RoundtripCheck(unittest.TestCase):

def test_roundtrip(self):

'''from_roman(to_roman(n))==n for all n'''

for integer in range(1, 5000):

252

numeral = roman8.to_roman(integer)

result = roman8.from_roman(numeral)

self.assertEqual(integer, result)

1. The existing known values don’t change (they’re all still reasonable values to test), but you need to add a

few more in the 4000 range. Here I’ve included 4000 (the shortest), 4500 (the second shortest), 4888 (the

longest), and 4999 (the largest).

2. The definition of “large input” has changed. This test used to call to_roman() with 4000 and expect an

error; now that 4000-4999 are good values, you need to bump this up to 5000.

3. The definition of “too many repeated numerals” has also changed. This test used to call from_roman() with

'MMMM' and expect an error; now that MMMM is considered a valid Roman numeral, you need to bump this up

to 'MMMMM'.

4. The sanity check loops through every number in the range, from 1 to 3999. Since the range has now

expanded, this for loop need to be updated as well to go up to 4999.

Now your test cases are up to date with the new requirements, but your code is not, so you expect several

of the test cases to fail.

253

you@localhost:~/diveintopython3/examples$ python3 romantest9.py -v

from_roman should fail with blank string ... ok

from_roman should fail with malformed antecedents ... ok

from_roman should fail with non-string input ... ok

from_roman should fail with repeated pairs of numerals ... ok

from_roman should fail with too many repeated numerals ... ok

from_roman should give known result with known input ... ERROR

to_roman should give known result with known input ... ERROR

from_roman(to_roman(n))==n for all n ... ERROR

to_roman should fail with negative input ... ok

to_roman should fail with non-integer input ... ok

to_roman should fail with large input ... ok

to_roman should fail with 0 input ... ok

======================================================================

ERROR: from_roman should give known result with known input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest9.py", line 82, in test_from_roman_known_values

result = roman9.from_roman(numeral)

File "C:\home\diveintopython3\examples\roman9.py", line 60, in from_roman

raise InvalidRomanNumeralError('Invalid Roman numeral: {0}'.format(s))

roman9.InvalidRomanNumeralError: Invalid Roman numeral: MMMM

======================================================================

ERROR: to_roman should give known result with known input

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest9.py", line 76, in test_to_roman_known_values

result = roman9.to_roman(integer)

File "C:\home\diveintopython3\examples\roman9.py", line 42, in to_roman

raise OutOfRangeError('number out of range (must be 0..3999)')

roman9.OutOfRangeError: number out of range (must be 0..3999)

254

======================================================================

ERROR: from_roman(to_roman(n))==n for all n

----------------------------------------------------------------------

Traceback (most recent call last):

File "romantest9.py", line 131, in testSanity

numeral = roman9.to_roman(integer)

File "C:\home\diveintopython3\examples\roman9.py", line 42, in to_roman

raise OutOfRangeError('number out of range (must be 0..3999)')

roman9.OutOfRangeError: number out of range (must be 0..3999)

----------------------------------------------------------------------

Ran 12 tests in 0.171s

FAILED (errors=3)

1. The from_roman() known values test will fail as soon as it hits 'MMMM', because from_roman() still thinks

this is an invalid Roman numeral.

2. The to_roman() known values test will fail as soon as it hits 4000, because to_roman() still thinks this is

out of range.

3. The roundtrip check will also fail as soon as it hits 4000, because to_roman() still thinks this is out of range.

Now that you have test cases that fail due to the new requirements, you can think about fixing the code to

bring it in line with the test cases. (When you first start coding unit tests, it might feel strange that the code

being tested is never “ahead” of the test cases. While it’s behind, you still have some work to do, and as

soon as it catches up to the test cases, you stop coding. After you get used to it, you’ll wonder how you

ever programmed without tests.)

255

roman_numeral_pattern = re.compile('''

^ # beginning of string

M{0,4} # thousands - 0 to 4 Ms

(CM|CD|D?C{0,3}) # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 Cs),

# or 500-800 (D, followed by 0 to 3 Cs)

(XC|XL|L?X{0,3}) # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 Xs),

# or 50-80 (L, followed by 0 to 3 Xs)

(IX|IV|V?I{0,3}) # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 Is),

# or 5-8 (V, followed by 0 to 3 Is)

$ # end of string

''', re.VERBOSE)

def to_roman(n):

'''convert integer to Roman numeral'''

if not (0 < n < 5000):

raise OutOfRangeError('number out of range (must be 1..4999)')

if not isinstance(n, int):

raise NotIntegerError('non-integers can not be converted')

result = ''

for numeral, integer in roman_numeral_map:

while n >= integer:

result += numeral

n -= integer

return result

def from_roman(s):

.

.

.

1. You don’t need to make any changes to the from_roman() function at all. The only change is to

roman_numeral_pattern. If you look closely, you’ll notice that I changed the maximum number of optional M

characters from 3 to 4 in the first section of the regular expression. This will allow the Roman numeral

256

equivalents of 4999 instead of 3999. The actual from_roman() function is completely generic; it just looks for

repeated Roman numeral characters and adds them up, without caring how many times they repeat. The

only reason it didn’t handle 'MMMM' before is that you explicitly stopped it with the regular expression

pattern matching.

2. The to_roman() function only needs one small change, in the range check. Where you used to check 0 < n

< 4000, you now check 0 < n < 5000. And you change the error message that you raise to reflect the

new acceptable range (1..4999 instead of 1..3999). You don’t need to make any changes to the rest of the

function; it handles the new cases already. (It merrily adds 'M' for each thousand that it finds; given 4000, it

will spit out 'MMMM'. The only reason it didn’t do this before is that you explicitly stopped it with the range

check.)

You may be skeptical that these two small changes are all that you need. Hey, don’t take my word for it;

see for yourself.

you@localhost:~/diveintopython3/examples$ python3 romantest9.py -v

from_roman should fail with blank string ... ok

from_roman should fail with malformed antecedents ... ok

from_roman should fail with non-string input ... ok

from_roman should fail with repeated pairs of numerals ... ok

from_roman should fail with too many repeated numerals ... ok

from_roman should give known result with known input ... ok

to_roman should give known result with known input ... ok

from_roman(to_roman(n))==n for all n ... ok

to_roman should fail with negative input ... ok

to_roman should fail with non-integer input ... ok

to_roman should fail with large input ... ok

to_roman should fail with 0 input ... ok

----------------------------------------------------------------------

Ran 12 tests in 0.203s

OK

1. All the test cases pass. Stop coding.

257

Comprehensive unit testing means never having to rely on a programmer who says “Trust me.”

10.3. REFACTORING

The best thing about comprehensive unit testing is not the feeling you get when all your test cases finally

pass, or even the feeling you get when someone else blames you for breaking their code and you can

actually prove that you didn’t. The best thing about unit testing is that it gives you the freedom to refactor

mercilessly.

Refactoring is the process of taking working code and making it work better. Usually, “better” means

“faster”, although it can also mean “using less memory”, or “using less disk space”, or simply “more

elegantly”. Whatever it means to you, to your project, in your environment, refactoring is important to the

long-term health of any program.

Here, “better” means both “faster” and “easier to maintain.” Specifically, the from_roman() function is

slower and more complex than I’d like, because of that big nasty regular expression that you use to validate

Roman numerals. Now, you might think, “Sure, the regular expression is big and hairy, but how else am I

supposed to validate that an arbitrary string is a valid a Roman numeral?”

Answer: there’s only 5000 of them; why don’t you just build a lookup table? This idea gets even better when

you realize that you don’t need to use regular expressions at all. As you build the lookup table for converting

integers to Roman numerals, you can build the reverse lookup table to convert Roman numerals to integers.

By the time you need to check whether an arbitrary string is a valid Roman numeral, you will have collected

all the valid Roman numerals. “Validating” is reduced to a single dictionary lookup.

And best of all, you already have a complete set of unit tests. You can change over half the code in the

module, but the unit tests will stay the same. That means you can prove — to yourself and to others — that

the new code works just as well as the original.

258

class OutOfRangeError(ValueError): pass

class NotIntegerError(ValueError): pass

class InvalidRomanNumeralError(ValueError): pass

roman_numeral_map = (('M', 1000),

('CM', 900),

('D', 500),

('CD', 400),

('C', 100),

('XC', 90),

('L', 50),

('XL', 40),

('X', 10),

('IX', 9),

('V', 5),

('IV', 4),

('I', 1))

to_roman_table = [ None ]

from_roman_table = {}

def to_roman(n):

'''convert integer to Roman numeral'''

if not (0 < n < 5000):

raise OutOfRangeError('number out of range (must be 1..4999)')

if int(n) != n:

raise NotIntegerError('non-integers can not be converted')

return to_roman_table[n]

def from_roman(s):

'''convert Roman numeral to integer'''

if not isinstance(s, str):

raise InvalidRomanNumeralError('Input must be a string')

if not s:

259

raise InvalidRomanNumeralError('Input can not be blank')

if s not in from_roman_table:

raise InvalidRomanNumeralError('Invalid Roman numeral: {0}'.format(s))

return from_roman_table[s]

def build_lookup_tables():

def to_roman(n):

result = ''

for numeral, integer in roman_numeral_map:

if n >= integer:

result = numeral

n -= integer

break

if n > 0:

result += to_roman_table[n]

return result

for integer in range(1, 5000):

roman_numeral = to_roman(integer)

to_roman_table.append(roman_numeral)

from_roman_table[roman_numeral] = integer

build_lookup_tables()

Let’s break that down into digestable pieces. Arguably, the most important line is the last one:

build_lookup_tables()

You will note that is a function call, but there’s no if statement around it. This is not an if __name__ ==

'__main__' block; it gets called when the module is imported. (It is important to understand that modules are

only imported once, then cached. If you import an already-imported module, it does nothing. So this code

will only get called the first time you import this module.)

So what does the build_lookup_tables() function do? I’m glad you asked.

260

to_roman_table = [ None ]

from_roman_table = {}

.

.

.

def build_lookup_tables():

def to_roman(n):

result = ''

for numeral, integer in roman_numeral_map:

if n >= integer:

result = numeral

n -= integer

break

if n > 0:

result += to_roman_table[n]

return result

for integer in range(1, 5000):

roman_numeral = to_roman(integer)

to_roman_table.append(roman_numeral)

from_roman_table[roman_numeral] = integer

1. This is a clever bit of programming… perhaps too clever. The to_roman() function is defined above; it looks

up values in the lookup table and returns them. But the build_lookup_tables() function redefines the

to_roman() function to actually do work (like the previous examples did, before you added a lookup table).

Within the build_lookup_tables() function, calling to_roman() will call this redefined version. Once the

build_lookup_tables() function exits, the redefined version disappears — it is only defined in the local

scope of the build_lookup_tables() function.

2. This line of code will call the redefined to_roman() function, which actually calculates the Roman numeral.

3. Once you have the result (from the redefined to_roman() function), you add the integer and its Roman

numeral equivalent to both lookup tables.

Once the lookup tables are built, the rest of the code is both easy and fast.

261

def to_roman(n):

'''convert integer to Roman numeral'''

if not (0 < n < 5000):

raise OutOfRangeError('number out of range (must be 1..4999)')

if int(n) != n:

raise NotIntegerError('non-integers can not be converted')

return to_roman_table[n]

def from_roman(s):

'''convert Roman numeral to integer'''

if not isinstance(s, str):

raise InvalidRomanNumeralError('Input must be a string')

if not s:

raise InvalidRomanNumeralError('Input can not be blank')

if s not in from_roman_table:

raise InvalidRomanNumeralError('Invalid Roman numeral: {0}'.format(s))

return from_roman_table[s]

1. After doing the same bounds checking as before, the to_roman() function simply finds the appropriate value

in the lookup table and returns it.

2. Similarly, the from_roman() function is reduced to some bounds checking and one line of code. No more

regular expressions. No more looping. O(1) conversion to and from Roman numerals.

But does it work? Why yes, yes it does. And I can prove it.

262

you@localhost:~/diveintopython3/examples$ python3 romantest10.py -v

from_roman should fail with blank string ... ok

from_roman should fail with malformed antecedents ... ok

from_roman should fail with non-string input ... ok

from_roman should fail with repeated pairs of numerals ... ok