Development Tools for Larger Projects

Once you’ve mastered the basics, you’ll find your Python programs becoming substantially larger than the examples you’ve experimented with so far. For developing larger systems, a set of development tools is available in Python and the public domain. You’ve seen some of these in action, and I’ve mentioned a few others. To help you on your way, here is a summary of some of the most commonly used tools in this domain:

PyDoc and docstrings

广告：个人专属 VPN，独立 IP，无限流量，多机房切换，还可以屏蔽广告和恶意软件，每月最低仅 5 美元

PyDoc’s help function and HTML interfaces were introduced in Chapter 15. PyDoc provides a documentation system for your modules and objects and integrates with Python’s docstrings feature. It is a standard part of the Python system—see the library manual for more details. Be sure to also refer back to the documentation source hints listed in Chapter 4 for information on other Python information resources.

PyChecker and PyLint

Because Python is such a dynamic language, some programming errors are not reported until your program runs (e.g., syntax errors are caught when a file is run or imported). This isn’t a big drawback—as with most languages, it just means that you have to test your Python code before shipping it. At worst, with Python you essentially trade a compile phase for an initial testing phase. Furthermore, Python’s dynamic nature, automatic error messages, and exception model make it easier and quicker to find and fix errors in Python than it is in some other languages (unlike C, for example, Python does not crash on errors).
The PyChecker and PyLint systems provide support for catching a large set of common errors ahead of time, before your script runs. They serve similar roles to the lint program in C development. Some Python groups run their code through PyChecker prior to testing or delivery, to catch any lurking potential problems. In fact, the Python standard library is regularly run through PyChecker before release. PyChecker and PyLint are third-party open source packages; you can find them at http://www.python.org or the PyPI website, or via your friendly neighborhood web search engine.

PyUnit (a.k.a. unittest)

In Chapter 24, we learned how to add self-test code to a Python file by using the __name__ == '__main__' trick at the bottom of the file. For more advanced testing purposes, Python comes with two testing support tools. The first, PyUnit (called unittest in the library manual), provides an object-oriented class framework for specifying and customizing test cases and expected results. It mimics the JUnit framework for Java. This is a sophisticated class-based unit testing system; see the Python library manual for details.

doctest

The doctest standard library module provides a second and simpler approach to regression testing, based upon Python’s docstrings feature. Roughly, to use doctest, you cut and paste a log of an interactive testing session into the docstrings of your source files. doctest then extracts your docstrings, parses out the test cases and results, and reruns the tests to verify the expected results. doctest’s operation can be tailored in a variety of ways; see the library manual for more details.

IDEs

We discussed IDEs for Python in Chapter 3. IDEs such as IDLE provide a graphical environment for editing, running, debugging, and browsing your Python programs. Some advanced IDEs (such as Eclipse, Komodo, NetBeans, and Wing IDE) may support additional development tasks, including source control integration, code refactoring, project management tools, and more. See Chapter 3, the text editors page at http://www.python.org, and your favorite web search engine for more on available IDEs and GUI builders for Python.

Profilers

Because Python is so high-level and dynamic, intuitions about performance gleaned from experience with other languages usually don’t apply to Python code. To truly isolate performance bottlenecks in your code, you need to add timing logic with clock tools in the time or timeit modules, or run your code under the profile module. We saw an example of the timing modules at work when comparing iteration tools’ speeds in Chapter 20. Profiling is usually your first optimization step—profile to isolate bottlenecks, then time alternative codings of them.
profile is a standard library module that implements a source code profiler for Python; it runs a string of code you provide (e.g., a script file import, or a call to a function) and then, by default, prints a report to the standard output stream that gives performance statistics—number of calls to each function, time spent in each function, and more.
The profile module can be run as a script or imported, and it may be customized in various ways; for example, it can save run statistics to a file to be analyzed later with the pstats module. To profile interactively, import the profile module and call profile.run('code'), passing in the code you wish to profile as a string (e.g., a call to a function, or an import of an entire file). To profile from a system shell command line, use a command of the form python -m profile main.py args... (see Appendix A for more on this format). Also see Python’s standard library manuals for other profiling options; the cProfile module, for example, has identical interfaces to profile but runs with less overhead, so it may be better suited to profiling long-running programs.

Debuggers

We also discussed debugging options in Chapter 3 (see its sidebar Debugging Python Code). As a review, most development IDEs for Python support GUI-based debugging, and the Python standard library also includes a source code debugger module called pdb. This module provides a command-line interface and works much like common C language debuggers (e.g., dbx, gdb).
Much like the profiler, the pdb debugger can be run either interactively or from a command line and can be imported and called from a Python program. To use it interactively, import the module, start running code by calling a pdb function (e.g., pdb.run("main()")), and then type debugging commands from pdb’s interactive prompt. To launch pdb from a system shell command line, use a command of the form python -m pdb main.py args... (see Appendix A for more on this format). pdb also includes a useful postmortem analysis call, pdb.pm(), which starts the debugger after an exception has been encountered.
Because IDEs such as IDLE also include point-and-click debugging interfaces, pdb isn’t a critical a tool today, except when a GUI isn’t available or when more control is desired. See Chapter 3 for tips on using IDLE’s debugging GUI interfaces. Really, neither pdb nor IDEs seem to be used much in practice—as noted in Chapter 3, most programmers either insert print statements or simply read Python’s error messages (not the most high-tech of approaches, but the practical tends to win the day in the Python world!).

Shipping options

In Chapter 2, we introduced common tools for packaging Python programs. py2exe, PyInstaller, and freeze can package byte code and the Python Virtual Machine into “frozen binary” standalone executables, which don’t require that Python be installed on the target machine and fully hide your system’s code. In addition, we learned in Chapter 2 that Python programs may be shipped in their source (.py) or byte code (.pyc) forms, and that import hooks support special packaging techniques such as automatic extraction of .zip files and byte code encryption.
We also briefly met the standard library’s distutils modules, which provide packaging options for Python modules and packages, and C-coded extensions; see the Python manuals for more details. The emerging Python “eggs” third-party packaging system provides another alternative that also accounts for dependencies; search the Web for more details.

Optimization options

There are a couple of options for optimizing your programs. The Psyco system described in Chapter 2 provides a just-in-time compiler for translating Python byte code to binary machine code, and Shedskin offers a Python-to-C++ translator. You may also occasionally see .pyo optimized byte code files, generated and run with the -O Python command-line flag (discussed in Chapters 21 and 33); because this provides a very modest performance boost, however, it is not commonly used.
As a last resort, you can also move parts of your program to a compiled language such as C to boost performance; see the book Programming Python and the Python standard manuals for more on C extensions. In general, Python’s speed also improves over time, so be sure to upgrade to the faster releases when possible.

Other hints for larger projects

We’ve met a variety of language features in this text that will tend to become more useful once you start coding larger projects. These include module packages (Chapter 23), class-based exceptions (Chapter 33), class pseudoprivate attributes (Chapter 30), documentation strings (Chapter 15), module path configuration files (Chapter 21), hiding names from from * with __all__ lists and _X-style names (Chapter 24), adding self-test code with the __name__ == '__main__' trick (Chapter 24), using common design rules for functions and modules (Chapters 17, 19, and 24), using object-oriented design patterns (Chapter 30 and others), and so on.

To learn about other large-scale Python development tools available in the public domain, be sure to browse the pages at the PyPI website at http://www.python.org, and the Web at large.