Beginning Python (eBook)

Magnus Lie Hetland is an experienced Python programmer, having used the language since the late 1990s. He is also an associate professor of algorithms at the Norwegian University of Science and Technology, having taught algorithms for the better part of a decade. Hetland is the author of Practical Python and Beginning Python, first and second editions, as well as several scientific papers.

Contents at a Glance 5
Contents 7
About the Author 25
About the Technical Reviewer 26
Preface 27
Introduction 29
Chapter 1: Instant Hacking: The Basics 31
The Interactive Interpreter 31
Algo . . . What? 32
Numbers and Expressions 33
Hexadecimals Octals and Binary 35
Variables 35
Statements 36
Getting Input from the User 37
Functions 38
Modules 39
cmath and Complex Numbers 40
Back to the __future__ 41
Saving and Executing Your Programs 41
Running Your Python Scripts from a Command Prompt 43
Making Your Scripts Behave Like Normal Programs 43
What About Double-Clicking? 44
Comments 44
Strings 44
Single-Quoted Strings and Escaping Quotes 45
Concatenating Strings 46
String Representations, str and repr 46
Long Strings, Raw Strings, and bytes 47
Long Strings 47
Raw Strings 48
Unicode, bytes, and bytearray 49
A Quick Summary 52
New Functions in This Chapter 53
What Now? 54
Chapter 2: Lists and Tuples 55
Sequence Overview 55
Common Sequence Operations 56
Indexing 56
Slicing 58
A Nifty Shortcut 58
Longer Steps 59
Adding Sequences 60
Multiplication 61
None, Empty Lists, and Initialization 61
Membership 62
Length, Minimum, and Maximum 63
Lists: Python’s Workhorse 64
The list Function 64
Basic List Operations 64
Changing Lists: Item Assignments 64
Deleting Elements 65
Assigning to Slices 65
List Methods 66
append 66
clear 67
copy 67
count 67
extend 67
index 68
insert 69
pop 69
remove 70
reverse 70
sort 70
Advanced Sorting 71
Tuples: Immutable Sequences 72
A Quick Summary 74
New Functions in This Chapter 74
What Now? 74
Chapter 3: Working with Strings 75
Basic String Operations 75
String Formatting: The Short Version 75
String Formatting: The Long Version 77
Replacement Field Names 77
Basic Conversions 78
Width, Precision, and Thousands Separators 79
Signs, Alignment, and Zero-Padding 80
String Methods 82
center 83
find 83
join 84
lower 84
replace 85
split 85
strip 86
translate 86
Is My String … 87
A Quick Summary 87
New Functions in This Chapter 88
What Now? 88
Chapter 4: Dictionaries: When Indices Won’t Do 89
Dictionary Uses 89
Creating and Using Dictionaries 90
The dict Function 90
Basic Dictionary Operations 91
String Formatting with Dictionaries 93
Dictionary Methods 93
clear 94
copy 94
fromkeys 95
get 95
items 97
keys 97
pop 97
popitem 98
setdefault 98
update 99
values 99
A Quick Summary 100
New Functions in This Chapter 100
What Now? 100
Chapter 5: Conditionals, Loops, and Some Other Statements 101
More About print and import 101
Printing Multiple Arguments 101
Importing Something as Something Else 102
Assignment Magic 103
Sequence Unpacking 103
Chained Assignments 105
Augmented Assignments 105
Blocks: The Joy of Indentation 106
Conditions and Conditional Statements 106
So That’s What Those Boolean Values Are For 106
Conditional Execution and the if Statement 108
else Clauses 108
elif Clauses 109
Nesting Blocks 109
More Complex Conditions 109
Comparison Operators 109
The Equality Operator 110
is: The Identity Operator 111
in: The Membership Operator 112
String and Sequence Comparisons 112
Boolean Operators 113
Assertions 114
Loops 115
while Loops 115
for Loops 116
Iterating Over Dictionaries 117
Some Iteration Utilities 117
Parallel Iteration 117
Numbered Iteration 118
Reversed and Sorted Iteration 119
Breaking Out of Loops 119
break 119
continue 120
The while True/break Idiom 121
else Clauses in Loops 122
Comprehensions—Slightly Loopy 122
A Better Solution 123
And Three for the Road 124
Nothing Happened! 124
Deleting with del 124
Executing and Evaluating Strings with exec and eval 126
exec 126
eval 127
A Quick Summary 128
New Functions in This Chapter 129
What Now? 129
Chapter 6: Abstraction 130
Laziness Is a Virtue 130
Abstraction and Structure 131
Creating Your Own Functions 131
Documenting Functions 132
Functions That Aren’t Really Functions 133
The Magic of Parameters 134
Where Do the Values Come From? 134
Can I Change a Parameter? 134
Why Would I Want to Modify My Parameters? 136
What If My Parameter Is Immutable? 139
Keyword Parameters and Defaults 140
Collecting Parameters 142
Reversing the Process 144
Parameter Practice 146
Scoping 147
Recursion 150
Two Classics: Factorial and Power 151
Another Classic: Binary Search 152
A Quick Summary 155
New Functions in This Chapter 156
What Now? 156
Chapter 7: More Abstraction 157
The Magic of Objects 157
Polymorphism 158
Polymorphism and Methods 159
Polymorphism Comes in Many Forms 160
Encapsulation 161
Inheritance 162
Classes 163
What Is a Class, Exactly? 163
Making Your Own Classes 163
Attributes, Functions, and Methods 165
Privacy Revisited 165
The Class Namespace 167
Specifying a Superclass 168
Investigating Inheritance 169
Multiple Superclasses 170
Interfaces and Introspection 171
Abstract Base Classes 172
Some Thoughts on Object-Oriented Design 174
A Quick Summary 175
New Functions in This Chapter 176
What Now? 176
Chapter 8: Exceptions 177
What Is an Exception? 177
Making Things Go Wrong . . . Your Way 177
The raise Statement 178
Custom Exception Classes 179
Catching Exceptions 179
Look, Ma, No Arguments! 180
More Than One except Clause 181
Catching Two Exceptions with One Block 182
Catching the Object 182
A Real Catchall 183
When All Is Well 183
And Finally . . . 185
Exceptions and Functions 186
The Zen of Exceptions 186
Not All That Exceptional 188
A Quick Summary 189
New Functions in This Chapter 190
What Now? 190
Chapter 9: Magic Methods, Properties, and Iterators 191
If You’re Not Using Python 3 191
Constructors 192
Overriding Methods in General, and the Constructor in Particular 193
Calling the Unbound Superclass Constructor 195
Using the super Function 196
Item Access 197
The Basic Sequence and Mapping Protocol 198
Subclassing list, dict, and str 200
More Magic 201
Properties 202
The property Function 203
Static Methods and Class Methods 204
__getattr__, __setattr__, and Friends 205
Iterators 206
The Iterator Protocol 206
Making Sequences from Iterators 208
Generators 208
Making a Generator 209
A Recursive Generator 210
Generators in General 211
Generator Methods 212
Simulating Generators 213
The Eight Queens 214
Generators and Backtracking 214
The Problem 215
State Representation 215
Finding Conflicts 216
The Base Case 216
The Recursive Case 217
Wrapping It Up 219
A Quick Summary 220
New Functions in This Chapter 221
What Now? 221
Chapter 10: Batteries Included 222
Modules 222
Modules Are Programs 222
Modules Are Used to Define Things 224
Defining a Function in a Module 224
Adding Test Code in a Module 225
Making Your Modules Available 226
Putting Your Module in the Right Place 227
Telling the Interpreter Where to Look 227
Packages 228
Exploring Modules 229
What’s in a Module? 229
Using dir 229
The __all__ Variable 230
Getting Help with help 230
Documentation 231
Use the Source 231
The Standard Library: A Few Favorites 232
sys 232
os 234
fileinput 235
Sets, Heaps, and Deques 237
Sets 238
Heaps 240
Deques (and Other Collections) 241
time 242
random 243
shelve and json 246
A Potential Trap 247
A Simple Database Example 247
re 250
What Is a Regular Expression? 250
The Wildcard 250
Escaping Special Characters 251
Character Sets 251
Alternatives and Subpatterns 251
Optional and Repeated Subpatterns 252
The Beginning and End of a String 252
Contents of the re Module 253
Match Objects and Groups 255
Group Numbers and Functions in Substitutions 256
Finding the Sender of an Email 258
A Sample Template System 260
Other Interesting Standard Modules 264
A Quick Summary 265
New Functions in This Chapter 266
What Now? 266
Chapter 11: Files and Stuff 267
Opening Files 267
File Modes 267
The Basic File Methods 268
Reading and Writing 269
Piping Output 270
Reading and Writing Lines 271
Closing Files 272
Using the Basic File Methods 273
Iterating over File Contents 274
One Character (or Byte) at a Time 274
One Line at a Time 275
Reading Everything 275
Lazy Line Iteration with fileinput 276
File Iterators 276
A Quick Summary 277
New Functions in This Chapter 278
What Now? 278
Chapter 12: Graphical User Interfaces 279
Building a Sample GUI Application 279
Initial Exploration 280
Layout 282
Event Handling 282
The Final Program 283
Using Something Else 285
A Quick Summary 285
What Now? 285
Chapter 13: Database Support 286
The Python Database API 286
Global Variables 287
Exceptions 288
Connections and Cursors 288
Types 290
SQLite and PySQLite 290
Getting Started 292
A Sample Database Application 292
Creating and Populating Tables 293
Searching and Dealing with Results 294
A Quick Summary 295
New Functions in This Chapter 296
What Now? 296
Chapter 14: Network Programming 297
A Handful of Networking Modules 297
The socket Module 298
The urllib and urllib2 Modules 299
Opening Remote Files 300
Retrieving Remote Files 300
Other Modules 301
SocketServer and Friends 302
Multiple Connections 303
Forking and Threading with SocketServer 304
Asynchronous I/O with select and poll 305
Twisted 307
Downloading and Installing Twisted 308
Writing a Twisted Server 308
A Quick Summary 310
New Functions in This Chapter 311
What Now? 311
Chapter 15: Python and the Web 312
Screen Scraping 312
Tidy and XHTML Parsing 313
What’s Tidy? 313
Getting Tidy 315
But Why XHTML? 316
Using HTMLParser 316
Beautiful Soup 318
Dynamic Web Pages with CGI 319
Step 1: Preparing the Web Server 319
Step 2: Adding the Pound Bang Line 320
Step 3: Setting the File Permissions 320
CGI Security Risks 321
A Simple CGI Script 321
Debugging with cgitb 322
Using the cgi Module 323
A Simple Form 324
Using a Web Framework 326
Other Web Application Frameworks 326
Web Services: Scraping Done Right 327
RSS and Friends 327
Remote Procedure Calls with XML-RPC 328
SOAP 329
A Quick Summary 329
New Functions in This Chapter 329
What Now? 329
Chapter 16: Testing, 1-2-3 330
Test First, Code Later 330
Precise Requirement Specification 330
Planning for Change 332
The 1-2-3 (and 4) of Testing 333
Tools for Testing 333
doctest 333
unittest 335
Beyond Unit Tests 338
Source Code Checking with PyChecker and PyLint 338
Profiling 341
A Quick Summary 342
New Functions in This Chapter 343
What Now? 343
Chapter 17: Extending Python 344
The Best of Both Worlds 344
The Really Easy Way: Jython and IronPython 345
Writing C Extensions 348
A Swig of … SWIG 349
What Does It Do? 350
I Prefer Pi 350
The Interface File 351
Running SWIG 351
Compiling, Linking, and Using 352
A Shortcut Through the Magic Forest of Compilers 353
Hacking It on Your Own 353
Reference Counting 354
A Framework for Extensions 355
Palindromes, Detartrated1 for Your Enjoyment 356
A Quick Summary 358
New Functions in This Chapter 359
What Now? 359
Chapter 18: Packaging Your Programs 360
Setuptools Basics 360
Wrapping Things Up 362
Compiling Extensions 363
Creating Executable Programs with py2exe 364
A Quick Summary 365
New Functions in This Chapter 365
What Now? 365
Chapter 19: Playful Programming 366
Why Playful? 366
The Jujitsu of Programming 366
Prototyping 367
Configuration 368
Extracting Constants 368
Configuration Files 369
Logging 371
If You Can’t Be Bothered 372
If You Want to Learn More 373
A Quick Summary 373
What Now? 374
Chapter 20: Project 1: Instant Markup 375
What’s the Problem? 375
Useful Tools 376
Preparations 376
First Implementation 377
Finding Blocks of Text 377
Adding Some Markup 378
Second Implementation 380
Handlers 380
A Handler Superclass 381
Rules 382
A Rule Superclass 383
Filters 383
The Parser 384
Constructing the Rules and Filters 385
Putting It All Together 388
Further Exploration 392
What Now? 393
Chapter 21: Project 2: Painting a Pretty Picture 394
What’s the Problem? 394
Useful Tools 395
Preparations 395
First Implementation 396
Drawing with ReportLab 396
Constructing Some PolyLines 397
Writing the Prototype 398
Second Implementation 400
Getting the Data 400
Using the LinePlot Class 401
Further Exploration 403
What Now? 403
Chapter 22: Project 3: XML for All Occasions 404
What’s the Problem? 404
Useful Tools 405
Preparations 406
First Implementation 407
Creating a Simple Content Handler 407
Creating HTML Pages 409
Second Implementation 412
A Dispatcher Mix-In Class 412
Factoring Out the Header, Footer, and Default Handling 413
Support for Directories 414
The Event Handlers 415
Further Exploration 417
What Now? 417
Chapter 23: Project 4: In the News 418
What’s the Problem? 418
Useful Tools 419
Preparations 419
First Implementation 419
Second Implementation 421
Further Exploration 428
What Now? 429
Chapter 24: Project 5: A Virtual Tea Party 430
What’s the Problem? 430
Useful Tools 430
Preparations 431
First Implementation 432
The ChatServer Class 432
The ChatSession Class 433
Putting It Together 435
Second Implementation 437
Basic Command Interpretation 437
Rooms 438
Login and Logout Rooms 438
The Main Chat Room 439
The New Server 439
Further Exploration 445
What Now? 445
Chapter 25: Project 6: Remote Editing with CGI 446
What’s the Problem? 446
Useful Tools 446
Preparations 447
First Implementation 447
Second Implementation 448
Creating the File Name Form 449
Writing the Editor Script 449
Writing the Save Script 450
Running the Editor 452
Further Exploration 453
What Now? 454
Chapter 26: Project 7: Your Own Bulletin Board 455
What’s the Problem? 455
Useful Tools 456
Preparations 456
First Implementation 457
Second Implementation 462
Writing the Main Script 462
Writing the View Script 463
Writing the Edit Script 465
Writing the Save Script 466
Trying It Out 467
Further Exploration 469
What Now? 469
Chapter 27: Project 8: File Sharing with XML-RPC 470
What’s the Problem? 470
Useful Tools 471
Preparations 472
First Implementation 472
Implementing a Simple Node 472
Trying Out the First Implementation 477
Second Implementation 478
Creating the Client Interface 479
Raising Exceptions 480
Validating File Names 480
Trying Out the Second Implementation 485
Further Exploration 485
What Now? 485
Chapter 28: Project 9: File Sharing II—Now with GUI! 486
What’s the Problem? 486
Useful Tools 486
Preparations 486
First Implementation 487
Second Implementation 489
Further Exploration 491
What Now? 492
Chapter 29: Project 10: Do-It-Yourself Arcade Game 493
What’s the Problem? 493
Useful Tools 494
pygame 494
pygame.locals 494
pygame.display 494
pygame.font 495
pygame.sprite 495
pygame.mouse 495
pygame.event 496
pygame.image 496
Preparations 496
First Implementation 497
Second Implementation 500
Further Exploration 511
What Now? 511
Appendix A: The Short Version 512
The Basics 512
Functions 514
Objects and Stuff . . . 515
Some Loose Ends 518
Appendix B: Python Reference 520
Expressions 520
Statements 530
Simple Statements 530
Expression Statements 530
Assert Statements 530
Assignment Statements 530
Augmented Assignment Statements 530
The pass Statement 531
The del Statement 531
The return Statement 531
The yield Statement 531
The raise Statement 531
The break Statement 532
The continue Statement 532
The import Statement 532
The global Statement 533
The nonlocal Statement 533
Compound Statements 533
The if Statement 533
The while Statement 534
The for Statement 534
The try Statement 534
The with Statement 534
Function Definitions 535
Class Definitions 535
Index 536