Tag Archives: python

Did you use compressing while using resources from your scripts and projects? Many data files are compressed to save disc space and compressed requests over the network saves bandwith.

This article gives some hints to use the “battery included” power of python for the handling of compressed files or use HTTP with gzip compression:

1. reading log files as GZIP or uncompressed
2. using files directly from ZIP- or RAR compression archive
3. use gzip compressing while fetching web sites

reading GZip compressed log files

The most common use case for reading compressed files are log files. My access_log files are archived in an extra directory and can be easy used for creating statistics. This is done normally with zgrep/zless and other command line tools. But opening a gzip-compressed file with python is build-in and can be transparent for the file handling.

import gzip, sys, os

if len(sys.argv)<1:
    sys.exit()
try:
  fp = gzip.open(sys.argv[1])
  line = gzip.readline()
except IOError:
  fp = open(sys.argv[0])
  line = fp.readline()

while line:
  ...

The tricky part is the IOError line. If you open a plain text file with the GzipFile class it fails while reading the first line (not while calling the constructor).

You can use the bz2-module with the BZ2File constructor if you have bzip2-compressed files.

Reading a file directly from zip archive

If you need a british word list your linux can help (if the british dictionary is installed). If not you can get it from several sources. I choosed the zip-compressed file from pyxidium.co.uk and included it in a small python script.

Zip-files are a container format. You can put many files in it and have to choose which file you want to decompress from the ZIP-file. In the example I will fetch the ZIP achive into memory and unzip the file

en-GB-wlist.txt

directly.

import zipfile, os, StringIO, urllib

def openWordFile():
    #reading english word dictionary
    pathToBritishWords = "/usr/share/dict/british-english"
    uriToBritshWords = "http://en-gb.pyxidium.co.uk/dictionary/en-GB-wlist.zip"

    if os.path.isfile(pathToBritishWords):
        fp = file(pathToBritishWords)
    else:
        #fetch from uri
        data = urllib.urlopen(uriToBritshWords).read()
        #get an ZipFile object based on fetched data
        zf = zipfile.ZipFile(StringIO.StringIO(data))
        #read one file directly from ZipFile object
        fp = zf.open("en-GB-wlist.txt")
    return fp

#read all lines
words = openWordFile().readlines()

print "read %d words" % len(words)

If you want to read directly from a RAR file you have to install the rarfile module.

sudo easy_install rarfile

The module use the command line utility rar/unrar, but the usage is the same like the zipfile module.

import rarfile
rf = rarfile.RarFile("test.rar")
fp = rf.open(“compressed.txt”)

Use gzip compression while fetching web pages

The speed of fetching web pages has many parameters. To save the important parameter band width you should fetch http resources compressed. Every modern browser support this feature (see test on browserscope.org) and every webserver should be able to compress the text content.

Your HTTP client must send the HTTP header “Accept-Encoding” to offer the possibility for compressed content. And you have to check the response header if the server sent compressed content. A web server can ignore this request header!

import urllib2, zlib, gzip, StringIO, sys

uri = "http://web.de/index.html"
req = urllib2.Request(uri, headers={"Accept-Encoding":"gzip, deflate"})
res = urllib2.urlopen(req)
if res.getcode()==200:
    if res.headers.getheader("Content-Encoding").find("gzip")!=-1:
        # the urllib2 file-object dont support tell/seek, repacking in StringIO
        fp = gzip.GzipFile(fileobj = StringIO.StringIO(res.read()))
        data = fp.read()
    elif res.headers.getheader("Content-Encoding").find("deflate")!=-1:
        data = zlib.decompress(res.read())
    else:
        data = res.read()
else:
    print "Error  while fetching ..." % res.msg
    sys.exit(-1)

print "read %s bytes (compression: %s), decompressed to %d bytes" % (
    res.headers.getheader("Content-Length"),
    res.headers.getheader("Content-Encoding"),
    len(data))

As a developer I did not found the automatic support for gzip-enabled HTTP requests for HTTP clients in different libraries. And python dont offer the support build-in too. Copy/paste this lines in your next project or convert it in your favorite language and your HTTP request layer will become faster.

conclusion

One disadvantage: your software will consume some percent more cpu to decompress the data on-the-fly and will be slower on your local machine. Python use the c-binding to the zlib and is fast as any other component and in a network environment you can messure the benefit.

To work with lists many c/c++ programmer use a simple for-loop. I will try to explain how this can be made a little bit easier. For that I use the self-explaining script language python.

Here is a simple example to filter all odd integers from a list in simple python 2.x syntax.

def filter_odd(myList):  
   result = []               # initialize the result list  
   for num in myList:               # iterate over the list  
     if num % 2 == 1:        # modulo 2 return 1 for odd integers
        result.append(num)   # add to the result
   return result                 

The function call will return a list containing only odd integers (you can use the result of the modulo operation directly without comparing with 1). Try it directly on a interactive python console (as an applet on secapps.com).

>> print filter_odd([1,2,3,4,5]) 
[1, 3, 5]

try it functional

The function filter_odd could be a more generic method, if the condition could be a variable or used as a parameter:

def condition_odd(num):        # the condition as separate function
  return num % 2 == 1
#condition_odd

def generic_filter(condition, myList):
  result = []
  for num in myList:
    if condition(num):         # the same condition like the first example, only as parameter
      result.append(num)
  return result
#generic_filter

We can define a condition as a function and use it as a parameter in the generic_filter implementation to filter the values of list myList. Ok, you don’t have to define a generic filter function, python offers the filter function in the standard language.

>> print filter_generic(condition_odd, [1,2,3,4,5])
[1, 3, 5]
>>print filter(condition_odd, [1,2,3,4,5])
[1, 3, 5]

The new implementation does not say what to do, it say how to use the condition on a list and has much less code compared to first example.

Passing a function as a parameter is a key aspect of functional programming. But the condition_odd function will only be used for the line with the filter call. The next step will show you, how to use this function parameter without defining it separately.

>> print filter(lambda x: x % 2 == 1, [1,2,3,4,5])
[1, 3, 5]

The first parameter is the lambda construct: Create a temporary, anonymous function with x as parameter and return the result of the term “x % 2 == 1“. I don’t have to define a function and I can use the filter in one line. Save namespace with one-time functions and code lines.

list comprehension

One-line solutions are fine to write short code, but it also has to be readable. If the reader doesn’t know what filter and lambda means, it can be confusing.

An other alternative for the one-line filter solution, and more confusing in python, is

list comprehension</cite>. You can write the above example:

<pre class="prettyprint">>> print [x for x in [1,2,3,4,5] if x % 2 == 1]
[1, 3, 5]

The result contains items from the constant list, filtered with the term in the if-condition. [Updated] You have to use the exact syntax, not the simple variant like filter.

more functional helper: map and reduce

There are some other list functions. First the map function with calls for every item in the list a function. In a example we will double every integer from the list. This could also be written as a list comprehension.

>>print map(lambda x: x*2, [1,2,3,4,5])
[2, 4, 6, 8, 10]
>>print [x**2 for x in [1,2,3,4,5]]
[2, 4, 6, 8, 10]

Second function is reduce. It get a function and a list as parameter and will call the function with the first two items from the list. Then the result and the next item will be computed and so on. A simple exapmle will calculate the sum of a number list:

>>print reduce(lambda a, b: a + b, [1,2,3,4,5])
15

This simple example can also resolved with the build-in sum function.

two examples

Now we will try some real world examples - project eulers first problem: Find the sum of all the multiples of 3 or 5 below 1000. Simply to resolve with a for-loop:

1. the condition term is x%3==0 or x%5==0
2. filter the values beetween 0 and 1000
3. calculate the sum

def eulerOne(myList):
  mySum = 0
  for num in range(1000):
    if num%3==0 or num%5==0:
      mySum += num
  return mySum

But we want to do it with an one-liner, with functional programming or a list comprehension:

>>print reduce(lambda a,b:a + b, [x for x in range(1000) if x % 3 == 0 or x % 5 == 0])
233168
>>print sum([x for x in range(1000) if not (x%3 and x%5)])
233168

The second line show a shorter version with sum and a little bit boolean algebra but it will not so clear readable.

As a second example filter all persons who older than 18:

>>persons = [{'age': 5, 'name': 'peter'}, {'age': 20, 'name': 'paul'}]
>>print  [x['name'] for x in persons if x['age'] > 18]
['paul']
>>print map(lambda x:x['name'], filter(lambda x: x['age']>19, persons))
['paul']

summary

Here the summary with the python doc-strings to the new learned functions:

filter(function or None, sequence) -> list, tuple, or string

Return those items of sequence for which function(item) is true. If
function is None, return the items that are true. If sequence is a tuple
or string, return the same type, else return a list.

map(function, sequence[, sequence, ...]) -> list

Return a list of the results of applying the function to the items of
the argument sequence(s). If more than one sequence is given, the
function is called with an argument list consisting of the corresponding
item of each sequence, substituting None for missing values when not all
sequences have the same length. If the function is None, return a list of
the items of the sequence (or a list of tuples if more than one sequence).

reduce(function, sequence[, initial]) -> value

Apply a function of two arguments cumulatively to the items of a sequence,
from left to right, so as to reduce the sequence to a single value.
For example, reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) calculates
((((1+2)+3)+4)+5). If initial is present, it is placed before the items
of the sequence in the calculation, and serves as a default when the
sequence is empty.

lambda - syntax

normal function definition:

def name(arguments):
  return expression

anonymous lambda function definition:

lambda arguments: expression

[Updated]: filter and map are still in python 3000 in the iterator variant, the reduce-function is moved in the functools module.

• Origins of Python's "Functional" Features
• Natural Language Processing with Python
• Python 3 for Absolute Beginners (Expert's Voice in Open Source)