Generating a list of users

There are lots of ways of putting together a list of Twitter users. If you’re on Twitter, you could use the list of your followers or the list of those you are following. You could extract user names from a list of search results, the public timeline or a twitter directory. There are lots of options. The following code uses a regular expression to extract the user names from a wefollow page.

import re
import urllib
 
def getWefollowTwitterUsers(category = "tech"):
        users = []
        url = "http://wefollow.com/twitter/"
        url += category
        html = urllib.urlopen(url).read()
        users = re.findall("""nofollow">(.*?)</a></strong>""", html)
        return users
 
if __name__ == "__main__":
        print getWefollowTwitterUsers()
 
Output:
['kevinrose', 'google', 'LeoLaporte', 'mashable', 'TechCrunch', 'Veronica', 
'alexalbrecht', 'ev', 'patricknorton', 'Scobleizer', 'woot', 'ijustine', 'timoreilly', 
'guykawasaki', 'engadget', 'CaliLewis', 'chrispirillo', 'wired', 'ryan', 'sarahlane', 
'ambermac', 'ginatrapani', 'tferriss', 'fforward', 'mollywood']

Retrieving a list of messages for each user

Each user’s messages are available in an RSS feed of the format http://twitter.com/statuses/user_timeline/.rss?count=[1..200]. The count parameter is optional and controls the maximum number of messages contained in the feed. The following code uses Universal Feed Parser to extract the entries from the data feed.

import feedparser
 
# [...]
 
def getUserMessages(user):
        url = "http://twitter.com/statuses/user_timeline/" + user + ".rss?count=200"
        feed_data = feedparser.parse(url)
        return feed_data.get("entries", [])

Generating keyword scores

The following code goes through a user’s messages, breaks them into fragments and counts the number of instances for each encountered word.

def getKeywordScores(user, messages):
        keywords = {}
        blacklist = ["a", "an", "by", "on", "that", "the", "these", "this", "those", "to"]
        # and many more words
        blacklist.append(user)
        for message in messages:
                tweet = message["summary"]
                words = re.split(" ", tweet)
                for word in words:
                        word = re.sub("^\W*", "", word)
                        word = re.sub("\W*$", "", word)
                        if word.startswith("http://"):
                                continue
                        word = word.lower()
                        if word in blacklist:
                                continue
                        if not word:
                                continue
                        count = keywords.get(word, 0)
                        keywords[word] = count + 1
        final_keywords = {}
        for k in keywords:
                if keywords[k] > 1:
                        final_keywords[k] = keywords[k]
        return final_keywords

Computing similarities

The code to compute similarity scores and the ideas behind that are presented in Programming Collective Intelligence. The source code for the book is available online. The relevant pieces are in chapter2/recommendations.py – sim_distance() (Euclidian Distance), sim_pearson() (Pearson Coefficient) and topMatches(). The latter compares one user to all others and returns the list of n most similar users along with their respective similarity scores.

Similar users

The following code brings it all together and demonstrates how we can show users that are similar to a specific one, given the computed dictionary of keyword scores.

from recommendations import sim_pearson, sim_distance, topMatches
# [...]
if __name__ == "__main__":
        users = getWefollowTwitterUsers()
        # add my own
        users.append("abendig")
        print users
        user_keywords = {}
        for user in users:
                print "processing data for:", user
                messages = getUserMessages(user = user)
                user_keywords[user] = getKeywordScores(user = user, messages = messages)
 
        # Similarity between the first user and three others
        print sim_pearson(user_keywords, users[0], users[1])
        print sim_pearson(user_keywords, users[0], users[2])
        print sim_pearson(user_keywords, users[0], users[3])
 
        # My top three matches
        print topMatches(user_keywords, "abendig", n = 3, similarity = sim_pearson)

Here is the output that this produces (at the time of this writing):

['kevinrose', 'google', 'LeoLaporte', 'mashable', 'TechCrunch', 'Veronica', 'alexalbrecht', 
'ev', 'patricknorton', 'Scobleizer', 'woot', 'ijustine', 'timoreilly', 'guykawasaki', 
'engadget', 'CaliLewis', 'chrispirillo', 'sarahlane', 'ryan', 'wired', 'ambermac', 
'ginatrapani', 'tferriss', 'fforward', 'mollywood', 'abendig']
processing data for: kevinrose
processing data for: google
processing data for: LeoLaporte
processing data for: mashable
processing data for: TechCrunch
processing data for: Veronica
processing data for: alexalbrecht
processing data for: ev
processing data for: patricknorton
processing data for: Scobleizer
processing data for: woot
processing data for: ijustine
processing data for: timoreilly
processing data for: guykawasaki
processing data for: engadget
processing data for: CaliLewis
processing data for: chrispirillo
processing data for: sarahlane
processing data for: ryan
processing data for: wired
processing data for: ambermac
processing data for: ginatrapani
processing data for: tferriss
processing data for: fforward
processing data for: mollywood
processing data for: abendig
0.693852667302
0.57137732992
0.350957713398
[(0.85762813072101673, 'ginatrapani'), 
(0.81973579573386002, 'CaliLewis'), 
(0.81455896587667598, 'timoreilly')]

The results suggest the users ginatrapani, CaliLewis and timoreilly as related to abendig based on the available data and thus maybe worth following.

Next

This showed an example of directly applying code and ideas from the book Programming Collective Intelligence to Twitter users and their message streams. This is of course also pretty simplified. User similarity is an interesting problem though.

There are lots of ways to make this more useful. The realtime nature of the message streams should be taken into account. Users’ posting frequency may matter. Also, people’s interests certainly change. Overall similarity is useful, but similarity based on time ranges could also be interesting.

URLs that are included in the messages are currently mostly ignored. It would of course make a lot of sense to include them (don’t forget to deduplicate the various URL shortener versions of the same URL) to be able to take into account that several people may be talking about the same articles.

Simple keyword counts are pretty crude. Semantic analysis of the messages would be useful to get an indicator of whether two people are talking about similar things even though they are using different words, if their opinions are similar, and so forth.

Oh, and scale it up to include millions of users.

The frequent items problem is to process a stream of items and find all those which occur more than a given fraction of the time. It is one of the most heavily studied problems in mining data streams, dating back to the 1980s. Many other applications rely directly or indirectly on finding the frequent items, and implementations are in use in large scale industrial systems. In this paper, we describe the most important algorithms for this problem in a common framework. We place the different solutions in their historical context, and describe the connections between them, with the aim of clarifying some of the confusion that has surrounded their properties.

Some of the interesting bits here are that the data stream will easily contain millions (or billions) of items and the algorithm will typically only get to take one look at each item as it comes up in the stream.

Space-Saving

In this post I focus on the Space-Saving algorithm and provide an implementation in Python. The algorithm itself is originally described in Efficient Computation of Frequent and Top-k Elements in Data Streams [PDF] by Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi:

We propose an integrated approach for solving both problems of finding the most popular k elements, and finding frequent elements in a data stream. Our technique is efficient and exact if the alphabet under consideration is small. In the more practical large alphabet case, our solution is space efficient and reports both top-k and frequent elements with tight guarantees on errors. For general data distributions, our top-k algorithm can return a set of k’ elements, where k’ ≈ k, which are guaranteed to be the top-k’ elements; and we use minimal space for calculating frequent elements. For realistic Zipfian data, our space requirement for the frequent elements problem decreases dramatically with the parameter of the distribution; and for top-k queries, we ensure that only the top-k elements, in the correct order, are reported. Our experiments show significant space reductions with no loss in accuracy.

The algorithm basically works like this: The stream is processed one item at a time. A collection of k distinct items and their associated counters is maintained. If a new item is encountered and fewer than k items are in the collection, then the item is added and its counter is set to 1. If the item is already in the collection, its counter is increased by 1. If the item is not in the collection and the collection already has a size of k, then the item with lowest counter is removed and the new item is added, with its counter set to one larger than the previous minimum counter.

Here is some pseudo code to make this clearer:

SpaceSaving(k, stream):
collection = empty collection
for each element in stream:
    if element in collection:
    then collection[element] += 1
    else if length of collection < k:
        then add element to collection, collection[element] = 1
    else:
        current_minimum_element = element with lowest count value in collection
        current_minimum = collection[current_minimum_element]
        remove current_minimum_element from collection
        collection[element] = current_minimum + 1

The straightforward approach

A first, easy implementation would use a simple hashtable, such as in the following piece of code:

def space_saving_frequent_k1(k, stream, debug=False):
        def get_smallest_key(d):
                """
                Given dictionary d, returns the key associated with
                the lowest value in the dictionary.
                """
                min_key = None
                for key in d:
                        if not min_key or d[key] < d[min_key]:
                                min_key = key
                return min_key
 
        counters = {}
        for element in stream:
                if counters.has_key(element):
                        counters[element] = counters[element] + 1
                elif len(counters) < k:
                        counters[element] = 1
                else:
                        current_minimum_key = get_smallest_key(counters)
                        if current_minimum_key:
                                counters[element] = counters[current_minimum_key] + 1
                                del counters[current_minimum_key]
                        else:
                                counters[element] = 1
        return counters

This works for smaller data sets and particularly, when there is never a need to find that smallest element. Otherwise however, (repeatedly) retrieving the element with the minimum count remains a comparatively costly challenge.

Stream-Summary

When describing the Space-Saving algorithm, the authors also introduced the Stream-Summary data structure (inspired by work in Frequency Estimation of Internet Packet Streams with Limited Space [PDF]), which groups elements with equal values together (in buckets) and allows quick retrieval of the element with the lowest count.

Here is a diagram of this structure, using three buckets and a total of six elements (E1-E6).

Buckets are stored in a list sorted by the buckets’ respective values. Each bucket maintains knowledge of associated elements. Each element in turn maintains a pointer to its bucket. The latter is implemented using a simple hashtable. If an element’s count needs to be increased, the element is removed from its current bucket and added to the neighboring bucket with value one greater than the previous one. If no such bucket exists, it is inserted in the bucket list. Empty buckets are removed.

The Python implementation using the Stream-Summary data structure may then look like this:

class Bucket(object):
        def __init__(self, value = 1, elements = []):
                self.value = value
                self.elements = []
 
        def __str__(self):
                return "%s: %s" % (self.value, str(self.elements))
 
        def append(self, element):
                self.elements.append(element)
 
        def first_element(self):
                if self.elements:
                        return self.elements[0]
                else:
                        return None
 
        def has_elements(self):
                return len(self.elements) > 0
 
        def remove(self, element):
                self.elements.remove(element)
 
class StreamSummary(object):
        """
        Maintains a dictionary of elements and a list of buckets. Each element
        points to a (parent) bucket.
        The bucket list is sorted based on the buckets' values. Each bucket also
        maintains a list of elments.
        This has the effect of grouping elements with equal values in buckets.
        """
        def __init__(self):
                self.elements = {}
                self.buckets = []
 
        def __len__(self):
                return len(self.elements.keys())
 
        def __str__(self):
                result = ""
                for b in self.buckets:
                        result += str(b) + " "
                return result
 
        def add_element(self, element):
                """
                Adds an element and ensures it's assigned to the correct bucket.
                """
                if not self.elements.has_key(element):
                        if not self.buckets or self.buckets[0].value != 1:
                                self.buckets.insert(0, Bucket())
                        self.elements[element] = self.buckets[0]
                        self.buckets[0].elements.append(element)
 
        def increase_element(self, element):
                """
                Increasing an element's value also means assigning it to the
                correct bucket. That can result in creating a new bucket and/or
                removing an empty one.
                """
                current_bucket = self.elements[element]
                bucket_index = self.buckets.index(current_bucket)
 
                if len(self.buckets) == bucket_index + 1:
                        self.buckets.append(Bucket(value = current_bucket.value + 1))
                elif self.buckets[bucket_index + 1].value > current_bucket.value + 1:
                        self.buckets.insert(bucket_index + 1,
                                            Bucket(value = current_bucket.value + 1))
                current_bucket.remove(element)
                self.elements[element] = self.buckets[bucket_index + 1]
                if not current_bucket.has_elements():
                        del self.buckets[bucket_index]
                self.elements[element].append(element)
 
        def has_element(self, element):
                return self.elements.has_key(element)
 
        def get_minimum(self):
                if self.buckets:
                        return self.buckets[0].first_element()
                else:
                        return None
 
        def replace_element(self, old_element, new_element):
                """
                Replaces an existing element with an entirely new element in
                the old element's bucket.
                """
                self.elements[new_element] = self.elements[old_element]
                self.elements[new_element].remove(old_element)
                self.elements[new_element].append(new_element)
                del self.elements[old_element]
 
def space_saving_frequent_k(k, stream):
        summary = StreamSummary()
        for element in stream:
                if summary.has_element(element):
                        summary.increase_element(element)
                elif len(summary) < k:
                        summary.add_element(element)
                else:
                        current_minimum_key = summary.get_minimum()
                        if current_minimum_key:
                                summary.replace_element(current_minimum_key, element)
                                summary.increase_element(element)
                        else:
                                summary.add_element(element)
        return summary

For larger data sets, where k is noticeably smaller than the number of distinct elements in the set, the Stream-Summary data structure proves advantageous.

Onward

There is a lot of ongoing research in this problem area. This article is clearly just barely offering a small (and simplified) glimpse. Explore the research. Find out what real-world applications use some version of this as part of their problem solving approach. Applications can be found in web access log processing, search applications, mining of real-time message streams, and so forth.

1. Is one string a circular rotation of a second string?
2. Is one string a substring of a (potentially rotated) second string?

Previously discussed Z-values are used for a solution.

Circular string rotation?

Given a prefix p of string t and a suffix s of string t, such that p + s = t, then a circular string rotation r is a string of the form s + p. Thus, it is a string that consists of the suffix of string t, directly followed by the prefix of string t, such that the resulting string r has the same length as the original string t: |r| = |t|.

Example:

Original:
t = “abcd”

Rotation:
r = “cdab”, with p = “ab” and s = “cd”

How can one string be shown to be a circular rotation of a second string?

If it is, then the lengths of the two strings t and r have to be identical. Also, since t = p + s and the rotation r = s + p, then 2r = r + r = s + p + s + p = s + t + p. Thus, if r is a rotation of t, then t is included completely within 2r. Not only that, but if it is shown that r is a rotation of t, then if t is found within 2r, more specific information can be shown about the rotation, because p and s can easily be indicated, too.

Assuming access to a fast string matching routine, the mechanism to answer the question should be straightforward:

If |t| = |r| and t in r+r
Then Circular Rotation
Else No Circular Rotation

Saving space

It may not be desirable to create a new string that contains twice the data of r, particularly assuming very large instances of strings r and t. Since r = s + p is available though, it is easy to imagine what 2r would look like, without actually creating a new string consisting of 2r.

The previously mentioned Z algorithm can be heavily modified to

1. operate on two strings, and
2. allow for Z-values up to maximum of the string length, regardless of the index position, effectively allowing for a substring to begin towards the end of the string and continue at the beginning of the string.

getMaxSuffixZ

Let getMaxSuffixZ be a process that takes as input two strings, t and r and returns a maximum Z-value <= |t|. This involves calculating Z-values for each position in r and returning the largest one. In this case the Z-values are not calculated based on a prefix of r. Rather, string t is considered the prefix. So, if r is a rotation of t and t = p + s, then r = s + p, so maxZ should return |p|. It then only makes sense for maxZ to pick that maximum Z-Value that represents a substring that stretches to the end of r.

Here is the modified implementation:

def getMaxSuffixZ(p, s):
	result = {}
	l = 0
	r =  -1
	for k in range(0, len(s)):
		if k > r:
			zk = 0
			for si in range(0, len(s)):
				if k + si < len(s) and \
					si < len(p) and \
					p[si] == s[k + si]:
					zk += 1
				else:
					break
			if zk > 0:
				r = zk + k - 1
				l = k
		else:
			kOld = k - l - 1
			zOld = result[kOld]
			b = r - k + 1
			if zOld < b:
				zk = zOld
			else:
				zk = b
				for si in range(b, len(s)):
					if k + si < len(s) \
						and si < len(p) \
						and p[si] \
						== s[k + si]:
						pass
					else:
						break
				zk = si
				r = zk + k - 1
				l = k
		result[k] = zk
		if zk == len(p) - k:
			return zk
	return 0

This returns the expected results at least for this sample set:

assert getMaxSuffixZ(s, "") == 0
assert getMaxSuffixZ(s, "a") == 0
assert getMaxSuffixZ(s, "b") == 0
assert getMaxSuffixZ(s, "defabc") == 3
assert getMaxSuffixZ(s, "defabcd") == 0
assert getMaxSuffixZ(s, "abcdef") == 6
assert getMaxSuffixZ(s, "fabcde") == 5
print 'all tests passed.'
 
all tests passed.

Let z = getMaxSuffixZ(t, r). z is the prefix of t (and the suffix of r). If z > 0 then t[0..z-1] = r[z..|r|]. Similarly getMaxSuffixZ(r, t) yields the prefix of r (and the suffix of t).

s = "abcdef"
assert getMaxSuffixZ("", s) == 0
assert getMaxSuffixZ("a", s) == 1
assert getMaxSuffixZ("b", s) == 0
assert getMaxSuffixZ("defabc", s) == 3
assert getMaxSuffixZ("defabcd", s) == 0
assert getMaxSuffixZ("abcdef", s) == 6
assert getMaxSuffixZ("fabcde", s) == 1
print 'all tests passed.'
 
all tests passed.

If r is a rotation of t, then getMaxSuffixZ(t, r) + getMaxSuffixZ(r, t) = |t| = |r|. If t = r, then getMaxSuffixZ(t, r) = getMaxSuffixZ(r, t) = |t| = |r|. This leads to the following flow:

If |t| = |r|
Then
    zTr = getMaxSuffixZ(t, r)
    If zTr = |t|:
    Then Circular Rotation (t = r)
    Else
        If zTr = getMaxSuffixZ(r, t)
        Then Circular Rotation
        Else No Circular Rotation
Else No Circular Rotation

getMaxZ

Let getMaxZ have the same attributes of getMaxSuffixZ, with one exception: When finding matching substrings that begin at position k and reach to the end of the string s, the algorithm continues comparisons at the beginning of string s up to a maximum of position k-1. It no longer makes sense to stop at the first substring that reaches to the end of the string, as later substrings could have a longer reach into the beginning of the string. It does make sense to return a result as soon as a z-value is found that equals the length of the prefix.

def getMaxZ(p, s):
	result = {}
	l = 0
	r =  -1
	maxZk = 0
	for k in range(0, len(s)):
		if k > r:
			zk = 0
			for si in range(0, len(s)):
				if k + si < len(s) and \
					si < len(p) and \
					p[si] == s[k + si]:
					zk += 1
				else:
					break
			if zk > 0:
				r = zk + k - 1
				l = k
				if r == len(s) - 1:
					r2 = 0
					for si in range(0, k):
						if zk < len(p) and \
						p[zk] == s[si]:
							zk += 1
							r2 += 1
						else:
							break
		else:
			kOld = k - l - 1
			zOld = result[kOld]
			b = r - k + 1
			if zOld < b:
				zk = zOld
			else:
				zk = b
				for si in range(b, len(s)):
					if k + si < len(s) \
						and si < len(p) \
						and p[si] \
						== s[k + si]:
						pass
					else:
						break
				zk = si
				r = zk + k - 1
				l = k
				if r == len(s) - 1:
					r2 = 0
					for si in range(0, k):
						if zk < len(p) and \
						p[zk] == s[si]:
							zk += 1
							r2 += 1
						else:
							break
		result[k] = zk
		if zk >= len(s) - k:
			if zk > maxZk:
				maxZk = zk
		if zk == len(p):
			return zk
	return maxZk

Tests show the correct results:

s = "abcdef"
assert getMaxZ(s, s) == 6
assert getMaxZ(s, "fabcde") == 6
assert getMaxZ(s, "fgabcde") == 6
assert getMaxZ(s, "a") == 1
assert getMaxZ(s, "ba") == 2
assert getMaxZ(s, "") == 0
assert getMaxZ(s, "xyz") == 0
assert getMaxZ(s, "defabcabcdabc") == 6
assert getMaxZ("", "") == 0
assert getMaxZ(s, "defabc") == 6
print 'all tests passed.'
 
all tests passed.

Following the above manner, the resulting flow looks like this:

If |t| = |r| and maxZ(t, r) = |t|
Then Circular Rotation
Else No Circular Rotation

Matching substrings in string rotations

Given the above discussion, a mechanism can be shown that allows the matching of substrings in circular string rotations. To illustrate the problem, in a string t = “abcd” it would then be possible to find substrings such as “abc”, “cda”, “da”, “dab”, etc.

The above algorithm can be applied directly:

Let t = text and p = substring to search
If getMaxZ(p, t) = |p|
Then p found in t
Else p not found in t

Likewise, getMaxZ can be applied to this problem directly:

def showFound(p, s):
	if getMaxZ(p, s) == len(p):
		print "%s found in %s" % (p, s)
	else:
		print "%s not found in %s" % (p, s)
 
showFound("ab", s)
showFound("fa", s)
showFound("abcdef", s)
showFound("x", s)
showFound("abc", "")
showFound("efabcd", s)
showFound("efgabc", s)
showFound("abd", s)

The results look correct.

ab found in abcdef
fa found in abcdef
abcdef found in abcdef
x not found in abcdef
abc not found in
efabcd found in abcdef
efgabc not found in abcdef
abd not found in abcdef

It would make sense to add modifications to allow detecting all instances (up to a specifiable number) of substring matches along with their positions.

Next?

Run-time analysis. And lots of code cleanup!

A string prefix P is a substring of S that begins at S[0]. The Z algorithm calculates for each position k > 0 in S the maximum length of P that is matched by a substring starting at k. The algorithm performs in linear time by keeping track of previously calculated values and recognizing, if the character at a currently examined start position is within a previously detected substring.

An example

Here is a string S and its Z-values for each index position k:

S = ‘aabcaabxaaaz’

Step by step

Let’s look at a these one at a time to show how easily the Z-values can be computed.

k = 1

Then s[0] = s[1], but s[1] != [2], so Z[1] = 1. The matched substring has the boundaries l = 1 and r = 1, so s[l..r] = s[1..1] = ‘a’

k = 2

Then k > r, which means s[2] is outside the previously discovered substring. s[0] != s[2], so
Z[2] = 0.

k = 3

In the same manner, if k = 3, s[0] != s[3], so Z[3] = 0.

k = 4

Then s[0] = s[4], s[1] = s[5], s[2] = s[6], but s[3] != s[7], so Z[4] = 3. The matched substring has the boundaries l = 4 and r = 6, so s[l..r] = s[4..6] = ‘aab’

k = 5

Then k <= r: s[5] is within a previously discovered substring. The previously discovered substring starts at k - l = 5 - 4 = 1. The Z-value found at that position is Z[1] = 1. Since that value is smaller than the length of the remaining substring S[k..r], there is no need to perform other character comparisons. Z[5] = Z[1] = 1. l and r remain unchanged.

k = 6

In the same manner, if k = 6, then Z[6] = z[2] = 0.

k = 7

Then k > r, s[0] != s[7], so Z[7] = 0.

k = 8

Then k > r, s[0] = s[8], s[1] = s[9] and s[2] != s[10], so Z[8] = 2. The matched substring has the boundaries l = 8 and r = 9, so s[l..r] = s[8..9] = ‘aa’.

k = 9

Then k <= r, so s[k] is within a previously discovered substring. The previously discovered substring starts at k - l = 9 - 8 = 1. The Z-value found at that position is Z[1] = 1. That value is not smaller than the length of the remaining substring S[k..r] = s[9..9] = 'a', so additional character comparisons need to be performed. Let b = r - k + 1 = 9 - 9 + 1. Z[9] will be >= b. s[b] = s[1] = s[k + b] = s[10], but s[2] != s[11], so Z[9] = 2. The new substring has the boundaries l = 9 and r = 10.

k = 10

Then k = r, so s[k] is within the previously discovered substring of s[l..r]. The z-value for that string is Z[k - l] = Z[10-9] = Z[1] = 1. Since that value is equal to the length of the remainder of the substring s[k..r], additional comparisons are performed, but since s[2] != s[11], Z[10] = Z[1] = 1.

k = 11

Then k < r and since s[0] != s[11], Z[11] = 0.

Implementation

Here is the implementation of the algorithm (as outlined in the book) in Python:

def getZ(s):
        result = {}
 
        l = r = 0
        for k in range(1, len(s)):
                if k > r:
                        zk = 0
                        for si in range(0, len(s)):
                                if k + si < len(s) and \
                                        s[si] == s[k + si]:
                                        pass
                                else:
                                        break
                        if si > 0:
                                zk = si
                                r = zk + k - 1
                                l = k
                else:
                        kOld = k - l
                        zOld = result[kOld]
                        b = r - k + 1
                        if zOld < b:
                                zk = zOld
                        else:
                                zk = b
 
                                for si in range(b, len(s)):
                                        if k + si < len(s) \
                                                and s[si] \
                                                == s[k + si]:
                                                pass
                                        else:
                                                break
                                zk = si
                                r = zk + k - 1
                                l = k
                result[k] = zk
        return result

That code deserves some cleanup, but it does yield the correct result:

s = 'aabcaabxaaaz'
z = getZ(s)
for k in range(1, len(s)):
	print k, z[k]
1 1
2 0
3 0
4 3
5 1
6 0
7 0
8 2
9 2
10 1
11 0

Next?

The Z algorithm can be used as a precursor for additional string analysis. With little modification it can also be changed to work as a simple, linear exact matching algorithm.

s = "this is a test"

I want to get a list of all the 2-word pairs from the string.

Here is one attempt:

import re
re.findall('[a-z]+\s[a-z]+', s)

The result is not satisfactory:

['this is', 'a test']

findall() returns non-overlapping matches. In this context this means that the pair “is a” will not be returned, since “is” was already matched in the “this is” string.

This returns a complete list of pairings:

words = re.findall("[a-z]+", s)
maxPos = len(words) - 1
currentPos = 0
while currentPos < maxPos:
    print words[currentPos: currentPos + 2]
    currentPos += 1

Here is the result:

['this', 'is']
['is', 'a']
['a', 'test']

Generalizing this to make it usable for n-grams of sizes other than two:

def getNGrams(words, n):
    maxPos = len(words) - n + 1
    currentPos = 0
    while currentPos < maxPos:
        print words[currentPos: currentPos + 3]
        currentPos += 1

Here is how that works for n = 3:

>>> getNGrams(words, 3)
['this', 'is', 'a']
['is', 'a', 'test']

It is probably more interesting to have that functions return a list of n-grams along with frequency information.

Also, this seems reasonably quick, using larger strings (> 200,000 words). Still I wonder, if this can be accomplished using only regular expressions.

import pickle
allPickles = ''
allPickles += pickle.dumps('test0', protocol = 0)
allPickles += pickle.dumps('test1', protocol = 1)
allPickles += pickle.dumps('test2', protocol = 2)

This would result in the following data:

"S'test0'\np0\n.U\x05test1q\x00.\x80\x02U\x05test2q\x00."

Here is an easy way to break the string apart into individual pickles:

pickles = []
import StringIO
sio = StringIO.StringIO(allPickles)
while sio.len > sio.pos:
    currentPos = sio.pos
    pickle.load(sio)
    pickles.append(sio.buf[currentPos:sio.pos])

Here is the resulting list:

["S'test0'\np0\n.", 
'U\x05test1q\x00.', 
'\x80\x02U\x05test2q\x00.']

Just to verify:

for item in pickles:
    print pickle.loads(item)

The result is correct:

test0
test1
test2

If this displays with correct syntax highlighting on the blog, then this was also a successful test of the wp-syntax plugin.

def dow(m, d, y):
    if ((m == 1) or (m == 2)):
        m += 12
        y -= 1
    return (d + 2 * m + 3 * (m + 1) / 5 +
    y + y / 4 - y / 100 + y / 400) % 7

The returned integer is a value between 0 (Monday) and 6 (Sunday).

Combine it with a list of day strings:

days = ('Monday', 'Tuesday', 'Wednesday',
'Thursday', 'Friday', 'Saturday', 'Sunday')

Like in the following example to display the name of day, given a date:

days[dow(6, 28, 2006)]

I found the algorithm in Dr. Dobbs Journal issue 229 (April 1995) and the article features a very nice description of how the formula was derived. The original C implementation of the algorithm is about as readable as the above pieces of code.