The software is available for download and experimentation. I would love to try it out but the requirements make it sound like that will have to wait until I can come up with a Windows machine.

This looks very interesting though.

Generating a list of users

There are lots of ways of putting together a list of Twitter users. If you’re on Twitter, you could use the list of your followers or the list of those you are following. You could extract user names from a list of search results, the public timeline or a twitter directory. There are lots of options. The following code uses a regular expression to extract the user names from a wefollow page.

import re
import urllib
 
def getWefollowTwitterUsers(category = "tech"):
        users = []
        url = "http://wefollow.com/twitter/"
        url += category
        html = urllib.urlopen(url).read()
        users = re.findall("""nofollow">(.*?)</a></strong>""", html)
        return users
 
if __name__ == "__main__":
        print getWefollowTwitterUsers()
 
Output:
['kevinrose', 'google', 'LeoLaporte', 'mashable', 'TechCrunch', 'Veronica', 
'alexalbrecht', 'ev', 'patricknorton', 'Scobleizer', 'woot', 'ijustine', 'timoreilly', 
'guykawasaki', 'engadget', 'CaliLewis', 'chrispirillo', 'wired', 'ryan', 'sarahlane', 
'ambermac', 'ginatrapani', 'tferriss', 'fforward', 'mollywood']

Retrieving a list of messages for each user

Each user’s messages are available in an RSS feed of the format http://twitter.com/statuses/user_timeline/.rss?count=[1..200]. The count parameter is optional and controls the maximum number of messages contained in the feed. The following code uses Universal Feed Parser to extract the entries from the data feed.

import feedparser
 
# [...]
 
def getUserMessages(user):
        url = "http://twitter.com/statuses/user_timeline/" + user + ".rss?count=200"
        feed_data = feedparser.parse(url)
        return feed_data.get("entries", [])

Generating keyword scores

The following code goes through a user’s messages, breaks them into fragments and counts the number of instances for each encountered word.

def getKeywordScores(user, messages):
        keywords = {}
        blacklist = ["a", "an", "by", "on", "that", "the", "these", "this", "those", "to"]
        # and many more words
        blacklist.append(user)
        for message in messages:
                tweet = message["summary"]
                words = re.split(" ", tweet)
                for word in words:
                        word = re.sub("^\W*", "", word)
                        word = re.sub("\W*$", "", word)
                        if word.startswith("http://"):
                                continue
                        word = word.lower()
                        if word in blacklist:
                                continue
                        if not word:
                                continue
                        count = keywords.get(word, 0)
                        keywords[word] = count + 1
        final_keywords = {}
        for k in keywords:
                if keywords[k] > 1:
                        final_keywords[k] = keywords[k]
        return final_keywords

Computing similarities

The code to compute similarity scores and the ideas behind that are presented in Programming Collective Intelligence. The source code for the book is available online. The relevant pieces are in chapter2/recommendations.py – sim_distance() (Euclidian Distance), sim_pearson() (Pearson Coefficient) and topMatches(). The latter compares one user to all others and returns the list of n most similar users along with their respective similarity scores.

Similar users

The following code brings it all together and demonstrates how we can show users that are similar to a specific one, given the computed dictionary of keyword scores.

from recommendations import sim_pearson, sim_distance, topMatches
# [...]
if __name__ == "__main__":
        users = getWefollowTwitterUsers()
        # add my own
        users.append("abendig")
        print users
        user_keywords = {}
        for user in users:
                print "processing data for:", user
                messages = getUserMessages(user = user)
                user_keywords[user] = getKeywordScores(user = user, messages = messages)
 
        # Similarity between the first user and three others
        print sim_pearson(user_keywords, users[0], users[1])
        print sim_pearson(user_keywords, users[0], users[2])
        print sim_pearson(user_keywords, users[0], users[3])
 
        # My top three matches
        print topMatches(user_keywords, "abendig", n = 3, similarity = sim_pearson)

Here is the output that this produces (at the time of this writing):

['kevinrose', 'google', 'LeoLaporte', 'mashable', 'TechCrunch', 'Veronica', 'alexalbrecht', 
'ev', 'patricknorton', 'Scobleizer', 'woot', 'ijustine', 'timoreilly', 'guykawasaki', 
'engadget', 'CaliLewis', 'chrispirillo', 'sarahlane', 'ryan', 'wired', 'ambermac', 
'ginatrapani', 'tferriss', 'fforward', 'mollywood', 'abendig']
processing data for: kevinrose
processing data for: google
processing data for: LeoLaporte
processing data for: mashable
processing data for: TechCrunch
processing data for: Veronica
processing data for: alexalbrecht
processing data for: ev
processing data for: patricknorton
processing data for: Scobleizer
processing data for: woot
processing data for: ijustine
processing data for: timoreilly
processing data for: guykawasaki
processing data for: engadget
processing data for: CaliLewis
processing data for: chrispirillo
processing data for: sarahlane
processing data for: ryan
processing data for: wired
processing data for: ambermac
processing data for: ginatrapani
processing data for: tferriss
processing data for: fforward
processing data for: mollywood
processing data for: abendig
0.693852667302
0.57137732992
0.350957713398
[(0.85762813072101673, 'ginatrapani'), 
(0.81973579573386002, 'CaliLewis'), 
(0.81455896587667598, 'timoreilly')]

The results suggest the users ginatrapani, CaliLewis and timoreilly as related to abendig based on the available data and thus maybe worth following.

Next

This showed an example of directly applying code and ideas from the book Programming Collective Intelligence to Twitter users and their message streams. This is of course also pretty simplified. User similarity is an interesting problem though.

There are lots of ways to make this more useful. The realtime nature of the message streams should be taken into account. Users’ posting frequency may matter. Also, people’s interests certainly change. Overall similarity is useful, but similarity based on time ranges could also be interesting.

URLs that are included in the messages are currently mostly ignored. It would of course make a lot of sense to include them (don’t forget to deduplicate the various URL shortener versions of the same URL) to be able to take into account that several people may be talking about the same articles.

Simple keyword counts are pretty crude. Semantic analysis of the messages would be useful to get an indicator of whether two people are talking about similar things even though they are using different words, if their opinions are similar, and so forth.

Oh, and scale it up to include millions of users.

Semantic email addressing [PDF] aims to solve this problem:

Email addresses are a means to an end. The goal is usually not to send an email to a particular address, but to a particular person. You want to say hello to your friend Steve or send a message to the VP of marketing at Microsoft or to the head caterer for your wedding. Ideally, you could send a message to a person just by entering his or her name, position, or some other descriptive attribute. If a person’s email address changes, the email system should send to the new address automatically. If the person matching a description differs over time, the email system should send to the person currently matching that description.

In the given example, the user would be able to get answers to his or her questions by reaching out to the people with the fitting skill sets without previously having known those people: The email system can decide, who the most appropriate receivers of the messages are.

I cannot help thinking that Aardvark was at least a little inspired by the ideas behind semantic email addressing. Their process is simple: Users send in questions (using email, twitter, IM, etc.), Aarvark routes the question to another user is (hopefully) qualified to answer it and the user will eventually receive a response, often just a few minutes later. In this social search approach, Aardvark accomplishes the job of finding information by finding the right people who can provide it. The service has received very good press and was recently acquired by google.

Twitter seems like it might be a good platform for this problem area. If someone has a public twitter feed, they are essentially broadcasting their updates to the open stream and anyone can see them. It is probably safe to assume, they are at least open to the idea of talking to strangers/responding to messages from people they do not already know.

How could one go about finding the best people to message though? One method is certainly to search the message stream for specific keywords and basically manually look for people who might be active in areas of interest. You can also search in and add yourself to one of the many directories that are being developed.

But, if I simply need to talk to someone and ask them “May I ask you a question about XYZ?” then clearly, a) broadcasting my question hoping that someone will answer could be very inefficient and b) first researching who the best person might be for my question(s) puts all the burden on me.

What if the user could simply send out the question and the system would ensure that the most appropriate people see it?

The basic idea here is this: The user submits the question (along with a set of keywords) to his or her software. The software has analyzed other users’ message streams, extracted keywords, etc. and generated a knowledge base. If the query can be confidently matched to another user, a message is generated and send to that user. The message will be visible to that user as a regular name mention and they can choose whether to engage in that conversation.

Some of the obvious challenges:

• Generating of meaningful keywords/subject areas based on a person’s message stream.
• Successful matching of queries with users.
• Establishing an effective communication protocol that does not easily lend itself to abuse, i.e. spam.

A lot of web-based social networks are great at helping you connect with people you already know. Twitter makes it easy to connect with new people. The outlined approach (or a variation thereof) might be a good way of further supporting creation of those new connections, based on areas of interest.

In sci-fi films, when anyone gives a computer emotions, it all goes horribly wrong. The computer becomes vain, doubtful and irrational and Armageddon by wayward technology is only narrowly avoided.

This is not surprising – science fiction has been informing us and becoming part of our culture for a while. It is increasingly really all around us: We Are Living in a Sci-Fi World.

Affective computing is an intriguing concept though:

Affective computing is a branch of the study and development of artificial intelligence that deals with the design of systems and devices that can recognize, interpret, and process human emotions. It is an interdisciplinary field spanning computer sciences, psychology, and cognitive science.

Imagine educational software that modifies its teaching style depending on the user’s mood. Cars that communicate with other drivers, if its driver is angry, intoxicated or talking on the phone. Music players could adjust their playlist based on the listener frowning, smiling or similarly expressing themselves. Email clients could disable the send button, if the user is clearly upset and about to send out an email he or she may regret later.

A lot of different uses are conceivable here and this could contribute to much more personalized computing experiences.

Modern laptops and desktop computers are typically already equipped with microphones and cameras. Future operating systems may well feature a mood evaluation component and search engines may take information from that component as part of the search query. Similar scenarios are conceivable for other types of web-enabled applications.

Imagine logging in to Facebook some evening and finding a notification “John has been having a bad day. Check in with him to make sure he’s okay.” Intriguing.

And at least a little bit eerie.

I found particularly interesting the discussion around talking to develop understanding:

Around 1980, as computer science undergraduate students at the University of Cambridge, my friends and I noticed a strange phenomenon that we called expert programmer theory. When one of us had trouble getting our programs to work, we’d describe the nonfunctioning state of our code to each other over coffee. Quite often, we’d realize in a flash what was wrong and how to solve it. These epiphanies were quite independent of the other person having any real understanding of our problems—the listener often seemed little wiser about the subject.

I have experienced similar scenarios and this can be both relieving (finally solved the problem!) and frustrating (why didn’t I think of this a few minutes ago?).

Explaining something to another person or even an object can help the person’s own understanding. Wray points out that it is helpful, if we can talk to an expert, even if that expertise is large based on perception. The main reason seems to be that that person would be more likely to ask us deep questions that we can ponder or that may influence our thinking.

The ability to ask questions that are most appropriate for the given situation seems most valuable: Questions that don’t require too large a leap, but rather motivate the person to advance just a little further – questions that stimulate thinking.

What if software that we use daily asked us questions?

Lots of scenarios are conceivable, but here is one example. Imagine a news website that attaches to each article a module that contains at least one interesting question, such as “Do you think this policy change will effectively solve problem XYZ?”, “What do you think of senator X’s position on Y?”, “What if the economic situation in Y would change in Z way?” and so forth. These would be meaningful questions, based on the content of the article and meant to stimulate intelligent discourse (readers could leave responses and discuss amongst themselves). These questions would also ideally be automatically generated.

If we can accept that good questions at the right time can help our understanding and that deeper understanding is generally a good thing, then I think we will benefit from giving software more of an ability to ask questions – for our own benefit.

I do a significant portion of my shopping on the Internet and I have come to appreciate customer reviews, recommender systems and many other features that have become common at a lot of online stores. I often take information provided by those systems into account when making buying decisions. I have gotten used to those features and – as I realized on that day at the mall – I miss them in their absence.

Usually, I am content to satisfice, but when I am in the store without access to those familiar features, I feel a bit deprived, as if one of my senses were shut off. I like that imagery, too: The idea of an additional sense, based on Internet data is an intriguing one.

The following video shows how MIT’s Sixth Sense may have the potential to act as an additional sense to equip you with the features that you may have gotten used to on the Internet.

I wonder when we will routinely wear devices that integrate cameras, microphones, displays/projectors, etc. and that continuously scan our surroundings and have the ability to feed us data about it back in real time. It could be a version of Sixth Sense using discreet packaging.

Quick access to product reviews, as we look at a book or CD in a store sounds like a useful feature. Maybe sunglasses (and their integrated display) could provide directions as we are walking. It could also display quick stats regarding our surroundings, incl. a warning of nearby danger. The potential for applications seems endless.

In the meantime, I am of course still stuck with unfinished shopping.

A lot of interesting subjects are covered in the presentation. Here are references to projects or papers that are mentioned:

• Seam Carving for Content-Aware Image Resizing [PDF] by Shai Avidan and Ariel Shamir presents a smarter method of image resizing. Speed of processing of modern computers greatly helped with the development of this algorithm.
• Scene Completion Using Millions of Photographs by James Hays and Alexei Efros is only possible and successful because of its large data sets.
• The More Data vs Better Algorithms slide is from Michele Banko’s and Eric Brill’s 2001 paper Mitigating the paucity-of-data problem: exploring the effect of training corpus size on classifier performance for natural language processing.
• Canonical image selection from the web by Shumeet Baluja, Yushi Jing and Henry Rowley compares low level features of image result matches for given queries to rank the images.
• Learning people annotation from the web via consistency learning by Jay Yagnik and Atiq Islam uses Eigenface representations and large collections of images to annotate them.
• Audiovisual Celebrity Recognition in Unconstrained Web Videos [PDF] by Mehmet Sargin, Hrishikesh Aradhye, Pedro Moreno and Ming Zhao uses both face and speech recognition to detect celebrities in videos.
• How to Write a Spelling Corrector by Peter Norvig provides spell checking in just over 20 lines of Python.
• Google made an n-gram corpus publicly available at Linguistic Data Consortium.
• Google’s Flu Trends is based on search data.
• Google Sets allows generating of sets of expressions similar to an initial set of expressions.

Also discussed: Text segmentation, statistical machine translation, MapReduce, Web bias and more.

Intuitively, I think, we tend to read that question as Are great programmers born xor made? – understanding it such that it is either one or the other. I believe that is false: It is not one or the other; it is both, at least to some degree. However, the ratio is important.

Innate ability, such as a baseline degree of brain capacity is absolutely required, maybe measured as at least average IQ. That baseline or innate ability is the smaller part of the whole.

I would argue that it helps to be strong at abstract and critical thinking, logic, mathematics, pattern matching/prediction, memory and recall, and so forth. However, those are largely skills. They serve as very useful prerequisites or corequisites, but they are learnable. The same is true for other skills in software development, such as deep understanding of programming language usage, the ability to follow code style guidelines, writing good unit tests, coming up with “clean” designs, etc.

From what I understand, deliberate practice is key to acquiring expertise in an area. I provided some notes on that in Accelerated Learning with AI systems?, though with a slightly different angle. Mary Poppendieck very much relates this principle to software development in her presentation Deliberate Practice in Software Development that she gave at Agile 2009.

When I first taught myself programming (Turbo Pascal, if you are curious), it felt like it came easy to me. It was also great fun, which served (at least partly) as motivation for me to learn and experiment more, eventually turn it into a profession.

If you either “have it” or “don’t have it,” then there does not really seem to be a chance for greatness for the person who is missing that innate ability. On the other hand, if training/deliberate practice can play such a significant role, then there are options: The opportunity of a new challenge. I think, this should be very encouraging.

Practice.

Except instead of passion, the authors deliberately use the German word leidenschaft. Leidenschaft combines the words leiden (to suffer, experience pain) and schaffen (to make, create, achieve).

People passionate about their pursuit are willing to suffer in the process.

I believe that the connotations of the word leidenschaft have changed a bit and modern day usage is much closer to the positive aspects of a passion: To pursue an activity or subject with great interest, dedication or enthusiasm. Still, I think it is instructive to keep that implied duality in mind, not just as an introspective exercise to more fully understand oneself, but also to find concrete opportunities for growth.

It is worth examining software/technology and evaluating how different products strengthen the positive aspects of a passion and how they help deal with potential negatives. Then, do something about it.

The frequent items problem is to process a stream of items and find all those which occur more than a given fraction of the time. It is one of the most heavily studied problems in mining data streams, dating back to the 1980s. Many other applications rely directly or indirectly on finding the frequent items, and implementations are in use in large scale industrial systems. In this paper, we describe the most important algorithms for this problem in a common framework. We place the different solutions in their historical context, and describe the connections between them, with the aim of clarifying some of the confusion that has surrounded their properties.

Some of the interesting bits here are that the data stream will easily contain millions (or billions) of items and the algorithm will typically only get to take one look at each item as it comes up in the stream.

Space-Saving

In this post I focus on the Space-Saving algorithm and provide an implementation in Python. The algorithm itself is originally described in Efficient Computation of Frequent and Top-k Elements in Data Streams [PDF] by Ahmed Metwally, Divyakant Agrawal, and Amr El Abbadi:

We propose an integrated approach for solving both problems of finding the most popular k elements, and finding frequent elements in a data stream. Our technique is efficient and exact if the alphabet under consideration is small. In the more practical large alphabet case, our solution is space efficient and reports both top-k and frequent elements with tight guarantees on errors. For general data distributions, our top-k algorithm can return a set of k’ elements, where k’ ≈ k, which are guaranteed to be the top-k’ elements; and we use minimal space for calculating frequent elements. For realistic Zipfian data, our space requirement for the frequent elements problem decreases dramatically with the parameter of the distribution; and for top-k queries, we ensure that only the top-k elements, in the correct order, are reported. Our experiments show significant space reductions with no loss in accuracy.

The algorithm basically works like this: The stream is processed one item at a time. A collection of k distinct items and their associated counters is maintained. If a new item is encountered and fewer than k items are in the collection, then the item is added and its counter is set to 1. If the item is already in the collection, its counter is increased by 1. If the item is not in the collection and the collection already has a size of k, then the item with lowest counter is removed and the new item is added, with its counter set to one larger than the previous minimum counter.

Here is some pseudo code to make this clearer:

SpaceSaving(k, stream):
collection = empty collection
for each element in stream:
    if element in collection:
    then collection[element] += 1
    else if length of collection < k:
        then add element to collection, collection[element] = 1
    else:
        current_minimum_element = element with lowest count value in collection
        current_minimum = collection[current_minimum_element]
        remove current_minimum_element from collection
        collection[element] = current_minimum + 1

The straightforward approach

A first, easy implementation would use a simple hashtable, such as in the following piece of code:

def space_saving_frequent_k1(k, stream, debug=False):
        def get_smallest_key(d):
                """
                Given dictionary d, returns the key associated with
                the lowest value in the dictionary.
                """
                min_key = None
                for key in d:
                        if not min_key or d[key] < d[min_key]:
                                min_key = key
                return min_key
 
        counters = {}
        for element in stream:
                if counters.has_key(element):
                        counters[element] = counters[element] + 1
                elif len(counters) < k:
                        counters[element] = 1
                else:
                        current_minimum_key = get_smallest_key(counters)
                        if current_minimum_key:
                                counters[element] = counters[current_minimum_key] + 1
                                del counters[current_minimum_key]
                        else:
                                counters[element] = 1
        return counters

This works for smaller data sets and particularly, when there is never a need to find that smallest element. Otherwise however, (repeatedly) retrieving the element with the minimum count remains a comparatively costly challenge.

Stream-Summary

When describing the Space-Saving algorithm, the authors also introduced the Stream-Summary data structure (inspired by work in Frequency Estimation of Internet Packet Streams with Limited Space [PDF]), which groups elements with equal values together (in buckets) and allows quick retrieval of the element with the lowest count.

Here is a diagram of this structure, using three buckets and a total of six elements (E1-E6).

Buckets are stored in a list sorted by the buckets’ respective values. Each bucket maintains knowledge of associated elements. Each element in turn maintains a pointer to its bucket. The latter is implemented using a simple hashtable. If an element’s count needs to be increased, the element is removed from its current bucket and added to the neighboring bucket with value one greater than the previous one. If no such bucket exists, it is inserted in the bucket list. Empty buckets are removed.

The Python implementation using the Stream-Summary data structure may then look like this:

class Bucket(object):
        def __init__(self, value = 1, elements = []):
                self.value = value
                self.elements = []
 
        def __str__(self):
                return "%s: %s" % (self.value, str(self.elements))
 
        def append(self, element):
                self.elements.append(element)
 
        def first_element(self):
                if self.elements:
                        return self.elements[0]
                else:
                        return None
 
        def has_elements(self):
                return len(self.elements) > 0
 
        def remove(self, element):
                self.elements.remove(element)
 
class StreamSummary(object):
        """
        Maintains a dictionary of elements and a list of buckets. Each element
        points to a (parent) bucket.
        The bucket list is sorted based on the buckets' values. Each bucket also
        maintains a list of elments.
        This has the effect of grouping elements with equal values in buckets.
        """
        def __init__(self):
                self.elements = {}
                self.buckets = []
 
        def __len__(self):
                return len(self.elements.keys())
 
        def __str__(self):
                result = ""
                for b in self.buckets:
                        result += str(b) + " "
                return result
 
        def add_element(self, element):
                """
                Adds an element and ensures it's assigned to the correct bucket.
                """
                if not self.elements.has_key(element):
                        if not self.buckets or self.buckets[0].value != 1:
                                self.buckets.insert(0, Bucket())
                        self.elements[element] = self.buckets[0]
                        self.buckets[0].elements.append(element)
 
        def increase_element(self, element):
                """
                Increasing an element's value also means assigning it to the
                correct bucket. That can result in creating a new bucket and/or
                removing an empty one.
                """
                current_bucket = self.elements[element]
                bucket_index = self.buckets.index(current_bucket)
 
                if len(self.buckets) == bucket_index + 1:
                        self.buckets.append(Bucket(value = current_bucket.value + 1))
                elif self.buckets[bucket_index + 1].value > current_bucket.value + 1:
                        self.buckets.insert(bucket_index + 1,
                                            Bucket(value = current_bucket.value + 1))
                current_bucket.remove(element)
                self.elements[element] = self.buckets[bucket_index + 1]
                if not current_bucket.has_elements():
                        del self.buckets[bucket_index]
                self.elements[element].append(element)
 
        def has_element(self, element):
                return self.elements.has_key(element)
 
        def get_minimum(self):
                if self.buckets:
                        return self.buckets[0].first_element()
                else:
                        return None
 
        def replace_element(self, old_element, new_element):
                """
                Replaces an existing element with an entirely new element in
                the old element's bucket.
                """
                self.elements[new_element] = self.elements[old_element]
                self.elements[new_element].remove(old_element)
                self.elements[new_element].append(new_element)
                del self.elements[old_element]
 
def space_saving_frequent_k(k, stream):
        summary = StreamSummary()
        for element in stream:
                if summary.has_element(element):
                        summary.increase_element(element)
                elif len(summary) < k:
                        summary.add_element(element)
                else:
                        current_minimum_key = summary.get_minimum()
                        if current_minimum_key:
                                summary.replace_element(current_minimum_key, element)
                                summary.increase_element(element)
                        else:
                                summary.add_element(element)
        return summary

For larger data sets, where k is noticeably smaller than the number of distinct elements in the set, the Stream-Summary data structure proves advantageous.

Onward

There is a lot of ongoing research in this problem area. This article is clearly just barely offering a small (and simplified) glimpse. Explore the research. Find out what real-world applications use some version of this as part of their problem solving approach. Applications can be found in web access log processing, search applications, mining of real-time message streams, and so forth.