email: ceberhardt@scottlogic.co.uk
on Google+
UPDATE: I have posted the sourcecode for this control on codeproject.
Recently I have been wondering about the wealth of information that can be gleaned from the 2.5 million programming question on Stack Overflow. A few weeks back I found a tag trending tool, which can be used to measure the rise and fall in popularity of tags over time. Whilst this is a great little tool, I am sure there is much more that can be done with the freely available Stack Overflow data, for example, exploring the relationships between the many technologies people ask questions about.
On a recent trip to Copenhagen I decided to put my hours of travelling time to good use and create a Silverlight application that plots the relationships between the various tags. I created an application that downloaded the 1,000 most recent questions via the Stack Overflow API and plotted the relationships between the 20 most popular tags, as seen above.
The graph is constructed as follows:
- The size of each segment is proportional to the number of questions relating to the tag, i.e. android and java are the most popular tags.
- Connections between tags indicate questions that have been tagged with both technologies. The thickness of the connection indicates how many questions share these two tags, i.e. jQuery and JavaScript tags appear together quite often.
- Each segment is coloured based on the number of connections it has, red for many connections, blue for few.
The ordering of segments can be changed using the drop-down control. Probably one of the most interesting views is the one where related tags are clustered. This is done by assigning a ‘weight’ to the current configuration of the graph by summing the length of all connections, with connections that cross the centre of the circle adding most weight. An iterative process is used to minimise the overall graph weight by moving each segment a few steps left and right, until the least ‘weighty’ configuration is found. This is the one where each tag is most closely related to its neighbours.
When clustering is applied we can see small ‘pockets’ of related technologies, with the following patterns emerging
- The two most popular tags, Java and Android, are very closely related to each other, but have very few other relationships.
- iOS, Objective-C and iPhone form a close-knit group. However, Objective-C questions are sometimes also tagged with C#, C and C++.
- C#, .NET and ASP.NET are clustered, however C# has links with many other tags
- The strongest relationship is between jQuery and JavaScript, probably due to jQuery having become the de-facto framework for JavaScript development, being used on 53% of websites.
- There is a large cluster of connected web technologies, CSS, HTML, JavaScript, jQuery, reflecting the mix of technologies involved in creating web sites and web applications.
- Python, whilst being a popular tag, has very few relationships, only being weakly linked to PHP.
I am planning on tidying up the code for this visualisation, making it more generic, allowing it to be used to graph other datasets. Let me know if you are interested in this!
Here is the same graph, but showing the top 30 tags, again, more interesting relationships start to emerge:
Finally, thanks to Chris P., Adrian C. and Graham O. for their ideas and input!
Regards, Colin E.
Tags: charts, silverlight, visualisation
Great post! High performance diagramming is something most component vendors lacking… Visiblox should add these kind of visualizations to their toolbox as well
Funny, just found this visualization in an article on Swiss design…
Article:
http://www.smashingmagazine.com/2009/07/17/lessons-from-swiss-style-graphic-design/
Vizualization:
http://www.flickr.com/photos/kdj/2844449995/
Thanks Filip – that’s a great article. I love Swiss-style.
Cool!!
a suggestion: you could label the small block like: ‘Eclipse’ along the radius. also camel Case would make it even better. e.g. ‘JavaScript’.
[...] Data visualization of StackOverflow (link) [...]
On the ‘top 30′ charge, choose clustering, then look at the relative positions of CSS and SPRING (lower right). Their location seems reversed; do you know why the clustering algorithm made a mistake there?
Another example of swapped ordering is ASP.NET-MVC-3 and LINUX (lower left), also which appear that they would be better clustered if their positions was reversed.
‘charge’ = chart, ‘positions was’ = ‘positions were’. Blarg. Can’t communicate tonight.
And the odd ordering is gone! Was it just me, or did something get fixed?
No, it’s still there, I’m just an idiot. And I’m talking to myself, on a moderated blog where the blogger is sleeping. Which makes me weird, as well as mistaken.
Hi Myrddin,
Thanks for your (many) comments, they made me chuckle! I am cleaning up the code and publishing it. I will check for the errors you reported.
Colin E.
I note that “Silverlight” doesn’t appear as a tag…
No – it doesn’t. Nor does Flash, Java Applets or Java-FX … or any other plugin technology.
I have no Microsoft platform to view this on. :-/
Silverlight works on the Mac too…
Where can I download the code for this one, looks interesting
Hi Sacha, glad you like it. I wrote the code whilst travelling to / from Copenhagen, so it isn’t really in a state I would like to share! (seriously, if it got out, my reputation would be in tatters
). I plan to clean it up next week and write-up on codeproject.
Colin E.
let me know when you do I really really like this one. I like things that move weirdly (don’t like spiders though)
ha ha … I’m with you on the spiders thing.
[...] Visualising StackOverflow Tag Relationships with Silverlight [...]
Awesome – I really like the hovers that highlight a tag’s relationships.
You should check out the d3 project for more visualization examples:
http://mbostock.github.com/d3/ex/
I could see a circle packing graph for related tags, too.
Hi Jarrod, thanks for the link to D3, I’d not seen that one before – some great visualisations in there.
Very cool. Was this done using Visiblox?
Nice visualization!
Excellent!
Thanks Michael – glad you like it
[...] Visualising StackOverflow Tag Relationships with Silverlight (Colin Eberhardt) [...]