Ultra-large-scale Sites

This became a research area almost by accident. I had to write an article on distributed systems in the context of computer science and media which - due to work on much research and work on secure systems - turned out to be too late for publishing. During a sabbatical term I started to extend the article. At that time mayspace, facebook, flickr and others were growing like crazy and it was clear that distributed systems played a central role in this: the content side of media production turned out to become distributed as well through Web2.0. And the single user in front of a PC became a distributed node within her groups, linked through many differnt relations and communication media.

I developed a big interest in researching and describing the interconnections between socially and technically distributed systems, between groups and their social sites which developed their own interconnnections.

The presidential election in the states showed at the same time the growing importance of social sites and their infrastructure which had to scale immensely during thw whole phase. And it was then that I discovered Todd Hoff's excellent portal on scalability issues: HighScalability.com. His continuously growing collection of articles on the technical infrastructure of large-scale sites like facebook or flickr provided me with the necessary input for my work. And I noticed that sometimes it was rather hard to understand what the chief architects of famous sites really meant when talking about problems and solutions.

And that was the birth of my current research: I am trying to find the core architectures, methods, components and algorithms that make systems scalable. Research questions are e.g. whether one can build extensible and scalable kernel and grow them incrementally into large scale sites. Or whether "revolutionary" phases are needed which will require complete re-writes.

But there is an educational and very practical goal besides my research interests: To enable beginners and professionel developers to build ever larger but still maintainable systems, using the experience from ultra-large-scale sites. In my course on ULS Systems at digicom I wrote that even if you do not need to build a system on such an extreme scale, you will benefit greatly from knowing those architectures. Only they make scalability problems really visible.

Currently me and my students of the system engineering and middleware courses are experimenting with large-scale infrastructures, measuring and optimizing etc. And at the same time we continue to look at core algorithms and problems like time in virtual systems, failure detection and basic architectural components. In other words the second half of the book is currently developing and sections on MMOGs, Clouds etc. are being added

The resulting book is under Creative Commons license and will be Open Accessed. After taking so much from Todd Hoff's site and the input and papers from my students I considered it only decent to give the work back to the public.

You will find here a growing set of materials, research papers etc. on the topic of ultra-large-scale sites and their use in social media.