The Next Big Thing

Ok, so it’s been a while since I paid any attention to this blog - not that I have nothing to say, but rather that I couldn’t decide what demands the most attention. I also want to ease up on the hostility expressed in a number of previous posts, as there’s probably enough people complaining about the industry without me making more noise..

So I think what I need to do is focus on a single topic for posting about, and in the spirit of this blog’s title I have decided to document my experiences creating the Next Big Thing in software.

For quite a number of years (perhaps as far back as the beginning of my career about ten years ago) I have been particularly unimpressed with the features provided by the second most ubiquitous software category (the web browser being the first), the email client. Over the years this dissatisfaction has grown to include all so-called “groupware” clients, notably infamous among these being Microsoft Outlook. So in my spare time over the years I have explored and experimented with what it would take to write an Outlook killer (for want of a better term). Being that I have used Java almost exclusively for my entire career, I had confidence in language features and the growing list of third-party libraries to be able to produce a complex rich client.

With an attention span challenged at the best of times, often my focus has spun off on tangents - particularly when Java proved to be too immature for certain critical features of an email client (a situation that may be argued still exists today). Occasionally these tangents have proved to be not a total waste of time, with projects such as iCal4j and mstor resulting from these divergences.

The long-term goal however, remained, and in the last couple of years a number of developments have arisen within the Java community that I believe have been of critical importance to developing complex rich client Java applications:

• First, there have been dramatic improvements with desktop Java performance in Java 5, and even more so in Java 6. Hopefully we can expect even greater performance from the upcoming Java 6 Update N, and Java 7
• Second, I believe modularisation plays a key part in building complex systems, whether they be client- or server-side. Taking modularisation to the next level is the Open Services Gateway Initiative (OSGi), which I think could change the way we develop applications in the future
• Finally, and I really mean finally (!!), we now have a better way to build Swing GUI interfaces in JavaFX Script. Admittedly this technology is still very new, and to be honest the implementation is not even finished yet, however with my limited exposure I am already feeling much more comfortable with writing complex rich client interfaces

And so armed with a new set of technologies, I am once again attempting to make progress on this Next Big Thing. Along the way I will be documenting progress, insights and frustrations, in the hope that something of value may find it’s way into this blog. Stay tuned.

When designing a software architecture it is important to maintain a focus on the problem you are trying to solve. Without this focus you will most likely diverge from the original goals, resulting in a solution that may not represent the best approach to solving the problem at hand. Often is the case that JEE architects lose sight of the problem domain in favour of the technologies employed in the end solution.

The Java (2) Enterprise Edition (JEE) is partly to blame for this loss of focus due to the rigid structure defined in the specification. When a framework such as JEE imposes such tight control over application design, typically you will see every application implemented the same way. The primary goal of JEE was to provide a standard approach to enterprise application development that, whilst particularly useful for those unfamiliar with enterprise architectures, has resulted in a single “flavour” of application design that is not necessarily optimal for all problem domains.

Consider banking systems. Aircraft control systems. All of these problem domains will typically find certain architectures more favourable than others. Embedded systems obviously aren’t interested in JEE-sized applications, whilst enterprise applications need enterprise-quality scalability. From this kind of logic has emerged the different flavours of Java: JSE as the basis for all things Java, the superset JEE, and the subset JME. Unfortunately this model brings an assumption that problem domains only differ in terms of size, power and scalability, which is clearly not the case. If we consider the needs of an insurance company, in terms of size, power and scalability it may not be very different from the needs of an online bookstore, however the architectures most optimal to each domain are bound to be quite different.

If we consider the online bookstore example, you might assume that they may require some kind of supply-chain management software to handle the placing of orders through to delivery of the product. In addition to the typical enterprise-scale requirements, such a system might also be centered around a messaging or ESB technology. Alternatively, if we look at the insurance company’s requirements, we might find a need for a policy management system, responsible for generating premiums and managing renewal periods. While the policy management system may also depend on messaging technologies for communication with legacy systems, the focus of such a system is more likely to be the business logic and underwriting rules necessary for policy management. Now of course a JEE application can be implemented and deployed for each of these particular scenarios, however in doing so we lose focus on what is important to each of these systems. At the same time we are deploying technologies that aren’t even used in the problem domain, simply because they come bundled with the application server, further increasing the “clutter” and distracting us from the end goal.

Without having a clear focus of the goals of a software system, we find ourselves simply “shoe-horning” the implementation into the simplest configuration of a JEE application. Of course some JEE architects do try to model the application on the problem domain, being so bold as to implement multiple EJB modules in a single enterprise application. Multiple web applications are less common, typically because inter-web application communication is so difficult (something that portals tries to rectify, however the fact that it was “taped on” as an afterthought isn’t really optimal), which ultimately means that web applications are huge and unmanageable, even if the architect has the foresight to separate the business logic from the user interface.

In fact many JEE architects have discovered that the most configurable way to implement an enterprise application is as a collection of JEE applications. Each application might contain a single EJB module, or a web application, or web services. This approach will only become more prevalent with the popularity of Service Oriented Architectures (SOA). Assuming this trend does continue enterprise applications will ultimately become redundant, serving only as a defunct wrapper to the actual module contained within. Add this to the list of other failed technologies (Entity and Stateful EJBs), and JEE starts to become made up of more legacy technologies than useful ones.

There is a lesson to be learned from JEE, and that is if you try to be everything to everyone you will end up being not particularly useful to anyone. As this realisation is made more apparent with every new release of the JEE specification, we will ultimately see a rejection of JEE as a whole, instead being replaced with architectures employing only those technologies required to meet the goals of the project. Such architectures will not necessarily resemble a traditional JEE design, but will rather be designed specifically for the problem domain. With the resurgence of such architectures of many different flavours will come a renewed demand for true software architects, and will hopefully result in software designs more suited to the task at hand.

If there is one thing in abundance in software development it is opinions. From which technology to use (languages, frameworks, libraries, etc.), to architectural design (everyone’s an architect!), to the user interface layout, it is never hard to find opinions.

One thing that is rare however, is a firm grasp on reality when it comes to these opinions. It is common for developers to have difficulty conceptualising how long it will take to implement a requirement. As a result they will often “aim too high”, setting goals that are surely not achievable within the given time constraints. The end result will be that even if the requirement is implemented, it will usually be over time, over budget, and of poor quality due to the pressure of looming deadlines and the abandonment of adequate testing (which is, without fail, always the first casualty).

Also common amongst software developers is the resume-friendly implementation. How many times have you seen an implementation use AJAX where it is completely unnecessary? I know that I have seen a few. Sure a technology might be cool, and developers do need to keep their skills up to date, but is it the right tool for the job? Will it realistically integrate with existing infrastructure with the given time-frames? Using the wrong tools to implement production systems is ultimately irresponsible, as it can eventually lead to lower quality software and higher maintenance costs.

Typically you might expect a greater level of responsibility from software architects, however this is not always the case. Unlike developers, architects need to be aware of the entire system infrastructure, and as a result may make safer decisions when it comes to technology choices. However when it comes to the system design, more often than not the architecture will include needlessly complicated configuration options, allowing for the greatest number of scenarios possible. Maybe the architect’s model is sufficiently complex to allow it to be used in many different configurations, but how many configurations do you actually need? If the software is designed properly, shouldn’t you be able to add support for additional configurations at a later time as required? This kind of unnecessary complication is also often a major factor in implementation issues.

One of the biggest failings of the modern software development process is the disregard for Human/Computer Interaction (HCI) experts. Once upon a time people with a good knowledge of interface design were employed to do just that. More recently the system users feel they are qualified to do it themselves (how hard could it be?), leading to glaring inconsistencies and ultimate confusion for the end user (”this isn’t what we wanted” - “but you designed it yourself!”). Or in the slightly less disastrous situation the developers are given free reign to design, and whilst the user interface design may be fine for developers, will it be easy to understand for the end users?

Having access to a diverse range of opinions is ultimately a good thing, but you also need people that can adequately filter these opinions in order to recognise the approaches that are in fact realistic and will achieve the desired goals. Such an ability takes a good deal of self-discipline and objectivity, which is unfortunately not something that is all too common in Information Technology circles.

All software has a lifespan. The three primary factors that control this lifespan are the popularity of the software; it’s usefulness, and the lifespan of the controlling company. Whilst the first two factors are common to all software, the third is one of the primary differentiating factors between proprietary and Open Source software.

A measure of the success of Open Source software is to gauge the likelihood of the software to survive, if the originating company and/or individuals suddenly ceased all support for it. Is the community surrounding the software strong enough to keep it afloat, or is the community also dependent on the software authors contributions?

Comments by Mark Shuttleworth, founder of Ubuntu Linux, whereby he states that Canonical (his company and origin of Ubuntu Linux) is now just a small part of the Ubuntu community, are a good example of how software can out-grow it’s origins and effectively sustain itself. Such a scenario is often quite difficult to achieve however, and software must certainly be both popular and useful in order to have any hope of surviving. But it is worth remembering that irregardless of how popular and/or useful proprietary software is, it’s lifespan will always be limited to that of it’s originating company. This aspect alone demonstrates that if you must be dependent on any software, a preference for Open Source software is a much safer option in the long term.

An Open Letter to the Maven Central Repository Maintainers:

My first impressions of Maven were negative to say the least. Where previously I had seen all manner of Open Source projects loosely following the same conventions, suddenly I found projects that all looked the same. Their websites were almost identical, there deliverables were all following the exact same convention, and it was becoming increasingly difficult to tell them apart. Obviously there are a number of benefits that come from such uniformity, however all I could see was that Maven was infringing on the creative process of these Open Source projects.

Coding is, without doubt, a creative process and Open Source is perhaps the most visible expression of that process. Open Source offers software makers an outlet for their creative expression, with the only real tangible goal being the evolution of the software itself. Unlike many other creative activities however, software typically has a purpose beyond its creation: applications are designed to provide benefit to the end-user, and software libraries are designed to be useful to other software makers. Without these less tangible goals, many software projects would not have a reason to exist.

There is perhaps no better example of this than the wide array of Open Source Java libraries available today. These libraries are now at the heart of most Java-based projects, and I would be very surprised to learn of any Java project that did not depend on Open Source in some form. With these dependencies being such an integral part of the Java development process, we definitely need conventions that simplify the management of such dependencies. In the past most projects have used Ant as the basis of their build, package and distribution process, a tool that has worked reasonably well due to different projects adopting similar conventions which we all understand. However, Ant does not address the issue of managing the growing list of dependencies we use in our projects, which has always been essentially a manual process. Upgrading to newer versions of dependencies can be quite a headache in Ant-based projects, depending on how sophisticated your build scripts are. We have also found that in order to provide a consistent environment for these projects, we have been forced to store these dependencies in our source control repositories, for a lack of a better place.

So the rise of dependency management tools in Java projects is inevitable, due to the obvious benefits they provide to the development process. But where Maven appears to go beyond the aims of other dependency management tools is that it attempts to impose strict controls over the build, package and distribution process. At least, those were my first impressions anyway.

Since exploring Maven2 I have been pleasantly surprised by the fact that the configuration defaults are just that - defaults! Just about anything can been overridden in a Maven project descriptor (POM), allowing the developer to retain a certain level of creative freedom over their project. Obviously the package and distribution conventions cannot be altered to a large degree, and generally this is a Good Thing, as interoperability with other projects (third-party libraries, etc.) should remain of primary importance when developing Java-based projects. So to this end I have come to think of Maven as offering a benevolent dictatorship for Java dependencies, whereby strict control is only imposed for the benefit of the dependency management process.

Recently however, I have come to a different conclusion, one which I have been dreading since I first noticed Maven. It seems to me now that this dictatorship is perhaps not as benevolent as I first thought. This new conclusion is derived from a recent discussion had with one of the Maven Central Repository maintainers, in an attempt to load my Open Source project libraries into the master repository for distribution to the general public. It appears that the Maven repository maintainers are not happy with the chaos and disorganisation in the master repository, and have decided to impose even greater control over how you distribute your project. In the past you were welcome to contribute your project using any Group Id/Artifact Id, as long as it was unique! Now this is very important for distinguishing between one project and another, and to this point has served Maven’s goal of distribution reasonably well. Now it seems the repository maintainers are not happy with this approach, as identified in this note on their website:

IMPORTANT considerations about the groupId: it will identify your project uniquely across all projects, so we need to enforce a naming schema. For projects with artifacts already uploaded to ibiblio it can be equal to the previous used, but for new projects it has to follow the package name rules, what means that has to be at least as a domain name you control, and you can create as many subgroups as you want. There are a lot of poorly defined package names so you must provide proof that you control the domain that matches the groupId. Provide proof means that the project is hosted at that domain or it’s owned by a member, in that case you must give the link to the registrar database (whois) where the owner is listed and the page in the project web where the owner is associated with the project. eg. If you use a com.sun.xyz package name we expect that the project is hosted at http://xyz.sun.com.

Now, aside from the fact that they are basing this restriction on the Java package name convention that is not enforced, we must question what is the perceived benefit of this draconian measure. Will it help Java developers find the dependencies they require? No. Does it make it any easier to navigate the master repository? Not at all. Does it provide any real tangible benefit to Java developers? Definitely not! The only reason I can see for this is that the repository maintainers felt that the structure was a bit messy!

Some may argue that it helps to identify the correct Group Id where you know the domain under which the project is hosted. This is simply not true, as assuming that the project is actually hosted in the Maven repository you can be assured that instructions on the Group Id/Artifact Id will be provided somewhere on the project site (especially if they choose to use Maven’s uniformly-generated site!). In fact there is not even a guarantee that the Group Id will be the exact reverse of the host domain name anyway, as is stated in the Java Package Name convention from which this restriction is derived:

The name of a package is not meant to imply where the package is stored within the Internet; for example, a package named edu.cmu.cs.bovik.cheese is not necessarily obtainable from Internet address cmu.edu or from cs.cmu.edu or from bovik.cs.cmu.edu. The suggested convention for generating unique package names is merely a way to piggyback a package naming convention on top of an existing, widely known unique name registry instead of having to create a separate registry for package names.

Consider where Java would be today if you had to prove ownership of a domain prior to publishing any code using the reversed domain name package convention? Sure, it would be near impossible to implement such a restriction, but if Sun (or anyone else) had even attempted to try you can be assured that Java would not be as widely accepted as it is today. There are many reasons why Java has become such a popular language, but I think you’ll find that the primary reason for its success is that it is freely available and doesn’t impose too many restrictions on a developer’s creative freedoms.

Do we see such restrictions imposed elsewhere on the Internet? Take this Blog as an example. Do I need to own the domain www.thenextbigthing.com to publish it? You’ll find many blogs on the Internet with the same title, and in hindsight that probably makes it a poor choice in distinguishing my own blog, but it is my absolute right to publish this blog wherever I see fit, as long as the address is unique! Let me make this point absolutely clear: the content of this blog does not determine the URL used to locate it! So how can it possibly be suggested that Maven Group Ids must be derived from ownership of a domain?

Let’s say I moonlight as a Cyber Squatter in my spare time. I manage to get a hold of the domain maven.com. Does that mean I am now free to publish my Java projects using the Group Id com.maven? Even if my project is hosted on Sourceforge and neither myself nor my project have absolutely anything to do with that name, other than I got lucky at an auction?

Alternatively, how about a graduate developer that really digs the Open Source ideals and wants to start a new project. Initially she likely won’t own a domain, but doesn’t feel comfortable using an Open Source host such as SourceForge. No problem, she has broadband and a home server where she can host her project at: http://32.254.123.91/projects/foo. The added benefit of self-hosting is that it provides experience with the intricacies of site management. She also likes the idea of Maven as it gives her an easy way to get the project started and distributed to others. How exactly would you propose she can get her project into the master Maven repository?

I realise that these enforcement policies are not maliciously intended, but rather come from a lack of consideration of the wide variety of styles and opinions in the greater community. We see similar motives in the Linux community, with attempts to create the one true Linux, which always fail miserably. This can be attributed directly to the different styles and goals of the wider community, and such diversity should be celebrated rather than suppressed. Maven surely provides great benefit to the Java development community in the management of dependencies, but it’s maintainers need to understand that restrictions imposed simply to maximise order and minimise chaos with no real benefit to the wider community, will fail just as miserably as any political dictatorship has and always will.

Increasingly we see the use of dependency management tools becoming a part of mainstream software development. A core feature of many dependency management tools is the use of a repository (or set of repositories) that provide access to published artifacts. Perl has CPAN, PHP has PEAR, and Java has Maven.

Maven solves a number of problems that become more visible as projects become larger and more complex. First, there is the issue of managing an increasing number of dependencies. As a Java developer today you would be crazy not to make use of the large number of Open Source libraries available, however it does come with the challenge of managing all of those libraries - especially as there are frequently new versions released. Maven simplifies such management by allowing you to describe the dependencies (artifact id and version) from which it can automatically download and include in your classpath. Upgrading to a new version can be as simple as updating the version number in your project descriptor.

Not only does Maven improve the management of third-party dependencies, but it also provides an improved approach to generating internal artifacts and dependencies. In the past source code repositories have been somewhat misused as storage for dependencies also (e.g. JARs). Maven helps to move both internal and third-party artifacts out of the source code repository into a common artifact repository. This reduces the burden on your SCM system allowing it to focus solely on source artifacts rather than build artifacts.

Another, perhaps intangible benefit of a dependency management tool such as Maven is that it can be used to encourage regular commits to source repositories. Modern IDEs allow cross-project references in the workspace, which means that you can code for days in multiple projects without needing to commit anything to your SCM repository. The reasons why this is bad should be reasonably obvious (especially in multi-developer teams), so I won’t go into details. Maven can help to discourage this practice by restricting the use of cross-project workspace references and relying solely on build artifacts from the repository. If these artifacts are built by an automated build tool (e.g. cruisecontrol), developers will be forced to commit individual project changes in order to build the artifacts and use the changes for work in other projects. This may seem counter-productive, and it would probably not be prudent to enforce this with an iron fist, but it will help developers to focus on fully testing project changes without being distracted by work in other projects.

There has always been resistance to change of the status quo, and the introduction of dependency management tools is no different. Many still argue that Ant is sufficient for doing whatever you want in a Java development environment, and that tools such as Maven are too restrictive and impose too many rules on build processes (I know because I once held these views myself). However once you become more familiar with dependency management tools you begin to see the bigger picture, in that where Ant provides a small piece of the puzzle (namely building), tools such as Maven aim to help manage much more of the entire development process.

Recently we have seen real momentum developing behind Open Source software, largely as a result of the realisation that in the pursuit of software quality, cooperation is much more effective than competition. The promotion of Open Formats is another movement underway that, whilst not as publicly recognised as Open Source, is perhaps of more importance to the wider community.

In the early days of computing, data portability was never really an issue. After all, you had your computer, you had the software to run on the computer, and that was about it! But with the increasing popularity of computer networks, inlcuding the biggest network of all: the Internet, we have found value in the ability to move data from one computer to another without it losing its value.

Network connectivity and the increased need to share data has worked to the advantage of proprietary format owners, specifically because data in a proprietary format has no value without the proprietary software used to access it. This has resulted in software spreading in a viral nature, following the data from computer to computer, creating a false economy based not on the quality of the software, but on the value of the data that is dependent on the software. Perhaps the most visible example of this is Microsoft’s Word which, whilst is globally recognised as poorly implemented software, remains the defacto standard for editing documents purely through the proprietary nature of it’s popular .doc data format. As a result of this popularity we find that we are not only forced to use inferior software, but due to constant changes in the .doc format, we must also upgrade this software whenever a new version is released in order to achieve seamless document sharing capabilities.

More importantly, we are now discovering that whilst we may be capable of archiving our data efficiently, there is no guarantee that we will be able to interpret that data in the future, as the software we use today will most likely be obsolete. For data archived in Open Formats we at least have the public specification available and, as a worst-case scenario, would be able to implement software that could interpret our archived data. For proprietary formats we have no such specification, rendering our data worthless.

Proprietary formats are specifically served to us like any other addiction: purely in the interests of those profiting from that addiction. Whilst these formats can be a difficult habit to break, it is essential to the health of your data that you find an alternative.

In the search for methodologies and patterns that help to build better quality software, we find that our methods continue to evolve at a blistering pace. Whether its the leap from procedural to object-oriented design, or the difference between the waterfall and prototyping approaches, all share the common goal of building robust, complex software systems. Whilst many great advances continue to be made, one thing that eludes us in the search for the perfect development strategy is how to future-proof a software architecture. We like to think that we will have all the requirements and intended uses of the software prior to design and implementation. In reality however, this is almost never the case. Requirements change, focus shifts, often to the point that the final requirements bear little resemblance to those the original design was based on. Ultimately a good design should be able to handle such a shift in requirements, however even the best designs will not cope with a major refactoring of the intended uses of a system.

So how can we protect our software designs from such a seachange in required functionality? In addition to accepted practices and methodologies, we should try to incorporate the following principles into our architecture:

• Think Small Usually when we design architectures the components tend to be large and provide support for a multitude of features. It is not uncommon to even have a single component representing the front end, back end, or business rules of a system. Combined with the principle that a good design will explicitly identify each component’s roles and responsibilities, we find our components are too rigid and not really amenable to changes in these roles and responsibilities. The solution to this problem is to build smaller, more specific components. Rather than components handling many different tasks, we should design the architecture in such a way that each component is responsible for handling a single task. Such tasks may include support for a specific protocol or data format, a service used to access an external or legacy system, or even data structures specific to a particular feature in the system. The important thing to remember is that a component should only handle a single task rather than a collection of them as is the popular approach.

• Separate APIs from Implementation Another recognised good practice is the use of Interfaces when designing an API allowing components to communicate with each other. By using interfaces we are effectively defining the contract a component holds with the rest of the system, irregardless of the implementation details. Whilst this allows us to provide alternate implementations of the API without further changes to external components, we still generally need to provide an entirely new version of a component - even if the API itself has not changed. Although the API is not bound by the implementation, usually the entire API (interfaces and implementation) are bundled together anyway. The solution then is to separate the API definition from the implementation and provide them as two distinct components. By doing this the API specification does not require updating where only the implementation requires changes. Additionally, alternative implementations of the API may also be deployed independently, thus maximising the possible re-use of a component API.

• Loose Coupling A major contributing factor to the complexity of software are the dependencies each component has on its peers. Aside from the careful design decisions required to ensure no two components are dependent on each other (i.e. cyclic dependencies), typically where even a single dependency is not resolved the entire system will fail. To avoid architectures easily susceptible to such a catastrophic failure we need to maximise decoupling of components. This does not necessarily equate to minimising dependencies across components, but rather is more concerned with ensuring that such dependencies impose as little restrictions as possible. This is the promise of Loose Coupling, whereby individual components may operate independently or as a part of a larger system constructed of similarly loosely coupled components. One popular approach to incorporating Loose Coupling in software design is the Inversion Of Control pattern (also known as Dependency Injection). A number of frameworks based on the Inversion Of Control pattern are now available, which assist in the “wiring together” of individual components to construct a complete system. This encourages greater encapsulation in components and simplifies other aspects of quality assurance such as unit testing.

By adhering to these principles we are able to design software that can adapt to just about any change in direction required of it. No longer do we need to throw away an entire system and start from scratch, as we will be able to incrementally upgrade and replace individual components without detriment to the rest of the system. This approach also enables us to extend the functionality of the system by “wiring in” new components as required. Best of all, these principles ensure that the software architecture is built from clearly defined design contracts and encourages maximim re-use.

Some now speculate that the first real “thinking” machine will be the meta-computer, the Internet. This suggestion bears a particular resemblance to the suggestion made by Douglas Adams in The Hitchiker’s Guide to the Galaxy, that the Earth is really a supercomputer designed to calculate the question to the answer to the meaning of life, the universe and everything. In Adam’s plot we are also a part of the machine, an integral part of what makes this mega-computer function.

If we consider that the Earth is in fact itself a massive organism, consisting of the various biological, ecological and industrial spheres, the logical deduction is that the Internet is in fact a massive supercomputer consisting of the various Personal Computers and Servers that connect through it. But the real intelligence of this supercomputer comes not from the PCs that are connected to it, but rather the users of those PCs and how we use them to communicate via the Internet.

Consider the flow of information over the Internet when a natural disaster - such as the 2004 Tsunami - occurs. If we are to perform a search for information regarding Tsunami, we can see a general impression of how we collectively think about the event, and how it has affected the entire world.

What we are really seeing every time we hit Google is a primitive interface to the collective human intelligence evolving on the Internet. In its current form we still must manually decode this intelligence (i.e. sort the wheat from the chaff), but you can expect new interfaces in the future that don’t just give you references to words and phrases, but rather actually understand what you are asking.

There have already been a number of attempts to provide a more user-friendly search interface (e.g. Ask Jeeves), but these approaches were ultimately performing the same search as everyone else. What is required is not an improved way of specifying what we are asking, but rather an improved analysis of the results that are returned. Current search techniques provide very little analysis of a collection of results as a whole, which is where the real intelligence of the Internet lies.

Whilst current search engines may provide results ordered with considerably advanced techniques, generally these results are provided as a flat listing, or possibly grouped together by host name. Future interfaces will find improved ways to group together “clusters” of results, providing a more hirearchical relationship between web pages based on this emerging intelligence.

Ultimately we are (inadvertently) building an Intelligent Internet, all we need to do is recognise that intelligence and everything else will happen automatically.

The Open Source movement has long been associated with socialist ideals such as communism - typically by companies who see Open Source as a threat to their own business. The irony is that driving factors behind Open Source are the very reason they even have a business.

In the consumer software industry the quality of software has not been the primary differentiating factor in consumer choice. As end-users we cannot see how a software title was built, what development patterns were employed, or even what language was used to build it. More importantly we cannot see how many unresolved bugs there are, for which we leave it to the discretion of the software developer to inform us (if they are even aware themselves!).

And so in the absence of a qualitative analysis of software (excluding user interface design), we must rely on a quantitative assessment, or a comparison of features. Software features have come to be known as innovation in the software industry, possibly as it conveys some kind of magical ability to invent something entirely new. In reality much of this innovation is not in fact entirely new, but rather a rehash of old ideas or, more likely, a feature copied from someone else. Now contrary to so much of the discussion surrounding software, there is nothing at all wrong with this. In fact it is in the very nature of software development to recognise a good idea and re-use it. Perhaps it is the common approach of claiming innovation, without due credit to the originator of the idea that creates so much hostility. Whatever the cause of this slightly misguided animosity, we should not deny that re-use of innovation, whether it be an idea or an actual implementation, is an integral part of software development.

Recognising that features are the differentiating factor in software, many companies have adopted a two-fold approach of adding as many features as possible, and protecting those features from getting into the hands of the competition. Now of course noone can stop another from reimplementing a feature found in competing software (although they do try!), but you can protect your own implementation, which at the very least makes it more difficult for others to duplicate. Or at least it used to. Nowadays we find large companies have vast teams of developers that can implement anything that you ask for. The hard part is knowing what you want to implement, and when all else fails, just look to a competitor for inspiration.

So whilst this feature arms-race continues amongst the largest companies, we see the intended effects of feature dominance becoming more neutralised. There is however another, possibly unintended, effect, which is that smaller companies and individuals are becoming increasingly locked out of the competition. If you cannot afford an army of programmers to keep up with the feature sets of the most popular software, how will you ever compete in a feature-dominated industry?

In the face of such marginalisation many have now put their faith in Open Source. Open Source provides individuals and smaller companies with that “army of programmers” needed to compete with the larger companies. But perhaps a more interesting aspect of Open Source is that it does not discriminate. The largest of companies are also able to incorporate Open Source software into their business model, which then begs the question, is Open Source really changing anything? To which the answer is an emphatic yes. If Open Source software is able to keep up with the feature arms-race of the largest companies, putting such features in the hands of everyone, there will no longer be a feature arms-race, or at least not on the scale that it is today.

So what does this mean for the future of software? Well for a start it means less emphasis on features, leaving room for a more qualitative analysis of software. Aside from giving equal access to feature implementations, another driving factor of Open Source is the qualitative analysis that goes with it. As financial reward is less of a concern when developing Open Source, we place greater emphasis on writing better code, so as to gain the reward of respect, recognition and popularity of our software. With quality being such a major emphasis of Open Source, proprietary software developers will be compelled to prove a similar quality of software in order to compete.

Some larger companies have suggested that by neutralising their competitive edge through feature dominance, Open Source is stifling innovation. On the contrary, by giving everyone a level playing field Open Source has once again made it possible for anyone to innovate. These larger companies seem to have the idea that they are the leaders of the industry and as such should be responsible for all innovation. In reality, as we are shown time and again, real innovation in software comes from individuals, not from task groups and think tanks. Such individuals may work for large companies, small companies or no company at all. Wherever they may be, we cannot force, or even predict, who will innovate next. All we can do is create the best possible environment for encouraging such innovation, and the best possible environment today includes Open Source.