Struggling toward the future

Diego posted something interesting last weekend (I meant to link to this last week, such is life) called two steps forward, one step back which talked about the high degree of coupling software tends to create. There are some interesting ideas there to think about.

Many of us have teased windows developers because of all the weird DLL interdependencies which have popped up over the years (commonly known as DLL hell by users) and I was certainly bad about it. On the Mac, we had these issues, but generally worked around them using things like Gestalt (a dynamic system used in classic MacOS to express versioning and capability advertisement) and code that dealt gracefully (or at least reasonably) with missing features. Crashing is not an option.

In the last few years, I've grown to know and despise the java variant (aka .jar file hell) which doesn't happen on the users system (typically), but happens none the less (on the server side) by virtue of java based system growth. The interwoven set of dependencies is ever growing as application containers grow and change and the number of applications one might deploy do likewise. Eventually, old code has to be re-written (or refactored) to deal with a changing API landscape.

It's also true of most of the core applications I use every day. They're dependent (largely) upon a collection of system and open source libraries which change over time. Often giving, sometimes taking away. The features available in the libraries influence to the capabilities in the applications written on top of them.

While Diego seemed to be thinking mostly about UI, I learned a long time ago that UI's are simply one aspect of an application (or engine) API. I'm not certain what caused that point of view to exist, although moving to a separation between data and presentation helped I tend to think adding AppleEvents was the final catalyst (to create an AE suite for an application meant creating a generic object like view of your data).

Forgetting this concept by ignoring a rational data model at some level leads to long term nasty issues. I've been there and wear the scars. By assuming that new features can always bubble up, he also ignores the history we have to guide us. The compromises created by known structure sizes (and data) in legacy operating systems are now legend. Concepts directly expressed in shared data structures are nearly impossible to fix which leads to a more layered approach and software based compartmentalization (if you don't need to know, you aren't going to be allowed to know). Finding the right model for inclusion is hard and fraught with its own set of problems.

Big systems (like Longhorn) tend toward being largely monolithic. While that carries a lot of positives (yes, we can depend upon this functionality) it also has drawbacks. Especially when it comes to file system changes like WinFS. Some of the biggest hurdles Apple had to overcome to get MacOS X out the door were file system related. While most of the details of the file system(s) are hidden from most applications (making it easy to deal with the majority of apps), there are driver level APIs that allow applications to know quite a bit along with the drivers themselves and the driver model they sit on top of. Creating a new set of layers that wraps (and maps) everything done today while adding a whole level of functionality is an immense task and I wasn't surprised to see it go sliding back a bit on the schedule.

File system stuff is usually opaque for a reason (to hide details). WinFS seems to destroy that concept by being not only quite expressive at a high level, but also by seeming to allow links back and forth across layers. Microsoft has a huge task ahead of themselves here and decoupling it from Longhorn will likely make WinFS better (and perhaps Longhorn in the interim).

I'm not sure where all of this leads, but it's good food for thought.

I've been wrestling with these problems for a few months and not gotten as far as I'd like while pondering a major data store transition. Some applications (generally data consumers) can deal with minimal APIs. Others have a need for complex and expressive data which needs to stay consistent regardless of any re-organization underneath them. Writers have to deal with all of it, being cognizant of past, current and future relationships and dealing with them appropriately. Finding the right balance and a way to make the transitions is the really hard part.