A Need to Blog

I was counting the other day, and came to the realization that, over the last couple of years, I have ended up blogging in about ten different places. Yet despite that, I also made the rather disturbing observation that while I am regularly writing articles for O'Reilly (about fifteen to twenty a month, which, when you get right down to it, is an incredible amount of writing), for XMLToday.org (which is focused on XML issues), for DevX, for EMC/Documentum and elsewhere, what I didn't have was a place where I could just blog for myself.

That's when I remembered this blog. It's not a Drupal instance, just a simple Blogger app, but I'm coming to understand that perhaps this is not a bad thing - I need something that I can periodically just write to without having to get sucked up in the mechanics of blogging. I've learned to really like Drupal, but sometimes you need to just walk away from the code and concentrate on the message.

In the case of Metaphorical Web, that's precisely what I intend to do. This can be considered my kick back my feet and just write whatever I want to write blog. I might talk about code, or economics, or just how I'm feeling that day, and I make no promises that you're going to learn anything here, other than maybe the occasional odd rumination.

Ironically enough, as a professional writer, I'm beginning to realize the real value of such a blog. One of the real challenges that you face as a pro writer or journalist is that overall you are always writing for others. You have to consider every thing you say in terms of the editorial message of the site, the audience involved, the needs to promote this or that conference or book or product. Especially in the age of tightening belts, you also have to make sure that what you're writing has value with every article.

Yet here's a secret for you - no writer can be "on" all the time. That's not to say that a writer can't consistently write articles or stories or what not. That part of writing is a lot like what a long distance runner goes through, just building up your endurance so that you can crank out an article every day or two, two to three stories a week. Yet even the best writer (just like that endurance runner) will have off days, will produce a piece or two of forgettable crap for every good story, and the really incredible pieces will be balanced by the occasional article that frankly should have been left exposed on a rock to do a humane death.

What's more, that writer periodically needs to write to no one in particular just to get the frustrations off his chest, to say the things he daren't say when writing for pay, to have, well, a journal. In its own way, writing is an addiction. The more you write, the more that the pathways in your brain see writing as the mechanism for expressing itself, and as a consequence you find that its often difficult to use the medium of speech because your brain is wanting to shape things into words on a page or pixels on a screen.

I've been at a number of author readings and been on panels at everything from science fiction conventions to international conferences with other writers, and one of the things that I've noted is that most writers tend to be naturally taciturn and withdrawn, though certainly capable of speaking eloquently when called upon to do so. At first, my thought was that the profession tended to draw introverts to it, in great part because introverts tend to live more in their head than via interactions with others.

Yet over the years, I've also begun to see that the process of writing reinforces this introversion, makes it stronger. Writers are aloof not because they believe themselves above other people, but rather because writing crowds out speaking and other human interaction, and as a consequence, the skills for dealing with other people become rusty and frequently mechanical, as writers find themselves having to remember how to do these things.

This isn't unique to writers, of course. Creators in most endeavors tend towards this way of thinking. Artists use different pathways, and as a consequence, the way that they view life differs somewhat, but at the same time, most artists become artists because art is the mechanism that they use for communication. Musicians, good musicians, similarly become wrapped around their music - and find that the channels over which they communicate dominate their interactions.

Once consequence of this is that there is a world of difference between communicating with someone in your modality of expression, and someone who simply "appreciates" that modality. I can communicate with my eldest daughter along a channel that others can't, because my daughter has the artist "genes" - the combination of talent and the overriding compulsion to draw and paint - that I have (I was very much the same as she was at her age -I was always drawing). I can communicate with my wife along the writing channel because she is a writer herself, which provides a shared set of referents or symbols (and experiences) that would be lost among non-writers.

Yet the irony is that I'm a lousy critic, which is I suspect also true for most creatives. An artistic critic is someone who looks for meanings and interpretations in a "work of art", as if there was actual intent there to provide such symbolism. I remember in high school one time, an elderly English teacher gave us a test which included the question "Why did Shakespeare write Romeo and Juliet?". My response, for which I received a rather stern lecture, was "because he had to pay the rent."

Writers write for public consumption not to load their works with deep symbols and meanings, but because they've discovered that the voice in their head that demands expression can be occasionally harnessed to pay the bills. The voice, the compulsion, to write, is still there, of course. They would write regardless, just as the artist will draw or the musician play, even if there was no audience. It's their language.

Yet this harnessing the writing impulse to pay the bills has a darker side as well. When everything has a deadline, what this means is that the temptation will be strong to do nothing but write for public consumption, even if what you're writing holds no great interest to you. The stories that you used to write get left undone because the clock is ticking and you have to get three articles on the latest news du jour written by the end of the day, you have to get the next chapter to the manual completed by mid-next week, the interview you did has to be transcribed and re-edited before the next conference. All are important, all pay the bills, but the music, the creativity that you once enjoyed as part of the writing process gets lost - your ability to express yourself gets lost in the requirement of expressing the needs of others.

There's another corrosive aspect of commercialization: you began seeing other writers not as people with whom you have a deeper understanding based upon your art, but as competitors for the same audience, the same revenues, the same lucrative barely minimal writing contracts. You don't dare do any but your best work because if you fail, you're toast. Unfortunately, this typically means that you also don't experiment or take risks, both critical for improving your craft, because the perceived cost of failure becomes too high.

I'm not sure there's necessarily a morale here, though I do have a suggestion to writers, (though it applies just as readily to artists, musicians and other creatives) from a writer who is rediscovering this for himself:

Always leave a certain space for yourself; block out a chunk of time in the week that is devoted to your play time, your experimentation time. If you write news for a living, use this time to work on a novel without the expectation that it will ever see print. If you're a technical writer, spend some time composing poetry, playing with the way that words sound and feel. If you're working on a book, take some time to write an essay about the coolness of the spring morning, or a random character portrait of someone you see in a coffeeshop.

Minimize the interruptions around you during this time, and do not, regardless of what else you do, use this time for paid work. This time is the equivalent of working out at the gym (something else you should do, for what its worth) in that it is not time that is owned by someone else but is necessary for your own sanity. This time takes precedence over everything - even if you have a critical deadline, take this time for yourself, because there will always be critical deadlines, and just as working out physically can often help relieve a lot of the physical stress that you face and make it easier to get things done, so too is this creative exercise time necessary to cut down on the mental stress that you face.

My suspicion, when it's all said and done, is that when I finally die, it will be the work I do during this time, rather than the marketing document for client X due next week, that will define me as a writer. Creativity is rare not because people aren't creative ... most people have a streak of creativity in them ... it's rare because people become so obsessed with the need to do their "work" that they fail to take the time necessary to actually be creative for its own sake, rather than in the service of some larger goal.

Take the time, it'll make you a better writer.

CDF: The common format you've never heard of

Quick! Do you use the Compound Document Format?! You, know, CDF … surely you use CDF, right?

Chances are pretty good that you have no idea about what I’m talking about. Everyone knows Microsoft’s word document format and Adobe’s PDF, chances are pretty good that if you’re reading this on XML.com you’ve heard of ODF and OOXML, especially after the fairly rancorous discussions about ISO status for these two formats. Yet CDF, hmmmm … that’s a rough one. Didn’t it belong to Corel, once upon a time?

Okay, now, how about this one … do you work with (or even just read) XHTML? Probably, if you’re involved in XML work, your HTML conforms to a great degree to the formal XHTML standard. Good enough. How about CSS 2.1? Sure, who doesn’t. Okay, here’s a biggy -how about XMLHttpRequest? You do AJAX work? Good for you. XForms - well, that one’s a little less prominent, but yeah, it’s beginning to appear. SVG? Hmmm … again, kind of touch and go, but even after a few hard years SVG’s by no means dead yet. The occasional XSLT - Google’s doing some nice work in getting their Javascript based AJAXSLT up and running, which means that those few browsers that don’t support XSLT (and XPath) natively should be able to support it via an AJAX layer. Oh, and perhaps through in XSL-FO for good measure, as it continues its quiet but relentless march into becoming a mainstream format.

So, do you work with CDF? You betcha. The Compound Document Format was set up as a way of tying together at a minimum all of those technologies described above into a single cohesive whole. Put another way, it’s a fancy way of describing the core suite of W3C document standards into a cohesive whole, although it does place some fairly minor requirements on usage in order to provide a consistent standard.

CDF was in the news recently with the implosion of the Open Document Foundation, originally established to endorse ODF, though in its death throes it briefly highlighted the CDF format as perhaps a better format for documents than either OOXML or ODF. This is admittedly one of those areas where it may be justified in looking at XHTML especially and going “huh”? How can that be a full document format - it’s used for web pages, after all - you wouldn’t want to use it to mark up a full book, would you?

Document formats are a lot like religions - people are ready to defend them to the death if need be, yet at the same time it becomes easy to dismiss certain religions that don’t even seem to be religions at all (such as my personal favorite, the rather philosophical Tao). Could you mock up a brochure in XHTML and CSS? Actually, it turns out that its surprisingly easy to do just that - especially if you throw a little SVG into the mix and allow the possibility of embedding XHTML within SVG (for all those odd little bits of rotation and other special effects).

How about linking between blocks of physically disconnected content? That’s what hyperlinks are for, after all. CSS gives you at least a dozen different unit references, lets you control borders, margins, padding, kerning, image placement, color management, and so forth. There are pieces I wish could work better - I’d love to have the capability of defining a color in SVG then referencing it within my XHTML document via CSS; full support of CDF will likely allow that. If you jump just a little beyond the current CSS 2.1 spec you even have some fairly decent support for columns of text, not to mention tabular structures and even VoiceML support. Moreover, consider XInclude support, something that really, really needs to be a part of every browser implementation (though its fairly trivial to write AJAX classes that let you create similar bindings).

These are all disparate documents, not a single “document” akin to Word or the ODF Writer format. However, even that’s not quite true. The effort of the CDF working group has been to essentially standardize on the way that web documents can be “bound together” into what appears to be a cohesive “whole”.

Part of this is accomplished through the use of a standard called the Web Integration Compound Document (or WICD). This standard provides a number of both new features and clarifications:

• an extended definition for the HTML object,
• the integration of Scalable Vector Graphics (SVG) with XHTML and other documents,
• establishes the and elements,
• establishes the characteristics of hyperlinks and focus across the boundaries of namespaces,
• defines the nature of focus across multiple embedded documents,
• introduces support for animation and synchronization, building on the older SMIL and SVG standards, and
• includes support for SVG fonts.

Two additional WICD standards - WICD Mobile and WICD Full - extend these with a few other features, most notably ECMAScript 3rd Edition (4th edition is currently still under development), the XMLHttpRequest to support most AJAX applications, XHTML 1.1 and the CSS 2.1 specification. Additionally, CDF working group also define two “modes” of operation - Compound Documents by Reference (CDR), in which internal content to the document is provided via reference links, and Compound Documents by Inclusion (CDI) in which internal content is rendered into the containing document directly in a different namespace.

Admittedly, I think it can be argued that the W3C effort could do with a few less acronyms (and perhaps a bit more of a PR effort) but overall, what is happening with CDF is a very critical - and welcome - evolution of HTML. The web needs to be more than just static web pages - this has been demonstrated by the continuing strength of the Web 2.0 meme, that documents should be able to talk back to the server and interact with them at higher levels than simple links for refreshing content. Commercial vendors would love to fill that space, and by doing so regain for themselves control over the underlying technologies that make up the web. Yet the message coming about with CDF is both simple and profound: the web grew up on the strength of seemingly simple technologies - HTML, CSS, JavaScript. These technologies are still around, they’re just maturing as people come up with new ideas about what the web can do, the pieces slowly unfolding as we progress further into the realm of the Internet as operating system.

Already, much of CDR has been implemented in the more sophisticatedly forward browsers. Opera 9.5 has a rather extensive support for most of CDF core and Firefox 3.0 is moving in that direction (though the biggest area of weakness is in SVG animation support). JustSystems, a company that has a huge presence in Japan but is only now just (sorry) beginning to make an impact outside of that country, has been working towards a CDF platform for a number of years, and has one of the more expressive (and impressive) displays of how compound documents COULD work (I will no doubt by writing more about JustSystems over time). Both Sony and Nokia have WICD implementations working (as prototypes) on certain of their mobile phone chipsets, with similar announcements from Abbra Vidualize and BitFlash, both makers of mobile graphical chipsets, while Sun is partnering with OpenWave to create a formal WICD implementation in line with JSR 290: JavaTM Language & XML User Interface Markup Integration.

This is no guarantee, of course, that WICD will catch on, or even become a household name (even in geek households), but unlike other technology most of what’s involved already exists, and has already been proven every day millions of times a day. Already, the number of HTML documents that exist dwarf (by a few orders of magnitude) the total number of Microsoft Word documents. As editing increasingly moves onto the web, its safe to say that the document of choice will be neither ODF nor OOXML, both of which gain their power on the basis of supporting legacy word processing systems. Instead, what seems to be emerging from the W3C is something that is not an office suite because it didn’t evolve from one, but that nonetheless is capable of most if not all of the same functions that office suite documents pose.

Moreover, if you come to realize that XHTML by itself is NOT the only targeted compound document (indeed, the specification is rather clear that it is intended to be extended to other formats, from XSL-FO to DocBook to the aforemention ODF or OOXML) then what becomes clear is that the ability to integrate content itself can be standardized, and like many other W3C formats, this move to the metadata level may very well provide the necessary differential to make the technology succeed in even the most competitive of milieus.

So, don’t worry - if you’re not using CDF yet … you will be.

Kurt Cagle is an author and chief architect of Metaphorical Web (http://www.metaphoricalweb.org). He lives in Victoria, BC, where he is making it his mission to visit every Starbucks on Vancouver Island.

Will Wall Street Tremors Bring Another IT Recession?

Greetings, one and all. I'm bowing to a reality here that Google, in its oh so infinite search engine wisdom, is still finding this particular website more often than it is my Drupal (www.metaphoricalweb.org) site. For the moment, I shall maintain both sites, and see if perhaps something comes of it one way or t'other.

Lately, I find that far more of my reading has centered on economic news rather than upon technical, though to be honest the intricacies of many of the "investment credit vehicles" that have been foisted off on the public frankly put the Obfuscation C Coding Contest winners to shame. In most cases, when a financial fraud is perpetrated, one of the hallmarks that jurors use as evidence is the degree to which illicit cash flows are threaded above, below, and through more mundane transactions. The intent of fraud is simple - make people believe that they are financially safe when in fact their life savings are being purloined.

What your seeing each day in the big financial papers is disclosures of "write-offs" of hundreds of billions of dollars of money by banks, mortgage institutions, hedge funds and similar financial players, largely because these organizations had become engaged in the practice of ... well, fraud.

Mortgage brokers initiated liar loans with people who clearly had no means of paying back even the teaser rates, then got those loans off their books and into "structured investment vehicles" that sliced and diced these shaky loans and, with the assistance of ratings organizations turned these loser portfolios into alchemical gold by building highly leveraged hedges against them.

The banks that purchases these would then get them off their books by selling them to investors, who generally had no idea that the "risk" that they were buying was not safe, conservative debt but rather investments into millions of loans that had about 50/50 chance each of going into default.

Not surprisingly, while many of the mortgage companies and investment banks that perpetrated this nonsense are either in bankruptcy or in financial distress, most of the CEOs who should have put on the brakes early have reaped hundreds of millions of dollars and golden parachutes that will likely see them well into their retirement years.

Of course, the same practices that made this possible also have laid the same insidious seeds within most other credit companies, including credit card issuers and corporate paper investors. As the domino of subprime mortgages fall, its already begun to press auto loan issuers, and there are indications that the commercial credit market is also seizing up as dubious corporate investing also is beginning to look ominously diseased.

Now, I am not an economist - I'm a programmer. While I find this a fascinating exercise to watch in system theory, ultimately the questions that I'm having to ask have more to do with the simple question of how it will affect me, my clients, and the IT industry in general.

Five years ago, the Tech Recession hit. If you were in financial services, you might be forgiven in saying "recession? We're in a recession?!" The effects on most other sectors was comparatively light, though it was devastating if you happened to be in IT. Fully 40% of IT jobs were shed at the height of the recession, programmers who had been buying Starbucks coffee and making six figure incomes suddenly found that the paper options they'd been paid in were worthless, they'd given the best, most productive years of their lives and their most innovative works for comparative peanuts, and they were living in their parent's spare bedroom.

Not surprisingly even now, five years later, a lot of programmers have become skittish about accepting stock options and they have become considerably more parsimonious about how they spend their money (and what jobs they accept in the first place) - and there are comparatively far fewer people entering into the IT field. They've also become very good at watching economic trends, and most programmers that I know who went through that period are getting rather twitchy about now.

Right now, the ill winds of recession are blowing most heavily through the housing sector, especially in the US. One of the things to consider about most recessions is that it is actually quite rare for a recession to affect the entire economy. Instead, in most cyclic recessions, what you have is a situation where supply has temporarily exceeded demand, and the excesses of that supply need to be worked off.

The IT recession of 2002-2004 was a fairly classical one - there was too much investment in IT, both in terms of jobs and in terms of technology, and it took about eighteen months and some fairly severe slashing before demand could catch up with supply. By 2004, even though tech was still technically (sorry) in a recession, there were already signs that things were beginning to recover, and the market very quickly went from too few jobs for the number of workers to too many - to the extent that finding good, qualified people is becoming quite challenging.

The housing sector is going through its own recession, but unfortunately it will take more than eighteen months for demand to catch up with supply; there are a number of reasons for this, from speculators flipping houses who are now left with too many properties and no buyers, to houses going into foreclosure due to onerous adjustable mortgage rate resets, to a longer term demographic switch away from large, costly-to-heat houses in the face of higher energy costs and shrinking families.

Yet by itself, the recession that results from this will also would be at least somewhat self-contained, if an oversupply of houses was in fact the only real problem. Unfortunately, what is now emerging is that this is the domino falling that seems to be catching other dominoes with it. Risk is being repriced much higher, because the same lending practices that made it possible to get a loan for a half-million dollar house with an income of less than $20,000 was also at work in the hedge-fund space, the commercial paper space (which affects corporate buy-outs, business expansion), credit card paper, student loans, re-insurers ... indeed, just about everyone out there is suddenly discovering that the foundations that they were building on was made not of concrete, but quicksand, which has many of the same properties but a rather disturbing tendency to be lethal.

This is the real recession, and the last time that something even vaguely similar happened, Herbert Hoover was president. Credit, which has replaced cash in just about every transaction that occurs today, is rapidly disappearing. Indeed, it is worth it to understand that credit is itself simply a measure of risk. "I will let you take out a loan because I believe you to be a trustworthy individual," the banker should say, "you pay back your loans on time and reliabily, you have open accounting, you have shown good judgment in how you spend your money. "

The problem comes with the next part - "Even the most reliable person occasionally runs into problems. As a banker, I should be compensated for the small chance that you will be unable to pay your loan under the terms you and I set out. That compensation is the interest on the loan."

So what's a reliable rate of interest? This is where things get tricky. The more risky the investment, the higher the rate of return should be, so the higher the rate of interest, and the harder it becomes to qualify for a loan. The problem of course is that the banks make money on that interest, and in good economic times it becomes easy to fudge the credit-worthiness of an individual or two, if higher returns is the desired goal. Of course, once risk becomes high enough, it becomes far more profitable to initiate the transaction (be it mortgage, credit card application, or small business loan) then sell off that risk to someone else. That proved SO profitable that the process got repeated many times, with the same questionable risks getting packaged and repackaged to the point where there was no real way of determining where the risks were - or, more to the point, what they were.

Now, that whole edifice is going through a slow motion crash, and despite everything you hear in the Wall Street Journal or CNN/Money, it is going to continue to crash into other sectors. Right now, it really hasn't fully impacted the consumer (though there's another factor working there) but it will. If no one has any idea what the real value of risk is anymore, no one will initiate loans. Companies are already finding it increasingly difficult to get funding that used to be routine. Companies can't expand beyond the reserves that they already have on hand, and already accountants at many companies are beginning to question just how stable those reserves are. More than one company has been forced to delay even paying their employees or their accounts because their banks have frozen all accounts.

In the late 1930s, the FDIC was established in order to avoid a credit crunch like the one that wiped out the economy in 1929. It stipulated that banks always maintain a fixed percentage of assets (originally 10% of total capital lended) in order to insure that in the event of a bank run, there was enough operating capital to keep the bank afloat until help could be arranged. Additionally, many banks were required to honor the first $100,000 of deposited funds. The problem is that lax regulatory oversight has effectively eviscerated most of the FDIC's mandate, to the extent that it is unlikely that most banks could in fact meet 1% of total capital extended if they had to.

Of course, in order to look like they were in fact complying with these strictures, many financial institutions put much of their assets off the books into secondary corporations that were not then subject to the limitations, meaning that the actual debt that a financial institution had was generally far greater than what they showed on the books. So long as no one looked too closely (and when the money was great enough people would not look too closely) this meant that banks could make incredible returns by lending money that, for all intents and purposes, didn't really exist.

The problem is a Schroedinger Cat crisis. In quantum mechanics, there is a classical thought experiment in which a single atom of uranium is placed in a special trap that would, when the atom decayed, release a toxic gas that would kill a cat locked in a box. Quantum theory states that what is at work here is a wave function, and as such, the cat could simultaneously be both alive and dead, and the only way that you could actually tell (well, besides the obvious yowling of a pissed boxed cat) was to open up the lid. Once you had opened the lid, then you had collapsed the wave function, and there could be only one solution.

The banking industry has been build on the same wave functions, and so long as no one looked in the box, then there could be an infinite amount of money there. Once you open the box, unfortunately, reality steps in, and the reality is considerably more disappointing than anyone could have hoped.

So what happens this time around to IT. The credit crunch will hit in a number of ways:

• No money available for expansion of companies, beyond existing funds. Too many companies have been "living beyond their means", using their existing reserves as collateral against loans. As those loans begin to dry up, they are forced into using those reserves, which often may be tied up in existing investments and thus are less than completely liquid. When you are forced to sell to meet costs, you often get a far less than ideal value for your assets, especially when everyone else is doing the same thing.
• Illiquid third party investment. VCs and other investment companies will have more demand on their moneys from more traditional sources, which may be good for the VCs (in the short term) but it also dries up much of the money that IT startups in particular count upon. Moreover, many of those VCs represent funds that may also be fairly heavily hedged, and so the same repricing of risk that is going on in the commercial sector will make it harder for them to pull together the funds necessary to complete deals.
  
• Strapped consumer spending. Don't pay too much attention to all of the sighs of relief after Black Friday sales. People are strapped for cash, and they are as attracted to a bargain as anyone. The problem will come soon enough, as people can no longer take out much money via a second mortgage, will be facing stagnant wages and unemployment in December (one of the highest months for force restructuring) and will find it difficult to get credit extensions on their credit cards. I expect that consumer spending will plummet in the next couple of months. Sales from Walmart are done from last year, but so are the sales from high end luxury brands. This will mean that spending on consumer software and hardware will also in general be declining pretty dramatically in the next few months, which directly impacts people in IT.
  
• Declining dollar. As the economy becomes more distressed, calls will be made increasingly to cut interest rates. The problem with that is that the dollar's value is declining relative to other currencies as rates go down, making American goods and services cheaper ... but also making American companies cheaper. This will reduce the attractiveness of outsourcing (a definite plus for developers, who've often had to compete with foreign developers) but will make it more likely that companies in other countries may start outsourcing to American IT workers. On the other hand, this will likely only happen once American wages become competitive with those in countries like China or India, which means that programmer's standards of living will likely be dropping for some time before you see a massive influx of capital, and that capital will likely mean that the "parent company" of your company is not even in your hemisphere.
  
  Note that the carry trade in currencies will likely intensify as people move money out of both hard assets (short term, commodities will likely stall somewhat, though the long term outlook is still somewhat bullish) and financial assets (nearly all classes, long term) but seek even some returns. The downside is that this will likely lead to greater instability in the currency markets, with the possibility that one or more countries could "crash" - the US being at a very real risk of this.
  
• Energy costs. IT has generally been a net consumer of power, though that's changing. In many ways, IT has been working towards applying technology to reduce their energy footprint for financial reasons rather than ecological ones, but of all departments in a given company, IT is often the "greenest". On the other hand, a significant amount of energy costs are due to a declining dollar (along with considerations for peak oil and other geopolitical issues) which means that the relative cost of oil (and hence gas) will affect the prices of goods that need to be transported (nearly everything at this stage) and hence will mean that computer infrastructure (which has generally been declining at a rate consistent with Moore's Law) will likely start getting more expensive again relatively speaking (though it may end up spurring additional investment into local hardware production infrastructure such as chip fabrication plants).
  
  The cost will also accelerate the decade long trend of distributing the corporations into islands of loosely connected individuals who may actually work half a world away from where they live. If your workers are spending four hours a day on the road and are forced to pay an increasing premium for the privilege, they will go elsewhere where they can telecommute instead.
  
• No (immediate) bloodbath in IT. Corporations are going to be considerably more loathe to trim their IT staff, in part because that staff has only begun to recover from the last bloodletting, and in part because in many cases existing infrastructure has been pushed for about as long as it can be, so the imperative for improving core infrastructure will likely remain strong even in the face of slowing sales. On the other hand, its unlikely that you will see a lot of companies do anything beyond low cost pilot projects for R&D, especially after the first quarter of next year. As the recession spreads to other industries, I think that IT could be vulnerable on an industry by industry basis, but some of that may be mitigated by investment from foreign investors seeking firesale prices on companies.
• Shakeout of vulnerable companies. There are a fair number of mid-sized IT companies in certain sectors that managed to survive the Tech Winter, but that are increasingly becoming vulnerable. Even a year ago, these companies would likely have been bought up by larger or more successful companies, but as the available moneys for buyouts dry up, what will more likely happen is that the less successful of these will be forced into bankruptcy instead, without the "Get Out of Jail" card that seems to be the privilege of companies in other sectors.
  
  The irony is that a lot of new startups may actually end up surprising relatively intact. While there is some fluff in the startup space, it never reached the level of 1999, and a lot of the companies that started in the last five to six years are generally very lean (and fairly heavily open source-based, from all indications) and for the most part are considerably more pragmatic about the market. On the other hand, there will be relatively few IPOs for the next several years out of this sector (though this has been a trend since around 2000).
  

In the face of all of this, what can the astute developer do? Having gone through this a few times now, I've found the following strategies can help.

• Save money when you can, and reduce spending on extravagances. If you currently have any debts (especially credit card debt) pay it off and get out of those cards, highest interest first. Debt ties you down and reduces your cash-flow, which will be threatened at the best of times.
  
• Stay as liquid as possible, and if possible diversify your funds out of US dollars, at least for the next year or so. If you can land one or more clients that are out of the US, even better, though try to negotiate in funds out of US dollars if at all possible. It's likely that 2008 won't be quite the rout for the dollar that 2007 has turned out to be, but it is also likely that you still will likely turn out better in foreign funds, even with conversion rate charges.
• On a similar note, for the short term, if you are invested in equities, resource and energy stocks will likely be the most solid for the foreseeable future, though you are looking there less for significant gains but to minimize losses.
• Diversify your client list as much as possible - and be fairly ruthless in negotiating. They will be. Assume the likelihood of a default by one or more clients is at least a possibility, and plan accordingly ... its possible that all of your clients may go belly-up at once, but in most instance, if you have three or four clients, you should be able to maintain steady work. On the other hand, work hard to deliver as promised ... the ones that do survive generally are ones you want to keep.
  
• If you do find yourself between clients, spend that time getting caught up on skills that you didn't have the chance to work on while you were too busy with clients. Unless you are fairly new to the programming field, you're best served by increasing the depth of your skill rather than the breadth, unless the shift to the new field is part of a larger move into that technology as a base. Be thinking a year ahead in terms of where tech will be.
  
• In boom-times, programmers generally find recruiters to be pests. In bust-times, recruiters find programmers to be pests. Find a few good recruiters at various companies and get to know them, even refer people to them when you're offered a job you can't or do not want to take. They'll be more inclined to use you when times get tough.
• Recessions generally happen for reasons that you have little or no control over, but it is easy to become stressed, bitter, depressed or angry during those times, often making it harder for you to get motivated to work or look for work if there isn't any. It's worth remembering that programmers in general usually tend to have very desirable skill sets. If you have the time, join and contribute to open source projects - not only will you keep your skills up to date and work on interesting (and occasionally even useful) projects, but you stand a better chance of making contacts, developing your interpersonal network and gaining early proficiency in new technology by doing so. It will also let you combat the tendency to want to crawl up in a ball and whimper all day. Indeed, staying connected in general is a good idea.
• Keep alert and informed. In general, diversify your news sources (especially economic and political) and verify what's going on with one from another. Assume that no one knows the exact timing of when the "recession" or "recovery" first hits until after it has happened.

This was not meant to be a doom and gloom post. The coming recession is much like a Class 5 hurricane - it's coming, it will almost leave quite a trail of damage in its wake, but it will also have cleared the air enough for some major reforms to get instituted, and life will go on.

-- Kurt Cagle

Move to Metaphorical Web.org

Trying to stay current on the web can prove a challenge for anyone, but if you publish a blog, you're likely to find making sure all the links that you have out there are in fact pointing to the right place one of the biggest challenges. I've recently moved Metaphorical Web to a new Drupal-based site:

www.metaphoricalweb.org

both because I needed the ability to customize the site to display code samples and the ability to set up and maintain web feeds, external links and a forum that I couldn't get from the Blogger environment. I hope that you join me there.

-- Kurt Cagle

Migrating Metaphorical Web

I have started using a new blog software provide for The Metaphoric Web, incorporating it under M. David Peterson's UnderstandingX*L sites at http://www.understandingXML.com. Please join me there. I will be redirecting the Metaphorial Web URL (http://www.metaphoricalweb.com) to that address by the first of the year.

-- Kurt Cagle

Take 2: Chaos and XML

When I wrote the last post, I didn't realize at the time that I was running about a 102 F temperature, and would end up spending the next three days in bed with alternating bouts of chills and sweating, talking in my sleep - it was not one of the more pleasant weekends I've ever spent. While the underlying concept of measuring complexity is sound, I erred on the definition of entropy. Entropy is not a measure of the number of states in a given system configuration, but rather a measure of the change in the number of states in a configuration over time - in a non-self organizing system, the potential number of states increases, consequently becoming more disorganized (or more properly, converting into heat).

The concept describing the log of the number of states itself is the multiplicity of a system. Multiplicity isn't of course the only measure of complexity, but its a pretty useful one. Think of the traditional model of chaos - the conversion of laminar flow to turbulance. Typically there the first state is simple laminar flow - each water molecule flows in a straight line. As the flow moves faster, interactions with the surrounding media become more prominant, causing first a split into two streams, then four, then eight, until eventually there are thousands of such substreams, and the water becomes turbulant (this is the typical model described by Lorentz equations, by the way).

I have to wonder, though, if in fact there is some underlying connectivity between language (whether human based or artificial, as in the case of XML) and chaos theory, by this same modelling. Specifically, as the multiplicity of a document schema increases, so does the turbulence induced by that schema. I'm not sure what that the interpretation of that turbulence might be, though informational noise would be an obvious candidate. A simple schema is unambiguous - the specific meaning of a given aspect (element content or attribute value) is clearly defined, within its own constraints. However, as the number of aspects increase, their potential for coupling increases, for two properties to both be dependent upon a third (potentially undeclared) property, or two elements may in fact describe the same property in different and potentially conflicting, ways. Ambiguity is a form of informational noise.

Keeping this short (for me) tonight. I will be migrating in the near future to a new website, though it should still be accessible via http://www.metaphoricalweb.com. Until next time ...

XML and Entropy

Lately, I've been spending a lot of time reading outside my usual diet of programming books,in great part because I find that inspirations often strike when you can think about different endeavors and how problems were solved in those. One particular book that has set me to thinking has been Minds, Machines, and the Multiverse: The Quest for the Quantum Computer by Julian Brown, an intriguing discussion both of quantum computers and the multiverse interpretation of quantum probabilities.

In one section, Brown discusses the role of thermodynamics in information theory, and more specifically, the role of entropy. Entropy is one of those concepts that has gained, over the years, and almost mystical aura about it, the basis for all Murphy's Laws, but in point of fact it is actually a pretty simple concept to understand - and it has definite applications to one of the central problems that I see with how business people utilize XML technologies.

Entropy, in its information theory form, is a measure of the total number of states that a given system can be in. For instance, think about two bytes ... this can hold up to 65,536 possible states. Typically entropy is measured as the logarithm (and in the case of information sets, the logarithm base 2, or log₂) so that the total entropy of that system of bits would be log₂(65,536) or 16, which is, not coincidentally, the number of bits in two bytes.

One of the challenges faced by IT professionals working with XML is trying to figure out which tools work best for the scope of XML you're going to be working with. DOM or SAX manipulation typically does not handle complex XML well, XQuery is perhaps better for slightly more complex XML but lacks the recursive templating structure that works best for documents. The difficulty comes in determing at what point an XML resource shifts from one area of complexity to another.

Every XML document has a schema that describes the structure. If you make a few basic assumptions, you can in fact get an idea about the number of states that the schema allows:

• multiplicities of an identical structure count only once if the upper limit of such multiplicities is "unbounded",
• PCDATA data is immaterial, whether as text or as attributes. However, an attribute with multiple NMTokens will be treated as having one state for each enumeration.
• alternatives within the schema will each be treated as separate trees for determining the total count of states.
• If a given element can contain another element of that same name, then the count stops at that second element. This avoids infinite recursion.

By this measure, a schema with no variability would have an entropy of log₂(1) = 0, a schema with one element of variability would have an entropy of log₂(2) = 1, two elements of variability would be log₂(3) = 1.585, and so forth. By this measure, most business documents (invoices and so forth) would likely have entropies in the neighborhood of 0 to 4, XML processes (such as an XSLT transformation) might be in the neighborhood of 10-12, and literary documents might have entropies in the neighborhood of 15-20. Keep in mind that these are logarithmic values - an entropy of 20 would correspond to 2 to the 20th states, or roughly 1,000,000 possible schema instances.

Entropy is important because it can better clarify the domain at which it is best to work with a given document. XQuery I think provides a good case in point here. XQuery supports XPath, and so it has some of the advantages that XSLT has, but it's not really all that useful for dealing with documents -- converting a DocBook document into WordML or vice versa would be impossible in XQuery, but for many business schemas with comparatively low entropies, XSLT is definitely overkill.

Lately, its become somewhat fashionable, especially among the business set, to deprecate XSLT in favor of XQuery for enterprise level applications. I see nothing wrong with this. XSLT is not always easy to work with, requires a different way of looking at a set of problems, and is probably not worth the effort or overhead for many types of transformations. That does not mean that XSLT does not have a very important place in the ecosystem, and its my own personal opinion that thinking that you have to choose either/or will limit you when you do have to deal with high entropy documents, as is the case with most document management applications.

The Business Case for XSLT 2.0

by Kurt Cagle

In my previous posting (Imports and Tunnelling in XSLT2) I started down a path that I've been planning on covering for a while: presenting a solid business case for migrating to XSLT2. When I first encountered XSLT, after an initial period of attempting to understand the paradigm, I found myself both impressed and disappointed. XSLT is an often underrated technology, in great part because it doesn't fit cleanly into the Algol-based model that is most commonly used today (C, C++, C#, Java, etc.).

I consider XSLT something of a jujitsu language - it is most effective when used sparingly, letting the XML itself do the heavy lifting with the XSLT providing just enough of the pivotal support to do incredible things. That's why it has quietly become the silent partner on any number of different platforms as XML becomes more pervasive on those platforms. It is used within any number of Java and .NET applications, just another piece of the infrastructure, though one that does a disproportionate amount of the real work in applications that are increasingly driven by dynamic GUIs and web services.

Yet what disappointed me about XSLT, especially the more that I had a chance to play with it, was the fact that it was deliberately shackled by its conception as an XML translator to HTML. You couldn't manipulate non-XML text with it, could only do a fairly limited number of strong manipulations (in a language that was, fundamentally, parsing text), you couldn't create intermediate nodes for processing, and things that should have been fundamental - the ability to create an indexed for-loop, for instance - necessitated some very ugly recursion that added considerable complexity to the language without a lot of reward.

I wasn't the only one who found this to be the case, by the way. Indeed, many developers have come to XSLT for its potential capabilities but found themselves so bogged down with the verbosity and complexity of XPath manipulations that they would soon beg to find some other, easier solution. This has, in turn, created something of a backlash to the language, and more than a few projects built around XSLT have consequently become management nightmares, because few developers wanted to develop the expertise to debug seemingly incomprehensible stylesheets, especially given that it fell into the "declarative ghetto" where salaries were often lower than for procedural programmers because of the bias to see XML expertise (and consequently XSLT) as being simply an extension of HTML expertise.

This motivated me to follow the development of the "next generation" of XSLT, with the hope that it might prove an improvement over what currently existed. XSLT 1.0 was not so much broken as incomplete, though there were some fundamental changes that needed to be made to the data model in irder to accomodate the additional changes. Thus began an arduous trek following the development of XSLT 2.0.

By the time that XSLT 1.0 came out, James Clark, the iconoclastic genius who created XSLT in the first place, had shifted his attention away from transformations and into schemas, eventually laying the groundwork for Relax NG. Meanwhile, Michael Kay, the author of the authoritative XSLT books for Wrox and the creator of the Saxon XSLT processor, took over the editorship of the XSLT working group, working in conjunction with people such as Jeni Tennison and Dimitre Novatchev to establish both a set of extensions to XSLT 1.0 under the EXSLT.org banner and ultimately a proposed XSLT 1.1 Working Draft by mid 2002.

However, a number of realizations about the depth of the problem with the data model (and consequently of XPath, which relies heavily upon this model) forced the withdrawal of the XSLT 1.1 Working Draft from the W3C and the formal establishiment of an XSLT 2.0 working group. The goal of this group was simple -- to do XSLT right, to fix some of the biggest problems of XSLT that came from being based upon certain intrinsic assumptions and to revise XPath so that it would be robust enough to handle a much wider range of problems.

Not Your Father's XSLT

The language that is emerging bears a number of broad similarities with XSLT 1.0, but underneath it is a considerably more sophisticated vehicle. Perhaps the biggest change has come in the introduction of sequences. A sequence is a linear list - a set of XML objects, numbers, text strings, and other objects that can be ordered and otherwise manipulated. In XSLT 1.0 (and more specifically within XPath 1.0), you could only work with node lists, and even though such lists could hold text content (what are called text nodes) these were still containers for content rather than the content itself. By generalizing the node-set into a sequence, several things were made to happen:

• Sequences could hold references to multiple distinct XML trees, something which was included as function in XSLT 1.0 (the document() function) but not in XPath 1.0.


• You could create temporary trees of XML from other XML operations, using this intermediate XML as a way to perform other transformations. Most XSLT implementations had implemented a "fiat" way of doing this (a node-set() function, typically) but the implementations varied and the underlying model was incompatible with this.


• Sequences made it much easier to eliminate duplicates and perform other logical operations on XML data, such as grouping (something that can be fiendishly difficult with XSLT 1.0).


• Sequences made it possible to create iterative loops (something analogous to for(i=0;i!=n;i++){do something with i;}). The to operator in XPath 2.0 let's you create constructs such as (1 to 5), which generates the sequence (1,2,3,4,5).


• Sequences also lay at the heart of another critical requirement for XSLT - the ability to parse strings into constituent pieces.

Once this foundational step was laid, the next stage in the process was to build up from that, adding new capabilities while still trying to retain as much of the power of the old standard as possible. This came about through the introduction of other innovations into the XSLT and XPath standards:

• Regular Expressions. Regexes (as they are often called) provide a powerful tool for both searching and manipulating patterns within text. XPath 2 now incorporates the Perl regular expression engine (with some minor additions), making it possible to convert text files into sequences and from there into XML documents. This key capability makes XSLT a particularly noteworthy candidate for handling compiling, something that will be discussed later.



• User Defined XPath Functions. The XPath Working Group established a formal mechanism for binding external functions into XPath, providing a clean, consisten means to build new functions XPath that could be written in C++, C#, Javascript, Java, Perl ... and XSLT. This dramatically reduces the amount of code necessary to invoke XSLT named templates (often by an order of magnitude or more) and also makes it possible to migrate XSLT from a Java based system to a C# based one without needing to change any XSLT - you'd just rewrite the external functions but keep the same function signatures.



• Conditional Expressions. With XPath 2.0, you can now write if/then/else and for() statements within XPath, making it possible to create much richer logic into the language. Not only does this also reduce the amount of verbosity in the language significantly, it also makes it possible to solve what was typically not even possible in XPath - such as adding taxes and discounts into item costs in an invoice before finding a total.



• Date/Time Manipulation. Date and time manipulation was something of a nightmare in XSLT 1.0, yet because of the importance of such information in transformations, there was a thriving industry in building work-arounds. Now such capability, including finding the difference between two dates or times, is built into the language.



• Complex Grouping. The data model in XSLT 1.0 made it very difficult to handle certain kinds of groupings, such as mapping the relatively flat structure of HTML to the group and section model of XSL-FO or DocBook. With sequences and regular expressions, generating such groups is now possible, especially in conjunction with certain additional XSLT 2.0 elements.



• Multiple Outputs. XSLT 1.0 was assymetric -- it was possible to pass in multiple XML documents through parameters, but it was not possible to produce more than one formal output. That's changed with XSLT 2.0. Now, you can write transformations that will generate any number of XML or text output formats, either to be saved to local storage or to be sent to external web addresses, depending upon security restrictions.



• Type Awareness. Perhaps one of the most contoversial aspects of XSLT 2.0 and XPath 2.0 is the introduction of schema-aware tranformations which were capable of validating and manipulating typed XML content from external XML objects. This is not an intrinsic part of the specification, however, so it is less likely that all XSLT 2.0 processors will be schema aware.

This combination of features fills most of the holes left from the XSLT 1.0 implementation and makes it possible to start thinking about XSLT sitting not just along the periphery of your operation, but right in the middle handling the processing of business logic.

XSLT For Businessmen

In a typical business, you buy or implement business software in response to your changing business needs. Much of this software is anticipatory in nature - the designers of the applications attempt to model ahead of time the scenarios that are most likely to occur in your business, and then build the business logic for these scenarios into the code itself.

Anticipatory design has a number of side-effects, few of them positive. For starters, the applicability of the software becomes a measure of the degree to which the application designers were successfully able to model the business processes that occur. When the modelling is close, the application integrates well into the work flow of the company. When the modelling isn't so close, the company is all too often forced to adapt to the workflow of the software, which introduces inefficiencies.

Moreover, over time, a company's business requirements change as the business itself changes. However, the software has almost certainly been written by someone who is no longer writing that particular piece of software -- best case scenario is that they are working on some other part, and consequently have to stop what they're doing to change the code. Worst case scenario is that your lead programmer is India (unless you are in India, in which case your lead programmer is in London), has long since left the company, and likely didn't document the code terribly well. Thus, over time, the software decays, until it is no longer useful to the company, forcing another massive expenditure into the whole in your business called IT Expenditures.

Finally, many such solutions are intimately tied not just to a particular operating system but a particular machine, and should something happen to that machine, your company could be left with a major problem.

All of these situations point out the limitations of anticipatory design, but such design is still the most prevalent because it 1) keeps software vendors in business, 2) keeps consultants in business, and 3) ultimately forces hardware upgrades, keeping hardware vendors in business. Of course, unless your business is specifically dedicated to keeping these three groups in business, such design often becomes a hidden tax on computer usage, a constant drain on expenditures that becomes very easy to accept as unavoidable. However, that cost really isn't as necessary as it may seem.

One of the great benefits of XML is the fact that it's use tends to encourage adaptive rather than anticipatory design. With adaptive design, the business logic of a company can be readily encoded in an easy to manipulate bundle of information which can work across any platform. Your code can generate your user interfaces in response to changes in data requirements, passing that information into transformations that can readily encode the business logic. Moreover, even the transformations themselves can be transformed, can be designed to change as business parameters change. In short, such systems, adapt to the changing requirements of the business.

XSLT 1.0 was an interesting first step in this process, but all of the points mentioned above - the complexity of the language, the verbosity of the code, and the often counterintuitive techniques necessary to handle frequent operations made it less than idea for this particular process. However, XSLT 2.0 is considerably simpler to follow, write, and maintain, can more reliably integrate with external processes and objects, and is able to handle multiple possible input and output forms at once.

As tools such as XForms (or some equivalent XML centric forms technology) becomes more prevalent, this also means that interface tools (and not necessarily just "web tools") will increasingly generate and send XML content directly rather than the more limited name/value pairs of HTTP (in essence what SOAP does via the agency of web services), and in general XSLT is a better tool for manipulating and extracting information form XML sources than DOM tools are ... if that extracted information is itself in XML format. In that respect, DOM can be thought of as being a binding mechanism that connects XML with other object representations (that is, other programming language data structures).

This use of XSLT within XML millieus is an important concept, with very broad implications. XSLT is not sexy. There are no marketing teams out there who put out multimillion dollar ad campaigns featuring well-coifed executives staring raptly at XSLT code on their laptops. Instead, XSLT is an infrastructure sort of thing, found deep within (a surprising number of) applications, increasingly taking over the task of document and object conversions that for years had been the domain of heavily sequestered filter writers. The application I'm writing this on right now, an HTML editor which I wrote, uses XSLT to convert between an editor component's internal representation and one of several XML formats -- including docBook, XHTML, Microsoft Word 2003's XML format and others. Yet without knowing that, you'd never even be aware of how critical that technology is, because it does exist so quietly.

Code Building Code

XSLT 2.0 will likely become much more pervasive, because it's domain of applicability is so much broader and because much of the design of the second version of the language is deliberately built around the manipulation of any textually represented object -- including other programming languages. Most programming languages have a very well-defined programming structure, independent of the commands themselves -- packages, classes, constructors, destructors, methods, properties, event handlers -- in most cases there are relatively few variations off of these basic entities, in great part because programming languages are process descriptions (at least imperative languages are).

XML in turn is a good language for the description of descriptions, and as a consequence, it can very readily incorporate larger functional blocks of code in a descriptive matrix. Once in that format, generating code in other languages becomes much easier using a language such as XSLT2, especially with the addition of regular expression tokenizing parsers. On the flip side, XSLT2 is also remarkably good at the inverse process -- parsing program language blocks and converting them into an XML representation. In short, XSLT2 could find itself becoming what amounts to a universal compiler/decompiler, at first into intermediate forms such as Java or C#, and then with increasing frequency, directly into bytecode or intermediate language (IL) generators (this is especially significant of C#, which already maps many of its languages into a common IL format).

From an application standpoint, this raises the real possibility that this next generation of XSLT2 could in fact not only handle the processing of business logic, but actually generate just-in-time compiled code to more quickly execute such logic, and could perforce route it to the locations where such JIT code would be needed. User interfaces could be built on the fly, something especially critical in business applications where a significant portion of the programming work (and hence cost) that takes place is oriented to developing the screens whereby people interact with the data. The combination of XSLT2 and web services can also abstract connections into SQL and LDAP repositories, meaning both that such data sources become more interchangeable and that the way of accessing something as different as a relational database and an LDAP directory becomes irrelevant.

Finally, XSLT2 simplifies the way that information moves within your enterprise, ironically by moving away from what had been the cornerstone of programming in the 1990s - the object-oriented-programming paradigm. One of the difficulties that has emerged from OOP principles has been in determining the decomposition of a problem space into distinct classes. Programmers new to OOP (and in all too many cases not so new) have a tendency to want to model everything as a class, and as a consequence their application code begins to resemble the Harry Potter series -- full of wonder and magic, but with entirely too many pages for what is, fundamentally, a children's story. The problem with this is that each class has to be written and tested, not only in isolation but also in tandem, and a seemingly trivial change in a base class can have profound consequences for other classes built upon it.

XSLT 2.0, on the other hand, shifts the approach taken from building up this complex zoo of class critters and pushes it back towards an approach which is coming back into vogue with the advent of Linux: streams and pipes. A stream can be thought of as data moving between two processes, whether those processes be a file reader, a web server, a program, a database, a web client, or the interface to some piece of hardware. A pipe on the other hand, is the conduit which carries the stream. The OOP revolution placed a huge amount of significance on the node where pipes met, and tended to relegate the pipes and streams to secondary status, at best.

However, XML and the web is changing this. One effect of XML web services is to envision programs as being the transmission of streams of data to distinct end-points, URLs, without necessarily caring about what happens within that end-point. An object orientation gives many more points of access into an object, but typically at a cost of dealing with that object's specific data restrictions. With a web service, I can send a stream of information to a URL, and the process at that end will choose (if it is well designed) to either determine that it is valid and usable (there are processes that are designed to work with that stream at that node), that it is valid but not immediately usable (it is sent off to a different process which will attempt to rectify it into something of significance to the first process) or that it is invalid (whereupon notification of this lack of validity is sent back to the sender).

XSLT2 can handle all three potential conditions (though the case where the data is not well-formed XML gets a little complicated). Well formed XML has an associated namespace, and this namespace can actually be used by the XSLT itself to determine the proper handling of XML content, perhaps with the concept that such an XSLT could pass on the parameters acting upon the transformation into part of a SOAP message and then routing that message to the appropriate final transformation. in a purely asynchronous model (one where each node can act both as a transmitted of XML and a receiver of XML under separate processes), the routing XSLT does not have to be the XSLT that handles the final processing of the associated data -- or the one that communicates back to the client. While this model doesn't quite work in the fairly strongly synchronous browser model that most people connected to the web currently use, contemporary web clients are in fact shifting to an asynchronous model where it will work just fine.

An Open Letter

If XSLT2 is such a superior technology, why has it raised the ire of a few of the largest software vendors (most notably Microsoft)? This is in fact a case of the question providing its own answer. XSLT 2.0 provides an answer to many (perhaps most) of the complaints of XSLT 1.0, including the most damning ... that it is too conceptually difficult and verbose for programmers to learn. XSLT2 is more functionally oriented than XSLT1, making it easier for programmers more comfortable with languages such as Java or C++ to use.

XSLT2 also binds to external classes much more transparently, making it much easier to communicate with external processes within the environment, regardless of what that environment is (or what kind of box that environment is running on). It doesn't require an expensive suite of tools, compilers, and libraries of objects to work with it, and it is fundamentally oriented to manipulating XML data (though not exclusively) without the strong-typing limitations that come with Algol based languages.

XSLT is also considerably more secure, based upon what I'd call the potency argument. In essence, most binary objects contain not only their own state but also the mechanisms by which that state gets expressed to the outside world. In essence, these objects are potent - they have the ability to create side-effects that may not be obvious from their interfaces, as they have to have a certain level of interaction with the system in order to be able to function. In effect, the level of trust that these objects require of the system simply in order to operate is too high, forcing the creation of human trust authorities.

With XSLT, on the other hand, the streams of XML information coming in are pre-potent. They provide a description of state, but are reliant upon the XSLT that is resident within the server to handle the manipulation of that state, and correspondingly to specifically provide exceptions for handling things outside of the boundaries of safe behavior. It is consequently incumbent upon the maintainer of the system to choose the interpreters of that data, rather than placing the security demands upon the (non-technical) users of the applications.

Given all this, XSLT2 enabled systems could serve to signficantly erode the requirements for complex components that sit at the heart of most operating systems, a potential boon to open source systems such as Linux but one that could dramatically impede the ability to effectively sell subsequent implementations of Windows (or so at least one line of reasoning goes that I've seen bandied about). It makes code generation a just-in-time process and so effectively blurs the distinction between data and code, a distinction that Microsoft still makes even as it defines its own XML user interface language (XAML, which requires a healthy dose of C# "code-behind" in order to do more than trivial applications).

Microsoft has chosen to include XQuery 1.0 (a data-binding language that builds on XPath) but not XSLT 2.0 in Longhorn, citing everything from lack of customer interest to complexity in implementation to insufficient maturity on the part of the specification. They have even gone so far as to try to develop an alternative language, C Omega, which is supposed to provide a C# oriented approach to manipulating XML.

I've played with C Omega some - it is a reasonably good way to avoid some of the tedium of working with the W3C DOM, and it is certainly possible to use it for some of the same purposes that you'd use XSLT2, though it lacks the powerful recursive templating capability that I think gives XSLT most of its power. It presupposes that the appetite for XQuery will be strong enough that they can essentially build a hybrid language around it, though after having written two books on XQuery that have between them garnered less than the production costs for the books in question I'm much less inclined to agree, especially as XPath2/XSLT2 becomes much more functionally oriented.

At the last Sells Brothers XML Conference (which I would heartily recommend, by the way) I gave a talk on Saxon.NET, an open source project in which M. David Peterson has converted Michael Kay's superb Saxon 8.0 XSLT 2 implementation over to .NET, with Dr. Kay's approval. I'm using it now for a content management system, and it has performed far better than I had even hoped. At any rate, when the Microsoft representatives at the conference later asked at the crowd whether they would rather have work on XQuery or XSLT2, the number of people (in many cases customers of Microsoft) who wanted to see a new XSLT outnumbered those of XQuery by a considerable margin.

While I strongly support Mr. Peterson's efforts, I also would like to make a plea to Microsoft to reconsider your stance on this. I believe that the demand for a more powerful version of XSLT is out there, and that it is being driven by application developers who are building applications for Windows as well as elsewhere. It will become the de-facto standard within your competitors' business productivity suites, web clients, home entertainment applications and operating systems, because if you choose not to develop such a processor, others will provide .NET versions that will be used in place of your own offerings. You will have already done most of the hard work in implementing it, as the major portion of the changes that occur in XSLT 2 is due to the revision of XPath 2.0, which you are already developing to support XQuery.

To business decision makers reading this, chances are really good that you will likely never actually have to sit and look at a screen of XSLT 2. However, as with XML six years ago, XSLT 2 is a technology that will likely end up shouldering much of the day to day processing within your organizations over the course of the next five years -- it is a natural complement to XML, which has, like kudzu, pretty much taken over the data infrastructure of most applications it comes in contact with.

There's another factor that comes into play here from a business perspective. In 1993-4, an independent consultant could earn $250 an hour writing HTML content. Today, HTML is considered a basic skill that every graphic designer needs to learn as part of his or her doing their job, and HTML generation is mostly handled via automation mechanisms. XSLT serves many of the same purposes that tools such PHP, ASP.NET, Perl, and JSP serve today, but as the world continues its adoption of XML as the standard for conveying structured content, XSLT is becoming something of a lingua franca - a common tongue - that developers in this space are learning, and are finding in that learning that by intelligent application of that XSLT that particular skill is cleanly transferable from a Linux box running Apache using PHP and a Windows box running IIS and ASP.NET.

XSLT 2 is not a new language - it is XSLT cleaned up to handle the areas it should have been capable of handling before, with much less verbosity, more integration, more power and a considerably easier development and maintenance path. This means that the learning curve for those developers going from XSLT to XSLT 2 will be much less extreme than having to learn another language in toto. This in turn means that within a space of a couple of years, if not less, XSLT2 will likely be just another core skill that a developer should have, yet one that helps them write platform and language neutral code for dealing with all of the XML that is even now circulating through your business. With skilled programmers in any area once again beginning to demand a premium, the coming ubiquity of XSLT2 skills should help keep your labor costs down not just in your web development department, but throughout your organization.

/EndTag

There's a tendency of late for writers in the XML space to want to play down some of the more recent standards from the W3C, a tendency to go "ho-hum, it's just the sequel". I think this attitude can blind people to what is actually happening. XSLT 1.0 was a uniquely different and powerful solution, and after having worked with it on an almost daily basis for nearly the last decade my respect for its innovation has only grown. However, even four years ago I felt that it wasn't powerful enough, and the amount of customization on the part of XSLT processor vendors over the last several years to me is testament to that. XSLT 2.0 is not profoundly different from XSLT 1.0 - it's in fact almost completely backwards compatible.

It does, however, rectify the shortcomings that emerged from the first iteration of the language, and does so in a way to make it an astonishingly powerful language. History is full of such standards, such as SQL, where it took one or two iterations to handle the inevitable discovery process that is part of any great human endeavor. This tool, XSLT, is already becoming one of the core work-horses in most contemporary applications, even given that it was never originally conceived to do what it is called for. To move forward to a version improved by half a decade of insight and exploration is not only logical, it's good business.

Imports and Tunnels in XSLT2

Every so often, I discover that just when I think I've wrung everthing I can out of XSLT, it is still possible to do something subtle that changes my whole mindset about the language. To me, one of the hallmarks of greatness in a language is that very ability to surprise, to make even a somewhat jaded coder sit up and say "Hey, that's cool!".

I faced a challenge of that sort recently in an application I was working on. The issue that I was struggling with was that there are aspects to XSLT2 which are not quite polymorphic. Polymorphism, for those of you who might of slept through that particular aspect of OOP training, is the idea that you can have two distinct objects have a particular method which, when invoked, produce similar but not necessarily identical actions.

The canonical example of this comes with the Shape class, with the method draw(). If I have two child classes called Rect and Circle, each of which inherit the draw() interface from Shape, then I could create instances of these classes:

rect =new Rect(); rect.draw();

and
circle = new Circle(); circle.draw();

each of these objects will perform a draw operation relevant to its given class (drawing a rectangle or a circle, as appropriate). By creating such a set of conventions (and the notion of a consistent interface gained via inheritance) you can additionally work with a wide range of similar objects through the same interface without having to write special type handlers for each object.

The three pillars of OOP, inheritance, encapsulation, and polymorphism, can exist within XSLT, but only somewhat uneasily. Inheritance, for instance, can be gained by creating two stylesheets, the first containing a dominant set of templates, the second "replacement" templates that effectively replace the first if they have an equal or higher precedence match. The problem with this is that there is no real way of saying "utilize the replacement template, then invoke the original template", the analog to the super() function in languages such as Java, which invokes the parent's method when invoked the same method of a child object.

XSLT 2.0 does provide features to ameliorate this somewhat.The first involves a subtle shift (albeit one that can cause a lot of grief for XSLT 1.0 developers). In XSLT 1.0, imported stylesheets had a higher priority than importing ones. This made it possible to "subclass" templates easily, but provided no clear mechanism for being able to invoke the replaced templates.

However, in XSLT 2.0, the priorities have been switched, which means that the calling stylesheet's templates will have a higher priority than the called stylesheet's templates, unless the priority is explicitly set as an attribute as being different. This has the immediate effect of giving the illusion that imported templates don't work. That's not quite the case, however. Instead, such templates can now be invoked with the <xsl:apply-imports/> element. This causes the processor to only look at templates in the imported stylesheet when applying the rules, and it is this functionality that gives you "super()" like behavior.

For instance, suppose that you have a stylesheet that takes the value of an item, and if it is a number less than 0, renders that number in red (in the file accounting.xsl)

<xsl:template match="example">
<xsl:choose>
<xsl:when test="number(text()) != NaN">
<xsl:choose>
<xsl:when test="number(text()) ge = 0">
<span style="color:green"><xsl:value-of select="."/></span>
</xsl:when>
<xsl:otherwise>
<span style="color:red"><xsl:value-of select="."/></span>
</xsl:otherwise>
</xsl:choose>
</xsl:when> <xsl:otherwise> <xsl:value-of select="."/> </xsl:otherwise> </xsl:choose></xsl:template>

A second template for "example" in a calling document could handle placing this content into a table cell, using the <xsl:apply-imports> element, as follows:

<xsl:import href="accounting.xsl"/>
<xsl:template match="example">
<tr><td><xsl:apply-imports/></td></tr>
</xsl:template><xsl:template match="examples"><table> <xsl:apply-templates/></table></xsl:template>

If the source XML document had the form:<examples><example>125</example> <example>-150</example><example>Twelve</example></examples>

Then this will render as:<table><tr><td><span style="color:green">125</span></td></tr><tr><td><span style="color:red">-150</span></td></tr><tr><td>Twelve</td></tr></table>
125
-150
Twelve

Significantly, if no import document is specified, then this will act in the manner appropriate for <xs:apply-templates/> for processing child elements and text of the current context. In other words, without the imported stylesheet, this will render as:

<table> <tr><td>125</td></tr> <tr><td>-150</td></tr> <tr><td>Twelve</td></tr></table>

This fairly subtle change will have an enormous impact upon the way that you develop stylesheets, especially in conjunction with a number of the other features that are now in play, including the next bit of subtlety - tunnelling parameters.

Digging Those Tunnels

You have probably noticed that while its possible to talk about applying OOP techniques to XSLT, such concepts do not necessarily translate cleanly in a one to one manner to how XSLT itself works. One instance of this comes from the problems inherent with dealing with globals.

Global variables are bad in the way that cookies and sweets are bad ... one or two, taken sparingly, can improve your code without too many ill effects, but go overboard and you'll be seeing the results show up in bloated code (or at least a bloated waistline).

Normally, in OOP code, you define variables that are appropriate for a given class, and oany methods that are defined within that class have the variables available to them. You can also create public properties which appear like "public" variables that are available from outside the class, but in most languages, these "public" variables are being replaced by getter/setter methods that expose specific properties while at the same time providing a certain degree of insulation on these variables. Additionally, regardless of scope, variables defined in this manner may also be available to any derived class of the base class, through some variation of the friend keyword.

In XSLT, things work a little differently. You can define a global variable or parameter within the <xsl:stylesheet> element of a transformation, in which case the variables are available anywhere within the transformation. However, if you import a global variable from an external template into your transformation, that variable or parameter will be overridden by a variable in the calling document if they have the same name. In some cases, this may be desirable, but in others, the calling template may have defined these variables for handling some internal state within the transformation, at which point you're feeding gibberish into your transformations (and usually very hard to debug gibberish, at that, as the code is syntactically correct).

You could, of course, not use globals in the called routines (a good practice anyway) and instead choose only to define the parameters within the likely entry point of the called stylesheet and then pass them down from template to template. Unfortunately, this solution tends to be rather verbose, especially when dozens of such "internal variables" are needed. Especially if the called templates didn't require the parameters, this particular technique was really only worthwhile if you absolutely needed that data at some point.

In XSLT2, a different approach was taken. A new attribute, the tunnel attribute, is placed on parameters in matched templates (i.e., <xsl:template match="foo") elements, with tunnel = "yes" indicating that the parameter should participate in tunneling, and tunnel = "no" indicating that the parameter is not tunnelled. An apply-templates parameter in turn would require that the tunnel = "yes" attribute be added to the <xsl:with-parameter> element in order for that variable to tunnel.

So what exactly is this tunnelling? In essence, a tunnel indicates that if a template calls another template that does not have the parameter declared (and consequently does not use that parameter), and this template in turn calls a third template that does have tunnelled parameters, then the third (or fourth, etc.) template acts as if it was called with the parameters from the first template. This way, the only templates that receive the parameters are the ones that actually use them, and you are able to keep your code lean in mean in the process.

As an illustration, the following illustrates a "tunnelled" hello,world type transformation:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:template match="/">
<xsl:apply-templates select="foo">
<xsl:with-param name="username" select="'Aleria'" tunnel="yes"/>
</xsl:apply-templates>
</xsl:template>

<xsl:template match="foo">
<html>
<head>
<title>Hello, World!</title>
</head>
<body>
<xsl:apply-templates select="bar"/>
</body>
</html>
</xsl:template>

<xsl:template match="bar">
<xsl:param name="username" tunnel="yes"/>
<h1>Hello, <xsl:value-of select="$username"/></h1>
</xsl:template></xsl:stylesheet>

With the assumed input being:
<foo>
<bar/></foo>

In this particular case, the parameter "username" is invoked from the root template match for the "foo" template. However, foo has no need of the parameter, so it doesn't need to declare the parameter. The "bar" template, on the other hand, does need to use the parameter, so it creates a parameter declaration, including the critical tunnel = "yes". Without this, the XSLT processor would work on the assumption that the parameter was simply not invoked from the "foo" template, and as a consequence, the $username is assumed to be blank.

This form of parameterization can vastly simplify writing imported stylesheets. Typically such sheets will typically provide a single point of entry for subsequent transformations in the imported sheets(especially if the templates use the mode attribute), and rather than defining global parameters that could possibly be clobbered, the entry point template could define the parameters within its body as being tunnelled, making these parameters available to any member of that set of modal templates.

</wrapup>

h2>

Thus, in addition to making your code less verbose (and hence easier to both write and maintain) tunnelling can additionally serve to make your imported stylesheets more modular and classlike. In essence, this turns an imported stylesheet into something that has a number of the characteristics of classes within a more formal hierarchical language such as Java or C++, while at the same time retaining the templating architecture that makes XSLT such a powerful language in the first place.

Internet Time and the Long Now

I wish to thank Edd Dumbill for syndicating the Metaphorical Web on his superb blog aggregator site http://planet.xmlhack.com. It's an honor and a privilege, and I hope to be able to produce commentary in keeping with the luminaries on this board.

I've been pondering for a while the state of the XML world. XML has now been in existence as a standard for nearly seven years. XSLT and XPath for six, SVG for four and change. By the scale of Internet time (how retrograde that expression seems now) these technologies are no longer bright and shiny new, but are actually approaching middle-aged. Heck, Microsoft would have gone through two world changing marketing campaigns by now, and would be gearing up on the third. So why does it seem that XML is still just getting started?

Oh, you could single out all the usual suspects, of course. The W3C moving at a snail's pace, vendor lock-in keeping these open standards from reaching their true pace, the lack of marketing to push standards into the forefront of computing and so forth (all of which are true to a certain extent). However, over the last year or so another idea has occurred to me, one that is perhaps heretical in the world of nanosecond processors, yet something that has nagged and tugged at my consciousness for a while.

Daniel Hillis, the founder of Thinking Machines and an Imagineer for Disney for a number of years, has of late spent a great deal of time concentrating on another project of his, The Clock of the Long Now. This particular clock is unusual in that it is intended to exist for 10,000 years. To put that into perspective, 10,000 years ago Europe was beginning to repopulate after the last of the continent covering glaciers had retreated, and the dominant language was an early Mesopotamian ur-language which would in time branch into the Indo-European languages.

Thus, to build such a clock, you have to design it to be simple, to be self-maintaining (who knows how many centuries may pass between people even knowing of it) and to be intuitively obvious regardless of what culture finds it. In many ways, the Clock is the ultimate exercise in user interface design. It is also, in its own way, a remarkable statement about the nature of time itself and man's interaction with it.

So what does such a clock have to do with XML? In any period of intense innovation, a pattern tends to emerge, regardless of the technology in question. Most of this should be familiar as a variant of the adoption curve:

1. An innovation is created, usually a variant of some existing technology but with a twist that moves the technology in a completely unexpected direction.
2. "Hobbyists" emerge, the earliest adopters who work with out while the technology is still in rough stages. They often end up becoming the evangelists for that technology later on.
3. The technology attracts the attention of early investors, who see this as a way to start companies that exploit the technology. Such investors seldom come from the hobbyists, but they are keyed in enough to what where the hobbyists are most excited.
4. The technology goes into its hype phase, where marketers begin promoting it for everything from improving efficiency to curing the common cold. This phase usually involves the appearance of semi-nude women seductively wrapped around the piece of hardware, working at a computer with the software, or otherwise doing improper things with a technological implementation.
5. The backlash occurs as people realize that it does not in fact cure the common cold, may actual decrease efficiency in certain ways, and tends to be a big turnoff to attractive young women at bars (who are usually after the early investors, not the schleppy users).
6. Meanwhile, those people who utilize the technology for a living figure out whether the technology actually does meet a specific need of theirs, and in the process will usually provide feedback to strengthen the technology in question.
7. If the technology manages to get past this stage into adoption, its development shifts from paradigm shifting innovation to more stable improvements, especially if the same technology inspires other implementations/variations. This competitive phase usually lasts until one particular implementation manages to gain a 90/10 mindshare. This phase also usually sees the emergence of some body of standards that define interconnectivity of that particular technology (for instance, standard gauge sizes in railroads, standard socket types, HTML). These standards usually reflect the implementation of the dominant player.
8. The technology then is able to sit with comparatively minor changes for a significant period of time - years or even decades, but because they are based upon these standards, the dominant player also has the greatest to lose and least to gain in changing those standards to reflects shifts in new technologies.Standards that were based upon a given implementation consequently freeze that implementation, until eventually, the standard is out of sync with additional innovations.
9. The dominant standard-bearer can suddenly find its position upended very quickly -- within a couple of years -- and find that it is now in a position with an aging infrastructure and a massive installed base. It either abandons that base and moves on, putting itself into a much more vulnerable position, or it sticks to that base even as the base itself moves on to new technology. In either case, the technology may remain for a while longer in a kind of white dwarf state, slowly cooling to a brown cinder. Periodically, the technology may be resurrected for a revival (think of all of the coin-op video games which now exist as software emulations) but such technology holds entertainment value only (restoring an 1890s railroad train engine, not because it is even remotely competitive, but because it has sentimental value).

Given this birth to death cycle for most technologies, why does XML feel so different? My suspicion is that XML is different -- it is a technology for the Long Now. XML of course has its roots in SGML, a technology that was relatively obscure for most people outside of corporate librarians and academics but that can in turn be traced back to the work of Charles Goldfarb in the 1960s. This means that it in fact predates another "long-now" technology: the Unix/C duality. SGML was not intended as a means to gain temporary market share; indeed, the high cost of creating SGML implementations meant that it was really commercially viable only for the largest of organizations.

SGML is not an implementation-driven technology -- it was from the first a vehicle that required consensus, because it's focus was at the heart of document communication, which means that from the start it was intended to model the way that people think, not what is necessarily the best way for machines to store artificially rigid class constructions. Because it was a meta-language, SGML by its very nature was intended to be implementation independent, long before Sun's "Write Once Run Anywhere" slogan came into play.

In addition to this, SGML is declarative. It's operant verb is BE, not DO. By doing this, it was able to provide a level of abstraction that bypassed the 1000 class monstrosities that emerged over the course of the next three decades. That abstraction lives on with XML, to the extent that it is dramatically affecting both the theory and practice of coding.

XML, in turn, has been a refinement of this notion of working with abstractions through an abstract interface, with the underlying assumption that so long as the expected behavior that's agreed upon is met, that specific implementation is irrelevant. One effect of this has been the increasing dominance of XML abstraction layers that in turn push the imperative code -- the C++ and Java and Perl of the world -- down the processing stack, away from the business logic and closer to providing a common low-level substrate. XUL, SVG, XAML, XFaces, XForms, etc. all provide manifestations of this principle to some degree. Create a binding layer such as XBL, and you can hide the API even more, with the consequence that you can increasingly reproduce sophisticated sets of object code with XML representations. The imperative code never goes away completely (nor should it) but it becomes much less significant in the scheme of things.

As a consequence of this, while certainly there have been companies that have ridden some aspect of XML through the technology business cycle described above, I think that XML as a technology is acting very much like the rise of mitochondrial RNA arising during the Cambrian era - providing a mechanism that serves as a substrate for a whole new kind of life (or in this case for a whole new kind of programming paradigm). The problem from the standpoint of those in what had been the dominant paradigm - the framework based OOP system - is that such XML provides few of the features that traditionally make money for vendors -- lock-in of file formats, opacity of data structures, variations in access formats that provide advantages to one or another vendor, arcane API implementations, reliance upon a specific language, and so forth.

Web services make it possible to do things that would have been impossible otherwise (the dominant of which seem to be less providing unique data feeds and more performing transformations between varying schemas). XML based GUIs (and even full application sets) are now becoming the standard way of building human faces to applications, cutting down on the reliance of specific language toolkits. XML is even being used for discovery of non-XML APIs, something which usually indicates a transient phase (if you can discover non-XML APIs, you are invoking an abstraction layer within the interface, which in turn makes it easier to decouple the implementations from an imperative basis.

Microsoft has gone through several core changes of technology (and more marketing "initiatives") in the last 30 years, yet curiously few of them have a staying power beyond about five years. This is not a disparagement of their products (which for all of my grumpiness about Microsoft even I will concede they do produce very usable and professional products), but I think is more indicative of the commercial software vendor market (I could replace Microsoft with Sun or Hewlett Packard in the above statement and be equally correct.) Their definition of Internet Time gets conflated with their definition of Business Time, where next quarter is more important than next year, and five years out is an eternity.

Ultimately, though, Internet Time to me is something that isn't measured in nanoseconds and months between deployment cycles. If, as I do, you believe that XML is becoming the neural circuitry of the Internet, then I have to wonder if perhaps the real Internet Time is measured in decades, and maybe even centuries.

Indeed, when Danny Hillis finally gets around to inscribing the operating instructions of the Long Now Clock on its stainless steel base, perhaps, just perhaps, he should write it ... in XML.