November 29, 2007

CDF: The common format you've never heard of

Quick! Do you use the Compound Document Format?! You, know, CDF … surely you use CDF, right?

Chances are pretty good that you have no idea about what I’m talking about. Everyone knows Microsoft’s word document format and Adobe’s PDF, chances are pretty good that if you’re reading this on XML.com you’ve heard of ODF and OOXML, especially after the fairly rancorous discussions about ISO status for these two formats. Yet CDF, hmmmm … that’s a rough one. Didn’t it belong to Corel, once upon a time?

Okay, now, how about this one … do you work with (or even just read) XHTML? Probably, if you’re involved in XML work, your HTML conforms to a great degree to the formal XHTML standard. Good enough. How about CSS 2.1? Sure, who doesn’t. Okay, here’s a biggy -how about XMLHttpRequest? You do AJAX work? Good for you. XForms - well, that one’s a little less prominent, but yeah, it’s beginning to appear. SVG? Hmmm … again, kind of touch and go, but even after a few hard years SVG’s by no means dead yet. The occasional XSLT - Google’s doing some nice work in getting their Javascript based AJAXSLT up and running, which means that those few browsers that don’t support XSLT (and XPath) natively should be able to support it via an AJAX layer. Oh, and perhaps through in XSL-FO for good measure, as it continues its quiet but relentless march into becoming a mainstream format.

So, do you work with CDF? You betcha. The Compound Document Format was set up as a way of tying together at a minimum all of those technologies described above into a single cohesive whole. Put another way, it’s a fancy way of describing the core suite of W3C document standards into a cohesive whole, although it does place some fairly minor requirements on usage in order to provide a consistent standard.

CDF was in the news recently with the implosion of the Open Document Foundation, originally established to endorse ODF, though in its death throes it briefly highlighted the CDF format as perhaps a better format for documents than either OOXML or ODF. This is admittedly one of those areas where it may be justified in looking at XHTML especially and going “huh”? How can that be a full document format - it’s used for web pages, after all - you wouldn’t want to use it to mark up a full book, would you?

Document formats are a lot like religions - people are ready to defend them to the death if need be, yet at the same time it becomes easy to dismiss certain religions that don’t even seem to be religions at all (such as my personal favorite, the rather philosophical Tao). Could you mock up a brochure in XHTML and CSS? Actually, it turns out that its surprisingly easy to do just that - especially if you throw a little SVG into the mix and allow the possibility of embedding XHTML within SVG (for all those odd little bits of rotation and other special effects).

How about linking between blocks of physically disconnected content? That’s what hyperlinks are for, after all. CSS gives you at least a dozen different unit references, lets you control borders, margins, padding, kerning, image placement, color management, and so forth. There are pieces I wish could work better - I’d love to have the capability of defining a color in SVG then referencing it within my XHTML document via CSS; full support of CDF will likely allow that. If you jump just a little beyond the current CSS 2.1 spec you even have some fairly decent support for columns of text, not to mention tabular structures and even VoiceML support. Moreover, consider XInclude support, something that really, really needs to be a part of every browser implementation (though its fairly trivial to write AJAX classes that let you create similar bindings).

These are all disparate documents, not a single “document” akin to Word or the ODF Writer format. However, even that’s not quite true. The effort of the CDF working group has been to essentially standardize on the way that web documents can be “bound together” into what appears to be a cohesive “whole”.

Part of this is accomplished through the use of a standard called the Web Integration Compound Document (or WICD). This standard provides a number of both new features and clarifications:

  • an extended definition for the HTML object,
  • the integration of Scalable Vector Graphics (SVG) with XHTML and other documents,
  • establishes the and elements,
  • establishes the characteristics of hyperlinks and focus across the boundaries of namespaces,
  • defines the nature of focus across multiple embedded documents,
  • introduces support for animation and synchronization, building on the older SMIL and SVG standards, and
  • includes support for SVG fonts.

Two additional WICD standards - WICD Mobile and WICD Full - extend these with a few other features, most notably ECMAScript 3rd Edition (4th edition is currently still under development), the XMLHttpRequest to support most AJAX applications, XHTML 1.1 and the CSS 2.1 specification. Additionally, CDF working group also define two “modes” of operation - Compound Documents by Reference (CDR), in which internal content to the document is provided via reference links, and Compound Documents by Inclusion (CDI) in which internal content is rendered into the containing document directly in a different namespace.

Admittedly, I think it can be argued that the W3C effort could do with a few less acronyms (and perhaps a bit more of a PR effort) but overall, what is happening with CDF is a very critical - and welcome - evolution of HTML. The web needs to be more than just static web pages - this has been demonstrated by the continuing strength of the Web 2.0 meme, that documents should be able to talk back to the server and interact with them at higher levels than simple links for refreshing content. Commercial vendors would love to fill that space, and by doing so regain for themselves control over the underlying technologies that make up the web. Yet the message coming about with CDF is both simple and profound: the web grew up on the strength of seemingly simple technologies - HTML, CSS, JavaScript. These technologies are still around, they’re just maturing as people come up with new ideas about what the web can do, the pieces slowly unfolding as we progress further into the realm of the Internet as operating system.

Already, much of CDR has been implemented in the more sophisticatedly forward browsers. Opera 9.5 has a rather extensive support for most of CDF core and Firefox 3.0 is moving in that direction (though the biggest area of weakness is in SVG animation support). JustSystems, a company that has a huge presence in Japan but is only now just (sorry) beginning to make an impact outside of that country, has been working towards a CDF platform for a number of years, and has one of the more expressive (and impressive) displays of how compound documents COULD work (I will no doubt by writing more about JustSystems over time). Both Sony and Nokia have WICD implementations working (as prototypes) on certain of their mobile phone chipsets, with similar announcements from Abbra Vidualize and BitFlash, both makers of mobile graphical chipsets, while Sun is partnering with OpenWave to create a formal WICD implementation in line with JSR 290: JavaTM Language & XML User Interface Markup Integration.

This is no guarantee, of course, that WICD will catch on, or even become a household name (even in geek households), but unlike other technology most of what’s involved already exists, and has already been proven every day millions of times a day. Already, the number of HTML documents that exist dwarf (by a few orders of magnitude) the total number of Microsoft Word documents. As editing increasingly moves onto the web, its safe to say that the document of choice will be neither ODF nor OOXML, both of which gain their power on the basis of supporting legacy word processing systems. Instead, what seems to be emerging from the W3C is something that is not an office suite because it didn’t evolve from one, but that nonetheless is capable of most if not all of the same functions that office suite documents pose.

Moreover, if you come to realize that XHTML by itself is NOT the only targeted compound document (indeed, the specification is rather clear that it is intended to be extended to other formats, from XSL-FO to DocBook to the aforemention ODF or OOXML) then what becomes clear is that the ability to integrate content itself can be standardized, and like many other W3C formats, this move to the metadata level may very well provide the necessary differential to make the technology succeed in even the most competitive of milieus.

So, don’t worry - if you’re not using CDF yet … you will be.

Kurt Cagle is an author and chief architect of Metaphorical Web (http://www.metaphoricalweb.org). He lives in Victoria, BC, where he is making it his mission to visit every Starbucks on Vancouver Island.

November 26, 2007

Will Wall Street Tremors Bring Another IT Recession?

Greetings, one and all. I'm bowing to a reality here that Google, in its oh so infinite search engine wisdom, is still finding this particular website more often than it is my Drupal (www.metaphoricalweb.org) site. For the moment, I shall maintain both sites, and see if perhaps something comes of it one way or t'other.

Lately, I find that far more of my reading has centered on economic news rather than upon technical, though to be honest the intricacies of many of the "investment credit vehicles" that have been foisted off on the public frankly put the Obfuscation C Coding Contest winners to shame. In most cases, when a financial fraud is perpetrated, one of the hallmarks that jurors use as evidence is the degree to which illicit cash flows are threaded above, below, and through more mundane transactions. The intent of fraud is simple - make people believe that they are financially safe when in fact their life savings are being purloined.

What your seeing each day in the big financial papers is disclosures of "write-offs" of hundreds of billions of dollars of money by banks, mortgage institutions, hedge funds and similar financial players, largely because these organizations had become engaged in the practice of ... well, fraud.

Mortgage brokers initiated liar loans with people who clearly had no means of paying back even the teaser rates, then got those loans off their books and into "structured investment vehicles" that sliced and diced these shaky loans and, with the assistance of ratings organizations turned these loser portfolios into alchemical gold by building highly leveraged hedges against them.

The banks that purchases these would then get them off their books by selling them to investors, who generally had no idea that the "risk" that they were buying was not safe, conservative debt but rather investments into millions of loans that had about 50/50 chance each of going into default.

Not surprisingly, while many of the mortgage companies and investment banks that perpetrated this nonsense are either in bankruptcy or in financial distress, most of the CEOs who should have put on the brakes early have reaped hundreds of millions of dollars and golden parachutes that will likely see them well into their retirement years.

Of course, the same practices that made this possible also have laid the same insidious seeds within most other credit companies, including credit card issuers and corporate paper investors. As the domino of subprime mortgages fall, its already begun to press auto loan issuers, and there are indications that the commercial credit market is also seizing up as dubious corporate investing also is beginning to look ominously diseased.

Now, I am not an economist - I'm a programmer. While I find this a fascinating exercise to watch in system theory, ultimately the questions that I'm having to ask have more to do with the simple question of how it will affect me, my clients, and the IT industry in general.

Five years ago, the Tech Recession hit. If you were in financial services, you might be forgiven in saying "recession? We're in a recession?!" The effects on most other sectors was comparatively light, though it was devastating if you happened to be in IT. Fully 40% of IT jobs were shed at the height of the recession, programmers who had been buying Starbucks coffee and making six figure incomes suddenly found that the paper options they'd been paid in were worthless, they'd given the best, most productive years of their lives and their most innovative works for comparative peanuts, and they were living in their parent's spare bedroom.

Not surprisingly even now, five years later, a lot of programmers have become skittish about accepting stock options and they have become considerably more parsimonious about how they spend their money (and what jobs they accept in the first place) - and there are comparatively far fewer people entering into the IT field. They've also become very good at watching economic trends, and most programmers that I know who went through that period are getting rather twitchy about now.

Right now, the ill winds of recession are blowing most heavily through the housing sector, especially in the US. One of the things to consider about most recessions is that it is actually quite rare for a recession to affect the entire economy. Instead, in most cyclic recessions, what you have is a situation where supply has temporarily exceeded demand, and the excesses of that supply need to be worked off.

The IT recession of 2002-2004 was a fairly classical one - there was too much investment in IT, both in terms of jobs and in terms of technology, and it took about eighteen months and some fairly severe slashing before demand could catch up with supply. By 2004, even though tech was still technically (sorry) in a recession, there were already signs that things were beginning to recover, and the market very quickly went from too few jobs for the number of workers to too many - to the extent that finding good, qualified people is becoming quite challenging.

The housing sector is going through its own recession, but unfortunately it will take more than eighteen months for demand to catch up with supply; there are a number of reasons for this, from speculators flipping houses who are now left with too many properties and no buyers, to houses going into foreclosure due to onerous adjustable mortgage rate resets, to a longer term demographic switch away from large, costly-to-heat houses in the face of higher energy costs and shrinking families.

Yet by itself, the recession that results from this will also would be at least somewhat self-contained, if an oversupply of houses was in fact the only real problem. Unfortunately, what is now emerging is that this is the domino falling that seems to be catching other dominoes with it. Risk is being repriced much higher, because the same lending practices that made it possible to get a loan for a half-million dollar house with an income of less than $20,000 was also at work in the hedge-fund space, the commercial paper space (which affects corporate buy-outs, business expansion), credit card paper, student loans, re-insurers ... indeed, just about everyone out there is suddenly discovering that the foundations that they were building on was made not of concrete, but quicksand, which has many of the same properties but a rather disturbing tendency to be lethal.

This is the real recession, and the last time that something even vaguely similar happened, Herbert Hoover was president. Credit, which has replaced cash in just about every transaction that occurs today, is rapidly disappearing. Indeed, it is worth it to understand that credit is itself simply a measure of risk. "I will let you take out a loan because I believe you to be a trustworthy individual," the banker should say, "you pay back your loans on time and reliabily, you have open accounting, you have shown good judgment in how you spend your money. "

The problem comes with the next part - "Even the most reliable person occasionally runs into problems. As a banker, I should be compensated for the small chance that you will be unable to pay your loan under the terms you and I set out. That compensation is the interest on the loan."

So what's a reliable rate of interest? This is where things get tricky. The more risky the investment, the higher the rate of return should be, so the higher the rate of interest, and the harder it becomes to qualify for a loan. The problem of course is that the banks make money on that interest, and in good economic times it becomes easy to fudge the credit-worthiness of an individual or two, if higher returns is the desired goal. Of course, once risk becomes high enough, it becomes far more profitable to initiate the transaction (be it mortgage, credit card application, or small business loan) then sell off that risk to someone else. That proved SO profitable that the process got repeated many times, with the same questionable risks getting packaged and repackaged to the point where there was no real way of determining where the risks were - or, more to the point, what they were.

Now, that whole edifice is going through a slow motion crash, and despite everything you hear in the Wall Street Journal or CNN/Money, it is going to continue to crash into other sectors. Right now, it really hasn't fully impacted the consumer (though there's another factor working there) but it will. If no one has any idea what the real value of risk is anymore, no one will initiate loans. Companies are already finding it increasingly difficult to get funding that used to be routine. Companies can't expand beyond the reserves that they already have on hand, and already accountants at many companies are beginning to question just how stable those reserves are. More than one company has been forced to delay even paying their employees or their accounts because their banks have frozen all accounts.

In the late 1930s, the FDIC was established in order to avoid a credit crunch like the one that wiped out the economy in 1929. It stipulated that banks always maintain a fixed percentage of assets (originally 10% of total capital lended) in order to insure that in the event of a bank run, there was enough operating capital to keep the bank afloat until help could be arranged. Additionally, many banks were required to honor the first $100,000 of deposited funds. The problem is that lax regulatory oversight has effectively eviscerated most of the FDIC's mandate, to the extent that it is unlikely that most banks could in fact meet 1% of total capital extended if they had to.

Of course, in order to look like they were in fact complying with these strictures, many financial institutions put much of their assets off the books into secondary corporations that were not then subject to the limitations, meaning that the actual debt that a financial institution had was generally far greater than what they showed on the books. So long as no one looked too closely (and when the money was great enough people would not look too closely) this meant that banks could make incredible returns by lending money that, for all intents and purposes, didn't really exist.

The problem is a Schroedinger Cat crisis. In quantum mechanics, there is a classical thought experiment in which a single atom of uranium is placed in a special trap that would, when the atom decayed, release a toxic gas that would kill a cat locked in a box. Quantum theory states that what is at work here is a wave function, and as such, the cat could simultaneously be both alive and dead, and the only way that you could actually tell (well, besides the obvious yowling of a pissed boxed cat) was to open up the lid. Once you had opened the lid, then you had collapsed the wave function, and there could be only one solution.

The banking industry has been build on the same wave functions, and so long as no one looked in the box, then there could be an infinite amount of money there. Once you open the box, unfortunately, reality steps in, and the reality is considerably more disappointing than anyone could have hoped.

So what happens this time around to IT. The credit crunch will hit in a number of ways:
  • No money available for expansion of companies, beyond existing funds. Too many companies have been "living beyond their means", using their existing reserves as collateral against loans. As those loans begin to dry up, they are forced into using those reserves, which often may be tied up in existing investments and thus are less than completely liquid. When you are forced to sell to meet costs, you often get a far less than ideal value for your assets, especially when everyone else is doing the same thing.
  • Illiquid third party investment. VCs and other investment companies will have more demand on their moneys from more traditional sources, which may be good for the VCs (in the short term) but it also dries up much of the money that IT startups in particular count upon. Moreover, many of those VCs represent funds that may also be fairly heavily hedged, and so the same repricing of risk that is going on in the commercial sector will make it harder for them to pull together the funds necessary to complete deals.
  • Strapped consumer spending. Don't pay too much attention to all of the sighs of relief after Black Friday sales. People are strapped for cash, and they are as attracted to a bargain as anyone. The problem will come soon enough, as people can no longer take out much money via a second mortgage, will be facing stagnant wages and unemployment in December (one of the highest months for force restructuring) and will find it difficult to get credit extensions on their credit cards. I expect that consumer spending will plummet in the next couple of months. Sales from Walmart are done from last year, but so are the sales from high end luxury brands. This will mean that spending on consumer software and hardware will also in general be declining pretty dramatically in the next few months, which directly impacts people in IT.
  • Declining dollar. As the economy becomes more distressed, calls will be made increasingly to cut interest rates. The problem with that is that the dollar's value is declining relative to other currencies as rates go down, making American goods and services cheaper ... but also making American companies cheaper. This will reduce the attractiveness of outsourcing (a definite plus for developers, who've often had to compete with foreign developers) but will make it more likely that companies in other countries may start outsourcing to American IT workers. On the other hand, this will likely only happen once American wages become competitive with those in countries like China or India, which means that programmer's standards of living will likely be dropping for some time before you see a massive influx of capital, and that capital will likely mean that the "parent company" of your company is not even in your hemisphere.

    Note that the carry trade in currencies will likely intensify as people move money out of both hard assets (short term, commodities will likely stall somewhat, though the long term outlook is still somewhat bullish) and financial assets (nearly all classes, long term) but seek even some returns. The downside is that this will likely lead to greater instability in the currency markets, with the possibility that one or more countries could "crash" - the US being at a very real risk of this.
  • Energy costs. IT has generally been a net consumer of power, though that's changing. In many ways, IT has been working towards applying technology to reduce their energy footprint for financial reasons rather than ecological ones, but of all departments in a given company, IT is often the "greenest". On the other hand, a significant amount of energy costs are due to a declining dollar (along with considerations for peak oil and other geopolitical issues) which means that the relative cost of oil (and hence gas) will affect the prices of goods that need to be transported (nearly everything at this stage) and hence will mean that computer infrastructure (which has generally been declining at a rate consistent with Moore's Law) will likely start getting more expensive again relatively speaking (though it may end up spurring additional investment into local hardware production infrastructure such as chip fabrication plants).

    The cost will also accelerate the decade long trend of distributing the corporations into islands of loosely connected individuals who may actually work half a world away from where they live. If your workers are spending four hours a day on the road and are forced to pay an increasing premium for the privilege, they will go elsewhere where they can telecommute instead.
  • No (immediate) bloodbath in IT. Corporations are going to be considerably more loathe to trim their IT staff, in part because that staff has only begun to recover from the last bloodletting, and in part because in many cases existing infrastructure has been pushed for about as long as it can be, so the imperative for improving core infrastructure will likely remain strong even in the face of slowing sales. On the other hand, its unlikely that you will see a lot of companies do anything beyond low cost pilot projects for R&D, especially after the first quarter of next year. As the recession spreads to other industries, I think that IT could be vulnerable on an industry by industry basis, but some of that may be mitigated by investment from foreign investors seeking firesale prices on companies.
  • Shakeout of vulnerable companies. There are a fair number of mid-sized IT companies in certain sectors that managed to survive the Tech Winter, but that are increasingly becoming vulnerable. Even a year ago, these companies would likely have been bought up by larger or more successful companies, but as the available moneys for buyouts dry up, what will more likely happen is that the less successful of these will be forced into bankruptcy instead, without the "Get Out of Jail" card that seems to be the privilege of companies in other sectors.

    The irony is that a lot of new startups may actually end up surprising relatively intact. While there is some fluff in the startup space, it never reached the level of 1999, and a lot of the companies that started in the last five to six years are generally very lean (and fairly heavily open source-based, from all indications) and for the most part are considerably more pragmatic about the market. On the other hand, there will be relatively few IPOs for the next several years out of this sector (though this has been a trend since around 2000).
In the face of all of this, what can the astute developer do? Having gone through this a few times now, I've found the following strategies can help.
  • Save money when you can, and reduce spending on extravagances. If you currently have any debts (especially credit card debt) pay it off and get out of those cards, highest interest first. Debt ties you down and reduces your cash-flow, which will be threatened at the best of times.
  • Stay as liquid as possible, and if possible diversify your funds out of US dollars, at least for the next year or so. If you can land one or more clients that are out of the US, even better, though try to negotiate in funds out of US dollars if at all possible. It's likely that 2008 won't be quite the rout for the dollar that 2007 has turned out to be, but it is also likely that you still will likely turn out better in foreign funds, even with conversion rate charges.
  • On a similar note, for the short term, if you are invested in equities, resource and energy stocks will likely be the most solid for the foreseeable future, though you are looking there less for significant gains but to minimize losses.
  • Diversify your client list as much as possible - and be fairly ruthless in negotiating. They will be. Assume the likelihood of a default by one or more clients is at least a possibility, and plan accordingly ... its possible that all of your clients may go belly-up at once, but in most instance, if you have three or four clients, you should be able to maintain steady work. On the other hand, work hard to deliver as promised ... the ones that do survive generally are ones you want to keep.
  • If you do find yourself between clients, spend that time getting caught up on skills that you didn't have the chance to work on while you were too busy with clients. Unless you are fairly new to the programming field, you're best served by increasing the depth of your skill rather than the breadth, unless the shift to the new field is part of a larger move into that technology as a base. Be thinking a year ahead in terms of where tech will be.
  • In boom-times, programmers generally find recruiters to be pests. In bust-times, recruiters find programmers to be pests. Find a few good recruiters at various companies and get to know them, even refer people to them when you're offered a job you can't or do not want to take. They'll be more inclined to use you when times get tough.
  • Recessions generally happen for reasons that you have little or no control over, but it is easy to become stressed, bitter, depressed or angry during those times, often making it harder for you to get motivated to work or look for work if there isn't any. It's worth remembering that programmers in general usually tend to have very desirable skill sets. If you have the time, join and contribute to open source projects - not only will you keep your skills up to date and work on interesting (and occasionally even useful) projects, but you stand a better chance of making contacts, developing your interpersonal network and gaining early proficiency in new technology by doing so. It will also let you combat the tendency to want to crawl up in a ball and whimper all day. Indeed, staying connected in general is a good idea.
  • Keep alert and informed. In general, diversify your news sources (especially economic and political) and verify what's going on with one from another. Assume that no one knows the exact timing of when the "recession" or "recovery" first hits until after it has happened.
This was not meant to be a doom and gloom post. The coming recession is much like a Class 5 hurricane - it's coming, it will almost leave quite a trail of damage in its wake, but it will also have cleared the air enough for some major reforms to get instituted, and life will go on.

-- Kurt Cagle

September 23, 2007

Move to Metaphorical Web.org

Trying to stay current on the web can prove a challenge for anyone, but if you publish a blog, you're likely to find making sure all the links that you have out there are in fact pointing to the right place one of the biggest challenges. I've recently moved Metaphorical Web to a new Drupal-based site:

www.metaphoricalweb.org

both because I needed the ability to customize the site to display code samples and the ability to set up and maintain web feeds, external links and a forum that I couldn't get from the Blogger environment. I hope that you join me there.

-- Kurt Cagle