They're Movin' and Shakin' at the W3C

<metaphoricalweb>

Thanks for the many responses I've had to my post yesterday concerning Canada - some very good thoughts and suggestions, including the primary things one needs to remember when in Canada:

"Moving to canada is no big deal, as long as you learn that re is better than er and eh is a verb, noun, and adjective all in one."

Thanks, Ronan.

The political statement out of the way, its time I got back to why I write this column in the first place ... GOSSIP! Er..., no, XML. Same difference.

The XML Bernie Botts

Watching the W3C at work is very much like shaking out Bertie Bott's Jelly Beans (the kind from Harry Potter that my 11 year old daughter so loves torturing her old father with). Sometimes the specs released by the W3C are very good - a vanilla or taffy flavored jelly bean that's surprisingly delicious. Sometimes the specs are more like "vomit" or "dirt", though with annotations.

The last couple of weeks I'd have to say there were a delightfully high number of blueberry delight and banana split and the worst to come out was still on the order of "grass" - edible, just not all that exciting. We're past the OWL specifications (there's another Harry Potter column coming on if you're not very careful) which quite frankly read like Noam Chomsky had just engaged in a serious argument with Richard Feynman. You had a drought where for a while all that was brewing were small specs like Accessibility, important in their own right but like listening to a lecture about going on dates from your spinster great aunt. Just when I was beginning to despair that the entire friggin' box was filled with "library paste" jelly beans, along came a bunch of very tasty treats indeed.

• 9 November 2004: XML Binary Characterization Use Cases
• 9 November 2004: xml:id Version 1.0 - Last Call Ends 13 December 2004
• 8 November 2004: Semantic Interpretation for Speech Recognition - Last Call Ends 5 December 2004
• 5 November 2004: XSL Transformations (XSLT) Version 2.0
• 2 November 2004: Assigning Media Types to Binary Data in XML - Last Call Ends 24 November 2004
• 1 November 2004: Timed Text (TT) Authoring Format 1.0 – Distribution Format Exchange Profile (DFXP)
• 29 October 2004: Pronunciation Lexicon Specification (PLS) Version 1.0 Requirements
• 29 October 2004: XQuery 1.0 and XPath 2.0 Data Model
• 29 October 2004: XQuery 1.0 and XPath 2.0 Functions and Operators
• 29 October 2004: XML Path Language (XPath) 2.0
• 29 October 2004: XQuery 1.0: An XML Query Language
• 29 October 2004: XSLT 2.0 and XQuery 1.0 Serialization
• 27 October 2004: Scalable Vector Graphics (SVG) 1.2 - Last Call Ends 24 November 2004

This was just picked up from the Recently Published Working Draft Section for the last two weeks. To quote my four year old daughter "This is AWESOME!"

Even more is hidden in the details here. For instance, I note that in the XSLT Serialization specification, there's not one editor but three: Michael Kay (Saxonica), Norman Walsh (Sun), Henry Zongaro (IBM). A lot of my job entails reading behind the lines and trying to understand the significance, but to me this is a pretty strong indication that both Sun and IBM are watching XSLT2 VERY carefully; you don't put a powerhouse hitter like Norman Walsh (the author of the DocBook specification) on a specification unless you think it'll be important.

That you'd need a specification just for the serialization aspect of XSLT2 may seem a little odd as well, until you understand that one of the key new features in that specification is the <result-document> element. This makes it possible for an XSLT2 document to generate more than one output. The most obvious uses of this capability is to use XSLT2 to generate secondary XML documents, but much like input and output streams this is just one of many areas. Generation of SOAP messages, creation of non-XML source code from XML documentation files (especially with many of the other new features in that specification), and rerouting of messages to databases all fall within the province of this new feature. Combine that with the elimination of the tree fragment such that intermediate XML can be created within a transformation and manipulated directly (no more node-set() function!), regular expression support, and the ability to import text, and XSLT2 is beginning to look pretty damn brawny.

Serialization gives you the ability to do things such as to set the content encoding of a document, making it much easier to handle Unicode UTF-16 encoding. The specification also introduces character mappings, which gives you a way of avoiding the need to incorporate DTDs within source code (or the transformations themselves) in order to define entities. While there are some features of DTDs which continue to be useful, XML is slowly losing its reliance upon them.

The XPath 2.0 specification is similarly an important document, and the news on this one was perhaps not quite as sweet as I'd hoped. XPath is a major specification because it underlies not only XSLT but XQuery, XForms, and SVG's xXBL may have some dependencies in its next draft (see below). The last specification was labelled Last Call, which means that it is usually considered one step before a Proposed Draft -- it is an opportunity to put the specification in front of people and get feedback. There was a LOT of feedback this time around, enough that after some consideration, XPath has been taken out of Last Call Status and is once again strictly a working draft.

The issues that have necessitated this are fairly broad, due in part to the need to more properly specify the character model (part of the serialization above) and in part to rectify some of the thornier issues with type conversions. Another common feeling (one that I personally agree with) is the huge number of date-time functions and the feeling that many of these are redundant or readily derivable, especially given the extension mechanisms that exist in both XQuery and XSLT. There are also some indication that one of the original goals of XPath 2 - providing data aware type objects, has receded considerably as people have thought more about what specifically the specification was intended to accomplish. Finally, collations, which affect string order, have been put on the table as things which need to be more clearly defined before the specification can be published.

Regardless, this means that most of the specifications involved will likely be in limbo for at least another six months because of this delay. The one positive aspect about this is the fact that the specifications are not likely to change considerably, so that so long as you stay clear of the more problematic functionality, you can actually use the beta XSLT2 processors that are beginning to surface with some reliability.

SVG 1.2 Goes Into Last Call

After the last paragraph, I'll be cautious about saying too much about this one, though because it is for me a little closer to home I've been watching this particular battle rage for the last couple of weeks. SVG 1.2 is, to put it bluntly, what SVG should have been four years ago:

1. SVG1.2 defines a mechanism for dealing with wrapping text in paragraphs, and does so in a spectacular fashion by making it possible to flow text through irregular shapes, and from one shape to another.
   
2. SVG1.2 defines video and audio tags for creating robust multimedia.
   
3. SVG1.2 includes a binding mechanism called sXBL (SVG XML Binding Language) that makes it possible to use XML to build complex components.
   
4. SVG1.2 includes an editable attribute to make it possible to change text on the fly.
5. SVG1.2 incorporates a mechanism to create filter vector effects.
   
6. SVG1.2 includes sockets and HTTP support for building distributed applications.

All in all, it puts together enough substance to make it into a serious graphics substrate layer, with the one lone difference between it and a language such as XAML being that it does not yet address 3D. Of course, given that the next version of SVG will be version 1."3", I'd say this isn't that big of a problem.

The SVG 1.1. specification was revisionary - it clarified a few features that were not completely specified in 1.0. SVG 1.2 on the other hand is meant more as an addendum, shuffling new chapters into the thing that we call SVG. It is also a very controversial specification, as SVG is now beginning to seriously encroach into other areas such as HTML and XSL-FO. This is not so much true in terms of the specifications - HTML is a logical description of a web page while SVG is a lower level graphics description language - but when you combine SVG with sXBL you have what amounts to a mechanism by which you can use SVG to render that HTML in a pretty fair representation of a web page. In this way what SVG serves to do is replace not so much the specifications, but at least SOME functionality of the browsers themselves.

X.Org Does SVG

This has some very interesting implications, especially in conjunction with what is happening in Linux. A year ago, the XFree86 organization, the group responsible for the X graphical system on Unix and later Linux, split on the basis of changes desired by certain members to make the license non-GPL compliant. The GPL faction formed a new organization, X.org, and immediately set out to solve another problem with XFree86 - the glacially slow development pace of XFree. XFree86 is ancient technology - the first X implementation was created in 1984, twenty years ago, and although the framework was remarkably flexible there were several places in the underlying graphics model that were proving remarkably constrictive.

X.org used this break as a chance to rebuild much of the more troublesome aspects of X. While this effort is still ongoing, already there are a number of new features to X which are making developers salivate:

• Moving from a 24 bit to a 32 bit color space - the RGBA space - to provide 8 bits (256 levels) of alpha channel support. This makes it possible to push transparency directly into the chips, whereas before transformation had to be done much higher up the stack at the application level, resulting in far poorer performance.
• Establishing much finer invalid region support into the graphics layer itself, resulting in much faster screen refreshes - an absolute must for animation.
• More tightly integrating the 2D and 3D engines used by unix based systems, a critical feature for the Java Glass window initiative (and X3D).
• Creating more robust events that can be captured at the X layer, rather than needing to be placed up in the application stack.
• Incorporating hooks for SVG integration (yeah!!).
  

One implication of this is that SVG is bidding to become the equivalent to GDI under Microsoft Windows - in a role with Linux that'll compete directly with the XAML graphical layer under .NET. As SVG 1.2 will likely be a full specification about the time that most of the new X.org begins making its way into commercial applications, this implies the possibility that SVG 1.2 could become pervasive under Linux.

New Compound Document Formats Group Formed
One of the more intriguing ideas behind many of the presentation oriented specifications is the concept that you can embed SVG and MathML code within XHTML, possibly combining it with XForms content or other related specifications. Some browsers are already doing this, but when theory hits reality, inconsistencies between models can cause more headaches than expected.

In mid-October, the W3C announced the formation of the Compound Document Formats Working Group (CDF) specifically to provide ways to smooth that integration. The formation of this group sends a signal that the W3C is shifting from the development of base technologies specifications and to the interchange between the various specs, something which has needed to happen for a while. I suspect that interoperability will be the name of the game for the next couple of years, especially once the big specifications like XSLT2 and XPath2 finally go golden.

There has been a tendency in the past for specifications to occasionally develop potentially competing and certainly conflicting aspects with other specifications issued by the W3C, meaning that if you wished to use on technology you might potentially be locked out of using another, even though both are conformant with the W3C process. By getting this resolved, user agents of the future will be considerably more integrated with all of the W3C specifications, and the notion that all of these objects can effectively play within the same DOM-space is particularly exciting (your HTML controls the MathML, which in turn controls the SVG, populates the XForm data, and passes the results up to a pipe using the XML DOM).

Wrap-up
I've been busy on other projects and am beginning to wrap those up, freeing up more time for Metaphorical Web columns - so expect the activity to pick up again here. Is there something that your company or organization is doing in the XML space that you think is worth shouting about? If there is, contact me at kurt@kurtcagle.net with the inside scoop. Until next time, enjoy!

</metaphoricalweb>

Canada Bound?

The last couple of weeks have been a time of exhiliration and heartbreak, and have forced me to seriously examine what my feelings are about the US and my relationship to it. I've started this same blog four different times now, each time trying to frame what I wanted to say. In the interests of getting back to the task at hand - discussing the world of XML, it seems best to make only a statement on what has happened.

On November 2, 2004, the two party system in the United States effectively collapsed. One party, a party that it notoriously hostile to civil rights, remarkably ignorant about technology, inclined to view such technology principally as a means either to wage war or control the populace, now controls all of the political organs of our state - the executive branch, the legislative branch, the legal branch, the military and intelligence services, and an overwhelming majority of all governorships, and to a great extent the media.

This party is not the GOP - though they were the agents through which this takeover was affected. The people who wield the power within this organization are students of a particularly virulent brand of nationalism that was last adopted by the German National Socialists, the Nazis ... as they came to be known. It is a philosophy of empire, of hegemony, of conquest for the sake of power and control, a philosophy that has learned to cloak itself in Madison Avenue friendly soundbites and wrap itself in the mantles of the cross and the flag, but it is friendly neither to people strong in their faith nor strong in their patriotism.

I voted for Kerry, not because I found that his ideas would have radically improved things, but because the party that Bush represents is making things far worse. Many of you, no doubt, will disagree with me ... and I welcome such disagreements - that is the spirit of honest debate. However, there are many indications that such honest debate will become much rarer, as will the spirit of innovation which is so critical to this field, as we become a nation under surveillance, so fixated on the enemy without that we do not see the enemy within.

A democracy needs two things to survive: a system of checks and balances and free and fair elections. We no longer have the first - the people in power can make pronouncements and judgements without the threat of what would in the academic community be called peer review - the ability to provide a second set of eyes to determine whether a given law is really good or whether it simply represents the desires of the person who makes the law for personal gain or aggrandizement.

In programming circles, the equivalent concept is code review. The Open Source Software community understands the value of this - in many ways OSS is essentially code review write large. No company worth its salt should ever accept software that's passed directly from the developer to the company at large (or their customers/clients) without at least two more sets of eyes looking at it; companies that fail to do this will not stay in business for long. The analogy holds true in government as well.

Were the elections free and fair? I personally do not believe so -- there are simply too many abberations at all levels, some due to human intervention, some due to machines either badly or maliciously programmed, and an overwhelming number of them favor the Republican party. Again, in statistics, a sufficiently large number of random errors will be distributed along a Gaussian Bell curve, such that errors usually tend to cancel out, not heavily favor one candidate or another. This will not change who gets inaugurated in January, but it does bode ill for the faith that one can place in the election process, not just now, but for the foreseeable future. Given that this faith is the cornerstone of democracy, the belief that each person within the society has a vote of equal value to all other people in that society, to compromise it to this degree begs the question of whether the United States is in fact a democracy anymore.

Many people in IT tend to be libertarian, and are either passionate about politics or supremely indifferent about it -- there is no middle ground. Thus to talk about it in a programming forum is often considered to be in bad taste. However, to those who see no relationship between the two, consider that with this election, several things are now set in motion that will have an impact upon both IT managers and programmers:

• Enforcement of such legislation as the Digital Millennium Copyright Act will be tied much more closely with federal agencies, with the strong potential to restrict the free flow of all information (throttling the Internet) in the name of chasing pirates.
• A concerted effort will likely be made on the part of several corporations to declare the Gnu Public License (the GPL) invalid, possibly along either DMCA or anti-competitive means. Patent cases under this administration have been settled overwhelmingly in favor of the largest busines interest ... expect this trend to continue.
• There is already something of a brain drain going on as academics and high end developers leave the country, fearful of what they see already coming down the road and frustrated as universities become simply extensions of corporations because federal funding is drying up. Additionally, as corporations in the US have become leaner and meaner, they have typically slashed or eliminated outright their key R&D centers, expecting that they can buy the innovation on the open market. I see this assumption as being facile and short-sighted.
• There is an increasing reluctance on the part of world standards bodies to want to deal with American corporations in light of political and military actions of late. I've been to a number of standards conferences in the last year or so, some in the US, some outside of it, and find that there are fewer and fewer Americans drawn into them. The heighened military posture of the US has also meant that certain American companies, such as Microsoft, are increasingly being seen as proxies for American intervention, and are losing business in Europe, Asia, and South America because of it. I personally don't think this belief is necessarily valid, but there is definitely a backlash building against US products because of that perception, with software being a major part of that.
• I'm keeping my focus here strictly on what I see as the most conservative impacts that the new administration will have solely upon the IT sector. I think there will be many bigger effects, including a fairly severe recession by mid-to-late 2005 which will also affect the industry (though not as badly as the 2000-2003 Nuclear Winter did), but there are too many unknowns to make such guesses with any certainty.

Those of you who are long time readers of Metaphorical Web may remember my rhapsody on themes Canadian after the 2003 SVG Open conference. While the decision is yet to be made, I am leaning strongly toward emigrating there, to get involved with what I see as the vibrant open standards scene in Vancouver. The city has become a magnet for XML technologies, drawn by such luminaries as Tim Bray, Philip Mansfield, Paul Prescod, Ron DeSerrano and many others. It's also becoming a hub for software entertainment, a field that to date has been only peripherally influenced by XML, but which is probably due to be the next frontier for that particular set of standards.

Moving one's family to a new country (even one only a few hours drive away) is always a momentous decision; moving because your own country is undergoing what could be a dark and dangerous transformation is heartbreaking, because you have to struggle with the question of whether staying would make a difference. It is a decision I am still weighing, and would gladly welcome comments from others on both sides of the divide.

Conferences and Google

The joy of being a consultant is that frequently you end up being very, very busy. The downside is that you frequently end up very, very busy, and the other things that you are working on tend to get short-shrift. I have been crazy busy, good for the pocket-book, but bad for things such as Metaphorical Web. I'm taking a brief break here though to catch up on the web and post what I've been up to the last couple of weeks.

Chris Sells' Applied XML Development Conference

I was invited to Chris Sells' Applied XML Development Conference, considered by many to be one of the best conferences for covering advanced XML techniques (the other being the Extreme XML conferences), to present a paper on XSLT2. Others will have blogged this conference much better than myself, but I do have a few notes to present as well.

Geek Dinner

Chris Sells put on a good conference, something that isn't always easy (as a habitual conference speaker, something analogous to a drug addict but with presentation tools as the opiate of choice, I should know). It was definitely a Microsoft-oriented event, and I do have to admit that I found the Microsoft propaganda at the show tedious and heavy handed, but at the same time I knew going in that it was likely not to be an agnostic crowd.

The accomodations, the Skamania Lodge in Stevenson, Washington, were remote (about 40 minutes from Portland) but otherwise quite spectacular, showing off southern Washington State at its best with scenic vistas of mountains and the Columbia Gorge. The lodge was the epitome of rustic - I keep wondering whether there is a "rustic" design specialty out there, with classes in how to properly design fireplaces and log walls. I drove down with M. David Peterson, the developer of the Saxon.NET XSLT2 processor and my co-conspirator in crime in putting together the presentation, and we spent the three hour trip down riffing on the nature of meta-data and models.

We didn't go directly to the lodge, however, instead making our way to downtown Portland for the Geek's Dinner. This has apparently become a time honored process at the Sells Cons, in which many of the attendees showed up at a dinner hosted by Don Demsiak (DonXML) at an appropriate venue - in this case the food court at Lloyd Center Mall. While most of the Microsoft big-players didn't show up, just about everyone else did. It was fun, we got a chance to just chat and make jokes before going into the more formal venue of the lodge itself.

Once we got to the lodge and settled ourselves, I went down to the bar and had the privilege of having a beer with Tim Bray, the main architect of XML and the driving force right now of the Atom specification for syndication. While I've met Tim before, this was the first time that I had a chance to really talk with him, and we covered everything from SVG and XUL to the best micro-breweries on the West Coast (he recommends the Yaletown Brewing Company, just down the way from his offices in the Yaletown district of Vancouver). I had a lot of respect for Mr. Bray before I met him, and after this conference my respect has gone up immeasurably.

Blue Pill, Red Pill, Purple Pill

The conference itself got underway around nine with Tim giving the keynote address. Not unexpectedly, it was about blogs and blogging (and the exponential growth of same) but he did manage to get in more than a few digs about a number of standards that were in the works, spread his frustrations with Microsoft and the W3C evenly, and also indicated that the world was not just Microsoft. This was a challenge to subsequent speakers, many of whom were talking the MS line.

Chris Anderson, a key Avalon developer at Microsoft, spoke next about Everyone Hates XML, a play no doubt on "Everyone Loves Raymond" but that oddly continued a theme that was both a little depressing and fairly antagonistic throughout the conference. Anderson's talk actually focused more on XAML and the design decisions used in its development. I'll have to confess that some of the examples that he used, such as

<button id="foo" value="Button">
<button.background>
<solidcolor value="Blue">
</solidcolor>
</button.background>
</button>

(note, may not be completely valid, as I don't have the example handy)

rub me very much the wrong way, as it effectively places stylistic attributes into a position where they effectively create a potentially huge combinatorial set of elements; its not validatable, and as a consequence requires having a very complex runtime in the background to handle. This very quickly jumped into a free-for-all with the audience asking about what exactly was a tag, and I noticed after a while that it was only the Microsoft regulars who were trying to push it into a non-issue. A prediction - the lack of XML validation will prove to be the undoing of XAML in the long run. The talk was othewise quite interesting, however.

Patrick Cauldwell and Scott Hanselman presented Bringing Strongly Typed Business Objects to Legacy Financial Systems with XML Schema, a mouthful which nonetheless proved to be a useful talk. One of the central ideas in their work was the use of XML Schemas to generate Word documents describing a financial system, something which could in turn be used as the basis of legal documents. This is a very interesting idea, one that I've actually pushed before myself. To whit, a legal document serves two purposes -- defining the language (services) upon which two or more parties agree to specify the terms of contract, and the penalties for failure to complete these services on either side. A schema in that regard can effectively perform at least one part of that process, stipulating the terms of the agreement. As a consequence, I suspect that over time, schema documents will begin to take on a quasi-legal status.

Don Box and I have sparred in the past, though I also took advantage of what he thought was a "safe" venue to make some observations about the WS-* initiatives, then in their infancy, that he felt (with some legitimacy) to be unfair. Don's grown more confident in his own role at Microsoft, and while I'm not completely wild still about the WS-* concepts I will concede that they have a certain utility. His talk WS-Why? went over many of these specifications, using a (somewhat strained) island metaphor to go over the latest Microsoft initiatives in this area. My biggest gripe came from his tendency to talk around the purposes of some of the initiatives without providing a little better explanation of what they are intended to be used for - I think even many who were ardent Microsoft proponents in the audience felt a little lost on this. I did note with amusement that UDDI had been relegated to the status of an "oops" kind of idea, one that had some lofty goals but bad implementation, citing many of the same reasons that I've had for disliking that standard (the overbearing use of "business metaphors" for instance). For all that, the talk was a good one, and while I didn't get a chance to talk with him in detail, I do hope to do so in more favorable circumstances in the future.

Then there was the absolute, no question greatest talk of the entire conference: Using XML for Navy Missile Systems, by Whit Kemmey of the Department of Defense. Nuclear missiles. Many submarine shots. Discussions about coding practices aboard ultra-secure environments. The Hunt for Red October could only wish it had been this cool. Whit was very definitely a military programmer, very clean cut, the only one in the entire auditorium wearing a suit, but for all of that he had a wry and deadpan sense of humor that caught some of the brightest programming minds on the planet off-guard more than once. Some favorite quotes:

• Im a trenches guy. Im not a vendor. This stuff isnt for sale.
• Our software has never been used for its intended purpose, and hopefully never will be.
• Localization is not a big problem for us.

The Navy apparently uses a customized Unix solution on hardware that is running about five years behind the curve, largely because of the need to stress-test everything to insure that there are no unexpected gotchas (I'll leave it to you, gentle reader, to figure out the consequences of a system crash on a boatload of nuclear weapons). XML was used here to handle the generation and implementation of Standard Operating Procedures (SOPs) using LibXML as the central processor. There was a certain amount of astonishment at the way that it was being used by many of the Windows-oriented type, though the descriptions for actions used made sense in terms of needing to specify in great detail every single step of a process with such incredible consequences of something going wrong.

I did find it a little irritating at the tone used by more than a few of the commercial XML developers, a kind of mocking astonishment about the process of software development in the military, but having been in the Navy myself in the 1980s, I have to admit that more than a few of these people could have afforded to spend some time in the service to understand what writing mission critical software is REALLY about. When lives could be lost due to a software error, conscientiousness in coding is not only a nice to have but a necessity, a lesson that a few of those writing operating systems would be wise to consider.

Sam Ruby is an unheralded genius. Working for IBM, he also serves helping to put together the Atom specification, and is involved with both the KDE and Gnome groups. He was also easily one of the most depressing speakers there, chastising the crowd for the dangers inherent in many of the problems associated with web practices, from Unicode violations and errors to URL mis-encryptions in his talk XML is an Attractive Nuisance. These were all things that he's had to deal with when coding for Atom, but in the process of giving the talk he kept pointing out fundamental problems about the Internet itself. He did try to use the Matrix metaphor, though the relative disasters of the latter two movies ment that his demos didn't have quite the oomph I think he would have liked. I enjoyed the talk, and managed to identify many (though not all) of the erroneous examples he provided, but given his Matrix references I think more than a few of use were wondering not whether we should take the blue pill or the red pill, but who had a stash of The Purple Pill (Paxil, a commonly prescribed anti-depressant).

Daniel Cuzzolino (evangelist for Schematron) not surprisingly did a talk All About Schematron. For those of you unfamiliar with it, Schematron is a schema validation tool that uses XSLT and XPath to provide more sophisticated validation than can be handled with XML Schema Definition (XSD). Significantly, he brought up a number of features with Schematron that he wanted to be able to incorporate, but couldn't such as the use of regular expressions in his validation. After I gave my talk, we personally discussed Schematron in much more detail with an eye toward using XSLT2 for handling the next generation of the application.

My talk was next, and I will be discussing it in much more detail in the next issue of this blog. I'd put it at a middling success - I had too much material and being immediately before dinner, the attendees were more than a little distracted. If you can avoid it, try not to compete with food - empty stomachs make for wandering minds.

The Case Against XSD

After dinner, the speakers all headed up to the front of the dining hall for a round-table, and almost out of the chute the topic was the inadequacy of the XML Schema Definition Language (XSD).

A little background is in order here. While most of the W3C standards have had a certain degree of controversy associated with them, far and away the most controversial of the specifications has been XSD. It was begun about the same time that the XML standard itself was in its earliest stages, but even given that it took more than five years of very contentious meetings to finally agree to it, with the most vocal proponents being the database vendors of Microsoft, Oracle, and IBM. The difficulty is not in the simple type definitions - how one defines primitives such as integers, floating point numbers, and so forth. Instead, the challenges have come in defining more complex types, such as elements with subordinate children in them. There is a lot of ambiguity in the specification, especially when it comes to these complex data types, and what makes matters worse is that the underlying data model of XSD favors a one tag = one object model that is increasingly being shown to be inaccurate in describing data models.

A subtle revolution is going on, and parts of it emerged in this conference. While there are many (especially those vendors) who want to declare the matter closed and have effectively turned a deaf ear to the plea for re-examining schemas, the most promising alternative candidate, Relax-NG (also given as RNG) has quietly been showing up in all sorts of interesting places. For instance, last year the SVG 1.2 specification was published using RNG and not XSD as the schema. Tools such as Oxygen are making RNG available as their primary validation scheme, and content management engineers, who have known for sometime the deficiency inherent in XSD, have been switching over DTDs to RNG and bypassing the XSD spec altogether.

Consequently, it was interesting to hear the number of people who began by saying that they didn't like XSD but would support it because its already baked into so many other specifications. In the .NET world this is probably true - Microsoft has taken to schemas with a vengeance, perhaps too much so in some respects, even though there are certain core technologies such as XAML that would be much more accurately described (with less ambiguity) in RNG. The Relax-NG schema works at a little lower level than schema does, and is more descriptive of contents; right now it is facing a VHS vs. Beta type struggle, but I don't think it's necessarily a good idea to write it off at this stage.

The rancor which this topic brought up raises a more subtle spectre that vendors especially should be mindful of - is it possible that people are beginning to react to XML technologies that they feel were forced down their throat by bypassing those XML structures in favor of more ad-hoc ones? There is little in the way of true empirical evidence, but there is a lot of anecdotal evidence that suggests this. SOAP has definitely achieved success in the hard business sphere, but is being adopted much less readily by many companies that are dealing with document content. WSDL (and the implied RPC model that it brings to the table) has likewise been successful in a much smaller niche than the proponents of these specs had hoped.

A word to such companies -- your customers are looking for the best solution to their dilemmas, not yours. While it is necessary to place a stake in the ground over a particular technology periodically, making the argument that it's too late to make changes will only make the foundation that these technologies are building just that much more fragile. It also may make your technologies out of synch with the rest of the world, and in a world that is becoming increasingly heterogeneous, this can result in some serious discontinuities to the bottom line when a critical mass of users of the alternative standard is reached.

Is RNG that much better than XSD? I'm still trying to ascertain that myself, though earlier indications are that it is a better schema language. RNG was designed from the bottom up to be a compelling language for defining schemas, XSD was designed from the top down as being a wish-list for vendor support of certain features. Having written a couple of books on XSD over the years, I think I can speak with some authority on this -- it has some REAL problems.

I think that right now there IS a window of opportunity to make changes in the underlying schema language. SOAP and Web Services has not taken off as explosively as the pundits would have liked, in part because of the schema issue, and even the WS-* standards being promulgated by Microsoft are still largely in a prototype and deployment stage.

To me, the best solution, admittedly one of the more complex, is to separate schema from such things as the WS-* standards, making the specific mechanism for validating dependent upon a user defined schema attribute. This approach is already being used with a lot of success in the XSLT space (where transformations beyond XSLT 1.0 can be used for processing just by changing an appropriate selectionNamespace attribute. Schemas, which in general should be much lighter weight than transformations, could easily adopt this approach.

There's a storm brewing there, and because it is something that is so fundamentally - the language to be used for defining things - all parties should spend some serious time asking themselves whether the disinclination to change the default schema lays as much in a level of intellectual laziness as it does worrying about cost. I would rather my children have the best schema language possible when their Internet comes along, rather than something that's a dark horse candidate proposed because everyone hates it equally.

Microsoft Propaganda

I enjoyed the controversies of the first day, even if they were more than a little contentious. That's what healthy scientific debate is all about, and at the level of this conference, I think the word scientifically can be legitimately used. There were multiple viewpoints, and the most refreshing ones were the ones that weren't peddling the next API or service pack upgrade.

I think that the conference to me slid into propaganda on the second day. Doug Purdy presented a piece on versioning with web services that was intriguing, even though I have to admit that it was a little too cut and paste-ish for my tastes. If this had been all of his talk, I think I would have enjoyed it immensely.

What I didn't enjoy was the ten minute "movie" thrown at us promoting Microsoft Development Services, based upon a take-off of the TV show Queer Eye for the Straight Guy, and portraying Linux and Java developers as being idiots who would be far more productive if they would only drink the Microsoft koolaid. To those of the faithful in the crowd, it was a funny piece, but to those of us who tend to straddle both worlds (and more, have moved away from Microsoft technology because of some of its inherent problems) the ad was insulting, and more, its placement within the Sells conference announced like nothing else that this was no longer a technology neutral forum but was instead a wholly owned subsidiary of Microsoft. Until that point, I had been enjoying the conference a great deal, but the movie soured it for me. Can the PR, Microsoft! The people who went to this conference wanted ideas and the give and take of ideas for fixing problems within XML, not captive advertising.

Vindication

On the other hand, one of my personal victories at this conference occurred when Neetu Rajpal, the Program Manager for Microsoft's XML program, was covering changes that would be introduced into VS.NET 2005 in the XML space. Much of it was welcome and overdue - better XML editing, a semi-decent XSLT editor (I still prefer OxygenXML or Stylus Studio, but that they had was a fair sight better than what currently passed for editors for these technologies in Visual Studio). Inferring schemas is also a nice feature, though again one that has made its way into most commerical XML editors.

What was of course NOT in there was any mention of XSLT 2.0. Microsoft has chosen not to support XSLT 2.0, not now at least, and when asked when by a member of the audience (not me, I promise) Neetu said that it wasn't planned for development any time soon. My talk, on XSLT 2.0, apparently did affect more people than I'd thought, because another member of the audience stood up and asked for a show of hands: who there wanted to see XSLT 2.0 and who wanted to see XQuery 1.0. Half of the audience raised their hand for XSLT, maybe 35% raised it for XQuery. That there are a lot more people using the technology than Microsoft wanted to acknowledge may make the bizDev people think twice about not wanting to adopt it.

Amazing Amazon APIs

Jeff Barr, of Amazon, regaled the audience with the full range of Amazon web services APIs that are currently being rolled out for business developers; some very interesting things going on in this space. Through these APIs (currently available freely to anyone who signs up with the Amazon development program) it is possible to query Amazon listings and customer contents, and to provide certain post-back information for tracking purposes with sales, in both their publishing and general retail divisions. They've also managed to partner with Google in creating a generalized search framework that is similarly scriptable through web services, making it possible to build very elaborate web queries that can then be tied into applications, not just web pages. Finally, they now provide a way to create interface bindings on their pages to build branded store presences, something that I predict will be VERY big.

I have to admit, after looking at this, I felt a strong urge to start creating my own store-fronts (and actually will likely do that with Metaphorical Web Publishing, a venture I will discuss in a subsequent blog). It's cool technology.

Conference Wrap Up

I needed to get an early start back, so missed a couple of the sessions. There were a few other sessions that I have heard from other participants proved to be just as provative as the ones I was able to attend, and if I can I will try to attend the conference again next year. I'd prefer to see the Microsoft aspect downplayed -- yes Chris Sells is now a Microsoft employee, but I think that this conference would have more credibility if it was less hostile to alternative viewpoints; additional sponsorship by Novell, IBM, Oracle, or the Linux center (which was located, ironically, less than an hour's drive from the conference) would also serve to make it more balanced and appealing to all XML developers, and may even lure a few of those hardy souls off in Mono land back into Microsoft's embrace.

Chris himself is a gentleman and a scholar, and he has a truly wonderful family of an age with my own. I wish him luck next year with the next Applied XML Developer's Conference, though this year will be hard to beat.

Google in Kirkland

I just learned this week that Google will be setting up a new development center in Kirkland, Washington, within a couple of miles where I live. The idea of having a Google output in the area makes me giddy -- in addition to possibly applying for a position there myself (okay, admit it, who wouldn't want to work at Google right now?) I get to see the fireworks as yet another major Internet player moves into the Puget Sound.

There's an interesting confluence going on right now. With Google in the neighborhood, the Silicon Forest of the Pacific Northwest is beginning to take on a presence not unlike that of San Francisco in the 1990s. Portland, once a major hub for hardware manufacturers, has quietly been evolving into a Linux powerhouse, with Linus Torvald's move there a few months ago adding the crown jewel for a region that is boasting a renewed IBM and Novell presence as well as bunches of Linux start-ups. Microsoft's influence is still strong in Seattle, but Amazon and now Google are going to change that dynamic considerably, with a shift away from desktop systems to the next generation Internet. Vancouver is becoming a mecca not only for games and movie production but also for UI technologies - several SVG companies (many also sporting XAML arms) are located between Yaletown and Belltown in Vancouver.

Makes me proud of my adopted home. Rain is good for developers.

Signing Off

Don Demsiak has called me the master of the long blog (thanks .. I think) but to keep this from turning into war and peace, I'm going to wrap it up here. Expect for things to pick up on Metaphorical Web, as both this conference and my latest development efforts have provided LOTS of fodder for me for the next few weeks. Until then, enjoy!

-- Kurt Cagle

Picking a Winner

One of the things that has often fascinated me is the way in which given kinds of technologies (and fads in general) spread throughout a population, a metric that I have often used as a way of gauging which technologies have potentially long staying power. This holds more than academic interest for me - as a writer of books, I often am trying to find those ideas, those memes, that will likely end up being popular enough that I can sell many copies of a book by the time it gets to market. Even if you are a writer of fiction, being able to spot these trends gives you a chance to get a book to market with a topic that's getting "hot", and also helps you to determine when a technology is "oversold" and consequently may be in danger of sitting on the shelves.

To that extent, I thought I'd dedicate my column today to a loose set of "rules" (observations really) that I use in looking into my crystal ball. Admittedly, what works for me not work for you, but I think most of these are pretty much common sense.

1. Open Standards Trump Closed Ones. Standards provide more than "playground principles" for getting people to play in a consistent manner - they also provide a way for smaller players to both avoid a lot of the conceptual research necessary to best describe a given technology, as well as insure that their product will have a market for what it produces. As standards typically form the basis for the way a given set of intellectual property evolves, open standards can also head off potential legal battles down the road.
   
   
2. Dead Tech Reincarnates in Software. Remember the drubbings that push channels, network computers, Hypercard, animated agents, and so forth received as they were pummelled into oblivion. Push lives on in spirit as Atom (nee RSS), network computers have morphed into cell phones and Blackberries, andHypercard was one of the big driving force behind the World Wide Web. If I was a betting man (and I am) I would wager that animated agents will come back into play very shortly as well via XML driven intelligent interfaces.
   
   One of the things that causes a technology to fail is that the technology becomes used first as a vehicle for promoting advertisements. Often the ideas themselves were pretty good (channels for instance), but because channels very quickly became associated with companies piping their ads to your inwilling desktop, push marketing became a term of loathing for most people. However, when people went back and looked at what made the technology possible (in this case the syndication offered by RSS) they often find novel uses for it, such as Weblogs. I keep an eye on any number of failed ideas and try to figure out whether the failure was due to the concept or to its execution, and if it looks like the latter, I'll put out a watch on the web and elsewhere to see if the core tech is showing back up.
   
   
3. Watch the Novice Developers. In the early 1990s, Microsoft came up with an exceptionally innovative product called Visual Basic. It combined an easy to use computer language with the ability to create native applications quickly in its core Windows operating system. Up until that time, Windows was not quite a slam dunk - it was fragile, was competing with other graphical interfaces (such as the Macintosh) and had a relatively small developer base made up of mid-level programmers working within C++. With VB, however, a whole new cohort of power users joined the ranks of "programmers" and there was a huge surge in the development community. Microsoft benefitted by this because it meant that they had, seemingly overnight, a whole forest of applications being developed for it, which of course strengthened the attraction to using this particular OS.
   
   However, starting in about 1997, several things changed. Microsoft shifted their focus away from the development community and toward the higher level IT managers, most of whom did not interact with the systems day to day. In the short term, this strategy paid off, as it sold more higher end server boxes. In the long term, though, Microsoft may have eaten its seed corn. Linux and the BSDs began to jump into prominence, and Java (and later XML) made it easier for developers to migrate off of the Microsoft ship. Suddenly, people were porting away from Microsoft in record numbers, driven mostly by the next cadre of young developers who cut their teeth on these new quasi-Unix systems.
   
   Young developers create software, and the most innovative software usually does not start in the boardroom but in the garage office. To see what things will be hot tomorrow, look at what the most common apps in beta are at Sourceforge or Fresh Meat. Chances are that the applications being development there by themselves will not be the ones that are hot, but applications like those probably will be.
   
   
4. Ignore the Natterings of the Press. I've worked for a number of tech magazines. They live and die off advertising. The mainstream tech reporters know that market well, and expend huge amounts of ink on the doings of Microsoft, IBM, Sun, Hewlet-Packard, Oracle, and so forth. These companies strip whole forests bare to get press releases out to the magazines, and the magazines oblige by writing good reviews about products in hopes of getting advertising money. Some are more objective about the news than others, but given tight deadlines and constrained budgets, the jump from press release to article is usually not far. As press releases are essentially advertisements indicating that a given product has reached (or is nearing) completion, it means that someone there must have been at least a year ahead of you at that stage.
   
   By the time it hits eWeek, information about trends is stale -- more or less. If you are in investor looking at getting into a given technology in the market, the magazines can give you a pretty good idea about what is beginning to bubble to the surface, but if you are a developer looking to catch the next big wave, or a writer looking to get his book pulblished in time for the market to take off, then you're probably late for the party once it makes the headlines.
   
   
5. Solutions in Search of Problems. One of the central questions I always have about a given technology is whether it is fufills an obvious need or whether it is a solution in search of a problem. "I need to make lots of money by selling this software" is not a "problem," though I am sometimes astonished at how many otherwise sane people believe otherwise. Visual Basic made it possible for people who otherwise didn't know the first thing about programming to write serviceable programs. In more contemporary terms, SVG is beginning to take off not because it has any immediate demonstrable impact in presentation graphics (ala Flash or Powerpoint) but because it makes intelligent graphics such as self-aware maps possible, and as people become more conversant with it, they are adapting it for other things.
   
   When evaluating a technology, the first thing you must ask is "How will this help me?" If, after some time spent looking at it you cannot formulate several different one paragraph answers, then the technology is not worth you investing your time on it. I will typically role play different potential users when looking at something new, and I figure if I can't see any obvious advantages in that role play session, then most other people won't either.
   
   
6. Coolness is a factor. There have been a few technologies that I've looked at and immediately responded, "That's cool!" This isn't necessarily due to eye candy - if it makes it possible for you to conceive of things that you absolutely want to do with it right now, even if it isn't quite ready for prime time yet, then you've stumbled on something that will likely be huge down the road. Additionally, this does not necesssarily have to be within the realm of computer technology; I read science and technology magazines extensively, looking not necessarily for what the latest gadgets or ideas are, but instead looking at what things a given innovation (say Bayesian computing) could open up down the road.
   
   
7. Know Your Own Limitations. I've worked with web technologies for eleven years, and XML technologies for eight. I know these areas very well, especially in the context of user interfaces. I know less about databases, though I have worked with them sporadically over the last several years, and I specifically disavow domain knowledge there. This is not to say that you should not try to learn as much about technologies that touch on your own area of expertise as possible (that's the way you learn more, after all) but you should put a confidence factor into any assessment you make based upon how far it is from your core competencies.
   
   
8. Watch the Edges. The most significant innovations do not occur well within the established body of a given area of knowledge. They occur at the boundaries between two otherwise disparate fields. Most new technologies come from people realizing that an often well established solution or methodology in one field can be adapted for use in a different one. Often that technology is a "bridge" a means of translating core concepts back and forth between the two domains -- such as the use of Analog to Digital to Analog subsytems used in modern car engines that originally had their foundation in sampling music. To that extent, seeing the metaphorical similarities between two systems can often let you apply expensively funded innovation in one area to work much less expensively in others.
   
   
9. Think Systemically. Progress is not linear. It branches and weaves, ebbs and flows, sometimes appearing to stagnate and then get a sudden burst of energy to seemingly come out of nowhere. When one particular technology (or company) dominates, it usually ends up creating back-pressure specific to that technology. In the absence of other factors, these usually balance out, but eventually some change in the technological environment will cause the two dominant technologies to shift with dizzying speeds. As a realpolitik example of this, several countries in the former Warsaw pact have undergone significant improvements over the years, making them competitive in a manner well outside of their relative population side. They've leaped ahead of much larger economies in areas such as telecommunication because they effectively found it more cost effective to scrap decades old infrastructure and start from scratch, gaining the advantage of new technologies without having to spend the time and money needed to develop the intervening stages. Look for factors like this when deciding where a break-out company may come from.
   
   
10. Take Time to Think. In our go-go world, there is a tendency to feel that if you are not at your desk ten hours a day, six days a week, then you are failing (and may even be considered a thief for taking money for work during those unoccupied moments.). Yet I find that by setting aside a period of an hour or so a day for "research", finding out about new technologies, working through patterns, doing something besides sitting and coding, you will generally be able to create better code with more applicability, will be able to better foresee the requirements that you will have, and maybe even be able to see ways around particularly thorny problems. Most of my best insights came to me while I was "taking a break" in this manner.
    
    I've long suspected that many of the problems that plague us today stem from a lack of reflection on the part of others in the past. Sometimes the best thing you can do is just turn off the TV, shut down the radio, and go take a walk.

I'd be interested in hearing about things that you do to help you "see the future". Until next time, enjoy!

-- Kurt Cagle

How "Widgety" should SVG Get?

Widgets have an interesting pedigree. Originally used as a humorous term for a small machine of some sort in a manufacturing process, widgets have made their way into the programming lexicon as a short-hand term for components used in software application development. These components are generally lightweight and are very seldom stand-alone, serving instead as some kind of snap-together Legos* to make application development go faster.

HTML has long had a staple of seven or so such widgets for building web forms, components that are fairly primitive and limited in their capability, but that serve to acquire enough information to make them both easily implementable on about any platform and basic enough to minimize cross-browser differences.

XUL, covered in my previous post, is a larger (albeit still finite) collection of widgets that provide enough flexibility to build what we consider traditional desktop applications. What differentiates XUL from HTML, however, is the presence of another layer, the XML Binding Language (XBL) that provides a set of interfaces for defining new Widgets using a largely XML based framework. XBL can both extend the functionality of existing XUL widgets (and HTML widgets, for that matter) and can define new ones from scratch.

This extensibility mechanism is something that developers should think hard about. Limited component architectures can only carry you so far. Eventually, you come across a need for a component that can't readily be built within the existing widget framework, or that needs to be skinned in a different way than the current set can handle. XBL provides a way of building not only structure but data structres, attaching (or removing) events, and metadata into new items introduced into any given namespace. In simple terms, this means that you can keep your core presentation language (be it XHTML, XUL, or SVG) as focused on its task as necessary, while adding in a reusable set of intermediate components that push into the application layer.

SVG is already an extraordinarily large and complex namespace, one that is proving difficult for many vendors to implement in its entirety. To add into this the requirement for additional core widgets puts two forces against one another that should be cooperating - the vendor community trying to put together decent workable implentations and customers/developers who want to see more functionality at the core level.

When talking about XML technologies, I've often differentiated between foundational schemas - ones that effectively define the toolkits that people use to make the web - and application schemas (the languages that use these primitives to build something useful). SVG is a foundational schema, perhaps even more so than XHTML is. In theory, you could replace XHTML with SVG (though it would entail a huge amount of SVG to do it), you couldn't replaces SVG with XHTML, however. XUL isn't foundational, though it isn't quite at the application level, either - it falls somewhere in between. Again, however, it is possible to create XUL with SVG (not necessarily practical, but possible) while the opposite isn't true.

And this is consequently where the crux of the debate about SVG widgets comes from. It is possible to create an editable text box in SVG (1.2), even if there wasn't an editable attribute in the spec itself. The attribute exists because while it is possible to build this kind of editor with SVG, the overhead of doing this in script is prohibitive (I speak from personal experience here). Beyond this one particular component, however, it is safe to say that most widgets could readily be created by some combination of SVG elements. This is the prevalent view that is spurring the recent developments in sXBL. Rather than establishing this additional set of widgets (an approach taken by the now defunct Corel SVG efforts in creating dSVG), sXBL takes the approach that components can be built using an XML language that would be far more flexible than any limited widget set.

XBL is far from perfect. I am dismayed by the overreliance in the original Mozilla implementation of XBL of RDF, which, while suitable for some applications, adds a layer of complexity to building apps that many developers are uncomfortable taking. I find RDF confusing, and I've worked with it on and off for the last four years. I'd also prefer to see XPath rather than CSS being used as the selector language, for reasons I've outlined before. However, all of this pales in comparison to the obvious benefits that an XBL of any sort provides.

One of the other facets that I think will become a much bigger factor within two-to-three years is the idea of incorporating SVG as an equal partner within the DOM. Adobe's SVG viewer is a powerful application, but so long as it is limited to being "in the box" it's utility will always be more for stand-alone applications rather than components, for memory reasons if nothing else. ASV used as a behavior for defining SVG code directly within a web page has much more potential for making SVG a foundation for Widget development, especially once you have an XBL-like mechanism to create SVG as shadow-entities of other XML constructs. I've played with this some in the Mozilla SVG builds, and there's more than a little power in being able to reference an SVG group in exactly the same way that you do an HTML element.

Right now, we tend to pre-impose what we know of established functionality on our ideas of what can be done with such technologies, but realistically, once you can break out of the box (one of the biggest advantages that SVG promises, when you get down to it) there will be a whole generation of web designers who will see existing web applications as being staid and dull. We need to get to that point, first, but it's coming.

-- Kurt Cagle

XUL Dialog sample from Who ya gonna call? XUL! 

Who ya gonna call? XUL!!

I seem to take a trenchant delight in working with languages and technologies that people have written off as being "dead". The XML User-interface Language (XUL) is one of those. In the long, slow decline of Mozilla from dominant browser to a technological backwater, an intriguing idea seemed to die with it. The idea was simple:

Create an XML based language that would describe a set of more complex interface components than those that shipped with the core HTML set. Use this as the foundation of an application framework, something that you could use to create such things as, well, web browsers. Call it the XML User-interface Language (or XUL, because it sounds very much like a demon out of Ghostbusters), and make it freely available.

This was a good concept, but Mozilla had long since lost its edge, Internet Explorer had become the dominant platform for web applications, and anyway, no one was really doing anything with XML on the browser anymore. And then something curious happened. The Mozilla team didn't give up. They kept pushing forward on new technologies, basing them largely on the W3C standards, though occasionally borrowing an idea from Internet Explorer that seemed pretty decent. They also worked to keep up with all of the major platforms (one of the beauties of code abstraction with XML), so that their browsers began to get better, incrementally at first, then faster in the last few years. XUL grew as well, as new components were integrated into the library, and the pieces began to play better with one another.

Last month, Firefox (a light-weight version of Mozilla 1.7) was announced as a release candidate, a near-final version of the application that would let people play with it in depth. As a web browser, it was pretty cool -- fast (though not as fast as it probably will be), easily extensible and skinnable, with a nod to the dominance of Google by the presence of a Google bar and a GMail extension, to RSS feeds by integrated webfeed support, and to XML with estensive XML and RDF functionality built-in (not to mention much more complete CSS support). This combination by itself was enough to make Firefox an intriguing prospect for me when I was looking at building a content-management-system client. I began to port over my code from Internet Explorer, surprised at the relatively minimal pain in doing so, but what I was developing was very much a traditional web page application. However, as I was reviewing the documentation, I kept coming across references to the foundation set used by Firefox ... XUL.

Okay, I have an admission to make. I'm not very good with C++ programming, at least as far as building windowed applications go. Oh, I understand the whole concept of pointers and references, templates make a certain amount of sense to me, and I can generally follow C++ code without a lot of effort, but I found the the whole reference counting, interface querying thing to be entirely too low-level for what I wanted to do - I may as well have been programming in assembler, for the amount of work involved. Perhaps that's why I gravitated to XML in the first place: I liked the notion of being able to abstract the pieces of an application, getting away from the routine and fairly ugly low-level programming that to me seemed should have readily been handleable by a decent compiler. Visual Basic was a big first step in this direction, but it took entirely too long for Microsoft to acknowledge the fact that VB was not a toy (and hence make it easier to access low level operations when need be). Ironically, while the revamping of the underlying interface description language (IDL) into the CLI (Common Language Interface), Microsoft walked away from the simplicity of VB, the thing that in fact made it appealing to beginning (and advanced but harried) programmers in the first place, in order to create VB.NET, which has all of the painful quirks of VB with little of the underlying simplicity.

XUL reminded me a lot of Visual Basic, but in a 2000's XML-ish sort of way. You can create web components with XUL (as I'll show shortly) but XUL ultimately is about application development - creating applications like web browsers, e-mail readers and editors, or even more staid applications such as sales force tools, systems monitors, accounting sofware, and so forth. It has a lot of the features that have become standard in application frameworks, such as layout tools with flexors that resize automatically as the page does, multi-column grid and list elements, toolbars and buttons, fully supported multi-layer menus and menu-popups, and one of the snazziest HTML editors I've ever seen, in or out of an application - and the beauty of all of this is that this capability is sitting within one of the best web browsers on the planet, meaning that once you complete your application, you can make it available as an extension to Firefox itself.

To illustrate the power of this, the following screen shot shows a XUL application I wrote for editing web content:


Screen shot of application built using XUL toolkit.

I didn't write the CNN site, by the way, though I did select the HTML content and drag it into the HTML editor en-mass, something I thought was incredibly cool. It took me about three days to write the app (though I'll admit that I'd done a lot of the groundwork before) and this while I was still learning the basic XUL API. At the moment I'm working at improving it, tying it into the CMS back end that I worked out, but the important thing to consider is that this is a "web application" - no C++, no VB, just a simple XML document and a block of Javascript code.

Actually, more properly, this is an extension. It is downloaded like any of dozens, and likely soon to be hundreds, if not thousands, of other extensions from the Mozilla site (though because of the nature of this application I can't release this version publicly .... working on an open source version, however). It becomes a part of Firefox, something called up from the Tool menu. Because it is part of the browser, this means that it is also within the secure local space, meaning that you can use this toolkit to create sophisticated, multi-platform applications that can be easily downloaded and updated.

This is not all that ideal for a generalized Internet scenario. It is, however, wonderful for the development of intranet (corporate wide) applications. These are the applications that frequently cause some of the biggest headaches for developers, who have to balance between the world of stand-alone applications and the limitations inherent in HTML web-portal applications. If, instead, you can straddle both worlds with an application framework that practically lives on the web, the potential is pretty much endless.

So what does this alien language look like? Here's a sample:

<window
id="findfile-window"
title="Find Files"
orient="horizontal"
xmlns='http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul'>

<vbox flex="1">

<description>
Enter your search criteria below and select the Find button to begin
the search.
</description>

<spacer style="height: 10px">

<groupbox orient="horizontal">
<caption label="Search Criteria">
<menulist id="searchtype">
<menupopup>
<menuitem label="Name">
<menuitem label="Size">
<menuitem label="Date Modified">
</menupopup>
</menulist>
<spacer style="width: 10px;">
<menulist id="searchmode">
<menupopup>
<menuitem label="Is">
<menuitem label="Is Not">
</menupopup>
</menulist>
<spacer style="width: 10px;">
<textbox id="find-text" flex="1" style="min-width: 15em;">
</groupbox>

<spacer style="height: 10px">

<hbox>
<spacer flex="1">
<button id="find-button" label="Find" default="true">
<button id="cancel-button" label="Cancel">
</hbox>

</vbox>

</window>

This is not the code for the editor, but rather for a dialog box. The root element for XUL is the <window> element, which serves the same purpose as <html> does for HTML. The other elements, including group boxes, menus, buttons and so forth, define specific widgets. For instance, the <button> element defines a button, including it's label, identifier, and default state.

The flex attribute is quite useful. It defines the degree to which the element will attempt to fill up the available space. A flex of 0 indicates that the element will take up the minimal space defined by it's default size, or by it's CSS indicated size if available. On the other hand, a flex of 1 indicates that the element will attempt to expand until other elements push back on it (for instance, a multi-line textbox will attempt to fill the entire window if it's flex is 1, stopped only by other elements. Two or more elements with the same flex will divide the space in half, while two elements of flex 1 and 2 respectively will each take up 1/3 and 2/3 of the available space as appropriate. This simple-seeming innovation can radically cut down on the amount of "resize" code that you have to write.

XUL applications can be run either from a user's local machine (through an RDF based configuration file) or it can be run from the web. The latter case puts some major restrictions of local file access, however, but other than that can work fine. You can embed XUL elements within HTML (for Firefox browsers only, of course) by using XUL's namespace:

xmlns='http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul'

or you can create XUL applications with XHTML interspersed. Typically, this will require that you modify your server software so that it can legitimately serve up XUL documents recognizable to firefox. In IIS, this would be done by adding the XUL mime-type:

application/vnd.mozilla.xul+xml

to the list of mime-types that IIS is aware of (select the properties pop-up menu item for your server in the IIS control panel, select the Mime-Types button, then add the appropriate associations for .xul as application/vnd.mozilla.xul+xml. With Apache, you add it to the associated mime-types by editing the httpd.conf file.

XUL Resurgent

Interest in XUL has picked up dramatically ever since Firefox came into its own as a separate application, and the combination of ease of use, cross-platform compatibility, intrinsic web awareness and a growing community of developers makes it a technology to watch.

One problem that XUL faces is that the documentation out of the Mozilla site is sketchy at best. However, XULPlanet at http://www.xulplanet.com has several good XUL tutorials and reference documents that are up to date, if not necessarily complete or exhaustive. I would strongly recommend if you're interested in XUL that you check out that treasure trove first.

XUL is, in many ways, analogous to Microsoft's XAML, and with the recent work on the part of the SVG community making its way into the Mozilla effort, I see XUL and XAML effectively going head to head as the first of a new breed of meta-application development languages. XAML is perhaps more exhaustive, but until the Mono effort gets further along, it is not as well represented on non-Microsoft platforms. Additionally, XAML puts a lot more of its emphasis on the use of C# code-behinds, an approach that makes sense for very large enterprise developers but seems like overkill for the average programmer.

XUL may not be your cup of tea, but you owe it to yourself to at least check it out ... as an application framework, it may very well become one of the big technology stories of 2005.

-- Kurt Cagle

The Era of the Mega-Storm

On a completely non-computer related note (there's been a lot of fodder for thought today), I noticed a headline that Ivan is coming back. Yeah, you read that right. Hurricane Ivan, one of the most lethal storms to hit the East Coast in a century, was pushed back out to the Atlantic, where it caught a cross stream, headed back into the Carribean, gathered strength, and seems about ready to pound the Gulf Coast as a tropical storm - oh, and the remnants of Hurrican Jeane seem poised to do the same thing.

Global warming is often taken by those unfamiliar with complex systems as a gradual warming uniformly across the world - temperatures go up by one or two or three degrees, which pushes summer up a little bit. However, this assumption can be dangerously naive. Global Warming is actually causing cooling to occur in certain portions of the world, even as the average temperature increases. What changes is something more important - the amount of energy in the system.

Complex systems are not random ones. Rather, they form regions of quasi-stability, where a particular whether pattern become apparent. Until fairly recently, the weather patterns in the Gulf of Mexico have had a certain pattern - storms would form in the Atlantic (largely because of heat, sand and contaminants picked up in the growing Sahara desert) and would move into the Caribbean, where the reasonably warm Caribbean water created a thermal gradient called a convection cell, in which cold air pushed down into a column of cold air, hit the warm Caribbean water, expanded outward as it was heated, rose in a donut shape back into the upper troposphere, where it then cools again and the pattern repeats. Because of Coriolis effects, this also begins spinning, and as more energy moves through the torus, the spin becomes faster. This is the basis of all cyclonic weather systems (i.e., most storms).

Once that Cyclone hits land, the energy that sustained it (the warm water) disappears, and the storm begins to lose energy. Usually, by the time it makes it way more than a few hundred miles, the storm has lost enough energy that it can't maintain the central column of air, and the storm dissipates. Usually. But not this time. This time, Ivan had SO MUCH energy, due to the extraordinary warmth of the Carribean, that it managed to maintain its cohesiveness even after making its way up to Pennsylvania, and once over the (warming) Atlantic, was able to reconstitute itself.

The Earth rotates, and that rotation in turn drags on storms and moves them into large circulation systems. Because most storms in the past haven't had that much energy, they fall apart far before they are significantly affected by the effect. Yet somewhere along the line, we may have entered into a new regime, a new semi-stable weather pattern, which is most ominous. If cyclones have enough energy to make a complete circuit of the Carribean circulation cell, then you have the possibility of a storm becoming a "repeat offender", persisting as a distinct entity for two, three, or four passes before entropy finally tears them apart.

What's worse, as storms re-enter the Carribbean, there is an increasing likelihood that a new storm can "cannibalize" the remaining energy of an older storm, with the effect that a Category 3 Storm gets bumped to Category 4 or 5 with regular frequency, and the likelihood increases of hypothetical Category 6, which would, by extension, have sustained winds in excess of 180 mph. To put this into perspective, Ivan's "sustained" winds topped out at 165 mph, but there were gusts within the storm that well exceeded 210 mph. A category 6 storm would have gusts in excess of 230 mph, which puts it into the regime of damage caused by a tornado (not to mention that storms of this magnetude would be spawning tornados by the dozens as such high "gusts" would be subject to incredible cyclonic pressures).

Average global temperatures have risen roughly 1 degree Celcius in the last twenty years. If existing trends continue, global temperatures will rise between three and ten degrees Celsius by the end of the century, with most computer models pointing toward the upper end of this range. That's energy going into creating even hotter water in the Caribbean and consequently even more powerful storms there. This raises the possibility of a permanent cyclonic band forming over the Southeast US, with hurricanes becoming weekly occurrences from March until November. Not a pleasant thought, to be sure ....

Patent Absurdity

We interrupt our regularly scheduled program for an important note from a commentator to this board:

Microsoft is trying to stop open formats with the accumulation of patents. In New Zealand, Microsoft have lodged the following patent applications:

Patent Number: 525857
Markup language and object model for vector graphics

Patent Number: 535067
Markup language data structure

The applications are in the examination stage. If they're approved we have three months in which to lodge objections. For more information contact the Intellectual Property Office of New Zealand (IPONZ):
http://www.iponz.govt.nz
info (at) iponz.govt.nz
hearings (at) iponz.govt.nz

Microsoft is also out to stop Open Office, as can be seen with this NZ patent application:

Patent Number: 525484
Patent Title: Word-processing document stored in a single XML file that may be manipulated by applications that understand XML

I have railed on the abuse of patents in the past, and had hoped after the debacle with ActiveX components in IE that Microsoft legal would have learned its lessons. However, with this announcement that appears not to be the case.

Let's get this straight. Microsoft was an initial signatory to the W3C SVG working group, where one of the requirements in joining the group was a discovery process indicating that the signatories would disclose any previous patented technologies that may in some way invalidate the claims by the W3C on the technology as a Royalty Free standard. Microsoft was one of two players (and I suspect was a bigger one) in the August 2001 debacle within the W3C that attempted to introduce (unsuccessfully) RAND (Reasonable and Non-Discriminatory) patents into the W3C structure, specifically to cover SVG, patents that would have introduced a license fee into the open standard itself.

Microsoft dropped out of most W3C efforts and started the WS-* standards group, which has to date produced one interesting security related specification and a great deal of otherwise useless crap. It did this in great part not to expedite the W3C standards process (any legitimate standard can take years to reach fruition, something they well know) but to avoid having to contribute to the open standards community and risk losing what they perceive of as valuable intellectual property to their competitors.

As far as the vector standard goes, Microsoft doesn't have much of a leg to stand-on. Adobe has a markup language and object model for vector graphics. It's called Postscript. While not XML, it predates ANY effort by Microsoft by decades, and from personal experience Postscript was in fact being used as an object language as long ago as 1983 (through the agency of Wolfram, the makers of Mathematica). Moreover, even in the realm of vector graphics languages, Adobe had a precursor to XML called PGML (Precision Graphics Markup Language) that predated Microsoft's only legitimate claim on an XML based vector language - VML (Vector Markup Language). Currently, while still available via a download, VML is largely considered now a historical oddity, and has all but disappeared from the Internet.

Microsoft also had another claimant to the vector markup language process to muddy things up - the ill-fated Chromeffects. It was an XML-based initiative to provide a 2D and 3D graphical and multimedia framework into Windows, way back Sound familiar? Can you say XAML, boys and girls? I knew you could. Chromeffects was too ambitious for its time, working far too slowly for anything even remotely resembling a real-time desktop system. Ironically, this may have been the reason for pushing Avalon to 2007, nearly a decade later. It was shelved in November, 1998.

Now here's where things get interesting. One of the requirements of the W3C's Recommendation process is that there must be two working implementations of a given Candidate Recommendation before it can be recognized as a full Recommendation. It's a fairly rigorous standard, and one of the things that it insures is that no one company can submit their technology into the W3C without at least one competitor having a chance at also being able to compete in the marketplace. It's one of those little subtlety things that has kept my admiration for the W3C very high, even when they do stumble.

Chromeffects never worked; it was withdrawn before reaching more than a very preliminary beta. Most patents require a minimum requirement of a demonstrable prototype, to insure that something that the inventor claims should work really does.

I'll not get into issues of fairness or unfairness here - patents in 2004 are intrinsically unfair, as they are typically used by corporations as a way of stifling competitors by unravelling the support skeins that others build their technology on. Even if the patent claims are disproved, the legal resources required to disprove the claim can significantly weaken competitors. My personal take is that patents on IP should be abolished, but that's not going to happen in the current culture of oligarchic fascism.

Open Office Targeted

The claim against Open Office.org (OOo) represents yet another attempt by Microsoft to eliminate by legal means what has proven to be a remarkably embarassing opponent otherwise. The use of the "single XML document" strategy in the OOo patent attack shows that Microsoft realizes it's case is extraordinarily weak there as well, perhaps weaker than it is for vector graphics.

Open Office made use of XML in a very innovative way - using XML as the way that information was stored for office applications, chief among them word processing. They chose to go with an array of five XML documents for efficiency - by breaking what could have easily been a single document into five they could maintain several smaller DOMs simultaneously for handling different aspects of an application. I've used OOo XML as the foundation of my own publishing efforts more than once, and the division makes a great deal of sense. They were also innovative in putting this XML together as a bundled package in a gzip format, another widely recognized open standard, making it possible to send related content without the need to encode binary information in XML, something that is both expensive and most cases unnecessary.

Microsoft has been criticized for years about not making their format user-accessible XML, especially after OOo beat them to the punch, and they would still probably not have done so had not Open Office begun to eat significantly into their sales of MS Office. They did have XML encoded within Office for metadata beginning in 2000, but even there it should be noted that this information was fiendishly difficult for users to extract and utilize - it was a marketing point with very little substance, and the company was roundly criticized for marketing their XML support in this manner when in fact they had nothing useful.

Open Office is a nightmare for Microsoft. It has improved dramatically in recent years, to the extent that it has begun to pull ahead of Microsoft in several key areas of development. It has forced Microsoft to drop the price point on their most profitable product several times, and they can't really win against a product that is available for free. The extensible nature of Open Office, while still too complex, is making it more appealing to those wishing to create specialized builds, while its lack of a commercial application has made it the darling of many a national or state government body with constrained resources and high needs. It has even made inroads in the one area where Microsoft thought it never would - the large-volume enterprise customers that Microsoft thought they had locked up after pushing Word Perfect down a hole.

Given all that, a legal attack via patent was pretty much inevitable, but it is also pretty baseless. Using the argument of single document XML belies the point that for XML, the distinction between one document and five interrelated documents is spurious - you can turn five into one by enclosing all of the documents into a single node in an XML tree. Given that XML is, itself, an open standard patented by the W3C, Microsoft's attempts to claim any kind of legal foundation there is laughable.

New Zealand?

The fact that Microsoft is choosing to pursue this in the country of New Zealand is also telling. Microsoft knows that it can't secure these patents in the United States, but if they can get an international patent they can effectively claim that it supercedes the US patent. I would like to think that the New Zealand patent office is able to smell this pile of dung for what it is, but perhaps Microsoft hopes that New Zealanders are technically unsophisticated. I rather doubt they are ... they've been burned by the promises of large corporations bringing "better"-run governments via privatization only to have the CEOs of the same corporations rob them blind, runing what had been a model democracy in the process.

Still, such patents can cause havoc, especially given the resources that a Microsoft can bring to bear once it has such a piece of paper in hand. I will be contacting the New Zealand patent office to offer my comments and would recommend to all my readers that you do the same. Microsoft has the resources to build world-class software applications, but there is a certain innate laziness that the company has in choosing to use legal means to stifle competition; these suits reflect badly upon it, and lay to rest any "outreach" efforts that Microsoft may be attempting to make into the Open Source community. Perhaps if they had to actually compete, Microsoft might finally end up making software that was simultaneously secure, stable and usable.

Kurt Cagle
Metaphorical Web Publishing
http://www.metaphoricalweb.com

SVG and the Search for <elegance>

After reading one comment to my previous post I thought it might be worth actually replying to it as part of the "Main Sequence". Much thanks to Mario for asking these.

Once upon a time, in the not so distant past, I considered myself a professional artist. I worked for a computer game company, drawing highly pixelated versions of Vanna White for one PC version of Wheel of Fortune. I worked as a computer graphic illustrator for an agency a few years before that, working on schematic maps and charts for annual reports of some fairly large companies. Thus I've always had a certain basic sensitivity to the needs and problems of artists, though of late my artwork tends more toward science fiction/fantasy than Fortune 500.

Artists, contrary to some popular perceptions, are seldom lazy people. A good artist can spend 50-100 hours working on a piece of work, making seemingly minute changes to a piece of work or spending hours doing nothing but stippling (putting dots of ink onto a page) or other very repetitive tasks. The difficulty for most artists comes from the fact that they become extraordinarily focused on what they are doing to the exclusion of everything else, which means that many of the mundane tasks that most people take for granted simply don't happen except in those rare intervals where the artist can surface. This can be taken by outsiders as being indicative that artists are typically very reserved (they generally are, except among themselves) and occasionally aloof.

In some respects, I could replace the mind-numbing (or soothing, depending upon temperament) process of meticulously creating a drawing with the act of writing quality algorithms, and the description of artist becomes one of programmer, or more properly, the aesthetic programmer.

There are a number of programmers who are more interested in the result than the process - these are programming engineers, who often tend to deviate very little from the established procedures for performing some action, and who as a consequence find the creative aspects of both art and programming subtly unsettling.

The other kind of programmer, the aesthetic-programmer, may not be the best programmers for putting together a human resources database system, but they are the ones that ultimately end up driving most of the new innovations in the field. I believe that this is because what they are seeking is that elusive quality called elegance.

Elegance is a difficult concept to explain, even to other programmers, because it is in effect a measure of the aesthetic qualities of a piece of code. Certain scientists understand this quality instinctively - high energy physicists for instance, who tend to operate on an extremely abstract level, and I think that one cannot be a mathematician without a desire to seek this elusive property. Einstein understood this with his elegant statement E = mc², and his more elegant though much less well known G^mn = 8pT^mn.

For an artist, elegance can be thought of as the perfect balance of color, composition, lighting, symbolism and realism. For a poet or writer, elegance is prose or poetry that has the right number of words, for the composer the right number of notes. Even for a master, elegance is elusive, a state that can only be achieved infrequently, but what gets created in the process often ends up setting the standards for what we consider the foundations of both art and science.

So where does SVG fit into all of this? Or put another way, where's the tag within SVG? It's not a part of the 1.0 or 1.1 draft, I know that for certain, and it's very likely that it will not manage to make its way into SVG 1.2 either. Okay, I'm being a little frivolous here, but there is a solid reason for me asking such a question. SVG is something of a platypus, ornithorincus anatinus (the name of which I remember, curiously enough, from a Mr. Roger's Neighborhood song). It is a graphics format. It is an animation format. It is an interactive GUI format. It is a DOM for performing integrated web services. It's becoming a publishing format. Like the duckbill platypus, it seems like it was stuck in some kind of bizarre transmogrifier ray, a la Vincent Price's The Fly, neither bird nor mammal but somewhere in between.

There's never really been anything like it, to be perfectly honest. Flash often comes to mind as the point of comparison, but in reality, Flash lacks the capabilities for abstraction that are intrinsic to SVG. Don't get me wrong on this - Flash is a very powerful tool for creating impressive looking graphic animations. The difference between Flash and SVG, however, is that Flash is a self-contained world; SVG on the other hand is beginning to shape up into an application that entwines itself within other specifications.

This will become more obvious when SVG moves more into the native space of browsers and operating systems, rather than being a plug-in. The significance of the Mozilla SVG effort, even at its current nascent stage, is that you can create interactive and animated graphics inline to other markup such as XHTML or XUL. This means, among other things, that the graphics on a page are immediately accessible as part of the DOM, are integrated into the whole fabric of a web page both programmatically and visually.

This DOM interaction has a lot of import. With Flash, it takes a certain amount of skill to write interfaces into the rest of the browser, and those interfaces are perforce very limited; in general, it is easier there to put most if not all of the interactive capability for the "page" in the Flash, bypassing the web page completely. Beyond such issues as text search or retrieving of meta-data for search engines, this process means that Flash ends up creating its own "browser" environment that can render a site useless to someone without the Flash plug-in (ask anyone on a typical Linux machine until recently what that means).

Native SVG support, which is on-going, changes that dynamic. Graphics can be changed at multiple levels of abstraction, and can produce a W3C consistent notion of the concept of level of detail (LOD) - initially a 3D concept where distant objects are rendered at the lowest possible resolution until a certain threshhold distance is reached, at which point a higher polygon mesh and higher res textures are used to render the object more realistically. LOD in a 2D sense means that information is only provided once the attention is focused on a particular area in a graphic, and hidden once that area is no longer the focus.

Graphic libraries can be easily created and integrated with SVG - not static graphics but dynamic ones generated from live data streams. Mapmakers have already discovered this use of SVG, but so far artists generally haven't. The distinction between graphic and multimedia presentation blurs as a consequence of this one facet of the language, as does the distinction between presentation and application. An SVG illustration, embedded within a web page (or a PDF), could read the story around it, could change from visit to visit, could customize itself to its audience. Moreover, it could leap out of its apparent visual boundaries and become an inhabitant upon the page itself, participating not only in the telling of the story, but in the reader's interaction with it.

Lest someone bring up to me the woeful tales of Bob and Clippy, the autonomous agents of Microsoft fame reviled pretty much universally, I'd content that the problem of such agents lay less in their concept and more in their focus. An agent is an extension of the reader, not of the system. As such, an agent is more like a pet, a playful kitten perhaps, than a majordomo - it will play in the corner of the screen, chase after screens, perhaps piddle on the desktop, but what it won't do is set expectations on the user by "waiting" on them. It's that elegance thing again - effective design means understanding the psychological driving factors of the people who use the technology, means putting in the extra effort to imbue agents with a sense of awareness.

The reason I'm focusing on agents here is precisely because all intelligent graphics ultimately are agents of one sort or another. They interact with the user, provide some sense of awareness with their surroundings, and can have memory. Sometimes they can be abstractions of living beings, but these are simply the most ostentatious of such graphics; one of the failures of Bob was the inability to recognize the universality of agents, concentrating upon the "reality" of the agent rather than upon the notion of intelligent graphics.

The key to elegance is to understand that everything has its place, its modality of expression, and the most elegant solutions are those in which the medium is utilized to its best effect. An SVG artist is creating intelligent graphics, autonomous, semi-aware agents, and the best artists in that medium will create art like no other, for it will be art that speaks to you, perhaps quite literally ... and if that's not the quintessence of elegance, I don't know what is.

-- Kurt Cagle