Metaphorical Web

Shared Spaces

2011-05-22T10:52:00.000-07:00

This is the world of Jayne Lucier:

Jayne is currently working towards a PhD in Computational Linguistics at the University of Washington. She's single, fit, twenty two years old, with short brown hair worn spiky, brilliant green eyes behind a pair of barely there glasses, a penchant for plaid skirts and vests, grandmother boots and backpacks over purses. With a little exaggeration, she could pass for a Miyazaki anime character, though there are hints of the more mature woman in her smile and posture.

Jayne lives in a shared space with three other girls - Tanya, Candice and Mykie - in which each has a separate room but they share a communal kitchen and den along with two shared bathrooms. The rent isn't cheap, even with the four of them sharing it, but in this day and age that's hardly unusual. The walls in the common rooms have artwork screens placed at convenient locations, with the rule being that while anyone can set any or all art at any time, they can't be something that would send the parental units screaming in anger or fear. A couple of other larger screens are for more traditional uses - gaming, news, the latest shows - and they're on a first come, first serve basis, though if a roommate doesn't reclaim it every hour it's up for grabs.

Every girl has a smart phone and a pad, which the shared space uses to determine who is where at any given point. The phone's usually considered a better bet - pads, while important, usually tend to end up in backpacks. Jayne's room also has a full screen on her desk, along with a wireless keyboard that she can use in lieu of her device's still cumbersome onscreen keyboard. Besides, with the keyboard it gives her more real estate to write on.

The morning ritual is usually pretty much the same - up at 7am when Jayne's pad communicates with the audio in the room, playing everything from Gustave Holst to Emilie Autumn from her playlist at random, into the bathroom to take a shower while the pad on the medicine cabinet reads out comments from the students' she's doing Teaching Assistant for, as well as any messages that came during the night. She can also bring up the house status, both from the sensors on the internal systems - #toilet 1 has a leak and needs to be repaired, report sent to building maintenance - and from the external ones: Candice->Mykie: The SalonCare Shampoo is MINE. You take it, you DIE!!!! . Yeah, Candice has issues.

The medicine capinet, cupboards, fridge and linen closet all have RFID scanners, as does her closet, though that one she could disable if she wants. She can see at a moment's glance what she has, where it is, how long it's been in there, what's likely close to needing to be replaced. The medicine cabinet informs her that she still needs to take four more doses of the Amoxycillin, which was prescribed to her after a fall cold turned into bronchitis, and grimacing, she pours out the foul smelling liquid into a small cup, drinks it, makes another face, and puts the lid back on, the top's click sending a signal via a simple active RFID transmitter in the bottle to decrement the count.

The shared apartment has its own online presence - all of the ones in the building do. It includes a message queue (which she moderates - as the oldest and farthest along in her studies, there was no question that she would) between each of them, a shared chat thread (which is usually pretty quiet, but sometimes can see a lot of activity, especially when Tanya brought Jack home a few weeks ago which immediately set of a bidding war. Jack never knew how close he came to having Mykie hijack him, but she settled for a pair of Avril tickets. It also contained their respective calendars, showing who had what duties when, who was out of the house, who needed a few moments of privacy with a significant other. It didn't always work, but there was enough incentive there to keep it up to date that the girls participated.

Jayne dressed, frowning at the long list of clothes that were currently marked as dirty. She'd need to do laundry tonight, or she'd be down to one particularly inappropriate pair of rave pants and a peekaboo mesh blouse for teaching in - probably not something that Dr. Hannaford would be particularly pleased to see her in - though no doubt a few of her more erstwhile students might.

"Reminder - laundry tonight," she said to the air, and the air dutifully recorded her reluctantly claimed penance. Somehow they never got around to automating laundry.

Dressed in a white blouse, forest green vest and green plaid skirt and knee-hi's, her hair up in a bun, she knew the image she projected, but frankly didn't care. It was her teaching image, and the one time she'd shown up in jeans and t-shirt (because she had forgotten to do laundry) everyone from her professor to half her students to her friends in Vladivostok and Stockholm had written notes of worry and concern.

Closing the door (it automatically locked when her phone left the room unless she voice-overrode it) Jayne wandered down the hallway, looking at the latest artwork choices of her roomies - Kandisky (Tanya), a DNAngels anime drawing (Mykie), Hot Firefighter for October (Candice - that one bordered a little too close to the edge of the parental unit rule) and Waterhouse's Lady of the Lake (hers). Candice splayed out on the couch, texting furiously with her ... girlfriend, Jayne remembered, the gender and identity tended to change weekly - while Mykie came out of her room with her oversized pad in her backpack and the zombie-look of Starbucks withdrawal plastered on her face.

Mykie was their resident artist, and the pad she carried was her canvas and palette as well. She'd taken one look at Jayne when she'd let the empty room (Kirsten had graduated the previous semester) and immediately pulled out her electronic pad, slapped a sheet of paper on it, and started drawing. The schoolgirl that had emerged from Mykie's pen had ended up both on her wall and on her "school wall", one if physical media, the other in virtual, and Mykie had been delighted to let Jayne use that as her "logo".

Tanya emerged from the refrigerator, a bowl of cereal and nearly empty jug of milk in hand.

"We're almost out of milk," she said around the cereal already in her mouth.

"Already on the delivery queue," Jayne replied, "and it's your turn to meet and greet the delivery man."

"Oh, da- ... sorry, I forgot. I've got another appointment."

"Did you calendar it?"

"Um, er ... no ..."

"Fine, then you either work it out with the delivery guy or you get to pick up the tab for all the groceries this week, 'cause I'm not going to have the milk spoiling and soggy ice cream because someone forgot."

"I ... okay. I'll see if I can reschedule."

"Good. 'Nuff said."

Jayne despaired at Tanya - she was a good kid, but she tended to have her head in the clouds, and they weren't electronic ones. She grabbed a sweet roll from the pantry, threw another one Mykie's way that was caught deftly as the artist headed out the door, and waved an unreturned bye to her remaining roomies.

As she walked out the door, the local shared space handed her off to the outside world (her choice, she usually wanted to maintain context awareness). She could still bring up the apartment's shared space if she needed to, but the phone typically latched onto whatever context (usually spatial, but occasionally temporal) was most appropriate for configuring the apps available to her. Jayne liked called her phone her context machine.

She grabbed the phone from her vest (she liked the vests because they included an outside pocket ideal for phones and inside one ideal for pads) and pressed the UW bus app, which picked up her location automatically and passed it to a central dispatch which in turn aggregated the calls and arranged for a least time path between pickup points. The app returned a modal notification: "Green Bus in 10 minutes". She'd bought a year pass, but if she hadn't, the app would have simply debited the fare from her account. The university had taken their time getting the system in place, but this had proved one of their biggest successes. It pushed ridership up dramatically while insuring that they didn't have buses driving empty when they didn't have to. There were still a few traditional loops, but they were being phased out as the number of non-online students dropped precipitously.

Jayne shared an office with Doug Stillwell, a PhD student in Computational Linguistics, though truth be told they could have just about hotelled the grad student offices, which were, for the most part, little more than closets. They'd already started doing that in other buildings, and she suspected that Informational Systems was probably next. When she needed to face-to-face with a student, Jayne usually preferred to meet them in the mini-cafeterias in the basement. The professors were still very jealous of their own offices, and could not understand why the younger generation seemed to be so blasé about giving up theirs. Of course, Jayne thought, the whole university institution was undergoing so many changes that it was likely not to exist in a remotely recognizable form ten years later - the Post-University Deconstructionist Movement, as the more philosophical members of the school called it.

The bus dropped her off at Mary Gates Hall, and she ambled to her classroom. She'd spent the evening grading papers online (for her XML class) then laughed quietly as she realized that there'd never been a paper anything - she'd read through each q&a pair as they came up on her screen, marked up wrong answers both inline and in a separate comments block and invited additional comments and clarification, gave each answer a rating, then summed up the rating for a grade. In a few places, she'd annotated with links to their textbooks at the appropriate paragraph or external links to the web, and in a few others she'd added video commentary directly, though she preferred not to do that - more than once, she'd found her face staring back at her on Youtube under the heading "Hot for Teacher", usually with a contemporary rock song playing in between her pauses. Never give students ammunition they can use against you.

She entered her classroom with ten minutes to spare, and stood in the doorway as Doug (soon, Dr. Doug, as she liked to tease him) finished up his own lecture on semantics. Doug was ... cute - puppy dog nerd cute, but was finally growing into his 6' 7" in frame. He'd never be a face person, but he was passionate about his research, was damned good with the electric mandolin, and not only could he get a computer to sit up and beg, but he looked good in a Steampunk outfit, as he'd shown when he arrived at the office after a weekend at a con. They'd already been out on a couple of dates, but neither were in any hurry.

The classroom itself was tiny, no surprise there, but what was surprising was that it was intended for a class of up to 300 students, even though there were only sixteen seats in the room itself. A teacher in this day and age was expected to be a television performer; the video setup in the backroom was mostly automated, though there was usually a kid - Steven or Kaitlin for her own classes - who sat and monitored all the classrooms just in case from a central station. The school, like most universities, had realized that students could record lectures and leave, and that seats in butts was becoming an increasingly anachronistic concept. Some students still showed up for class, but for the most part classes were taught remotely, and questions were handled asynchronously and offline.

As a consequence, her classroom was tiny, but her blackboard was infinite. The classroom's shared space had picked up her pad the moment she walked in the door, but wouldn't relinquish control until after Doug chose to, which he did with a flourish ...

"... and up next, the sexiest Computational Linguistics teacher on the UW campus, the ever lovely and charming Jayne Lucier."

"I'll get you for this," she mouthed at him as he walked past her, and he winked in reply.

Still a little flushed, she headed to the front of the room, her pad activating the university presentation app, automatically slaving the monitors to its controls. She rather liked the presentation that she'd come up with last night - her presentation skills weren't quite at the level of Mykie's, who could do absolutely incredible game quality work, but like most people her age she'd cut her teeth editing digital video and presentations - it didn't hurt that she had also paid Mykie a reasonable sum of money to help her put together clips for her talks.

The students filed in - she was female, she was pretty, and she liked interacting with live students, so she had a pretty full room - but she also noted that her class was currently being audited by 1,052 people worldwide, though she set a maximum of forty participating students just to keep ahead of the homework. The auditors could take the tests and would eventually see the answer keys so they could self correct, but they didn't get credit. Still, they could (and did, vigorously) participate in the chatter stream, and often times would provide help or assistance even as she was giving her talk (there were a couple of professors at MIT who interacted with her show regularly, as well as several engineers at Google, Microsoft and Intel, and she doubted she'd have much trouble landing a job or a post-graduate position once she was done with her own PhD).

The teleprompter monitor just outside of the camera's vision, which showed her with the composite background from her pad as compared to the green-screen behind her, flashed the ten second signal ... five, four, three, two, one ..

"Welcome to XQuery 102, Week Fourteen, class 25. I'm Jayne Lucier, your instructor. Last week, we examined the use of faceted search methods in order to ..."

This particular sketch - not quite a story, more a bit of exploratory narrative - originally came to me in a dream . It highlights both what could be in the very near term future and where we are not quite yet at. There are a few key highlights that should be food for thought:

Smart phones in particular truly are context machines - at any given point, they know who you are (assuming you are the owner of the phone), know where you are, and know when you are. This is a powerful combination. With these three pieces of information, shared spaces become possible. A shared space is place that can take advantage of that context to make facets of itself available. The applications that you need at home (whether shared or not) are different from those that you need at work or on the road.

Shared context can be combined with membership to identify groupings of services, and to provide appropriate levels of access for those services. Jayne and her roommates could purchase specific food for themselves as well as food used in common with that handled by automatic debiting, could have the food ordered in the background, could have it delivered at a time and space most appropriate for everyone (though someone still has to put it away :-). Shared spaces allow the access to display devices (the various public screens in the apartment), services (online delivery of movie content) and responsibilities (who gets to take out the trash or wash the dishes this week).

Such a shared space is a web presence (Facebook for places, if you will), and it's about more than just people - the toilet announcing that it has a leak, giving a telematic warning to the building's superintendent, or the shared space of an individual, the clothes in her closet and the medicines in her medicine chest. It's also a space that knows it's own physical boundaries - Jayne walking out the front door hands her off from the apartment's shared space to other broader shared spaces. She can still log into the more focused ones, of course, but ultimately it is the ability to move from one appropriate context to the next that makes remote devices so compelling.

Shared spaces also open up some interesting issues; monetary transactions become largely background issues. We're getting close to that now, but for the moment, such transactions are still largely at the institutional layer (and the need to wander around with dozens of cards). Things like QRCodes and Near Field Communication (NFC) should speed that along - the ability to go from visual to device to web is powerful, as is the ability to use the device as a physical replacement for credit cards, vendor cards, metro bus cards, membership cards and so forth. Of course, security becomes a much bigger concern as well - loss of a phone could be inconveniencing, theft of one could be financially (and reputationally) damaging, but I frankly do not believe that this factor alone will be sufficient to deter these technologies from moving forward.

Indeed, one area that I think QRCodes in particular could help is that it provides a powerful way to change your shared space. Go into a store and scan it's QRCodes, and all of the apps that are available for that store become visible, from deals to chat rooms to means of optimizing payment. Go to a convention and can it's QCode, and you then have the whole shared context for the convention's activities, feeds, maps, and presentations. Scan a course's QRCodes, and you have apps that provide links to syllabi, bios, resources, feeds.

In this respect a shared space is a portal, but its a portal of both links and applications, and what's more its a portable that knows you and can provide you services based upon access levels and need. Because it knows time, it can look at your calendar, it can update lists of things that need to get done, and it can even make inferences once enough history is gleaned.

One additional aspect of this is that we're moving to a world where physical proximity is no longer necessary in order to interact with the world, because we can replace this with information about context. Mass transit systems -- especially buses -- are comparatively inefficient - they still require that you move to an access point in the system in order to get on it or get off it, and that access point may be very inconvenient from where you need to go or be picked up from (this is especially true in suburban neighborhoods). On the other hand, by aggregating requests and using that to determine the optimal routes moves you to a door-to-door service while at the same time providing for the most efficient distribution of stops. While some communities have a similar system today that's phone based, for the most part these are limited to people with disabilities, and as such have a comparatively small participation rate.

Similarly, teaching, training, sales conferences, and the like have been moving to a distributed model for some time, though the public educational system is perhaps slower there. In many respects one of the central problems with distance teaching is the belief that it is necessary to build elaborate software presentations on top of content. That's certainly possible, and I think as we move forward the ability to present effectively online will need to become a much bigger skill-set for teachers at all levels, but at the same time, the rise of collaboration tools means that it becomes increasingly possible to use proven pedagogical principles in order to teach - even Q&A type essays can be handled in this manner, and over the web (and especially over mobile) this may in fact be the most effective means to teach complex content.

Additionally, there's a need to recognize that mobile devices (pads in particular) will become the de facto mechanisms for both giving and receiving instruction moving forward - and that these will be used asynchronously. We don't think with our butts. We think with our heads focused upon a particular instructor or information provider, and the reason that the lecture is still a fundamental tool in the teaching lexicon is that such a lecture is a means to achieve that focus.

Yet attending a lecture when awake, when distracted either mentally or physically (school chairs, anyone?), or when attempting to digest a previous lecture on a different topic, is usually counterproductive. By separating the presentation from the perception, it makes it possible for the student to determine the time when he or she is focused. It also makes it possible for the crowd sourcing phenomenon to extend beyond the immediate moment, giving it time for people to ask questions or make points, and for informational gestation to occur.

The idea behind putting this scenario out is to help understand where the technology is (and more importantly where it isn't and could be) and to envision given that what improvements in technology (or even what technologies themselves) are needed to achieve that technology. I hope to do this again in different contexts.

First published at http://blogs.avalonconsult.com/blog/e-learning/shared-spaces/

Bad Call? Microsoft buys Skype

2011-05-10T10:52:00.000-07:00

Software titan Microsoft just purchased Skype, whose voip-based services have made it one of the largest players in the web telecommunications space. The deal, for $8.5 billion in cash, provides a major benefit to Microsoft, which has struggled to remain competitive with their Live Meeting offerings and significantly expands their consumer base, but also indirectly provides benefits to Facebook, a Microsoft investee - by marrying Skype capabilities with Facebook's core systems, Facebook can get a significant leg up on phone connectivity between its members significantly expanding its standing as a social communications medium.

For Skype, the acquisition by Microsoft also places the company into a position where they can expand their offerings into the enterprise space that Microsoft has a major presence in, a market that Skype had difficulty penetrating before. This in turn provides a direct challenge both to Google, which has been trying to expand its Google Voice offerings to do the same (in conjunction with Google Docs and their email services), as well as Cisco's enterprise VOIP and virtual meeting software and hardware.

What I find intriguing about this particular buyout is that Skype will in effect become a separate division of Microsoft, one reporting directly to Steve Ballmer. Not only does this put one of their divisions almost completely in Silicon Valley, where Microsoft has had but a token presence until now, but it also emphasizes the underlying realization by the company that VOIP has become a major pillar and diffentiator for the largest software concerns, and needs to be treated as more than a minor offshot to their office strategy.

This last year has also seen an increasing validation of a strategy to try to keep companies intact with only a secondary ancillary branding as a Microsoft entity. This can be seen in Facebook, which bears little outward mark of being a Microsoft invested company, and ironically, if (and it's a big if) Microsoft can get Facebook and Skype to play well together, there may be some advantages to be had.

At the same time, the acquisition of Skype may also be a case of Ballmer chasing after brand and market share rather than technology, an approach which has burned him more than once. VOIP is reasonably well understood at this stage, and Skype's been sold once before because it couldn't make the revenue match predictions (it was losing money in its PC to PC communications, which of course was its prime attraction). Admittedly, it was still outcompeting Live Meeting, but there's a major question about whether the effort to integrate Skype into the Microsoft line-up (and the costs attendant with any such reorganization) may ultimately make this a losing proposition. If Microsoft reduces its service offerings there, it also reduces the appeal of the Skype service, and given the fairly mature state of the VOIP market, the primary paying customers may very well end up sticking with their dedicated providers.

In the end, I suspect that this will be modestly successful, but not a major game changer. The buyout helps Microsoft recover lost market share in a critical market if the integration remains minimal, but if Ballmer tries to bring Skype too much into the Microsoft fold he risks both customer and employee defections. Moreover, while voip should be a major part of a company strategy for a company the size of Microsoft, it may also prove a distraction to those areas where it needs to be far more focused, such as the related mobile market space, and even with a fair amount of cash still in the books, the cost of acquisition and integration is going to eat up a not insignificant part of that at a time when other markets are likely to be more profitable long term.

Moreover, there's the question of whether this deal was motivated more by the need for Ballmer to show himself as being aggressive in the marketplace than it was for the stockholders. Ballmer has been far less aggressive in the market space than Gates was, and has often been swayed more by the desire to get the hottest properties rather than the ones that made the most competitive sense for the company. Skype was an old maid - it had been sitting out in the marketplace for a while, represents older technologies, and really was most valuable for its installed customer base - most of which were looking to pay as little as possible to use its services. This doesn't really bode well for Ballmer moving forward - indeed, it may prove to be the final misstep in a series of questionable buyouts and investments.

A New Job and New Ramblings, or Information Architecture 101

2011-04-16T12:10:00.000-07:00

Hi! Chances are if you came from links for the MarkLogic site, you're probably looking for XMLToday.org (I run both) but if you're just here to see some rambling thoughts about information architecture from a graybeard, then no doubt you are in fact in the right place.

About the new job - about a year ago, I came out to Annapolis, Maryland in order to work as a contract data architect for Lockheed on the US National Archives ERA project. NARA was fun to work with. Lockheed, not so much. In January, in a cost cutting move, several consultants were let go. The usual scramble for work began, aided for once with a fairly healthy bank balance and being in a place where there is a hunger for XML architects. Eight weeks later, after a number of interesting encounters, I landed at Avalon Consulting. They had what I was looking for in a job - challenging work, decent pay, sane ideas about remote working and a lot of very bright people that liked to do good things in the information space. That it gave me a chance to work with MarkLogic again was a big plus, as I've rather become addicted to the platform.

I even have a nice shiny new title - Information Architect. I seem to be making a career out of taking jobs with titles that didn't even exist five years ago, which is why I've always found it amusing when interviewers ask me what I want to do five years from now. I think next time I'm in that position, I'm going to tell the interviewer that I hope to be a Paradigm Architect. What does a paradigm architect do? I have no frickin clue, but it definitely sounds cool.

So what about that Information Architect role? I have a theory about this. Humans have this obsession with building. Anthropologists seem to miss the point about humans being tool users - plenty of animals use tools, but only humans use those tools for the purpose of constructing buildings ... you don't see beavers out there with trawls spreading mortar on their dams, you don't see birds construct suspended baskets and scaffolding in order to build their nests. What's more, in every single animal except humans, the building of nests is something that seems to be specific to one overriding objective - to insure that the female and her young are protected while gestating and nursing.

Walk along the Mall in Washington DC or Fifth Avenue in New York and ask yourself whether in fact the role of all of those huge imposing edifices is to insure that someone's female and her young are protected. Uh huh. You answer likely was the same as mine. We build because we can, because it makes a statement about the builder, because it serves to house complex entities - governments, businesses, universities, marketplaces and so forth. It is our attempt to impose upon the natural world a sense of constraint, orderliness and predictability, in great part because our sense of reality is predicated upon the fact that, when reality is orderly, it means you can concentrate upon other things, but when reality is disordered, all of your attention is perforce placed just upon threat analysis.

The idea that information can be similarly ordered is hardly a new one. Large scale relational databases are a testament to such a philosophy - there are known schemas and relationships that exist within any given domain, and one can consequently order those schemas and relationships, describe them precisely down to data type and exceptions. The problem with this is similar to the problem that I see at the Madison Building of the Library of Congress, where I've been doing some consulting. The outside is a massive marble monument, a testament to the power of the written word and the making of informed decisions, and this is carried into the main atrium.

Yet beyond the power foyer, much of the building is occupied with dense arrays of small cubicles, and people who aren't essential to the day-to-day activities of the library are actually encouraged to work from home because it frees up space. The architecture of the building effectively limits the flexibility of the activities, because when the building was conceived, it's evolution was never really taken into account (this is a common problem with office space in DC in general; many of the buildings were created at a time when government was roughly a fifth the size it is today).

Many years ago, I wrote an essay entitled The Architect and the Gardner, and while my perspective has (hopefully) grown since then, I find that my belief in the validity of the underlying thesis really hasn't. Organizations evolve over time, but they build their information systems as if the organization is always static. The reality that they see is a snapshot of a changing culture, but just like the buildings which seem massive on the outside but are in fact cramped and limited in space on the inside, the information structures that they create also tend to reflect the business reality of the time, with no thought to the evolution of that business.

My own belief is that in many respects the information systems for an organization are a vital, living part of that organization. They follow cycles of creation, maturation, senescence and destruction. They evolve - both in terms of the nature of the data that passes through them and in terms of the ambient technological framework which dictates new content. Additionally, just as an organization has different types of culture depending upon where it is in its own cycle, so too do the information needs of that organization change based upon factors as diverse as current economic climate, stability and size of the company, regulatory regimes and so forth.

Taken in that context, the idea of "building" such an information system begins to look suspect. Rather, a good information designer should think more like a landscape architect than a building architect. From the outset there should be a recognition that information systems have a definite life cycle - growing seasons, if you will - and that a rich ecosystem will have everything from long term data systems that are much like the trees in a garden and that serve as the archival backbone of an organization, to intermediate systems that support working data management and act as perennials, to localized data systems that are brought up on an application basis then brought down when that application is no longer needed.

Yet even beyond that are transients - data that streams through the organization, that arrives from and goes to external services. Until comparatively recently these were seen as being more in the domain of the network manager, but while the network manager is concerned with throughput and access, the information architect is more focused on the content of those data streams, and insuring that the organization doesn't build dependencies upon those streams that are more than transient in nature themselves.

Like a gardner, the information manager has to spend a great deal of time not only planning but pruning - insuring that legacy data systems are either migrated into archives or brought down gracefully, such that interdependencies can be reduced, and providing growth avenues for data flows when they become a larger part of an organization's operational information.

One additional responsibility that an information architect has comes in helping to shape the shape or type of that data. An IA isn't a data modeler or ontologist (though he or she may wear that hat as well), but typically the architect will be called upon to set the ground rules upon which the ontologists build, as well as to determine which standards will be followed when working with content.

A big reason for that is that information systems, like gardens, generally will evolve on their own, but not necessarily in a way to help facilitate the useful or relevant information flow for the organization. Again, in the garden analogy - if you have good soil, sunlight and rain, you can get anything to grow in a garden, but it's more than likely that what will take root are weeds - information that takes up space in your physical systems but provides little benefit, and reduces the availability of resources for that information that is important. Ontologies run rampant can make accessing information difficult, ad-hoc or poorly designed schemas (or no schemas at all) can make interoperability difficult, too high a degree of coupling can cause siloization and make the sharing of physical resource problematic and so forth.

In many respects an information architect performs for the information in an organization the same role that a systems architect performs for the physical hardware and a software architect performs for processing systems. The three are complementary roles, and are still largely technical in nature, and in general they represent the technical systems designers for an organization. In particularly large information-centric businesses information architects may manage data designers, ontologists and search engineers, something I suspect will increasingly be the norm for organizations as they shift from process development to information management.

Granted, a lot of these are just my own observations. Roles emerge because there is a niche that needs to be filled in an organization, and given the increasingly dominant role that information organization and management (information gardening?) has in business today, it only stands to reason that the role of information architect is likely to be a critical one moving forward.

Now, about that paradigm architect position ...

Kurt Cagle is an Information Architect specializing in XML data systems, and works as a consultant for Avalon Consulting. He can be reached at caglek@avalonconsult.com

So What's the Problem?

2010-02-16T00:50:00.000-08:00

Recently, I stumbled across an interesting article on the X Prize for zero-emissions automobiles. While I think the article had some merit (I especially liked the concept of using the skin of an automobile to store and release energy), the associated commentary actually got me to thinking about a bigger problems: how do you get from the world as it exists today to one where we have far smaller dependency upon oil, are able to significantly curb carbon emissions and still have a society that is able to grow in terms of its intellectual innovations even as it meets the demands placed upon it by its population?

There are of course several camps of thinking here. The first is the BAU - business as usual - camp, which basically says that we can continue doing what we do today and expect some miracle solution to save us. In this scenario, each problem that emerges is essentially isolated and unrelated to any other, resources are limitless because there will always be some magical piece of technology that will keep us empowered and keep our environment clean and productive, and that you can create economies based upon the movement of electrons representing financial tokens. I refer to this world view as Kurzweil City, named after famed futurist Ray Kurzweil and his belief in the power of technology.

The second camp is the apocalypse view - world oil production is falling even while the population is rising, the environmental systems are under extreme stress and will soon result in massive climate change, and we've passed the event horizon. In this view, the end of civilization as we know it is nigh, and we will be in for a painful, long term depression that will ultimately end up with us huddled in our unheated houses, guns at hand, waiting for the morning as the ravening mobs strip the landscape like locusts under cover of night. The resulting Kunstlervilles, named for writer James Howard Kunstler, ultimately resemble nothing more than 18th century frontier towns set against a backdrop of rusting automobiles and decaying suburbs.

While my own views tend to be more Kunstler than Kurzweil, I think that both are missing the point. Humans adapt. It's what we've done successfully for more than three million years, and there's absolutely nothing to indicate that, as a species we've lost this ability to adapt. At the same time, we usually adapt best in times of maximum adversity ... we're not terribly good at anticipating the need for change, but once change has happened, we are actually quite good at seeing what the landscape looks like and changing our behavior accordingly. What that implies is that both Kunstler and Kurzweil are correct, but the scale on which they are correct are different.

Consider, for instance, the twin problems of climate change and peak oil, both of which I think are in play. It's quite probable that the global climate is in fact heating, but there are three questions that, in my mind, are still very much unanswered:

1) Have we accounted for all externalities? That is to say, can we see with any conclusive proof that there are factors that aren't anthropogenic that are causing the changes in the environment?

2) Are the models that we're creating sufficiently powerful to create an accurate (rather than politicized) view of the behavior of weather? and

3) Are we certain that non-linearities in the model won't ultimately end up inducing a negative feedback look that will dampen factors? I'm thinking most specifically of the North Atlantic Gyre here, and the potential for causing localized cooling effects that could ultimately dampen out the heating effects, though there are other feedback loops in the system that I don't think we've fully taken into account.

Climate change is very important to us right now because our civilization is remarkably sensitive to weather based disruptions, from hurricanes to the massive snow-storm that blanked the eastern US this year, while leaving the Olympics snow-free in Vancouver, but we need to understand that there is a distinction between such climate vulnerability and specific trends. Indeed, the most rational course of action is to develop plans for considering many potential futures, then to fine tune them as any given scenario comes to pass.

Peak Oil is another such issue, and likely a far more immediate one. Again, however, the problems that are associated with Peak Oil tend to be problems that have occurred because we've built an oil based economy, and that economy is becoming increasingly tenuous. I have concentrated on the historical aspects of Peak Oil in other posts as well potential effects down the road, but here I want to look at the question of how do we deal with scenarios where oil supplies become unreliable.

The BAU scenario is becoming evidently increasingly unsustainable long term, but the interesting thing is that slowly and far from uniformly, things are changing. One of the most obvious consequences of the oil-based economy is that it has shaped our cities. Most European and Asian cities were formed before the power resources inherent in oil were available, and as such tend to reflect a pedestrian model.

In North and South America, on the other hand, most of the population growth has taken place in the last one hundred years, and as such has fitted very closely to the rise (and to a certain degree fall) of oil as the primary motive and heat source. Small, largely isolated communities were connected first by rail, then by road, growing larger until they eventually overlapped other communities. Political autonomy in this communities was lost as the benefits of centralized authority took hold.

More importantly, work centers shifted out of these small towns to urban centers, in part because of economies of scale, in part because of the strong need for managers to have their primary operations personnel - financial systems, research and development, production and so forth - as close to the management structure as possible in order to minimize communication and resource management problems.

A certain amount of ego was involved as well - you aren't truly successful as a business unless you have a tall, new tower to dominate the landscape, a symbol (fraught with obvious symbolism) of the overarching control you have compared to everyone else. Because power was centralized, this meant that such psychological issues could readily be rationalized as business ones. Similarly, workers in this system were important only for their skills, and as such required constant babysitting and/or bullying lest they revert to the lying, cheating scum that management knew them to be.

This meant that the work was at an increasing distance from the homes. Here's where I break with Kunstler - I don't believe that the suburbs were created due to poor urban planning, but rather occurred because urban planning can at best only nudge the development of a city by zoning - and fails to take into account the fact that in many cases agglomerated townships had their own views of zoning and planning that were consistent at the time but failed to factor in being swallowed by larger entities.

People moved to the suburbs because of three factors:
1) Cheap oil made living at a distance still more affordable than living closer nearer to the urban center, and provided the opportunity to own nicer property at lower prices,
2) Once a family is established in a house, they will live there for a while, and will only move when economic benefits outweigh the benefits of maintaining residence. This meant that expanding populations will end living farther and farther away because there are simply not enough niches closer in, and this factor usually meant smaller communities getting absorbed into larger ones.
3) Cheap oil and the fact that that the US in particular was a net oil exporter meant that there was significant money for the development of large scale interstate projects and traffic artery construction.

This means that the underlying structure of most cities came about at a time when energy was cheap and money was plentiful, and are increasingly at odds with an era where energy is expensive and money is scarce. It is this mismatch, as much as any, that is causing so many problems in our society.

Yet it is worth appreciating the fact that most of this change has occurred within the last fifty years. While, for a certain generation, this may seem like the bulk of their lives, social memory usually doesn't get baked into stone until after a couple of centuries. In many places, such as here in Victoria, the pre-oil era didn't really end until comparatively recently, and there are many still alive who remember a time when the oil economy wasn't so heavy an influence. This means that for many people (especially the younger ones), the oil economy is itself a relatively transient event, which is what offers up some hope.

The year 2007 is notable for the fact that for the first time, Christmas sales online exceeded Christmas sales in the brick and mortar world. More books are now sold online than are sold offline, and an increasing number of those are eBooks. From the Internet today, you can provide your measurements to any number of vendors and get a full wardrobe sent to you within a few days, a wardrobe that likely is closer in style and fit than anything you could find in stores. Malls, once the center of retail activity, are increasingly becoming ghost towns, and in some places are being razed and left to go fallow again. Meanwhile online retail presences are growing, even in one of the harshest recessions in living memory.

Once critical business support centers - accounts receivable and payable, research and development, marketing, sales and even management - are increasingly becoming distributed, and in more and more cases are becoming virtualized altogether. The office towers, constructed in part because of the need to consolidate business functions, and in part as somewhat phallic representations of power and dominance, are emptying out, not just because businesses are folding but because businesses increasingly have no need for them - they're white elephants from a different era.

Intellectual services - legal services, medical advice, accounting, design, marketing, etc. - have been migrating to a virtualized state for the last couple of decades, as professionals in these fields chase the money to where the market is ... in this case online. Traditional publishing is in freefall because of this, but after a couple of decades of transition, online publishing is beginning to distinguish itself as a profit center, adjusted to the environment of the web.

Physical services - from hair care to acute medical care to restaurantiers and pubs - make up the bulk of the suburban businesses, but it's easy to underestimate their significance. These are services that effectively can't be virtualized (you can't get a haircut online), but can be distributed. What's more, these businesses are shifting away from the major corporate chains, which are facing increasing challenges because their primary benefit - an ability to distribute large amounts of food inexpensively through the use of uniform packaging, marketing and purchasing economies of scale - are now being challenged by the mix of higher oil prices and stricter environmental regulations.

Indeed, this last point needs to be reiterated. Many business analysts made the mistake of assuming that the primary reason that companies such as McDonalds proved so popular was because of their effective uniformity of marketing. I don't think that was actually the case, or at least all that major a part of it. The real benefit was logistical - by taking advantage of economies of scale due to inexpensive energy and oil availability, such companies could reduce the overall cost of a meal to well below what the local market could provide, forcing many of those companies out of business, except at the boutique end.

However, once that factor fades - once the real costs of shipping those Big Macs and Fries enter into the equation and the cost of mass producing the raw inputs for those meals rise, the local stores (with much shorter supply lines and generally fresher and healthier product) are more effectively able to compete. This is why the next decade will see a dramatic rise in the number of independently owned restaurants compared to the chains.

The same is true of the big box stores, save that their demise will likely come due to the Internet rather than competitive local stores. However, what's interesting here is that the goods being sold will be more locally produced, likely through networks of vendors working either independently or in conjunction with mediating merchants online.

Again, distribution and shipping costs play a big part here, and these will force a relocalization of both raw materials and finished goods. While the middleman may not be completely cut out of the equation, the chances are good that the middleman is likely not going to be spending large amounts of money on mall outlets when they can spend the money online to minimize distribution costs.

This raises questions about the viability of the Walmart model moving forward. Walmart is fundamentally dependent upon the oil economy model. It's goods are generally produced in low labor cost and low materials cost regimes, then distributed in bulk on oil-intensive transport tankers. They have been competitive because of their ability to undercut local producers. As that competitive edge disappears (and local vendors begin to become cost competitive again), expect the big box stores to be closed.

Manufacturing has similar patterns. Manufacturing overall tends to be local, and because of the potential environmental issues, manufacturing in general has been more distributed than other businesses. Because outsourcing of manufacturing is becoming increasingly expensive, local manufacturers should end up being far more competitive than they have been, and will better be able to supply localized economies.

What this means is that urban use patterns are now shifting - intellectual work no longer needs to be tied to an urban center (because these services are being done over the web), which strengthens local suburban communities. Manufacturing and physical services similarly are becoming more localized and shifting away from either urban centers or outsourced vendors. This will tend to strengthen suburban centers as the needs for such services rise, though such centers will likely become more municipal than commercial.

The final factor in all of this is energy production. As you shift towards more localized and distributed forms of energy production (and alt-energy forms), this tends to provide additional revenues that accrue at the local level. Large scale energy production - coal, oil, nuclear, etc. - on the other hand, favor transnational corporate control of energy resources, as does the oil distribution network (especially when the country is a net importer of oil). This money in turn usually cycles itself back into the local economy, where it can be used to continue upgrading existing infrastructure, setting up a virtuous cycle again.

This process won't happen evenly, and it definitely won't happen without some (perhaps a lot) of pain. It will give more power to specific regions, and metropolitan areas within regions, at the expense of the larger nation-state (the US, Canada and Mexico especially) and rural areas (which currently tend to have a disproportionate amount of influence in the US especially given their greater weight in the Senate).

It will also favor those regions that have the best mix of urban and rural distribution, available water supplies, ports and the like, while coming down especially hard on those regions that are missing key pieces of this mix (for instance, Scottsdale, AZ has few water resources and little in the way of arable farmland, even though they have the potential for solar energy, while areas like Seattle or San Francisco are reasonably well endowed with all of these).

However the speed at which this occurs will depend largely upon the effectiveness of the existing economic power structure in obstructing the changes. That it will occur is a given - these are very hard and fast megatrends at work here - but such an economic transition favors very different players, and the oil economy players have the advantage of incumbency at this stage.

I'm coming to believe that political considerations - the structure of power sharing within a society - are increasingly driven by economic considerations rather than vice versa, though politics generally dictates the degree to which economic costs are externalized to the society at large.

Many economies globally are now going through a period of economic decline as positions favoring the oil economy are being unwound, mostly due to the realization that the underlying assumptions of effectively limitless energy resources, which lays at the foundation of our current credit-based economy, are no longer valid. These assumptions were facile on the surface, but were vital for the oil economy to get access to the necessary funds that would finance the continued development of political structures and labile populations friendly to this economic viewpoint.

The very likely second leg of deflation, now beginning to unfold in Europe as the Maastricht treaty disintegrates though likely to come back round to North America as American and Canadian investments in Europe are hit by the largest margin calls in history, are only continuing the unwinding of these assumptions, further weakening the oil economy and likely causing the political unrest necessary to affect meaningful political change, even if such change ends up redrawing political boundaries to more accurately reflect economic rather than historical realities.

To that end, I'm increasingly of the opinion that while you will see political volatility and a great deal of economic instability over the course of the next fifteen to twenty years, these changes are the necessary precursors to the transition to a new economy and new political reality. It will force us to rethink a great number of assumptions, from the nature of work and the mechanisms of valuation to how we build our cities (and countries) moving forward. It will look superficially more agrarian, but will hide a sophisticated technological underpinning.

It will offer fewer opportunities for "getting rich", but potentially a healthier, more satisfying standard of living for a larger percentage of the population. It will not be uniform - by the end of the twenty first century it is very likely that there will be regions that are technologically and socially far more advanced while other regions may almost be feudal in nature, based largely upon choices made in the next decade.

This should be an interesting time.

Economic Hurricane Season

2010-02-13T02:47:00.000-08:00

This is my first blog on this platform in a while, though I will likely be writing more from here moving forward. My life seems to have moved into one of those strange interstitial periods where, despite the best of intentions, little seems to be happening. Over the years I've noticed that these are often gestational times, when the batteries are recharging and projects that had been sidelined or abandoned are re-examined and revitalized.

There are projects - I've actually been writing out chapters to a novel, though I'm nowhere near solid enough at this stage to even seek a publisher, assuming I don't just go the route that I think many others are now following of going directly to print (and eBook format) without the intervention of a formal publisher, at least to start.

While I won't go into details here, many of the ideas of the novel come about from my thoughts about where we're going, both technologically and societal, thoughts that I've expressed here and elsewhere in a more speculative mode. For those whom I've discussed this with before, this is essentially Open, Mark II, just cast into the mode of fiction rather than speculative analysis.

Regarding the "where are we going" part, what happens over the next couple of years is going to have major ramifications for the next thirty or more, probably more so than at any time since perhaps the early 1970s. The sovereign economic crisis in Greece will likely get resolved by some deft paper movement, likely at least officially out of the IMF, which in practice means that the US and Germany are effectively loaning Greece the money to try to get out of their hole.

Of course, this loan will only be effective if Greece is capable of reducing its current debt (on and off book) from 800% of GDP (yes, you read that right) to something closer to 75%. This means that all governmental expenditures stop - no health care, no unemployment insurance, no government subsidies, no education, no spending on infrastructure, and that taxes rise to 75% or more on all citizens.

Put another way, in order for any financial assistance to be even remotely useful in restoring the financial health of Greece, it will require cuts so draconian that people will be in the streets with torches and machine guns. It won't happen. This week has seen the application of a band-aid on a cancer patient.

Greece isn't alone in this. Portugal, Ireland and Spain are in only slightly less dire straights, and the political will to save Greece will likely mean that one or more of these other countries will almost certainly be left hat in hand at the curbside. And lest those in the US feel smug about their own situation, consider that both US and UK debt at this point has on and off-book debt obligations well in excess of 500% of GDP, and several subcountries within the US (California, New York, Georgia, etc.) are looking at remarkably similar debt ratios.

We're facing an ebb-tide right now. There's an interesting phenomenon that anyone who's stood in the water of a beach with sizable waves can attest to. If you stand out far enough in the water, you'll notice an interesting shear force that occurs as a wave crashes and advances along the beach. Even as it is advancing, there's a back flow that is occurring, moving away from the beach, near your feet.

In the US the backflow started about 1972, which was, not coincidentally also the time when oil imports exceeded oil exports in the US for the first time. The momentum of the breaking water caused the waves to continue to advance for some time, but people already noticed that it was becoming harder to feed a family of four, that despite increasing wages, the money didn't seem to go as far, and the trips to far off places that were part and parcel of the 60s and early 70s became increasingly expensive over time.

By the mid-1980s, women were coming into the workforce less because of feminism and more because the family increasingly needed that second income to continue the lifestyle that they had, and even then the lifestyle seemed to recede. The job guaranteed for life became two jobs, then four, then eight, and has now reached a point where employment itself is becoming a dicey proposition.

Populations are becoming older globally, though most especially in the so called first-world countries. People who retire pull money out of the system rather than contributing money into it, or at least would have if a couple of back-to-back recessions didn't proceed to pull out most of that money for them. And yes, financial managers are as aware of the demographics as anyone, and increasingly the questions need to be raised about exactly where all that invested capital ultimately ended up.

We're in full ebb-tide now - beyond the demographics, the realities of Peak Oil (which by many accounts the peak of which occurred between three and five years ago) indicate that we're in for a time of oil price rises and crashes globally, meaning that there's now a very real brake acting on the economy anytime it begins to show signs of health. In that kind of environment, especially with credit still effectively frozen, development of both oil exploration and extraction rigs and development of alternative energy forms slow to a crawl because the risk premium is too high. Government investments here will help some, but those investments will take a while to show any significant returns, and as such are vulnerable to political pressures.

And for those looking for a bottom to the housing market, consider that we've been in what amounts to the eye of a hurricane since March 2009. Subprime mortgage papers hit reset points from 2007 to 2009 as teaser rates jumped 100-300% once the grace period ended. These have largely reset (resulting in about 7% of all mortgages now either 90+ days in arrears or in foreclosure, up from 0.3% in 2004). However, most alt-prime mortgages are resetting between now and early-2012, as are a number of ARMs, along with a lot of commercial real estate that is increasingly being abandoned by tenants going out of business.

This second wave of the hurricane will hit an already badly weakened economy in the US and UK (and is a big part of the reason that it's already taking out the weaker economies along the Mediterranean). This is why, despite valiant efforts to keep the DOW propped above 10K, the inertia has shifted away from an explosively growing market to one that is a black swan away from a major nose dive. While it is just possible that we'll hit the best case scenario of the economy staying relatively stagnant for the next couple of years, most people already feel instinctively like we're already entering phase two of the first twenty-first century depression.

Yet, as odd as it may sound, the net effect of all this may ultimately end up proving to be positive. There have been two forms of globalization that have been taking place. The first has been the consolidation of financial power as the end game that started around the end of World War II. This gave the movers of money unprecedented control, and ended up with obscene fiscal imbalances between richest and poorest, both within countries and globally. This particular storm is destroying that power base, is eliminating the bonds of economic liberalization that overall have benefited the wealthy at the expense of the weak.

It's forcing us to rethink the notion of export oriented growth economies. At some times, growth economies are good - when a population on average is young, significant industrialization is needed to support the needs of those entering into the work force, is needed for educating the children and taking care of health and providing better infrastructure for transportation and resource distribution. When populations are old the societies contract; you need fewer goods and smaller houses, more medical services but less educational services. The economy should contract as well to reflect this. In the fullness of time this will change again, but those changes occur on the scale of decades, not years.

The second change has been globalization of communication. This is a trend I expect to continue, and it's also having far reaching effects, especially among the young. This is helping to deconstruct the command and control economy in place since the early 1950s, and replacing it with transient global webs of entrepreneurship and innovation. It will likely end up destroying education as we know it, education built largely upon the need to learn facts rather than analysis, but will replace it with something both richer and stranger than we can possibly anticipate.

It's also about to replace sovereign reserve notes with myriad local currencies, each jockeying for position on a second by second basis, their strengths and weaknesses mirroring localized economies and their ability to trade - and the soundness of their economic policies.

Phase transitions can be harrowing, sweeping away ideas that worked until they didn't that have become embedded into society, but they are also times of opportunity as new economic niches open up ... a lot like hurricanes, when you think about it.

Future Proof: Kurzweil Cities and Kunstlervilles

2009-05-07T10:08:00.001-07:00

According to a number of serious and well-intentioned articles written in the last century (not to mention the Jetsons), by 2010 we were all supposed to live in giant cities with mile-high sky-scrapers, flying cars, pneumatic tube transit, and robot servants. Admittedly, this view gave way to a somewhat more dystopian version around 1980 or so - think Blade Runner, for instance - in which overpopulation, over-urbanization and technology gone wild created a dark and threatening world.

Perhaps one of the biggest exponent of the power of technology to change the world for the better is Ray Kurzweil. Inventor, author, visionary, Kurzweil's central thesis has been that as Moore's law continues in its seemingly unrelenting progression, humanity and the machine will become ever more indistinguishable, and that ultimately there will be no problem that can't be solved with the suitable application of intellect.

At the other end of the spectrum for Kurzweil is James Howard Kunstler. Kunstler has been writing about the relentless spread of urbanization and the problems that will occur as systemic shocks - peak oil, peak water, climate change, aging populations and so forth - cause profound changes to the way that we build our cities, ultimately resulting in the destruction of the suburbs and the end of technological society as we know it. His vision of late 21st century life looks a lot more like the mid-19th century, and it is, curiously, surprisingly appealing even in its starkness.

I've dubbed these scenarios Kurzweil Cities and Kunstlervilles - views of optimism and pessimism that, when looked at through somewhat different lenses, could just as readily be interchangeable in terms of their values as utopias vs. dystopias.

As many people have noted, cities are organic over a large enough period of time. They exhibit emergent behaviors that seem eerily similar to the way that lower order life-forms act. They grow in response to available energy sources, expanding outward as energy enters the system, contracting back in on itself as energy leaves. Highways and streets are the arteries, carrying car and truck corpuscles from one part of the city to another. The nerves are the power and information conduits within the city. When cities collide, they either form systemic cells or absorb one another, the older former towns slowly losing their distinct identity over time.

This metaphor, or abstraction, is an important thing to keep in mind when looking at the future of cities. Cities grow in response to increases in population. This may seem obvious, but I'd contend that its actually a very subtle point - the larger the population, the more likely that the necessary number of interactions can take place to push the city to a new level of abstraction, while at the same time the greater the drain on the energy resources available to that city. In a city where the energy drain is higher than the energy sources (where energy can be physical energy such as electricity or the abstraction of energy in the form of money) the quality of life in the city drops - there are fewer job opportunities, the standard of living goes down, the government becomes more authoritarian, the ability to support urban services decline.

Technology cannot create new energy - it can only make it possible to use existing energy more efficiently, and always at the cost of powering the technology itself. This has always been the fundamental flaw of Kurzweil's vision - Moore's law does not come for free. Every generational doubling in processing power occurs because more energy goes into the technologies to make it happen. Costs for fabrication plants for creating new microprocessors increase geometrically as well - the cost to create a typical fab is now well into the billion dollars category.

The problem that's led to the current crisis is that the energy costs here are borrowed. Intel or AMD generally doesn't have anywhere near the amount of cash on hand to build new fabs. Instead, it borrows the money against future earnings - it is in essence borrowing energy that hasn't been created yet.

This future borrowing has been endemic in US culture for a long time. Cities (and larger geopolitical structures) generate their revenues in one of a few ways - they either take possession of a power or resource source and sell from that, they receive revenues from the state (which simply pushes the problem up a level of abstraction), they place a tax on the current revenues of its citizenry and associated companies, or they create bonds to borrow from the future earnings that the project in question will produce, adding in a premium in order to compensate the bond holders for the potential risk of the bond defaulting.

When the weight of such credit exceeds the potential of the system (as it exists) to pay back those loans, the system collapses. That's what is happening now. The system is becoming less energetic, and as such, the ability of the system to support its abstractions is diminishing. The US government is working diligently to prop up the system, but it's constrained by the same problems - any money that it creates is a promissory note on new energy production, despite the fact that energy production in the US has been declining since the early 1970s. It may be able to sustain the status quo for a while longer - but the next crash will likely be harder.

So is Kunstler in our future then? Not necessarily. Kunstler's central thesis is that an oil-dependent economy will eventually lead to collapse as the supply of oil continues to diminish relative to demand. Oil is important, both as a fuel source and as a resource for production of goods, but its important to differentiate these two use cases. The principle use of oil today is for transportation, moving things from point A to point B. If you can switch cars over to electric or electric-hybrid use this will significantly reduce demand on oil - perhaps even to the point where US production can easily accommodate all other uses of oil. Flywheel systems, and shock kinetics also add potential power, especially for larger vehicles that have more intrinsic momentum).

To do so, however, other changes become important. Electricity production needs to become more distributed. Efficiencies in solar power production are raising the possibility that cities can actually become net power producers - both with regional power "farms" and solar enabled houses and businesses. Beamed power - in which solar power collectors in space are used to create coherent microwave beams that can provide power to collectors even in cloudy areas, could dramatically increase capabilities. Geothermal taps, wind power, wave power and more efficient super capacitors make energy production in coastal areas more feasible. More efficient monitoring and routing of power (smart grid) can also insure that energy is made available to those places that have the largest demand, rather than getting wasted.

There are even places for such technologies as nuclear fission plants, which, despite the publicity of both Four Mile Island and Chernobyl, are generally much safer today. The principle problem here comes in taking care of the high upfront costs and the still troublesome waste disposal issues.

What this implies however is that the future will likely be neither Kurzweil cities nor Kunstlervilles. Instead, for a while it will be a mix of both - cities that can most effectively harness net energy production will thrive and grow, and the standard of living there will improve. Cities that can't will sink into slums and abandoned neighborhoods, crime will rise and people who can afford to leave will for those places that offer better standards of living. The dominant cities of the twenty-first century will be the ones that make the transition first, and it is likely that these cities will also end up creating stronger regional trading blocs that circumvent political boundaries (a case in point would be the Vancouver/Seattle/Portland corridor, which has the potential to become a cohesive political entity as energy and resource systems merge, despite crossing both state and national boundaries).

Indeed, this last point is worth reiterating - political boundaries may be conservative, but they also eventually snap in the face of energy flow structures. Regional trade and energy blocs are comparatively new abstractions, eddies along the older nationalistic boundaries. They will gain in cohesiveness over time, eventually overshadowing older nationalistic boundaries altogether. This means that, again taking the case of "Cascadia", while inhabitants of Portland, Seattle and Vancouver will continue being citizens of their respective states, provinces and countries, they will increasingly think of themselves as being Cascadians as trade and energy alliances build.

Ultimately, the cities of 2020 or even 2050 will likely end up being not that different from today, at least on the surface, though individually they may look quite different. Some places, like Detroit, may not even exist - it was conveniently placed in the 1920s through the 1960s to bring together the raw materials, energy sources (from Pennsylvania oil) and cheap labor to mass produce cars, as well as conveniently placed to distribute them. None of these factors are in play anymore, so the city is dying.

On the other hand, when a city enters into this mode, it is also, ironically, at its most fluid - investment terms are favorable, it's easier to raze dead neighborhoods, townships that are tethered to the city are able to break free and make more effective decisions at lower levels of abstraction, Detroit in 2050 may very well be a network of independent towns, each powering their own subgrids, each producing its own own products and services. Education and the arts may may very well be growth industries by that point, with energy production subsidizing the initial costs, and this is perhaps the real lesson to be gained from Kurzweil and Kunstler both - by moving off the oil grid, by moving away from a caustic and self-defeating consumerist culture, it may be possible that both scenarios come true; the future is a region full of universities towns and centers of learning and the arts - urban enough to bring together the necessary confluence of people but rural enough to sustain the agriculture basis of the region. I could live with that.

Future Proof: Freelancer

2009-05-07T10:07:00.000-07:00

I have been a freelancer for most of my working career. The specific jobs vary, of course - I've been a freelance writer, a freelance journalist, a freelance programmer, a freelance information architect, a freelance trainer, a freelance teacher - the list goes on and on. While there is a standing joke that freelance is another word for unemployed, I'd definitely have to disagree there ... I have had years where I've cleared six figures as a freelancer, though there have also been a few years where I've made just barely above the poverty line.

There are certain professions that lend themselves well to freelancing, most of them in the information sphere. Programming is a natural - projects have a beginning, a middle and an end. After the project is done, you may or may not need the skill-sets of the person involved. Ergo, freelancing. Writing is another intellectual pursuit that has a definite terminus. You teach classes for a quarter or two, but unless you're tenured there's no real advantage to keeping a teacher when all you're looking for is someone to impart this particular wisdom at this time.

There are similarly professions that don't lend themselves well to freelancing, though they are becoming rarer. Indeed, as I'm writing this, I'm scratching my head about what professions can't be done in freelance mode. And that, in a nutshell, may be the problem.

Full time work makes a great deal of sense in an industrial society - the need for producing X number of widgets per hour every day means that you need to have labor there every hour of that day, you need managers for that labor, then need managers for the managers. The cost of disruption in that labor is high - if someone quits then you have to get someone else trained up in that new role, and you have holes moving through the organization until you can find someone from the outside. This translates into significantly reduced productivity.

Most of the "benefits" provided by business have their origins in this mindset as well - health care originally made sense by having an onsite doctor or working closely with a nearby hospital, in great part because it made sense financially to insure that workers had as few disruptions due to illness or injury as possible. Pensions (and later investment vehicles) similarly emerged as a way of keeping employees long term - people were far less likely to quit for a competitor (and take potentially valuable information with them) if the company held on to their retirement savings. In general, retention was the rule.

However, today, this process is going in reverse. Businesses are disaggregating. Conglomerates are selling off or IPO-ing divisions because the costs involved in a large labor force are increasingly outweighing the benefits. Health care costs are skyrocketing as the workforce grows older, as the multiple layers of "managed care" extract ever larger portions of the pie, and as fewer doctors and nurses enter the field. Pensions had long been something of a running joke - a borrowable pool of funds for the company that was often used to invest in fairly risky investments, and as those investments failed to play out, companies are now faced with new retirees asking for their pension funds just as those funds have been wiped out by malinvestment and mismanagement.

What makes this worse is that the technologies are increasingly in place such that people no longer need to be in one place to work, and that if a person does in fact leave they seldom have the same negative impact on productivity (in the short term) that they once did - even if that person is a stellar performer. Longer term, of course, losing those start performers can be the death knell for a company, but the difference is that the impact is seldom felt for a while.

Thus, for many companies, the ongoing recession is a chance to reduce their existing obligations - purge their full-time ranks and then, as people become more desperate, rehire them on a contingency, freelance or part time basis. From an accounting standpoint, this is the best of all possible worlds - you don't need to pay for ever-increasing health care, don't need to make contributions into a pension plan that you know will never actually be fully capitalized, don't have to dilute existing stock, can hire more people when demand rises and can then lay them off when demand falls, either on a project basis or over the course of a general economy's rise and fall.

However, from the labor standpoint, this is also the worst of all possible worlds. As a freelancer, you are essentially running your own business, but almost invariably without the level business support that corporations routinely have. You become responsible for your own health care, for doing your own taxes (and usually get taxed at a fairly high premium for being "self-employed"), for your own retirement. Work becomes episodic and sporadic - you either are searching for new work or you are facing a glut that you can't fill, but you don't dare outsource it because you need the money to tide you over in the lean times.

Most independent freelancers compensate for the sporadic nature of the work by charging a premium for services - a contractor should, in theory, cost a company more than a full time employee short-term because the freelancer is paying for his or her overhead that would otherwise be paid for by the company. However, in practice, unless you are highly specialized, you are competing with a (currently growing) pool of similar contractors which mean that companies can effectively bid on competing contracts to keep these wages low.

This practice is exacerbated by agencies, which usually end up acting as a buffer between the freelance labor force and companies. Most of them may offer very short term benefits for the duration of the contract - minimal health care policies, for instance, that the employee usually has to purchase - but usually nothing beyond that. In exchange for that, they absorb that 20-30% pad that freelancers would otherwise save up for down-time, meaning that from the hiring company's standpoint, the labor is still expensive, but can be let go at a moment's notice without significant contractual problems - and because the agency itself can cap the wages, the wages are still less than they would be for an independent contractor.

Currently 29% of the workforce in the United States is contingency contract, up from 24% in 2005. That includes both part time workers (those that are deliberately held below the minimal 35 hour line that costitutes full employment) and freelances who may work 40 hours or more a week but are hired on a temporary basis. It's likely, as companies continue to shed jobs that this will grow to between 33 and 37% by 2015, meaning one in three people will be working outside of the established "safety net" of full or salaried employment, including a rising percentage of professionals - management executives, medical practitioners, financial services professionals, lawyers, engineers, marketing and communication specialists, designers, system architects and software developers, along with the whole plethora of "creatives" - artists, writers, musicians and so forth.

Of those, roughly 70% are female, which reflects less upon a bias against women (though that's there too) and more on the fact that women have entered the workforce more recently than men, are more likely to be in information-centric careers and are thus perhaps more indicative of future trends than men are. It's worthwhile noting that the percentage of contingency workers under the age of 40 is also much higher than it is for those older than forty, though how much of this is due to structural changes in the workforce vs. the fact that younger workers are more likely to have fewer commitments that make contingency work more attractive is hard to tell, save that the under-forty contingency percentage has been creeping up steadily for decades.

The question for policy-makers is what to do about it. First, its worthwhile to note that there is a world of difference between a freelance lawyer or programmer who has specialized knowledge and can usually afford to handle insurance, taxes and retirement savings and the part time Walmart worker who likely can't - and for clarification, I'll refer to the first as freelance workers and the second as contingency workers.

The freelancer in general should bite the bullet and incorporate as a small business, and press for better legislation to provide more legal rights to these microcorporations. I see this ultimately happening, especially as the number of web businesses rise - businesses that have significant "virtual" presence, but that may represent a group of one or a handful of active partners. Overall the IRS has taken a dim view of such small organizations, but they represent the bulk of all new incorporations, and as the force multipliers of technology have increased the ability of such small companies to have an oversized presence, it's likely that most of these businesses will stay small, people wise.

Unfortunately, contingency workers may not be as well positioned. In the 1930s, Franklin Delano Roosevelt worked with the Federal Reserve to create a plan to put Americans to work long term - an decision was made to allow for a certain amount of inflation in the monetary base in exchange for full employment. In essence, every year the monetary base was allowed to grow by between 2% and 3%, which devalued the dollar by a corresponding amount. Prices rose, and with them real wages dropped, but as more people were entering into the system at that point than leaving it, it meant that people entering the system were actually making marginally less that, in the aggregate, freed up a lot of capital, which in turn was used for starting new projects and hiring more people while keeping people happy that their wages (at least on paper) were stable or growing slightly.

This worked for a while because it was a Ponzi scheme - so long as the work population itself was growing, such a model was sustainable. However, in the last eighty years, the demographic pyramid has inverted, and there are now more people near the end of their career than there are starting out. Add into this the effects of technology in making work more efficient, and you get the rather ugly situation that we're in now - a situation where you have more people who are staying in the workforce at the higher wages that their skills and experience should support (and who are desperate now to refill their coffers after the last couple of years) and fewer people at lower wages that support the Ponzi scheme.

What this means is simple - many of the jobs that are being shed now by business are not coming back. Middle management has been hemorrhaging for the last two decades because the value provided by these managers is no longer as critical. Retail sales jobs are disappearing at a rapid rate as retail centers collapse in the face of low demand and Internet distribution. Manufacturing looks to be in its death throes in the US (if the bankruptcy of Chrysler and GM are any indication), and it is likely that moving forward the jobs being replaced will not be at the upper end but at the lower with new designers and engineers who are more familiar with cutting edge tools, fabrication methods, and technologies. Construction will likely be at an ebb for the next decade. There are more marketing people out there than there are markets, and again as new jobs do arise, they will be in areas where you have the young and savvy rather than the experienced.

What makes this worse is that even in those areas where growth may occur - health care, energy production, high speed rail, education and the like - you are generally going to be talking about specialist jobs requiring long term training - and an education/training system that is still bound up in a large corporate model. What's more, even if the education did exist, the absorption rate for these professions is comparatively small - you need more doctors, for instance, but if even one half of one percent of the unemployed work force were to go back and get medical degrees, it would easily swamp the field.

This will set in place the great forces of the next decade. Each recession is likely to result in higher unemployment than the previous one, while each recovery will see a smaller percentage re-employed. Against this will, paradoxically, be the growth of spot shortages in the labor markets in specialized areas - those that are capable of going freelance and are successful at it will end up creating loci of specialized job growth, but the growth will remain limited.

This is always a dangerous mix for political stability. Shadow economies usually emerge once you get a certain level of unemployment - people still have an imperative to survive, and will do so any way they can. Drug trafficking typically rises during recessions, even as prices for those drugs fall, because drug dealing provides not only income but organizational structure (albeit very dangerous organizations). Prostitution rises as well for much the same reason. Both left wing and right wing paramilitary organizations usually tend to do quite well during these periods, providing both places to live and organizations to be a part of, and such organizations, while possibly carrying out political agendas, usually provide "security" services to that same underground economy. The Internet in this case will likely only hasten this process; it is very easy to set up online communities and exchanges that can't be easily regulated, taxed or even monitored. As people become more desperate, expect that barter and trafficking on these sites will increase dramatically.

On the other hand, its also likely that a virtual side of the shadow economy will show up in online games and other environments. Already, there are people that are making a living either producing goods and services in games like World of Warcraft or Second Life, are playing automated gambling sites, or are fully engaged in eBay or other online markets. The irony here is that while this market is likely growing dramatically, its metrics are so different from those of the "real" world that it's hard to tell how many people who are technically unemployed are actually making a living there.

Note that these are also freelancers, though they don't show up in official measures as such - and there's a lesson to be learned from this. Over the next decade you're going to see a generation grow up on the Internet, learn to make a living there, and develop an entirely new conceptualization of business there. They're growing up in a grey area that's neither "corporate" nor governmental, becoming very entrepreneurial while at the same time working outside of the bounds of contemporary business.

Many (and certainly the best and brightest) of these younger men and women are going to grow up with nothing but disdain for the modern corporation. The more that they establish themselves on the Internet, the less likely that they are going to put up with office politics, small cubicles, long commutes, and the increasing uncertainty of job stability in an organization that could cut 10,000 jobs in one fell swoop. The talented ones will be on the cutting edge, creating new virtual company after virtual company, each staffed with perhaps a couple dozen people tops that communicate with one another from remote locations, each company with a killer product or idea that will chip away at market share of conglomerates piece by piece.

When the economy does improve, this generation will not come to work for the old corporations. The smart companies will change in response. Most won't. Many of these companies will sink into irrelevancy, no longer able to tap into a mindset that is radically different from anything that the senior managers can even begin to imagine. These people will have become used to starting with next to nothing and being exceptionally frugal - they will be anti-consumerist, highly innovative, and with very little use for traditional social structures.

Hiring managers, beware. The freelancer is about to take over your business.

Swine Flu: End of the MBA Farmer?

2009-05-07T10:06:00.000-07:00

While there are legitimate questions about the potential severity of swine flu, it is still a dangerous flu for a simple reason - most flu viruses in circulation are very minor variations on existing strains, which means that most people who get the "flu" end up with symptoms that have more to do with histimine reactions - runny eyes and nose, aching joints, maybe a day in bed feeling lousy - then they're past it.

Swine flu, otherwise known for its genetic markers as H1N1, isn't an existing, commonly circulating flu. It's relatively new although with very old antecedents, which means that most people have no immunity to it. This means that it will likely spread remarkably quickly, will leave a significant portion of the population sick with it, and could prove to be deadly even for adults.

What epidemiologists are discovering about this particular flu bug is very disturbing - first, that it is a variant of the Spanish Flu virus that accounted for more deaths worldwide than World War I, which was waging at the time. Spanish flu was extraordinarily virulent, and when it finally died out, it became very quiescent - effectively disappearing altogether from the cloud of seasonal viruses that normally lay people low in late winter.

However, in addition to this, it now appears that the term Swine Flu is more apt than was even apparent on the surface - Swine flu itself first appeared in hog factory farms in the 1990s, mutating rapidly in the high density "population" of pigs kept in tiny pens little larger than the pigs themselves. The flu wasn't lethal for pigs, and the particular strain of swine flu that did jump to humans was of a variant that didn't "catch", failing to reach critical mass or virulence to be a true pandemic.

The early 1990s also saw the graduation of a crop of new business school MBAs, instilled with a twin philosophy - automation was the wave of the future, and one could apply the new thinking of the 1980s to every business endeavor in order to transform these into hyper-efficient super businesses, including agriculture. "Archaic" farms that had established an understanding of animal husbandry over centuries were quickly put out of business and bought out by new "factory farms" that used a combination of technology, mass-injections of antibiotics, close-confinement of the "stock" and waste disposal being passed to the community.

The last issue eventually caused enough of a reaction that many of the now very wealthy agribusiness concerns realized that setting up factory farms in Mexico, which had far laxer environmental laws, lower labor costs and generally a less empowered populace, might actually prove more profitable (just as such farms had tended to relocate in states that had lower taxes, environmental restrictions and wages originally).

In the end, this strategy, while increasing the overall production of beef, pigs and chickens dramatically, also caused the price of these meats to drop fairly dramatically, further eroding the ability of other farms to compete and driving them out of business. Meanwhile, south of the Rio Grande, these elongated factory farms proved the ideal breeding ground for increasingly antibiotic strains of viruses. It was only a matter of time before such a strain would jump to humans (indeed, it's likely that Mexican workers at these plants were also virus laboratories, providing many more opportunities for animal to human transmission), and from there, additional vectors took it into the general population - other kids playing with the infected kids bringing home the virus usually without knowing they had it.

The epidemiology of viruses is well known, yet advanced knowledge of medicine isn't going to help when you have viral factories that speed up the evolution of viruses a thousand fold. Even if this particular virus proves not to be especially virulent, the next one or the one after that may well be. Perhaps it is time for us to start questioning whether the factory farms are in fact yet another artifact of the "greed is good" mentality that's proving to be so destructive to the rest of society. Beyond the ethical dilemmas of keeping animals in such conditions, these factory farms are increasingly proving to be businesses that do harm than good, and as such at a minimum need to be rethought in light of that, and perhaps even need to be abolished (not just moved to places where people can't protest them).

Chrysler, Hedge Funds and Contracts

2009-05-07T10:05:00.000-07:00

President Obama is beginning to look less like Franklin Delano Roosevelt, and a lot like his distance cousin Teddy. After several months of trying to come up with a viable solution for preserving Chrysler, yesterday the ailing car company went into formal bankruptcy, which means that the auto unions and the government at this stage essentially now own the company.

For the last week, Obama has been working with all of the major parties - automaker Fiat, unions, banks, hedge funds and similar investors and lien holders on the company, to try to stave off bankruptcy, while trying to keep from adding even more federal loans to the beleaguered company. In the end, while most parties agreed, the major hedge funds balked, demanding preferential treatment in terms of payback and seeking to get 2-3 times as much return on their investments as every other players. Obama finally lost his patience, ordered the company into bankruptcy, and effectively hit the reset button, wiping out several billions of dollars of equity outstanding as part of the process.

No doubt the financial industry and its captive press will scream bloody murder here, but the events of the last week represent the emergence of a new, big-money hostile political environment that will likely only strengthen from here.

In financial circles, one of the most sacrosanct documents is the contract. Filled with obscure legalize, most contracts are dense, deliberately obscure, and are often designed to seek the maximum possible advantage of one side over the other. Contractual obligations have played a major part in the most recent financial crisis, especially when such contractual obligations have overwhelmingly benefited the financial industry. A prime example of this was the defense given by investment banks that, even while being funded to the tune of hundreds of billions of dollars by the US government, paid out lavish bonuses to superstar investment bankers, analysts and C-level officers - they were contractually obligated to pay these bonuses, and as such couldn't go back on them.

Contracts are important. However, the problem with contracts is that while they may in fact describe the contractual obligations between two parties, there is *always* a third party involved. Call it the public good, call it government, call it society, but in all cases it should be seen as the interest that the rest of society has in insuring the peace and stability of that society. One of the underlying concepts that has taken place over the last forty years has been the rise of the doctrine of private business - that so long as companies do not in fact engage in specifically illegal activity by the letter of the law, government has no role in the contractual process - even if the companies engage in activity that violates the spirit of the law. Grover Norquist's famous quip about wanting to see government so small that it can be drowned in a bathtub is perhaps the most pithy encapsulations of this philosophy.

Tim Geithner, a former Federal Reserve regional governor, and Ben Bernanke, the current Federal Reserve chairman, have been steeped in this zeitgeist for so long that it is central to their world view. Barack Obama, on the other hand, has seen what happens when the rule of contract exceeds the rule of law, and as he becomes more comfortable with his own authority, is also beginning to exercise what may very well become known as the Obama Doctrine - that contracts that harm the public good even while being within the letter of the law can be abrogated by the third party in those discussions - the government, keeper of the public good.

There are many in the investment community (and in political circles) who are fearful that this approach will cause investors to not want to reinvest in the banks, for fear of their investments essentially being annulled. This has always been powerful weapon to wield against those who would seek to change the status quo; however, it is an argument increasingly without teeth. Those who invest should understand implicitly that no investment is guaranteed to be without risk, regardless of whether that investment is a few shares of penny stock or a sizable investment in a car company ... or a bank. The owners of a business who fail to press the people who manage that business to be more responsible, more innovative, more willing to respond to changes in demand, and more ethically responsible should hardly be upset when those companies fail.

Indeed, this is perhaps one of the fundamental problems that this society faces: there is an implicit assumption that one can make a business grow and thrive simply by pouring money into it, especially at the senior management levels. In essence, money is being used as a way to get reward - dividends - without otherwise having to do any work. It also has become a way of dodging the responsibility of managing that company well; rather than planning for changing environments, trying to produce better products and services, most senior managers have become adept at manipulating the markets instead to increase dividend yields for their owners.

It's increasingly obvious that sweat equity, which has long been a very secondary aspect of business, is once again coming into its own. To me Obama has just turned Chrysler into an object lesson, one that banks and financial institutions in general should pay a great deal to. It looks like the silent partner is beginning to speak up, and what he's saying is going to completely reshape the way that America does business.

Future Proof: The Disaggregation of Business

2009-05-07T10:04:00.000-07:00

91. Our allegiance is to ourselves—our friends, our new allies and acquaintances, even our sparring partners. Companies that have no part in this world, also have no future.

Cluetrain Manifesto

The following blog is written in support of Cluetrain Plus Ten, a celebration of the 10th Anniversary of the Cluetrain Manifesto.

The news today in the papers was rather stunning - the United Auto Workers union was buying part of GM and Chrysler. General Motors, once the largest and most powerful car companies in the world, is being sold to its workers because the company became too fixated upon the business of making money and not fixated enough upon the business of making cars. Presumably, those workers, who still are in the business of making cars, may actually understand where their priorities really are.

This process is going on everywhere. The newspaper publishing industry is disintegrating, not because there's not enough news, but because there's too much of it - millions upon millions of "citizen journalists" who are reshaping the fabric of news, armed with inexpensive camcorders and laptops and iPods. Big box stores are being replaced by hundreds of thousands of specialized retailers, operating over the Internet or with minimal brick and mortar presences. Office parks are emptying out, as the workers of the companies that used to be in them work from homes and coffeeshops and conferences a thousand miles away. The giant businesses loom over all of this like hulking dinosaurs, scary until you realize that most are dying, and that what you are seeing are the skeletal ribs of decaying corporate carcasses.

Recessions come and go (though most in the last eighty years have not been quite so bad as the current one) and in most of them, older, less efficient businesses often disappear along the wayside, beat out by newer, flashier, more nimble opponents. Yet it is likely that this time around, we're going to see a mass extinction event, because the very nature of business itself is changing.

The large-is-better business model evolved through much of the late 19th and 20th centuries because it was the most efficient mode for communication channels - a hierarchical business model is a network with a bias towards centralized information dissemination and execution. Direction was passed from a leader to his subleaders, who would then break down the tasks pertinent to their domain and pass it down to their respective subleaders, until eventually you had specific tasks assigned to individuals at the leaf ends of the network. It also had the advantage of working well in a geographically centralized manner - each subtree usually represented a geographic aggregation of some sort.

Additionally, such command and control structures had the additional benefit of pushing information back up through a series of management filters - if it was not perceived as being important enough to engage the time of a given lieutenant, it wouldn't pass beyond that lieutenant to his superiors. This meant that, in theory, only the most important information would make it up to the top, and the role of the centralized decision maker became at least somewhat rationale.

In practice, however, such filters also served to isolate these same decision makers from interacting with the outside world. Hierarchies by their very nature tend to promote privilege - the higher up the chain you are, the more you are rewarded, and in practice the less you are likely to interact with the people that actually use your business products or services - instead, you interact with your counterparts at other businesses or organizations. And as a consequence, hierarchies can become forts, with the leaders of the hierarchy only vaguely aware of (and usually far less mindful of) the actual work done collectively by the others in the organization that in turn pay his paycheck and bonuses - or of the people who pay for that work.

The hierarchical model is well suited for broadcast - information from a centralized source gets disseminated through the hierarchy, while the hierarchy in turn acts as a filter to analyze and consequently respond to this data in aggregate. This has the side-effect, however, of dehumanizing the response channel - you are less interested in whether Jane Doe was motivated by your messaging (advertising or otherwise), but far more interested in the fact that a 32 year old Caucasian single woman who makes $64,000 a year, lives in a $550,000 house and is a vegetarian purchased your product. Jane Doe is a person, the latter is a demographic profile that can be used to see whether Product X is successful in getting Jane Doe to fork over her hard-earned money.

The Internet establishes an alternative set of communication channels that are very different from the hierarchical model. In effect, it makes for ad hoc, collaborative, overlapping interest groups. It makes aggregate collectivist behavior far easier to accomplish, and it means that information can spread very quickly, as it passes from interest group to interest group through common members.

Most companies originally thought that such interest groups were a good thing - after all, most of marketing involves targeting your message toward a given interest group while trying to reduce the exposure of the message outside of that interest group (as the non-interested groups produces far fewer responses - it's not as cost effective to advertise to people who either lack the means or the desire to purchase your goods or services). Company X could market its new organic power bar to such interest groups, and expect a much higher conversion ... which in fact did happen.

What these companies were not expecting was that the members of this interest group would also pass negative information about the products (and the company) to one another ... and that they would talk back. This wasn't supposed to happen. If the power bar didn't taste very good, this was information that would spread just as quickly, and it was beyond the control of the company to fix. If the organic components really weren't, if the green message on the wrapper was at odds with the fact that bar was produced in a factory in China under less than ideal conditions, if the CFO was involved in an affair with the CEO's wife, all of this information would get passed on ... and the company had no way of controlling this back-channel communication.

Corporate communication is very impersonal - its intent is not necessarily to inform, but rather to protect the hierarchy - to promote the successes, to spin damaging news, to obfuscate the communication access to the primary decision makers and in general to reduce potentially embarrassing contacts between the decision makers and the outside world. The problem of course is that as the dialog channels between people improved, the cold, mechanistic nature of corporate speak also became far more obvious - and more sinister. People react negatively when they realize that communications are one-sided - that while there may be a semblance of human communication going on, there's actually no one on the other side that is in a position to actually do anything about it ... it's a waste of time.

Beyond this, corporations are made up of people, and when those people feel that they have been abused by the company, they now have at their tools powerful ways of disrupting those corporations. When people are laid off in a poor and demeaning way, when they are customers who have been "shafted", they will lose whatever loyalty they may have had to the company in question - and will become increasingly shy about giving loyalty to any corporation. They will develop ideas and tools outside of the context of companies - something especially significant because it is often those very ideas and tools that the company would otherwise turn into products and sell themselves. They will encourage others to boycott companies and suggest alternatives that will reduce sales for the company in question.

In one scenario I saw recently, a disgruntled former customer of a cable company established a website and devoted himself to convincing others to take their business elsewhere. In the end that one customer probably cost the company $1.5 million dollars in revenue, all over a cap on services that might have cost the company perhaps $30. Such anti-customers really didn't make much of a difference pre-Internet - the company could have acted with impunity because the real ability of that customer to affect the company was limited. Today, a single Twitter from the right person (who might either be the anti-customer or sympathetic to the anti-customer) could have hugely negative consequences for a company.

The real difference between a company and an interest group (a social community) is surprisingly small - usually an agreement for revenue sharing. This means that whereas fifty years ago it may have taken several thousand people to establish and run a business of any complexity, today you can get by with perhaps fifteen or twenty - which in turn mean that such companies need a much lower threshold of net revenue to be viable concerns. This is increasingly as true in capital intensive sectors as it is in information services. Componentization and modularization of parts in various sectors mean that you can construct and customize even durable goods at only a slightly higher margin than a much larger factory, and because you don't have the significant overhead associated with the larger factory, the marginal costs even out.

This means that, even as dinosaurs like GM thrash about in their death throes, there are dozens of smaller companies making specialty cars that are far more responsive to new technology and market demands, at a small fraction of the overall costs that GM needs to develop a given car line.

The upshot of this is that we are in for a long period of business disaggregation - where huge conglomerates spin off companies to sink and swim, where small, ephemeral companies navigate more effectively than large ones, where the distinction between consumer and producer becomes blurred to irrelevance. People won't be any less loyal, but they'll be loyal to those "projects" that they themselves have a controlling interest in. Brand names are only significant as ways of identifying those prosumers who are most adept at navigating this world, and are increasingly tied into the "personal brand" - "I trust Jane Doe because I can communicate with her, her ventures generally succeed, and she knows how to involve others in her ideas."

It should be an interesting decade.

Where has all the money gone

2009-05-07T10:03:00.002-07:00

I entered into an interesting twitter exchange recently, to whit:

BrendanWenzel: A lot of people have "lost" money, but who is gaining it all? Wealth is never destroyed, but transfered. Who is it being transfered to?
kurt_cagle:Actually, in this case, "wealth" is just being destroyed, because assets are being repriced downward.
kurt_cagle:Most real wealth was made 2004-2006 by people in top 1%; we're just now discovering the fact that we've been robbed.
BrendanWenzel: So you are saying that these worthless assets never had value and were just a tool to steal wealth?
kurt_cagle: ... a tool to steal wealth? Um ... yup, pretty much. Did any investment banker really produce $30 million worth of value? No.

The numbers vary - from $2 trillion dollars to more than $40 trillion dollars depending upon how measure it, but in any case, a lot of "wealth" has seemingly gone up in smoke in the last year. Retirement, pension and college funds have been cut by 50% or more, municipal bonds have turned to dust, treasuries at the local, state and national level are bare. The world has, seemingly overnight, gone from being hyperfrenetic with activity to being, well broke ... and broken.

The question that Brendan brought up is a sensible one - where did all that money go? Is there someone out there who's now sitting on a pile of everyone else's money? No ... and yes.

People, including bankers who should know better, tend to look upon money as being, well, solid. You work every day, you get a paycheck for your efforts that represents a contract with your employer. That contract is usually slanted toward the employer - you provide the labor, and at the end of two weeks or one month or some other milestone, the employer gives you a piece of paper transferring a certain amount of value from the companies earnings to you. You take this to the bank, the bank deposits it, and from there you can "spend" this value.

Suppose, however, that the company has not made this money in earnings yet. Instead, they went to a bank and said "give us a line of credit, here is our plan to make value in the future". The bank evaluates the plan and the individuals involved, and if it feels like the plan will return a reasonable amount of earnings within a reasonable time, it will give the company that line of credit - a form of a loan, along with a fee to be added in order to compensate the bank for the risk that the company won't in fact make these earnings over the stated time.

This means that the money that you are making is not based upon existing value, but upon future value production. In essence, the company is in turn taking a risk that you will produce, though it is usually a pretty safe one. If you don't, then you will no longer receive that compensation, and someone else will be hired.

Yet, say after a couple of years, the company is not making enough money - the guess that was made concerning the profitability of the venture was off. The company's already sunk money into infrastructure, into salaries for the people, into energy costs, and into intangibles - marketing efforts. The company can go back to the bank and ask for an additional loan, but the bank at some point needs to determine whether the ongoing effort will ever prove profitable - otherwise, it is simply throwing bad money after good. The bank decides that "no, we're not going to give you the loan" and assuming the company doesn't find other investors (typically with more stringent requirements because of increased risk) it will close its doors, and everyone will be out of a job.

Now in this particular venture, you may have made money - though much of that money went into paying off necessities - housing, transportation, energy, food, information access and so forth - so you may have actually just broken even or even fallen behind. However, when the company fails, it can't turn around and ask for that money back. It's been spent. The money that the bank has also loaned has been lost - the loan becomes non-performing, because it no longer generates revenue, and the bank also took a loss.

If the bank charges fees on the establishment of the loan, these fees are things that can be assessed early - at the time the loan is made. At some point, the bank manager might realize that taking the fees are less risky than waiting for the loan to mature. They sell the loan as a "security". Now, this security is still potentially valuable, because it represents a steady stream of income in interest, and an investor can buy the security as a long term performing vehicle - so long as the person or company who took out the loan can continue to make payments.

Now the bank, at this point, has been lobbying the government to let them sell these securities, and a particularly business friendly administration gives the go-ahead. All of a sudden, a bank can make a loan, pocket the fees for that origination, then sell the loan as a security taking additional fees. What this means is that the bank no longer has any real incentive to insure that the person or company taking the loan can actually pay it back, because by the time it becomes an issue, it will be someone else's problem. The bank has essentially siphoned off a fairly significant amount of money in the transaction without actually creating significant value.

What this means is that they will be encouraged to make many more loans, because there is no moral hazard if a loan goes bad. If the loan is a mortgage or a lease, the bank may also encourage the ones acting as brokers in the sales of these properties to try to get top dollar, because it increases the fees that they can take off the transaction. The mortgage broker sees no problems in that - he too gets a percentage off the top, so the more valuable the property, the more he makes. The county assessors that determine the baseline price will try to increase the property price as well, because that increases revenues in the tax coffer, and if tax revenues go up, well, its good for the city or county.

Now, normally, this breaks down if interest rates are high - because the person who actually commits to the purchase has to pay the interest on top of the agreed upon price and fees. However, if interest rates are kept generationally low, then even though the house may cost more, the individual payments may be smaller, especially if they can be spread out over a longer period of time. Then of course, you also have speculators who buy up properties with no intention of paying the long term price - they simply become brokers themselves, selling to someone else at a higher price in three or six month, because real estate prices always go up. The buyer may also simply not have the financial resources to purchase the property in the first place under normal circumstances, but with a bit of "creative accounting" they are encouraged to buy.

Now this chain goes all the way up and down - ratings agencies are encouraged to rate securities higher than they should be, corporate raiders use risky securities (junk bonds) to effectively purchase companies, replacing actual earnings with debits against future earnings. Stock brokers use this debt to leverage purchase of stock with very little actual money committed, and so forth.

All of this activity involves replacing existing earnings - real work - with promised earnings - credit, and because there is comparatively little oversite, the actual obligation on the part of the wage earners and company earnings climbs and climbs and climbs, until you get a situation where a person would have to work continuously, 24 hours a day, for century or more to produce the real work that's been obligated on her, usually without her direct consent. That's clearly unfeasible, and the system ultimately collapses as each company or person fails.

Debts that the banks and shadow banks hold have to be written off, rather than being treated like assets. This reduces the amount of money that the banks can commit to writing loans, and also instills a sense of hyperconservatism in extending new loans, because they can no longer service the old ones. This causes credit availability to collapse, which means that companies can no longer pay their workers (as the paychecks were paid from the loan which was to be repaid by earnings).

As workers lose their jobs, they cut back on their spending, which causes other companies to go out of business, which only exacerbates the situation. Companies are forced to lower their prices in order to move any product, and a deflationary spiral sets in. Everything loses value as the availability of money dries up and markets plummet.

Eventually, demand for goods reasserts itself, as things wear out, as population grows, or as people become less fearful about the future. However, the damage has been done - the negotatiated value of things have dropped dramatically, whether that's the cost of a new car or the cost of a stock, and people who purchased the stocks thinking that it was a safe investment now discover that they're holding worthless paper - the company has either gone out of business or, if it survived, now has a much smaller cash position and it will take time for it to get back to its earning potential, significantly reducing the long term return on investment.

So, given that, chances are pretty good that there's not one person out there who is now sitting on everyone else's money. The money never really existed, save in potentia. What disappeared was the expected potential of that future labor.

However, that doesn't mean there aren't scoundrels. Companies who buy and sell these securities have profited immensely by the transaction fees and bonuses, which also came from future earnings. It would be much like you being paid for the next thirty years of your wage earning time up front. If the business fails, it makes no difference to you - you've already been paid handsomely, and can turn around and spend that money any way you choose.

Yet that money has to come from earnings at some point, and it does. It comes from pension funds that fail, leaving people who have invested with nothing. It comes from reduced pay elsewhere in the industry, as credit has been compromised. It comes from tax revenues, which decline dramatically in a recession because people don't have the wherewithall to pay. In other words, the thirty million dollar "bonus" that the hedge fund manager or bank CEO takes home comes directly or indirectly from the earnings of others, who now have to work longer just to get back to where they are.

So, yes, it was a ponzi scheme, a bubble with a skim, caused by the greed of "financial professionals" and political officials, aided by tax cuts that were highly favorable to these same people, and a war that made it possible to hide similar fraud elsewhere. It is still going on, and it has bankrupted this country for years to come.Where has all the money gone

End of US Dollar as Reserve Currency

2009-05-07T10:03:00.001-07:00

I'm going to get off the meta-trends that I've been following throughout the week and get down to a list of things that I've been tracking myself. I'll probably make this a regular feature - there's so much going on right now that I doubt serious whether any one article would cover more than a small part of it, and trends can be disrupted (or just peter out) without actually amounting to any thing. I welcome feedback here on what you've been watching as well, as I think the best way you can become informed about the world is to get a different perspective from the one you currently have.

So, without further ado:

The US Dollar is losing its status as the world's reserve currency.

First, a quick definition here: in the 1940s, a decision was made on the part of the various world powers to establish the US Dollar as the reserve currency for the purchase of oil. What that meant was that if you wanted to buy a barrel of crude on the spot market, you could only purchase it in dollars, and if you wanted to sell that barrel, you similarly had to accept that money in dollars. This was essentially one of the key requirements for receiving aid via the Marshall Plan, something that was sorely needed in Europe at the time.

In essence, what this meant in practice was that a country that needed oil (and all countries need oil) had to maintain a certain amount of financial reserves in dollars. To get them, it either had to buy US goods, or it had to give the US a certain amount of gold at a fixed rate to buy these dollars, usually in the form of treasury bonds. However, on the flip side, a country could also, upon demand, sell their dollars back to the US for gold.

This had the immediate impact of swelling the US treasury, which was in fact one of the things that helped the country pull out of the depression of the 1930s. It also introduced a pernicious inflation through much of the 1950s in the US, as the US printed more and more such dollars in order to meet global demand. However, it also had a darker side effect for the banking industry - it kept their ability to leverage down to a very definite minimum, meaning they could originate very few loans - and it increased the demand for gold globally as other countries began to recover.

Presidents Eisenhower, Kennedy and Johnson were all forced to revalue the currency by increasing the price of gold, but diminishing actual reserves finally forced Nixon, in 1973, to "close the gold window" and declare that the US would no longer honor the gold cap but would let the dollar float. This (along with the effects that it had upon the oil producing states) ended up forcing the energy crisis of 1974-77 and the subsequent period of hyperinflation. It was also one factor leading to the creation of the Euro.

Inflation was ultimately tamed by Paul Volcker, the Federal Reserve Chair, who raised the prime lending rate dramatically in order to attract foreign investment, a strategy, which, while leading to a fairly severe recession in the short term, managed to accomplish the task, restoring confidence in the American markets and laying the groundwork for much of the long bull market that followed thereafter.

However, without some form of backing, the degree of confidence in the dollar had largely become a function of the degree of trust in the US economy. This trust began to be eroded in the wake of the savings and loan scandals of the late 1980s and of the implosion of specific hedge funds after the economies of a number of Southeast Asian countries collapsed in the late 1990s.

However, the last decade has seen the uncertainty turn into outright distrust as the Federal Reserve seemed to be deliberately manufacturing bubble after bubble in an effort to sustain an increasingly shaky financial system. The housing bubble in particular in particular had the effect of significantly raising questions about the strength of the dollar, and by mid 2007, the value of the dollar dropped fairly precipitously relative to other currencies. The Canadian Loonie, for instance, at one point briefly topped US1.10 = CAN$1.00, after started at about US$0.74 in 2002.

This was also reflected in the price of oil at the time. While oil prices were up fairly dramatically in Europe, they were up far more (percentage wise) in the United States (the US has a very low gas tax rate while Canada and most European countries have a much higher rate, meaning that absolute measures were actually much closer). The top of the oil speculation market came in the summer of 2007, though by the fall, the first inklings of problems within the mortgage sector were making themselves felt.

This isn't the place to go over the whole financial collapse between 2007 and 2009, that story is now familiar to most people. However, in its wake, a couple of very interesting things have happened. The first has been a massive flight to treasuries (the US Dollar) which has caused the unwinding of most of the currency advances as investors have moved out of equities and properties into a temporary store. For foreign investors, the assumption has likely been that it was wise to move out of falling markets into treasuries rather than repatriating those funds. However, this has also had the effect of creating a bubble in Treasuries.

However, a second factor that's come into play has been that China has been purchasing Treasuries in order to keep their currency, the Renminbi pegged to the US dollar in order to remain competitive in providing goods and services. This has meant that they have ended up purchasing roughly $1.5 trillion dollars in treasure currencies as of 2009 - money that in fact has largely been used (dubiously) for the Iraq war and for financing the mortgage bubble in the first place.

One way to think of a treasury note is to envision it as a stock warrant in USA, Inc. The warrant pays a dividend (interest) to the holder of that note. When a company goes public, it sells shares in the company, yet the company itself is worth only so much money (essentially some percentage of its potential lifetime earnings). This means that as the number of warrants issued rises, the individual return on those warrants drop - the warrants become worth less. At $1.5 trillion dollars, China holds roughly 10% of the total GDP of the US. If China was dump these holdings on the world market, the value of the dollar would collapse overnight, resulting within six months of extraordinarily high inflation - high double or even triple digit inflation.

China won't do that, because it would not only make its investment worthless, it would also completely destroy the economy of its largest market. However, it has all but stopped its own purchases of treasuries, and in the last month (March/April 2009) has devised a strategy which will let it significantly reduce its own exposure to American financial activities. It has contacted the International Monetary Fund, and asked for the creation of an IMF bond invoking what are called Special Drawing Privileges that essentially would make it possible to set up direct currency exchanges with other countries - most notably Brazil, Russia and India, which, with China, make up what have become known as the BRIC economies. It has also created additional currency swap agreements with countries such as Argentina, a number of countries in Central Africa, and Indonesia. Significantly, all of these are oil or other resource producers or act as brokers for same.

Put simply, the goal of the Chinese is very much in accord with a number of oil producing companies in the Middle East - reducing the historical role of the US Dollar as the global reserve currency. Given the antagonism that the US has engendered over the last decade, there is far more support for such an action than there ever has been in the past, and even though it is likely that Obama's overtures will likely mend a few fences, the real damage has been done to the trust of the dollar. Sometime within the next 3-5 years, it is likely that a global "reserve currency" will arise, one that consists of a basket of floating currencies and exchange agreements rather than any single country's currency. The US will likely be a part of that, of course, but it will no longer be the world's bank (or the world's first consumer) - and that has a number of potential implications for the US.

One of the largest is that the US credit card will officially be maxed out. The US borrowing debt ceiling has been a convenient fiction for a long time - as the US gets close to it, an act of Congress raises the ceiling. This was done largely on the strength of expected sale of US Treasuries. The world now has more US Treasuries than it could use, but once a basket currency becomes the norm, there will be far more interest in purchasing other countries' debt instruments, which means that demand for US Treasuries will likely remain depressed for some time.

This means that the account deficits explode, and there becomes no way of even paying off the interest on the debt. This will result in the reduction of the rating of US bonds. One effect of this is that relatively soon, taxes will have to be raised, and fairly dramatically, in order to finance any new expenditures. The US will have to raise its own interest rates in order to attract more investors, at a time when the economy will just begin its recovery. Defense expenditures will have to be significantly reduced, social entitlements will have to be renegotiated, and the ability of the government to act will be increasingly hamstrung.

This will also mean that the Americans will have to save more. At the moment, much of the effort in the recovery is going towards getting Americans to spend more, to increase the velocity of money in the system, but its becoming increasingly obvious that this isn't working. Now the problem with saving is that while it is prudent at the individual level, it reduces the amount of money available to create businesses (in the short term) and reduces tax revenues that derive from the acceleration of money in the system at the macro-level. In other words, a savings-oriented mindset is anathema to a consumer economy ... at least for a while.

However, my own take is that we're not going to end up going back to a 30's style "great depression", nor do I necessarily see the stark future outlined by people like James Howard Kunstler. We're in the midst of a major structural change in society, one fed in great part by the profound changes in information infrastructure and additionally shaped by a growing awareness about the fragility of the underlying ecosystem. The consumer culture of the 20th century is failing, but that doesn't mean that all of a sudden we should all become socialists or communists or cogs in some oligarchical brave new world order.

What it does mean is that we're making it up as we go along. One of the reasons for the original Bretton Woods accords was to attempt to guarantee, as much as possible, the notion of full employment, at the expense of inflating the currency. The reality now is that full time employment, at least in the traditional sense, is breaking down. We need to make some hard decisions about what constitutes a valid standard of living - and what constitutes an excessive one. We need to come to terms with information and reputation as forms of currency, with virtual currencies, and with the degree to which currency reflects value.

All of these things (and many, many more) will ultimately need to be determined as the world adjusts to a new abstraction of currency, and the more that efforts are made to return to an unsustainable status quo, the longer it will take before a true recovery can take place - and the more turbulent society will become.

Future Proof: Tornadoes and Turbulence

2009-05-07T10:01:00.000-07:00

When I was sixteen, I saw my first tornadoes. My family had decided to visit some friends in Cheyenne, Wyoming, so we loaded up the van and made the trip from Peoria Illinois, encountering unsettled weather conditions all during the two day trek. As we were unloading our bags, I noticed my father periodically staring up at the sky, which had taken on a peculiar green cast to it. A few minutes passed, the sky became darker and more ominous, then out of the maelstrom one, then two, until finally six twisters came snaking down from the clouds. For nearly half an hour we watched as the tornadoes ran along a couple of ridges, occasionally flattening one house then hoping over the next, until finally the storm lost enough strength and the twisters became wispier and more tentative until they finally faded altogether.

By the end, the twisters had collectively destroyed 200 homes, flipped numerous cars, caused significant damage to the governor's mansion, and threw a commuter plane through the door of an aircraft hangar. It was one of the most destructive tornadic storms to ever hit the state, and gave me a healthy appreciation for the power, majesty and mindless destructiveness of nature.

One of the most interesting facts about tornadoes is also one of the least appreciated. Tornadoes come from thunderstorms - as anyone who has seen the ominous green thunderheads can attest to, but they are not, in fact, part of the the powerful circulating cyclonic cell of hot moist air and cold dry air that make up most thunderstorms. Instead, they arise due to turbulence along the edges of this large rotating system. Thunderstorms can move quickly, and can rotate quickly. As they do so, they drag the air around them, but this drag is uneven ... and is influenced by such factors as the topography of the ground, the overall viscosity of the air and the formation of wind sheers and streams ahead of the storm - in other words, the environment external to the storm itself.

Tornadoes are directly related to the vortices you get when you run your hand through water in a trough or other constrained place - they are islands of temporary stability within an otherwise unstable environment. They also act to siphon off a lot of the potential energy within the storm itself, converting that energy into kinetic energy, which eventually dissipates as drag with the rest of the environment. Once the tornadoes release their energy, this also typically pushes the storm down a level of organization and energy to the point where it can no longer hold the water vapor that it is carrying, which then causes the heavy rains that usually follow such storms as the structure dissipates.

Every system, including systems of abstraction, requires energy of some sort to maintain it. Similarly, every system interacting with those things outside of that system create drag on the system, resulting in turbulence. Turbulence should be seen as the transfer of energy out of a system into the environment, and as such is very closely linked with thermodynamics. This holds as true of software and social structures as it does of physical systems, as long as you understand that in both cases what you are dealing with are systems of nested abstractions.

This doesn't mean that outside of every social structure there's a giant whirlpool or tornado waiting to happen. Rather, it's worth understanding that any system is made up of interacting parts that for the most part have achieved a fairly high degree of internal efficiency. One way of thinking about this is that the system has a certain momentum associated with it - energy and information moves through the system in such a way as to keep the system cohesive.

However, especially at the edges, this energy drags against the outside world, and in so doing, it creates pools of resistance, and counterveiling forces. Normally, such forces are comparatively small, and in many ways can actually contribute to the underlying cohesiveness of the primary system because they create a barrier of insulation against external stimulae or impulses - the turbulent counterflows absorb the attack, dissipating or at least blunting the impact upon the system. One way of thinking about this is that people may have a particular ambivalence about leader or political group, but they fear change from outside more than they do the status quo ("better the devil you know than the one you don't").

However, in the presence of other dynamic systems, sometimes the turbulence that emerges becomes large enough and cohesive enough to became stable in its own right, especially as one particularly stable "whirlpool" merges with another.

A good example of this can be seen in the rise of Open Source software. Microsoft in particular had managed to dominate the software sector by the mid-1980s, and with it the proprietary software model become the accepted mode of operation by the mid-1990s. However, Microsoft also ended up stirring both resentment among other development groups and concern among customers that were afraid of vendor lock-in.

This set up turbulence for Microsoft's "system". Any one piece of that turbulence - Linus Torvald's invention of Linux, the rise of Apache as an increasingly popular browser, the GNU GPL, Sun's releasing of the Star Office code as Open Office, and so forth, individually bled small amounts of energy from Microsoft, but nothing that seriously impeded its own growth. However, each piece of turbulence would interact with others, and after a while a new countervailing system emerged out of that turbulence. A tornado or whirlpool is a cohesive system that draws on the energy of the overall hypercell, and the larger or more powerful the tornado becomes, the more it bleeds off energy from the main cell. Open source soon began to bleed away the proprietary model that Microsoft most clearly embodied at the time, pulling in more developers, more investment, more potential users.

Up to a certain point, the energy entering into a system ends up as more turbulence and more quasi-stable neo-systems, as well as providing the necessary glue for smaller systems to merge into larger ones. However, there's a certain balance here - too much energy into an environment can prove disruptive overall as the turbulence makes it too difficult for new systems to maintain cohesiveness - the turbulence spawned subsystems are disrupted by their own turbulence (in essence, the market is boiling at that point). Too little energy, and you get systemic decay, where the least stable systems fall apart. Typically, transitions from one stage to another of abstraction involve energy exceeding or failing to reach a critical threshold for that system.

From the future analyst's standpoint, then one of the lessons to be learned is that when you look at what appears to be a stable system, look at where it is causing the most turbulence. At the moment, for instance, the whole of desktop computing is being challenged by the cloud, a universe of services that individually may not be a match for the corresponding desktop app, but that collectively are reshaping the programming paradigm dramatically. The traditional world of publishing is under assault from a myriad of social media applications that individually are not that threatening, but which together is forming a cohesive interactive system of its own that has traditional publishing on the ropes. In the near future, centralized power distribution is being challenged not by a single new power source but by a whole spectrum of technologies that each emerged in response to the problems that the existing grid failed to answer, and that collectively are creating a new system that is challenging most of the core assumptions about power distribution that have been considered "holy writ" since the 1920s.

In other words, when looking toward investing (whether time, money, career involvement and so often) look toward areas where countervailing technologies are emerging, and pay special attention to those that seem to develop easy synergies with other complementary technologies. In the energy sector, for instance, solar energy (photoVoltaics) including beamed microwave energy, geothermal pumps, intelligent energy routers, hypercapacitors, hybrid automobiles, maglev trains, recycled heat systems and wind farms together make up a cohesive system of technologies that are complementary to one another, and that collectively make up a self-reinforcing system. Individually, they won't replace the existing carbon-driven fuel system, but collectively, they may very well.

Future Proof: From Word of Mouth to the Open Book

2009-05-07T10:00:00.000-07:00

We are a talkative species.

If you take a look at the bulk of inventions produced in the last 10,000 years, they fall into four broad swaths - better ways to move things (and ourselves), better ways to protect ourselves, better ways to feed ourselves, and better ways to communicate with one another. Communication with one another is such a strong imperative that one of the harshest punishments that we can inflict on people is to deprive them of that communication - to put them in solitary confinement, to exile them to the wilderness, to "ex-communicate" them. In many primitive cultures, should a person commit manslaughter or some similar crime and get caught, they became "dead" - not killed in retaliation, but made a non-person that others were not permitted to acknowledge or speak to.

Because of that importance, how we communicate is a very significant thread for the future analyst to watch. The predominant communication channels that a culture uses will dictate its organizational structure, more so than any other factor. In hunter-gatherer societies, communication (beyond one-to-one local communication) is typically done communally, within groups. For formal communications - when decisions need to be made, for instance, or in the recitation of (and addition to) a community's memory structure - the role of speaker was typically formalized - the speaker was the one who held a given totem, or given the floor. Communication range was also limited to the speed at which a man or woman could walk or run.

Additionally, these early cultures typically made use of a living long term memory, usually via an oral "song" that kept intact the important stories, historical figures, legends, and constraints of the group. One of the fascinating things that neurologists have recently discovered is that musical memories are stored in a very different way than spoken memories are in the brain, and that such memories are typically retained much longer and with better fidelity (in great part because these memories are actually retained in the hippocampus and cerebrum rather than the cerebellum). Kinesthetic memories similarly tend to be retained far better as well. This may be why most people within these cultures were taught oral history as a combination of chant and dance - the body actually "remembers" this information at a deeper level than it retains speech.

Nomadic cultures made an important discovery - horses not only made for good food, but if you could manage to sneak up on a horse, it was possible to actually ride it. At first such rides were probably just short enough to put a spear in it, but after a while some genius realized that if they could actually control horses, they could go far faster than they could on foot. Beyond the obvious advantages from a food hunting perspective, one additional advantage was that horse-borne messengers could communicate far more effectively with people at greater distances. This made it possible to coordinate actions, and was in fact one of the first instances of hierarchical military structures - a warlord with one force could communicate effectively with additional forces under his captains, who could in turn coordinate their forces with lieutenants.

Agrarian communities developed a somewhat more defensive structure, designed primarily to keep these same nomadic cultures out. but also because the communication requirements of agriculture are broader. Farming is a chancy business - you're forced into defending a turf of land - running away isn't really an option unless you were willing to starve - so you needed to have ways of coordinating the troops (again, a hierarchical structure). However, you also needed to manage inventories, to determine how much of a given crop you needed to replant as seed, to set prices on the grains and other goods, and, ultimately how much to tax people for the services in order to make all of this possible.

As discussed in an earlier column, what this amounted to was the process of shifting abstraction levels. This can be seen in mythology. Tribal mythologies are very animistic - every grove, brook, wind and cave had its attendant spirit, but for the most part, those spirits simply existed - you acknowledged their existence and occasionally bribed them in order to insure success in your ventures, but there was little in the way of hierarchy.

Most agrarian societies, on the other hand, very quickly established hierarchical models - supreme gods, and then secondary and tertiary gods - that reflected the growing power of centralization in human hierarchies. The warlord became the god incarnate, and power became concentrated in bureaucracies - priestly castes, military castes, merchant castes.

It's perhaps not thus surprising that writing only came about with this shift in complexity; tribal societies have no need for writing, but agrarian ones have a large number of such needs. The emergence of writing was a radical change in human society - first because it meant that humans didn't need to expend as much of their thought processes on rote memorization, and likely for the first time could start thinking about information in a way that wasn't tied specifically with a generational oral record.

Indeed, one interesting speculation about this is that "spoken" language may actually only have emerged about the time that writing began, and that most languages prior to that were likely sung rather than spoken. One possible indication of this is to look at those cultures in the last hundred years where there was no formal written language and compare what happens before and after they are exposed to writing. Typically, children from these cultures who are then exposed growing up with writing tend to have far worse rote verbal memorization capability, though they have far better analytic ability.

It's worth noting that reading and writing also cause a significant change in the communication structures of a society. Writing is an asynchronous operation - information placed in writing does not need the speaker of the information - you could write "letters" that allowed (slow) communication between people who were not geographically close.

Additionally, and more subtly, it becomes possible to scan written information in a way that's simply not possible with speech. This in turn let to breaking blocks of narrative into smaller, more digestable portions, a process that almost invariably occurs as new media emerge. The earliest written narratives were literally epic in scale - they represented a story that might be told over several hours in an evening, because they were almost certainly based upon earlier oral stories. However, as writing became more sophisticated, it began to develop a recursive hierarchical structure of its own as people began to master the nuances of committing symbolic representations of meaning to a physical medium.

Most early literate cultures developed a "bible" of some sort, a written work usually attributed to divine provenance, that encoded the mythos (the legends, accepted history, genealogies and so forth) and ethos (the ethical rules or laws of that people that described what was acceptable and unacceptable within the society) of that culture. The Hebrew Torah, the Islamic Talmud, the Christian Old and New Testaments, the Hindu Mahabharita and Ramayana, all of these "books" emerged in cultures that had established active literary traditions, and more had them long enough to accumulate a body of related "subordinate books". Indeed, by some estimates the "Bible" alone represents the political and cultural selection of between 80 and 110 different books, depending upon the particular sub-branch of Christianity or Judaism, with another few dozen books that were in one version or another over the years but have been dropped.

Cultures of the Book illustrate how powerful the advent of writing was. With a single cultural canon, mores and ethics can be established independent of geography. For instance, the Old Testament represents the ethos and history of a desert-based culture. Desert cultures are typified by a nomadic existence, a male-dominated society where women were usually treated as chattel, a strong sense of hierarchy, a low premium placed on the value of human life, and a very competitive warrior ethos. Even the New Testament, which may have been influenced by the Dionysian Mysteries so prevalent in Asia Minor as the time, is still filtered through this desert culture filter.

Yet because of the "authority" that the book has compared to more transient oral traditions, Christianity was carried all the way to the wilds of Northern Europe, England, Scotland, Ireland and Wales, which previously had a forest culture structure - far more gender equality and egalitarianism, strong oral traditions but only very crude literary ones, a far higher sanctity of life, a much stronger clan or family basis, and so on. It's perhaps not surprising that so many of the heresies that the Catholic Church eventually had to stamp out come mostly from the north as a consequence, as there was a certain cultural schizophrenia that occurred as a fairly alien cultural outlook became overlaid upon a very different foundation.

The migration of book production to the north also brought about the next major evolution in communication - the shift from papyrus based scrolls to vellum-based books. Papyrus came from desert reeds, and thus, over time, became increasingly brittle - and usually could support only a minimal amount of pressure before it crumbled - thus papyrus scrolled around two rods, casette-tape style, was the most effective way to store it.

Vellum, on the other hand, was made from lambskin, which was far more plentiful in the north. Because of the curing process, vellum was remarkably resistent to fading or crumbling (indeed, many vellum books survive to the present day in very good condition. However, lambskin was, by its very nature, much more limited in dimension, which eventually led to using vellum leaves that were originally stacked together, then later sewn together, into a new arranged where the content was displayed in pages.

The introduction of new communication channels are quite frequently accompanied by significant upheavals in cultures, especially if the new channels are markedly superior to the old. The first Western printed work was the Gutenberg bible, produced in 1439. Printing was quickly picked up and improved by Italian Aldus Manutius in the mid 1450s, and in England by William Caxton and others in the 1470s.

One of Caxton's most significant innovation was actually a cost-saving measure - rather than a printer using a single folio page for a book (which resulted in very large books), he subdivided the folio page into quartos (quarters), and figured out how to orient the page so that such quartos could be more efficiently printed and bound. This essentially meant that you could produce four times as many "books" with the same effort, and it also represented the shift to the first truly portable book since the innovation of the scroll, which had the effect of lighting a fire under the nascent publishing industry.

This technology change was likely one of the major factors of the Reformation and the rise of Protestantism. Prior to this period, most bibles were owned only by churches or the very wealthy/powerful. With Caxton books (and a subsequent shift away from expensive vellum to cheaper cloth and wood pulp pages), bibles (and many other books) now moved into the realm of being affordable (albeit still expensive) for the average middle-class burgher or shop-keeper.

Martin Luther's innovation (and its worthwhile understanding that it was an innovation) was to translate the contents of these bibles from Ecclesiastical Latin into contemporary German. This has the immediate effect of letting ordinary people understand and interpret what had been, up until then, what had only been disseminated by priests and clergy. In modern parlance, Luther disintermediated the priests. This had the fairly immediate effect of subverting the legitimacy of the Catholic Church (especially in the North), and the rise of a new class of priesthood who adapted to the new technologies by shifting from the role of arbiters to the role of guides and interpreters.

Of course, the established order did not go quietly into that good night - it seldom does. Once a given communication channel stabilizes, a social order will tend to evolve around that communication channel, to become invested in it. This is especially true in those situations where the communication system is hierarchical and it meshes with a hierarchical mindset. Once you introduce a technology that had previously been available only to the gate keepers to everyone else - whether affordable books in the language that people spoke or low-cost publishing systems that bypass the established news providers, then the value of the existing services plummet, while those that master the production within the new media are able to establish new value measures.

What's more, invariably the first uses of a new medium are to recreate the dominant pattern of the old. The vast majority of all of the new works produced during the mid-15th century were bibles. Of course, this undermined the scriptora throughout Europe - a single bible might take a team of monks the better part of a decade to create, whereas while it may take only a few months to set and print a bible using a press, and once one bible was printed, dozens more could be printed until the first wooden type blocks wore out. Once people began experimenting with molten lead dies, this meant that hundreds of such books could be created.

Yet the real changes - the truly political ones, came as printers began to realize that while the demand for Latin bibles was high, it wasn't infinite, and eventually they began to turn to examine other uses. The translation of bibles into contemporary languages (the vulgate, or common, versions) became an act of defiance of the existing religious establishment - as well as a means of controlling the message by local kings and rulers trying to break the stranglehold that churches had held on their lands for years.

It also meant that other books were soon also published. Histories, books of poetry, philosophical tracts, and similar works emerged around this time, as the medium made such works economical to produce, and in so doing laid the groundwork for the birth of most forms of contemporary literature. In many ways, publishing in the period from 1470 to about the 1530 or so was as dynamic a period of time for innovation as the Internet would be five hundred years later. By the end of this period, the Reformation would be sweeping Northern Europe just as the Renaissance was sweeping southern Europe. The church, seemingly dominant and invincible in 1450, would be torn by strife and dissension as a thousand year old empire disintegrated.

There are a number of lessons to be learned here. Changing communication channels can have huge impacts upon society, something that we're only just really beginning to face today. It is a mistake to see the world of 2020 as being much like today, because the very structures that formed the foundation of the last couple of centuries is now being torn asunder in very much the same way. More on this in the second part of this blog post.

Future Proof:Catching Black Swans

2009-05-07T09:58:00.000-07:00

Recently an earthquake hit the town of L'Aquila in Italy, collapsing a number of buildings, killing more than a hundred people and leaving several thousands stranded. Last year Hurricane Ike slammed into Galveston, Texas, leaving many parts of the coastal community submerged. In the news of late have been the "one-time write-offs" that banks are taking because of the extraordinary credit crisis.

The one thing that all three of these things have in common is that they appear to be "Black Swans", a term that economist and mathematician Nassim Taleb used to describe events that seem wildly improbable, yet nonetheless do occasionally happen. The term derives from the saying that, as all swans are white, the term "as rare as a black swan" means something that is so improbable that there is no way that it could happen.. Of course, eventually a species of black swan was eventually discovered in Australia, which was Taleb's point - beware of assuming that simply because events are rare, they will never happen - and when they do happen, they tend to cluster.

To understand why black swans are not as rare as one may think (and to anticipate their appearance), it's worth going back to the abstraction model proposed a couple of posts ago. Almost all complex systems exist at multiple levels of abstraction. One way of thinking about this is to envision the human body as being made up of subsystems - organs - each of which in turn are made up of tissues. which are in turn made up cells. A person in peak health has all of her subsystems (organs) operating more or less optimally.

However, one day a carcinogenic agent enters the body - tobacco smoke, asbestos, environmental steroids, the list is rather depressingly long. A cell in her breast mutates in the presence of the carcinogen, losing the ability to "shut off and die", which most cells do after they've reached a point where their internal mechanisms are no longer sufficient to do the job efficiently. The cancer spreads, every time the cell would undergo normal meiosis. If the woman is lucky, a routine examination would find this cancer when it is fairly small, at which point the best solution is to remove the cancerous tissue.

If the woman is unlucky, the cancer would grow until it found a conduit (typically a lymph node or a blood vessel). Cancer cells that broke free from the mass would be transported by the conduit until it ends up somewhere else in the body, at this point it would attach itself to other tissue and continue spreading. The cancer cells crowd out other cells, choking off access to blood vessels or waste channels, and other cells either become cancerous in turn or they become necrotic - dying but not being removed by the body's defenses. The woman becomes tired more easily as energy that would normally be going to maintenance of the body is increasingly co-opted by the cancer cells. Tissue becomes tender and inflamed, and pain caused by cancer cells crowding in on nerve endings becomes more endemic.

If the cancer spreads to the lungs, then breathing becomes more difficult and becomes an emphysema. If it spreads into the lymph system then the woman has more trouble fighting off infections, and becomes sicker more often. If it spreads into the bone then normal stresses may cause the bone to snap.

The broken bone is a black swan event - it seems unlikely that a bone would normally break under typical stress actions, but in point of fact, this isn't a typical stress action. The system has been compromised, and the cancer has spread out in a spider-web like fashion through much of the body. An aggressive fight against the cancer by irradiating it or using poisons (chemotheraphy) may be able to remove the tissue, but typically it does so by further weakening the ability of the body to function.

The body does not die because of the cancer. Instead, the cancer causes each system in turn to become less efficient, and ultimately to fail because it can't get the energy necessary to continue. Once one organ fails, it increases the likelihood that other systems that are dependent upon that organ will also fail. The unfortunate woman dies of system failure.

This rather detailed and morbid description still serves as a metaphor for other systems. Complex systems are made up of simpler ones, which are made up of simpler ones still. Corruption usually occurs fairly far down in a given system, but most complex systems are generally fairly effective at catching and eliminating the obvious points of corruption (or worst case scenario, sequestering them off in isolation) . Corruption here simply means a subversion of the normal functions of that particular abstraction - an bridge inspector signing off an inspection report with only a peremptory check of the bridge, an employee stealing supplies from the supply cabinets, a student cheating on an exam or a businessman cheating on his taxes, a radical publishing seditious tracts, to name a few of the many, many examples.

Physical system analogs would be areas of snowfall on a mountain that gets more sun than normal but is also supporting other areas of snowfall, a particularly warm, dry, dust-laden wind coming off the Sahara into the Atlantic Ocean, the gradual creep of increasing temperatures in an area that hasn't faced them pushing flowering and bee pollenation behaviors slowly out of synch.

The point about most such corruption (i.e., regions of potential instability) is that, for systems in quasi-equilibrium, the corruption usually has comparatively little impact over the short term. Most systems have regulating mechanisms that tend to correct for such instabilities - the office manager notices that one department is using more supplies than the other, and a bit of surveillance reveals that one employee is using dramatically more than he should be. At that point, the employee is summarily fired, and a new employee hired to replace him, and the message is made clear - you steal, you're gone. This tends to move the system back into equilibrium.

The regulation and action is not a normal event - it is only undertaken when corruption is noticed. It's a small "collapse", one that may result in some disruption of activities and hence impact the efficiency of the abstraction - and for the employee it results in a significant disruption from the way things were. However, such feedback cycles normally keep the system relatively stable.

However, over time, the corruption can become more endemic, and at a higher level of abstraction. The managing bridge inspector is lax in checking on reports, and the inspectors under him avoid looking at those places on the bridge that are awkward to get to or would require getting especially dirty, the comptroller in a company works with one or two accounts to falsify the books, the teacher at a university starts accepting bribes and sexual favors for grades, a company provides campaign donations to a politician in order to give them a tax break or special legislative consideration.

Note in this case that there are two levels of abstraction involved in all of these scenarios. Generally the role of a manager is to act as a governor or regulator on the actions of others, to provide negative (damping) feedback to minimize corruption in a system. When that feedback is subverted, it amplifies the corruption rather than reducing it, and it makes it increasingly likely that the feedback will start to compromise the stability of the abstraction layer.

One of the more interesting phenomena that takes place in systems of abstraction is the paradox that the longer a system remains in equilibrium, the more likely that it will become unstable. In order to understand why, consider that most corrective feedback occurs only after a problem has reached a crisis point - the office manager finds that supplies she just ordered are gone, and she can't think of any legitimate reason why they would be. As she is responsible for her budget, she knows that she has less lattitude if excess pilferage is reducing her budget (and that she could face "corrective action" if such thefts continue to be unexplained.

However, as the organization gets larger, the office manager has more responsibilities, and tracking down pilferage drops down the list. The attitude begins to form that office supplies are fair game, and people begin more inclined to take supplies whether they need them or not - and those office supplies begin to move towards bigger ticket ideas like computers and projectors. Expense accounts start to become padded, and pretty soon begin to become a significant part of a person's income. Eventually the amounts begin to become high enough that it impacts the bottom of the line of the company, particularly if the comptroller and his friends in accounting are in on it (getting kickbacks for equipment that's disappearing).

What makes this worse is that it has gone from being an isolated instance to becoming pervasive and endemic. You can't fire everyone without bringing the company to its knees. Eventually you are forced to fire the comptroller, establish a new-tight accounting system for all internal goods and services, alienate a number of your employees who had come to see the office supplies as a right, and then spend several months searching for a new accounting team.

Stability breeds complacency, which breeds instability. Hyman Minsky, an economist in the 1960s, laid out this hypothesis for financial systems, but it holds in most complex multi-layered abstraction systems. Deregulation of the banking industry, low interest rates on the part of the Federal Reserve and a push towards home-ownership in the early 2000s meant that bankers could make higher risk mortgage loans to increasingly unqualified buyers then sell these loans to other financial institutions. These financial companies would combine these mortgages in novel (and dangerous) ways and sell them as financial vehicles to investors. The investors would then use these securities as collateral to build increasingly unsustainable leverages, while insurance companies sold "black swan" insurance that they never expected to pay off in order to make these securities palatable to accountants. Meanwhile, the real estate agents worked with the appraisal firms and builders in order to get the largest payback in fees, and homeowners in turn found themselves forced to take out ever larger loans for the same properties.

Low initial rates on loans were reset after a specific period to a much higher rate, and people began to fall behind on their payments, in time, the cascade of defaults and jungle mail cascaded through the system. The rapidly receding value of these assets caused a Minsky Moment in September 2009, as a key investment bank, Lehman Brothers, was allowed to go bankrupt. Because of the unwinding of the positions that Lehman had held, this created the financial equivalent of a heart attack as credit disappeared from the market overnight.

Starved for credit, companies could no longer sustain regular payroll, watched their energy supply (cash inflow) dry up as consumers pulled back abruptly in spending and soon were forced into rapid liquidation. Abstractions were unwound as energy (in the form of credit) disappeared from the system. Unemployment shot up as millions of people were forced out of work, accelerating the crisis, while attempts to recapitalize the banks have so far fallen short of solving the systemic problems.

Instabilities - turbulence - disrupted each layer of abstraction. This process is still ongoing, and will likely take one to two more years in order for the turbulence to dissipate to a level that new structures can start to form again, at a lower energy level.

The lesson for futurists - first, disruptions do not happen without reason. Most disruptions occur when a stable (complacent) abstraction becomes corrupt and brittle. In a recently stable scenario, external stimulae impinge upon the system all the time, but the system is resilient enough to ward them off. As systems become more mature they become more fragile, and their ability to adapt becomes increasingly compromised. Eventually, a stimulus occurs that causes a breakdown of a particular part of the system, and the system has become so interdependent that this shock then gets passed on, destructively, to other subsystems. The subsequent loss of system integrity can prove fatal, and the system will unwind to a less energy intensive state as energy bleeds into turbulence.

In general, you cannot predict what the shock will be that will ultimately send a system over the edge, and its futile to try. What's important is to examine whether, given a shock, a system is resilient enough to absorb it, or whether the shock will prove devastating. The role of both analysts (who are futurists) and regulators is to do the research to determine what organizations are too fragile, and then to examine the consequences that a shock to the system will have.

One final note here: a good place to look are organizations that are deemed "to big to fail". Most likely once a company (or a government) reaches that point, it is definitely overdue for an earthquake.

Future Proof: Abstraction Uses Energy

2009-05-07T09:56:00.000-07:00

Societies, just as software systems and other dynamic systems, have the potential to operate at multiple levels of abstraction. Government is an abstraction, for instance - you can't point to a particular building and say that is the government - it may house the officials and the record stores, but the government is in fact the interaction of a set of laws and rights with the participants of that government, and the officials in question essentially serve to make government happen. Government isn't a thing, its a process - and as such it uses energy.

Abstraction is the process of hiding complexity under a metaphor (or interface). Once the abstraction exists, it becomes a thing, and can interact with other things at the same level. This can be seen in the settlement metaphor. A person leaves a city and journeys into the wilderness. There he sets up a farm. Others, driven by the same pressures, set up farms nearby - the resource base is extensive (though not infinite). The farmers initially farm to produce enough goods to feed themselves, but eventually may reach a point where they produce more than they consume and they can start trading.

A trading post is set up within comfortable transportation range, and they are soon able to specialize in producing different products. Other amenities spring up - smiths to handle shoeing of horses and production of localized implements, chandlers to produce candles for light, coopers to build barrels and storage bins, taverns to provide food and drink (and occasionally companionship) and so on. A certain degree of energy is required to reach this stage, and the degree of dissonance rises with the number of interactions, until eventually, a group of people get together and set up a set of rules to create a "town", providing basic services (such as law enforcement) at the expense of certain activities (such as robbery or murder being "allowable"). It takes a certain level of energy to get to this new level of abstraction, but once it's achieved, it's stable - it's a quantum electron state within a metaphorical atom.

So long as the energy is supplied (taxes or services on the part of the members), the town continues to exist - and can even exist for a certain period of time after the energy is removed (thus, it's actually quasi-stable). This is why gold mining towns often died out - once the energy (in the form of gold sales) was removed, the town eventually couldn't maintain a sufficient level of complexity to continue. The building might persists, but no one was in them to do the business of governance.

Eventually, if the energy is maintained, other towns spring up in the same general vicinity. Advances in technology become possible as people specialize, and as edge effects motivate innovation. The towns begin to trade with one another, and as they grow, their boundaries eventually overlap. These towns (or townships, as they are now increasingly called) are able to produce more, are able to devote more energy to communal activities - the development of schools, of watersheds, of roads, of sanitation facilities. The towns become a city, forcing a higher level of abstraction (and more energy).

The interactions of cities with other cities is similar to, that of towns with other towns, but it is usually at a much higher level; ad hoc rules become the foundation of formal legal systems, universities spring up, as does the rise of a formal peace-keeping militia (a police force). At some point, if there are indigenous peoples in the same area, the expanding towns will eventually end up reaching one of three states - if they are at a sufficient level of abstraction to expend the energy, they will wipe out the indigenous population. If they aren't, then the will either assimilate the other culture and be changed by it in turn, or they will marginalize the other population - literally push them to the margins of the existing system. This is part of the reason why cultures with lower levels of societal abstraction (energy draws) tend to end up living in typically harsh and forbidding lands when such expansion occurs.

In time, the various cities in a region also form abstractions - provinces. Provinces (in the general sense, not specifically Canadian provinces) differ from cities in a qualitative way, and again they require a certain energy level to function properly. This nesting process can be carried up the line. Moreover, you can have other abstractions that have their own plateus - the structure of business, from sole proprieterships to transnational corporations, follow a very similar progression, and are in many ways fueled by the same energy derivatives - the movement of money in the economy.

It's worth addressing that issue here. Energy is required to convert resources into usable form, energy is required to transport those goods to either other processors or to consumers, who in turn use that energy to handle the processing of those goods either physically - a foundryman pouring steel into ingots, a truck drive driving those ingots to a construction site, a construction worker using the beams as piles the foundational support of an office building - or virtually - the bank manager who arranges finance for the office building, the graphic artist who puts together a brochure about the office building for prospective buyers, the programmer who writes the program that handles the finances. Money, then, can be seen as a proxy for either energy or resources (and usually a combination of both), while technology should be seen as a way to convert this money back into energy in some fashion.

Note, however, that this process of nested abstraction does not place all at once. A higher level of abstraction can only exist once the need for that abstraction arises - while you can call a new town in a virgin region a province, can draw maps, and put together flags, the abstraction level will remain that of a town until such time as you have enough of the actors (towns in this case) to even make a city, let alone a province. Similarly, a conqueror can take control over a number of cities in a region, but if he doesn't do anything to make those cities work together in a cohesive fashion (and doesn't supply the energy to make this possible on a continuous basis) then his empire will crumble within a few years.

So what happens when the amount of energy available drops? At first, not much. You get discontent among the participants, but there's also a certain amount of momentum that exists at each plateau that keeps the abstractions going. Call it cultural memory - its the reluctance that people have to give up what they have in terms of their own binding to the abstraction. Moreover, the energy levels of the highest abstractions tend to have higher priority than lower level abstractions. This means those at the lowest levels of abstractions usually feel problems first.

Individual towns first reduce staff and services, then reach the fateful day when they can't meet payroll, and disincorporate. Services disappear, and people leave. Opportunities diminish. In time, the town becomes a ghost town. The same things begin to happen at former townships that are now suburbs. The cost of commuting becomes too high compared to the benefit of living away from the action. Individual stores begin to fail, then malls begin to fall vacant. Housing prices drop, foreclosures rise, and crime levels (a break down of the authority of the state to maintain order) rise along with it. The wealthiest neighborhoods become increasingly isolated, gated communities that effectively maintain their own separate identity (energy levels are higher here) but the allocation of energy (via wealth distribution and services) to those outside these gated communities begins its own steady decline.

Eventually, whole sections of a city go dark, lives become increasingly desperate, and those with the means to do so flee the city to get closer to centers of power (and hence of energy). This urban flight only exacerbates the situation, diminishing the tax base. Deserted areas increasingly go to seed, their resources stripped, their infrastructure devastated. Communication and transportation routes begin to fail due to vandalism and lack of maintenance. Educational levels drop, and the few areas that remain at a higher potential eventually become separate towns Urban centers, may see a brief uptick in their fortunes as suburbanites move closer, but unless they can find a new equilibrium from alternative energy sources (either directly or indirectly) then they become more and more townlike themselves.

At the higher level of organization, the provincial control begins to falter as energy is shunted to the political center. Unrest begins to rise, and at the edges of the province, where control is weakest, if a particular group of cities are able to gain a new source of energy, then they may in fact attempt to break away. This will usually result in armed conflict and civil war, as provincial authorities end up investing ever more energy into building up the military and police, yet the more that regions break away, the smaller the tax base and the less energy that the capital can allocate. Eventually, the province fragments into smaller abstractions, and with that loses political power.

Again, note that the same thing is true for any form of abstraction. Small companies merge or are acquired to provide more services, branch offices are added, which eventually requires regional management, which in time extends out to extra-provincial activities. This brings more money in but also increases its need to sustain that monetary flow. This means that most companies ultimately can get only so big before they become unwieldy, unable to respond rapidly to change in the markets, and consequently energy outflow exceeds energy inflow.

During tough times, companies close branch offices, spin off divisions, become increasingly specialized - in other words, they shift down to the previous level of abstraction, even while potentially seeding their future competitors. This also affects business partner and supply chain channels. The best and the brightest - the innovators in the company, who are in effect energy multipliers, realize their opportunities are limited, and leave to form their own ventures. In other words, as energy recedes, things become more primitive.

Note that this isn't necessarily an up and down process - technology can make new sources of energy viable, reversing a decline, structural changes in the economy (which can create temporary drawdowns) can be resolved, particularly draining activities (such as wars) can end, reducing demands upon energy and so forth. Moreover, once a given abstraction level is achieved, even if it is temporarily lost, the energy necessary to rebuild is usually less than it was initially (less need for expensive innovation).

On the flip side, environmental degradation can serve as friction towards the utilization of new energy. Environmental degradation in this case usually means that the cost of extracting energy into usable forms goes up, and tends to be a major drag upon any system. One way of thinking about this is that such degradation is systemic turbulence. The role of both technology and environmental degradation will be covered in the next column.

Future Proof: Technology is an Accelerant

2009-05-07T09:55:00.000-07:00

One of the most common mistakes that people make when looking to "predict" the future is to concentrate on the technology as being the future. In the 1950s, cars and television were the hot new technologies, and the future was filled with cars that could go three hundred miles an hour, would drive by themselves, would fly and go underwater..

Of course, the reality has turned out a little different - Certain cars have certainly achieved speeds in excess of three hundred miles (largely by turning them into rockets), recently a "flying car" - more like a small plane with foldable wings - debuted, and there have been several stabs at cars that could at least go on water, if not necessarily under it.

Yet even after sixty years, the typical car doesn't go that much faster than they did in 1955, still doesn't drive by itself, and is still best optimized for driving on paved roads rather than the airway. So what went wrong? To better understand this, its worth stepping back and looking at what technology is ... and isn't.

People have a historical view about technology. The first automobile was built by Karl Benz in 1885. The first radio was built by Guglielmo Marconi in 1901, the Wright Brothers built the first airplane in 1903, Edison created the light bulb in 1879. History is filled with these kinds of milestones, yet in every one of these cases, there had been innovations, ideas, patents, and unsuccessful (and in some cases successful) trials by many others. In most cases, the information that sparked the "successful" invention was culled from previous theoretical work or the innovation of key subcomponents. Benz's work, which involved the creation primarily of the engine, could only have been accomplished thanks to the efforts of Siegfried Marcus of Vienna, who created an engine to propel a handcart that made use of gasoline. It was in his critical work of building what would eventually become the alternator, however, that proved the key for many people moving forward.

Innovation builds on innovation, and while from the historical perspective it can appear that technology seems to have appeared in a moment of inspiration, this inspiration almost invariably involved improving what was "almost" right by someone else, or by combining two other inventions in novel ways.

One of the key points about such innovations however, is that there is a definite level of diminishing returns on energy investment. For instance, while there were huge numbers of refinements between Karl Benz first car in the 1880s and the cars of the 1980s, most of the critical changes occurred toward the beginning of that period. Why? Primarily because the number of cars on the road expanded dramatically between 1880 and 1950, and each new car offered a new opportunity for innovation. Yet once the basic principles were worked out, the changes became more subtle - eeking out better speed, more fuel efficiency, improving the user experience - and ultimately became little more than cosmetic touches (fins vs. no fins, grills vs. hoods, that sort of thing). The technology had matured.

Yet at the same time, the specific role of the car was also becoming more clearly defined. There were cars that were capable of driving very fast - in excess of 200 miles or more an hour - but with the exception of racetracks, such cars would have been seen as dangerous to anyone who didn't have twitch reflexes to handle it, and even then the practical limitations of city transportation grids made it unwise to try to go even 100 miles an hour in most places.

Similarly the concept of a car that could fly was mitigated by the very real issue of coordinating existing flying traffic, safety issues, noise issues and a host of other problems. If you could only take off and land such a car at an airport, why not simply purchase a small plane for approximately the same price, something that's far more well designed for use within the existing infrastructure.

Technological change thus always emerges only in context, and it is shaped by that context.

Surprisingly enough, the last decade has probably seen far more innovation in automobiles than the last fifty years prior. The reason for that is two-fold - first, computer miniaturization reached a stage where it was feasible to create localized processing at key points within the system, to better regulate fuel consumption, engine heat, electrical systems, and so forth, as well as to provide a better interface with the rest of the world (think about the innovation of the GPS system for cars). By becoming increasingly intertwined with the operation of the car, the computer systems give the car a type of internal intelligence, making it more systemic, less mechanical.

You're beginning to see this manifest in things like early warning systems that can notify a driver of potential hazards in the environment and can even slow or stop the car if it appears that the driver is not responding to these hazards. Cruise control systems are becoming considerably more sophisticated as well, better handling the sometimes seasickness induced surges that older systems ran into. Navigational systems and real time telemetry can notify drivers where congestion is and suggest alternate routes. In short, cars are becoming considerably more intelligent - as one pundit put it, the car is the ultimate mobile computer.

On the other hand, system shocks due to oil disruptions are forcing the development of electric and electric/hybrid vehicles. Ostensibly one of the major reasons for this is to reduce carbon emissions, but while this is a "feel good" reason, the reality of high oil prices, while no longer as dominant a theme, has made electric car investment feasible - especially in conjunction with the increased computerization of such vehicles.

Most technologies are developed in order to make processes more efficient. "Labor saving devices" has been a major selling point for technology since the late 18th century. When you pay a person to work, what you are doing is asking them to commit their energy (and time) to process a particular resource into a different form. You're paying for them to expend their energy and their expertise (their technology) to accomplish a given task. In a factory setting in the 1900s, by creating the assembly line, you could also optimize this process by having people that were more proficient in a given technology do that technology, thus reducing the overall time to accomplish the task, letting you do more of those tasks.

As advances in metallurgy, in electrical systems, and in machine tooling swept the factories, machines augmented this specialization; making it possible for one man to do in a day what two men could do earlier, then what three men and so forth. In doing so, this had two effects - it meant that with the same number of men, he could accomplish far more in the same period of time, or it meant that he could use fewer men but reduce labor costs. In most cases, he could do both, especially as technological improvements came fast and furious. Ultimately, this reached a point where the amount of manufacturing capacity exceeded the available market, and the economy slowed until either the population grew to accommodate the higher demand, the price dropped to make the product affordable (tightening margins), or wages increased (wage inflation) to create new markets.

Technological innovation slowed over the first half of the twentieth century as more investment resulted in more marginal returns. Manufacturing reached a metastable point - until the innovations of the 1970s and 1980s. Robotics is the process of applying machine intelligence to mechanical systems. A contemporary car or truck is a robot. A commercial jet is a robot. Each of these are robots because there are autonomous decision making components that determine certain aspects of a given subsystem's operations.

Whereas before, mechanical systems replaced either the speed or the power with which a given individual could apply to a task, robotic systems effectively replaced the judgement and expertise of that individual, could do the work repeatedly, with little downtime, and could alert managers whenever something beyond its ability occurred. This led, not surprisingly, to the complete downfall of manufacturing as a form of employment.

Not surprisingly, as computer systems become more sophisticated, and processing levels of abstraction continue to climb, this is also destroying the service and management sectors of employment - in short destroying the existing corporate infrastructure. Corporations are using computers as mechanisms to enable the shift to ever higher levels of abstraction, but in the process are also reducing their own energy inputs as the number of people who are unemployed continue to climb, thus reducing the available capital flow to those companies.

Yet there are interesting developments along the margins - as technology becomes more pervasive, it also becomes more accessible. As technology multipliers climb, a handful of people can replace a thousand - so companies shrink, become more specialized, more distributed, more amorphous, more virtual. The energy costs of producing a good or service decline, meaning that while more can use the technology to convert resources into a usable form, , the margins to be gained by doing so eventually reach zero.

What's this mean to the budding futurist? When looking at a technology, examine its potential impact upon society, both in isolation, and in combination with other technologies - technologies that reinforce one another will ultimately beat out ones that don't. Understand that technology follows the curve in which science this year becomes engineering next year and technical maintenance the year after that. Most technology will ultimately end up eliminating more jobs than it creates, but in the process is also reducing the barrier to entry for nearly all fields, meaning that corporations also are forced to become smaller, more distributed and more virtual.This particular meme will play out for the next fifteen to twenty years before we reach a point of stability, and even technology which dramatically increases energy input will now only reinforce these trends.

Future Proof: The Future Comes From the Edges

2009-05-07T09:54:00.000-07:00

In software architecture circles, there's a rather curious term in vogue called "future proofing". Typically, when you are putting together an architecture, especially an enterprise level architecture, one of the major goals is to insure that, once laid down, it won't be necessary to come back in ten years time, tear the whole thing out and start from scratch. In essence, you want to set up the architecture so that, no matter what the future brings, the application or system can adapt.

It's a laudable goal in theory, but in practice, it's seldom an achievable goal. Part of the reason for this is that while architecture tends to be a top down imposition of ideas and structure, the future is notoriously bottom up. It creeps in through the cracks, like weeds pushing up through concrete pavement ... before you know it, the concrete has been reduced to rubble, and the seemingly soft and fragile weeds and saplings have become a wilderness.

This is a part of the reason that many very good architects - whether of buildings or of software - tend to have a touch of Zen master in them. They do not see the future as the enemy, but rather as a wild and unknowable force that, nonetheless, can be redirected to work the least amount of damage (and perhaps even to improve upon the existing infrastructure) over time. What this means is not anticipating the future per se - that really is an exercise in futility - but in understanding the characteristics of change and to build with these in mind.

Such thinking applies just as readily to urban planners, managers, and decision makers as it does to architects - too many choose to "make their mark" upon the future without realizing that doing so is an exercise in hubris, and usually doesn't end well.

I am beginning a series of essays on "predicting" the future that looks not at trends or even critical technologies to watch but rather that look at "techniques" - where to look for innovation, how to understand the growth and decline of systems, understanding how to pull the relevant from the irrelevant and so forth. Fans of systems theory will recognize most of these, as in general anticipating the future usually involves understands how trends and energy flows interact with one another and how layers of abstraction can be modelled and consequently separated from one another without losing out too much of the behavioral glue that nested abstractions bring about.

Each article in this series will be labeled as Future Proofing. Click on the Future Proofing keyword to see a list of all articles in the series and check back frequently, more will be added on a daily basis.

The Future Comes From the Edges

Any given structure that you build by its very existence establishes boundaries. So long as those boundaries remain inviolate, the structure will stand. However, those boundaries are also under near continuous bombardment as the outside world interacts with these boundaries. For instance, every day, a house's walls (and more especially its roof) are attacked by insects, by mold, by water, pollen spores, and other agents that are quite fully capable of interaction with these boundaries. Water pools under roof tiles, slowly dissolving nails and glue, until eventually a heavy wind rips one or two of these away.

Once the integrity of the house has been breached, the invasion mounts. Ants and termites get in, mold develops within wood beams and along exposed paper, increasing the breach and reducing the integrity even more. Stressed beams pop as rusting nails and staples crumble, sheet-rock and drywall become powdery and fragile, even mortar for bricks will eventually disintegrate. This process doesn't happen quickly, but it does happen - most houses constructed today, when left unmaintained, will become unlivable within a couple of decades, and will disintegrate completely within a hundred years.

The same process occurs in society, though it usually happens much more quickly because the boundaries involved are far more ephemeral in nature. Governments arise in order to insure that collective actions of a specific group can take place with as little organizational impedance as possible. While it is possible that the group is involved consists of all people within a given geopolitical boundary, in practice, governments usually serve to mediate conflicts within the most powerful members of the society - landowners, for instance, or investors, with the decisions to support other that are not in that group being made because of the potential that these non-group members have to disrupt the society.

However, these non-members also represent agents of change, for good or for ill. If they don't have representation in the group but the group has some effect upon their lives, then they will agitate for ways to gain representation. They operate at the margins - protests, work stoppages, subverting the dominant media, "terrorist" actions, warfare, interbreeding. The societal structures will remain change so long as the integrity of those structures remain solid, but eventually, at some point, a chink will develop. The organization/government/consensus must either adapt, integrating these new members and/or ideas, or it will disintegrate if the invaders have a more efficient mode of operation (more force or a better ability to exploit the existing environment).

From a system architect standpoint, this means designing architectures so that natural channels exist. If formalized mechanisms exist for recognizing and incorporating change exist, this makes the process less disruptive in the long term. This is called flexibility. Design so that the system can adapt to stress, perhaps even utilize it. People are beginning to experiment with building hi-rise towers that not only bend in the presence of high winds or earthquakes, but that can actually extract energy from the building as it does so to use for other purposes. Many social networks actually get their value from the change within their channels.

From a future watch perspective such edge considerations should therefore be subject to especially close scrutiny. Most intermixing of ideas occur at the boundaries (think of such boundaries as the edge of turbulence, where the inside environment mixes with the outside. When two cultures meet, they have to resolve differences, define new interfaces between one another, try to work out which works best on either side. This is the basis for innovation. However, it is also the place where the greatest strife is occurring, and the biggest threats to the company, as illustrated above. Look at mashups, where two unrelated technologies are combined in novel ways. Look at mergers, where the two organizations are roughly of the same size (same abstraction level, as discussed in the next section). Look at organizations that are on the verge of going from one abstraction level to the next (up or down). Look in cross discipline studies, where people from different disciplines begin comparing notes. This is always a good place to start when trying to assess the future.

An Experiment Towards the Future

2009-05-07T09:53:00.001-07:00

Social networks are finicky things - to develop a community of interest, you need to have both a compelling topic and a reasonable platform on which to host that community. I've been working with Drupal for several years now, and while I thoroughly love Drupal for its extensibility, in order to build a community of any complexity, you end up having to make so many modifications and add so many modules that the system can become glacially slow.

I stumbled upon Ning recently, though I've been on Ning networks before without realizing it, and found that I could put together a serviceable site that offered almost every capability that I'd been looking for in about twenty minutes. It's also separately hosted, which solves a lot of the concerns that I've had with my own self-hosted site at http://www.xmltoday.org, especially given the high cost of hosting and my own rather precarious financial state at the moment. Given those concerns, and of course the option of playing with a new social media environment, I decided to set up a Metaphorical Web ning site.

I want to open this up to participation by anyone interested in the primary topics of the site. What are those, you ask? (You did, really, I heard you about to say exactly that! Really!!). Well, what I'm going to cover here moves beyond what I cover on xmltoday.org, which focuses purely on XML issues, though I'll be covering that too. However, with Metaphorical Web, I have a somewhat broader agenda.

Specifically, the following topics come to mind:

The Metaphorical Web Side This is devoted more to future topics and issues
- The New Publishing - neo-journalism, social networks, CMS's, DITA, DocBook, Drupal, Twitter, SVG, etc.
- The Architect - discussions about the profession of information architects, software design, programming, methodologies, and so forth.
- Macro-Economics and Systems Theory - Everything from relevant economic discussions to software agents to the Sims
- Climate Change and Green Tech - A look at everything from climate change related stories (pro and con) to solar energy to infrastructure issues.
- TomorrowVille - A change to play futurist, looking at trends and patterns that will shape our future.
The XML Side This is devoted to XML related topics in particular
- Open Standards - W3C, OASIS, IETF, ECMA, and just about anyone else.
- XRX Technology - XQuery, XSLT, XForms, RESTful Services, XProc and other Xy things
- XBRL and Open Data - includes NIEM (and NIEM IEDPs), GML, ePub and other governmental XML standards, as well as looking at the intersection of IT and government.
- The Semantic Web - RDF and RDFa, classification, taxonomy, OWL, SPARQL, and the way that we organize information.

It's a fairly daunting list, and is no doubt incomplete. My desire here is to pull together people that are interested in the trends of tomorrow, from the technologies to the social issues to how we actually survive on this planet. Feel free to join the group, to blog, to create new discussions, to post your events or your photos, to make yourself at home. Think of this as a debate salon, a place to discuss the world with others, and to put in your two cents worth.

I will be maintaining the XMLToday.org site as well, but will continue the trend that I've started with it and keep it strictly focused on XML-related news exclusively. This site I'm hoping instead to focus on building community. Enjoy, kick back, and stay for a while.

Kurt Cagle
Managing Editor
The Metaphorical Web
http://metaphoricalweb.ning.com

A Need to Blog

2009-03-13T22:07:00.000-07:00

I was counting the other day, and came to the realization that, over the last couple of years, I have ended up blogging in about ten different places. Yet despite that, I also made the rather disturbing observation that while I am regularly writing articles for O'Reilly (about fifteen to twenty a month, which, when you get right down to it, is an incredible amount of writing), for XMLToday.org (which is focused on XML issues), for DevX, for EMC/Documentum and elsewhere, what I didn't have was a place where I could just blog for myself.

That's when I remembered this blog. It's not a Drupal instance, just a simple Blogger app, but I'm coming to understand that perhaps this is not a bad thing - I need something that I can periodically just write to without having to get sucked up in the mechanics of blogging. I've learned to really like Drupal, but sometimes you need to just walk away from the code and concentrate on the message.

In the case of Metaphorical Web, that's precisely what I intend to do. This can be considered my kick back my feet and just write whatever I want to write blog. I might talk about code, or economics, or just how I'm feeling that day, and I make no promises that you're going to learn anything here, other than maybe the occasional odd rumination.

Ironically enough, as a professional writer, I'm beginning to realize the real value of such a blog. One of the real challenges that you face as a pro writer or journalist is that overall you are always writing for others. You have to consider every thing you say in terms of the editorial message of the site, the audience involved, the needs to promote this or that conference or book or product. Especially in the age of tightening belts, you also have to make sure that what you're writing has value with every article.

Yet here's a secret for you - no writer can be "on" all the time. That's not to say that a writer can't consistently write articles or stories or what not. That part of writing is a lot like what a long distance runner goes through, just building up your endurance so that you can crank out an article every day or two, two to three stories a week. Yet even the best writer (just like that endurance runner) will have off days, will produce a piece or two of forgettable crap for every good story, and the really incredible pieces will be balanced by the occasional article that frankly should have been left exposed on a rock to do a humane death.

What's more, that writer periodically needs to write to no one in particular just to get the frustrations off his chest, to say the things he daren't say when writing for pay, to have, well, a journal. In its own way, writing is an addiction. The more you write, the more that the pathways in your brain see writing as the mechanism for expressing itself, and as a consequence you find that its often difficult to use the medium of speech because your brain is wanting to shape things into words on a page or pixels on a screen.

I've been at a number of author readings and been on panels at everything from science fiction conventions to international conferences with other writers, and one of the things that I've noted is that most writers tend to be naturally taciturn and withdrawn, though certainly capable of speaking eloquently when called upon to do so. At first, my thought was that the profession tended to draw introverts to it, in great part because introverts tend to live more in their head than via interactions with others.

Yet over the years, I've also begun to see that the process of writing reinforces this introversion, makes it stronger. Writers are aloof not because they believe themselves above other people, but rather because writing crowds out speaking and other human interaction, and as a consequence, the skills for dealing with other people become rusty and frequently mechanical, as writers find themselves having to remember how to do these things.

This isn't unique to writers, of course. Creators in most endeavors tend towards this way of thinking. Artists use different pathways, and as a consequence, the way that they view life differs somewhat, but at the same time, most artists become artists because art is the mechanism that they use for communication. Musicians, good musicians, similarly become wrapped around their music - and find that the channels over which they communicate dominate their interactions.

Once consequence of this is that there is a world of difference between communicating with someone in your modality of expression, and someone who simply "appreciates" that modality. I can communicate with my eldest daughter along a channel that others can't, because my daughter has the artist "genes" - the combination of talent and the overriding compulsion to draw and paint - that I have (I was very much the same as she was at her age -I was always drawing). I can communicate with my wife along the writing channel because she is a writer herself, which provides a shared set of referents or symbols (and experiences) that would be lost among non-writers.

Yet the irony is that I'm a lousy critic, which is I suspect also true for most creatives. An artistic critic is someone who looks for meanings and interpretations in a "work of art", as if there was actual intent there to provide such symbolism. I remember in high school one time, an elderly English teacher gave us a test which included the question "Why did Shakespeare write Romeo and Juliet?". My response, for which I received a rather stern lecture, was "because he had to pay the rent."

Writers write for public consumption not to load their works with deep symbols and meanings, but because they've discovered that the voice in their head that demands expression can be occasionally harnessed to pay the bills. The voice, the compulsion, to write, is still there, of course. They would write regardless, just as the artist will draw or the musician play, even if there was no audience. It's their language.

Yet this harnessing the writing impulse to pay the bills has a darker side as well. When everything has a deadline, what this means is that the temptation will be strong to do nothing but write for public consumption, even if what you're writing holds no great interest to you. The stories that you used to write get left undone because the clock is ticking and you have to get three articles on the latest news du jour written by the end of the day, you have to get the next chapter to the manual completed by mid-next week, the interview you did has to be transcribed and re-edited before the next conference. All are important, all pay the bills, but the music, the creativity that you once enjoyed as part of the writing process gets lost - your ability to express yourself gets lost in the requirement of expressing the needs of others.

There's another corrosive aspect of commercialization: you began seeing other writers not as people with whom you have a deeper understanding based upon your art, but as competitors for the same audience, the same revenues, the same ~~lucrative~~ barely minimal writing contracts. You don't dare do any but your best work because if you fail, you're toast. Unfortunately, this typically means that you also don't experiment or take risks, both critical for improving your craft, because the perceived cost of failure becomes too high.

I'm not sure there's necessarily a morale here, though I do have a suggestion to writers, (though it applies just as readily to artists, musicians and other creatives) from a writer who is rediscovering this for himself:

Always leave a certain space for yourself; block out a chunk of time in the week that is devoted to your play time, your experimentation time. If you write news for a living, use this time to work on a novel without the expectation that it will ever see print. If you're a technical writer, spend some time composing poetry, playing with the way that words sound and feel. If you're working on a book, take some time to write an essay about the coolness of the spring morning, or a random character portrait of someone you see in a coffeeshop.

Minimize the interruptions around you during this time, and do not, regardless of what else you do, use this time for paid work. This time is the equivalent of working out at the gym (something else you should do, for what its worth) in that it is not time that is owned by someone else but is necessary for your own sanity. This time takes precedence over everything - even if you have a critical deadline, take this time for yourself, because there will always be critical deadlines, and just as working out physically can often help relieve a lot of the physical stress that you face and make it easier to get things done, so too is this creative exercise time necessary to cut down on the mental stress that you face.

My suspicion, when it's all said and done, is that when I finally die, it will be the work I do during this time, rather than the marketing document for client X due next week, that will define me as a writer. Creativity is rare not because people aren't creative ... most people have a streak of creativity in them ... it's rare because people become so obsessed with the need to do their "work" that they fail to take the time necessary to actually be creative for its own sake, rather than in the service of some larger goal.

Take the time, it'll make you a better writer.

CDF: The common format you've never heard of

2007-11-29T22:21:00.000-08:00

Quick! Do you use the Compound Document Format?! You, know, CDF … surely you use CDF, right?

Chances are pretty good that you have no idea about what I’m talking about. Everyone knows Microsoft’s word document format and Adobe’s PDF, chances are pretty good that if you’re reading this on XML.com you’ve heard of ODF and OOXML, especially after the fairly rancorous discussions about ISO status for these two formats. Yet CDF, hmmmm … that’s a rough one. Didn’t it belong to Corel, once upon a time?

Okay, now, how about this one … do you work with (or even just read) XHTML? Probably, if you’re involved in XML work, your HTML conforms to a great degree to the formal XHTML standard. Good enough. How about CSS 2.1? Sure, who doesn’t. Okay, here’s a biggy -how about XMLHttpRequest? You do AJAX work? Good for you. XForms - well, that one’s a little less prominent, but yeah, it’s beginning to appear. SVG? Hmmm … again, kind of touch and go, but even after a few hard years SVG’s by no means dead yet. The occasional XSLT - Google’s doing some nice work in getting their Javascript based AJAXSLT up and running, which means that those few browsers that don’t support XSLT (and XPath) natively should be able to support it via an AJAX layer. Oh, and perhaps through in XSL-FO for good measure, as it continues its quiet but relentless march into becoming a mainstream format.

So, do you work with CDF? You betcha. The Compound Document Format was set up as a way of tying together at a minimum all of those technologies described above into a single cohesive whole. Put another way, it’s a fancy way of describing the core suite of W3C document standards into a cohesive whole, although it does place some fairly minor requirements on usage in order to provide a consistent standard.

CDF was in the news recently with the implosion of the Open Document Foundation, originally established to endorse ODF, though in its death throes it briefly highlighted the CDF format as perhaps a better format for documents than either OOXML or ODF. This is admittedly one of those areas where it may be justified in looking at XHTML especially and going “huh”? How can that be a full document format - it’s used for web pages, after all - you wouldn’t want to use it to mark up a full book, would you?

Document formats are a lot like religions - people are ready to defend them to the death if need be, yet at the same time it becomes easy to dismiss certain religions that don’t even seem to be religions at all (such as my personal favorite, the rather philosophical Tao). Could you mock up a brochure in XHTML and CSS? Actually, it turns out that its surprisingly easy to do just that - especially if you throw a little SVG into the mix and allow the possibility of embedding XHTML within SVG (for all those odd little bits of rotation and other special effects).

How about linking between blocks of physically disconnected content? That’s what hyperlinks are for, after all. CSS gives you at least a dozen different unit references, lets you control borders, margins, padding, kerning, image placement, color management, and so forth. There are pieces I wish could work better - I’d love to have the capability of defining a color in SVG then referencing it within my XHTML document via CSS; full support of CDF will likely allow that. If you jump just a little beyond the current CSS 2.1 spec you even have some fairly decent support for columns of text, not to mention tabular structures and even VoiceML support. Moreover, consider XInclude support, something that really, really needs to be a part of every browser implementation (though its fairly trivial to write AJAX classes that let you create similar bindings).

These are all disparate documents, not a single “document” akin to Word or the ODF Writer format. However, even that’s not quite true. The effort of the CDF working group has been to essentially standardize on the way that web documents can be “bound together” into what appears to be a cohesive “whole”.

Part of this is accomplished through the use of a standard called the Web Integration Compound Document (or WICD). This standard provides a number of both new features and clarifications:

an extended definition for the HTML object,
the integration of Scalable Vector Graphics (SVG) with XHTML and other documents,
establishes the and elements,
establishes the characteristics of hyperlinks and focus across the boundaries of namespaces,
defines the nature of focus across multiple embedded documents,
introduces support for animation and synchronization, building on the older SMIL and SVG standards, and
includes support for SVG fonts.

Two additional WICD standards - WICD Mobile and WICD Full - extend these with a few other features, most notably ECMAScript 3rd Edition (4th edition is currently still under development), the XMLHttpRequest to support most AJAX applications, XHTML 1.1 and the CSS 2.1 specification. Additionally, CDF working group also define two “modes” of operation - Compound Documents by Reference (CDR), in which internal content to the document is provided via reference links, and Compound Documents by Inclusion (CDI) in which internal content is rendered into the containing document directly in a different namespace.

Admittedly, I think it can be argued that the W3C effort could do with a few less acronyms (and perhaps a bit more of a PR effort) but overall, what is happening with CDF is a very critical - and welcome - evolution of HTML. The web needs to be more than just static web pages - this has been demonstrated by the continuing strength of the Web 2.0 meme, that documents should be able to talk back to the server and interact with them at higher levels than simple links for refreshing content. Commercial vendors would love to fill that space, and by doing so regain for themselves control over the underlying technologies that make up the web. Yet the message coming about with CDF is both simple and profound: the web grew up on the strength of seemingly simple technologies - HTML, CSS, JavaScript. These technologies are still around, they’re just maturing as people come up with new ideas about what the web can do, the pieces slowly unfolding as we progress further into the realm of the Internet as operating system.

Already, much of CDR has been implemented in the more sophisticatedly forward browsers. Opera 9.5 has a rather extensive support for most of CDF core and Firefox 3.0 is moving in that direction (though the biggest area of weakness is in SVG animation support). JustSystems, a company that has a huge presence in Japan but is only now just (sorry) beginning to make an impact outside of that country, has been working towards a CDF platform for a number of years, and has one of the more expressive (and impressive) displays of how compound documents COULD work (I will no doubt by writing more about JustSystems over time). Both Sony and Nokia have WICD implementations working (as prototypes) on certain of their mobile phone chipsets, with similar announcements from Abbra Vidualize and BitFlash, both makers of mobile graphical chipsets, while Sun is partnering with OpenWave to create a formal WICD implementation in line with JSR 290: JavaTM Language & XML User Interface Markup Integration.

This is no guarantee, of course, that WICD will catch on, or even become a household name (even in geek households), but unlike other technology most of what’s involved already exists, and has already been proven every day millions of times a day. Already, the number of HTML documents that exist dwarf (by a few orders of magnitude) the total number of Microsoft Word documents. As editing increasingly moves onto the web, its safe to say that the document of choice will be neither ODF nor OOXML, both of which gain their power on the basis of supporting legacy word processing systems. Instead, what seems to be emerging from the W3C is something that is not an office suite because it didn’t evolve from one, but that nonetheless is capable of most if not all of the same functions that office suite documents pose.

Moreover, if you come to realize that XHTML by itself is NOT the only targeted compound document (indeed, the specification is rather clear that it is intended to be extended to other formats, from XSL-FO to DocBook to the aforemention ODF or OOXML) then what becomes clear is that the ability to integrate content itself can be standardized, and like many other W3C formats, this move to the metadata level may very well provide the necessary differential to make the technology succeed in even the most competitive of milieus.

So, don’t worry - if you’re not using CDF yet … you will be.

Kurt Cagle is an author and chief architect of Metaphorical Web (http://www.metaphoricalweb.org). He lives in Victoria, BC, where he is making it his mission to visit every Starbucks on Vancouver Island.

Will Wall Street Tremors Bring Another IT Recession?

2007-11-26T20:40:00.000-08:00

Greetings, one and all. I'm bowing to a reality here that Google, in its oh so infinite search engine wisdom, is still finding this particular website more often than it is my Drupal (www.metaphoricalweb.org) site. For the moment, I shall maintain both sites, and see if perhaps something comes of it one way or t'other.

Lately, I find that far more of my reading has centered on economic news rather than upon technical, though to be honest the intricacies of many of the "investment credit vehicles" that have been foisted off on the public frankly put the Obfuscation C Coding Contest winners to shame. In most cases, when a financial fraud is perpetrated, one of the hallmarks that jurors use as evidence is the degree to which illicit cash flows are threaded above, below, and through more mundane transactions. The intent of fraud is simple - make people believe that they are financially safe when in fact their life savings are being purloined.

What your seeing each day in the big financial papers is disclosures of "write-offs" of hundreds of billions of dollars of money by banks, mortgage institutions, hedge funds and similar financial players, largely because these organizations had become engaged in the practice of ... well, fraud.

Mortgage brokers initiated liar loans with people who clearly had no means of paying back even the teaser rates, then got those loans off their books and into "structured investment vehicles" that sliced and diced these shaky loans and, with the assistance of ratings organizations turned these loser portfolios into alchemical gold by building highly leveraged hedges against them.

The banks that purchases these would then get them off their books by selling them to investors, who generally had no idea that the "risk" that they were buying was not safe, conservative debt but rather investments into millions of loans that had about 50/50 chance each of going into default.

Not surprisingly, while many of the mortgage companies and investment banks that perpetrated this nonsense are either in bankruptcy or in financial distress, most of the CEOs who should have put on the brakes early have reaped hundreds of millions of dollars and golden parachutes that will likely see them well into their retirement years.

Of course, the same practices that made this possible also have laid the same insidious seeds within most other credit companies, including credit card issuers and corporate paper investors. As the domino of subprime mortgages fall, its already begun to press auto loan issuers, and there are indications that the commercial credit market is also seizing up as dubious corporate investing also is beginning to look ominously diseased.

Now, I am not an economist - I'm a programmer. While I find this a fascinating exercise to watch in system theory, ultimately the questions that I'm having to ask have more to do with the simple question of how it will affect me, my clients, and the IT industry in general.

Five years ago, the Tech Recession hit. If you were in financial services, you might be forgiven in saying "recession? We're in a recession?!" The effects on most other sectors was comparatively light, though it was devastating if you happened to be in IT. Fully 40% of IT jobs were shed at the height of the recession, programmers who had been buying Starbucks coffee and making six figure incomes suddenly found that the paper options they'd been paid in were worthless, they'd given the best, most productive years of their lives and their most innovative works for comparative peanuts, and they were living in their parent's spare bedroom.

Not surprisingly even now, five years later, a lot of programmers have become skittish about accepting stock options and they have become considerably more parsimonious about how they spend their money (and what jobs they accept in the first place) - and there are comparatively far fewer people entering into the IT field. They've also become very good at watching economic trends, and most programmers that I know who went through that period are getting rather twitchy about now.

Right now, the ill winds of recession are blowing most heavily through the housing sector, especially in the US. One of the things to consider about most recessions is that it is actually quite rare for a recession to affect the entire economy. Instead, in most cyclic recessions, what you have is a situation where supply has temporarily exceeded demand, and the excesses of that supply need to be worked off.

The IT recession of 2002-2004 was a fairly classical one - there was too much investment in IT, both in terms of jobs and in terms of technology, and it took about eighteen months and some fairly severe slashing before demand could catch up with supply. By 2004, even though tech was still technically (sorry) in a recession, there were already signs that things were beginning to recover, and the market very quickly went from too few jobs for the number of workers to too many - to the extent that finding good, qualified people is becoming quite challenging.

The housing sector is going through its own recession, but unfortunately it will take more than eighteen months for demand to catch up with supply; there are a number of reasons for this, from speculators flipping houses who are now left with too many properties and no buyers, to houses going into foreclosure due to onerous adjustable mortgage rate resets, to a longer term demographic switch away from large, costly-to-heat houses in the face of higher energy costs and shrinking families.

Yet by itself, the recession that results from this will also would be at least somewhat self-contained, if an oversupply of houses was in fact the only real problem. Unfortunately, what is now emerging is that this is the domino falling that seems to be catching other dominoes with it. Risk is being repriced much higher, because the same lending practices that made it possible to get a loan for a half-million dollar house with an income of less than $20,000 was also at work in the hedge-fund space, the commercial paper space (which affects corporate buy-outs, business expansion), credit card paper, student loans, re-insurers ... indeed, just about everyone out there is suddenly discovering that the foundations that they were building on was made not of concrete, but quicksand, which has many of the same properties but a rather disturbing tendency to be lethal.

This is the real recession, and the last time that something even vaguely similar happened, Herbert Hoover was president. Credit, which has replaced cash in just about every transaction that occurs today, is rapidly disappearing. Indeed, it is worth it to understand that credit is itself simply a measure of risk. "I will let you take out a loan because I believe you to be a trustworthy individual," the banker should say, "you pay back your loans on time and reliabily, you have open accounting, you have shown good judgment in how you spend your money. "

The problem comes with the next part - "Even the most reliable person occasionally runs into problems. As a banker, I should be compensated for the small chance that you will be unable to pay your loan under the terms you and I set out. That compensation is the interest on the loan."

So what's a reliable rate of interest? This is where things get tricky. The more risky the investment, the higher the rate of return should be, so the higher the rate of interest, and the harder it becomes to qualify for a loan. The problem of course is that the banks make money on that interest, and in good economic times it becomes easy to fudge the credit-worthiness of an individual or two, if higher returns is the desired goal. Of course, once risk becomes high enough, it becomes far more profitable to initiate the transaction (be it mortgage, credit card application, or small business loan) then sell off that risk to someone else. That proved SO profitable that the process got repeated many times, with the same questionable risks getting packaged and repackaged to the point where there was no real way of determining where the risks were - or, more to the point, what they were.

Now, that whole edifice is going through a slow motion crash, and despite everything you hear in the Wall Street Journal or CNN/Money, it is going to continue to crash into other sectors. Right now, it really hasn't fully impacted the consumer (though there's another factor working there) but it will. If no one has any idea what the real value of risk is anymore, no one will initiate loans. Companies are already finding it increasingly difficult to get funding that used to be routine. Companies can't expand beyond the reserves that they already have on hand, and already accountants at many companies are beginning to question just how stable those reserves are. More than one company has been forced to delay even paying their employees or their accounts because their banks have frozen all accounts.

In the late 1930s, the FDIC was established in order to avoid a credit crunch like the one that wiped out the economy in 1929. It stipulated that banks always maintain a fixed percentage of assets (originally 10% of total capital lended) in order to insure that in the event of a bank run, there was enough operating capital to keep the bank afloat until help could be arranged. Additionally, many banks were required to honor the first $100,000 of deposited funds. The problem is that lax regulatory oversight has effectively eviscerated most of the FDIC's mandate, to the extent that it is unlikely that most banks could in fact meet 1% of total capital extended if they had to.

Of course, in order to look like they were in fact complying with these strictures, many financial institutions put much of their assets off the books into secondary corporations that were not then subject to the limitations, meaning that the actual debt that a financial institution had was generally far greater than what they showed on the books. So long as no one looked too closely (and when the money was great enough people would not look too closely) this meant that banks could make incredible returns by lending money that, for all intents and purposes, didn't really exist.

The problem is a Schroedinger Cat crisis. In quantum mechanics, there is a classical thought experiment in which a single atom of uranium is placed in a special trap that would, when the atom decayed, release a toxic gas that would kill a cat locked in a box. Quantum theory states that what is at work here is a wave function, and as such, the cat could simultaneously be both alive and dead, and the only way that you could actually tell (well, besides the obvious yowling of a pissed boxed cat) was to open up the lid. Once you had opened the lid, then you had collapsed the wave function, and there could be only one solution.

The banking industry has been build on the same wave functions, and so long as no one looked in the box, then there could be an infinite amount of money there. Once you open the box, unfortunately, reality steps in, and the reality is considerably more disappointing than anyone could have hoped.

So what happens this time around to IT. The credit crunch will hit in a number of ways:

No money available for expansion of companies, beyond existing funds. Too many companies have been "living beyond their means", using their existing reserves as collateral against loans. As those loans begin to dry up, they are forced into using those reserves, which often may be tied up in existing investments and thus are less than completely liquid. When you are forced to sell to meet costs, you often get a far less than ideal value for your assets, especially when everyone else is doing the same thing.
Illiquid third party investment. VCs and other investment companies will have more demand on their moneys from more traditional sources, which may be good for the VCs (in the short term) but it also dries up much of the money that IT startups in particular count upon. Moreover, many of those VCs represent funds that may also be fairly heavily hedged, and so the same repricing of risk that is going on in the commercial sector will make it harder for them to pull together the funds necessary to complete deals.
Strapped consumer spending. Don't pay too much attention to all of the sighs of relief after Black Friday sales. People are strapped for cash, and they are as attracted to a bargain as anyone. The problem will come soon enough, as people can no longer take out much money via a second mortgage, will be facing stagnant wages and unemployment in December (one of the highest months for force restructuring) and will find it difficult to get credit extensions on their credit cards. I expect that consumer spending will plummet in the next couple of months. Sales from Walmart are done from last year, but so are the sales from high end luxury brands. This will mean that spending on consumer software and hardware will also in general be declining pretty dramatically in the next few months, which directly impacts people in IT.
Declining dollar. As the economy becomes more distressed, calls will be made increasingly to cut interest rates. The problem with that is that the dollar's value is declining relative to other currencies as rates go down, making American goods and services cheaper ... but also making American companies cheaper. This will reduce the attractiveness of outsourcing (a definite plus for developers, who've often had to compete with foreign developers) but will make it more likely that companies in other countries may start outsourcing to American IT workers. On the other hand, this will likely only happen once American wages become competitive with those in countries like China or India, which means that programmer's standards of living will likely be dropping for some time before you see a massive influx of capital, and that capital will likely mean that the "parent company" of your company is not even in your hemisphere.

Note that the carry trade in currencies will likely intensify as people move money out of both hard assets (short term, commodities will likely stall somewhat, though the long term outlook is still somewhat bullish) and financial assets (nearly all classes, long term) but seek even some returns. The downside is that this will likely lead to greater instability in the currency markets, with the possibility that one or more countries could "crash" - the US being at a very real risk of this.
Energy costs. IT has generally been a net consumer of power, though that's changing. In many ways, IT has been working towards applying technology to reduce their energy footprint for financial reasons rather than ecological ones, but of all departments in a given company, IT is often the "greenest". On the other hand, a significant amount of energy costs are due to a declining dollar (along with considerations for peak oil and other geopolitical issues) which means that the relative cost of oil (and hence gas) will affect the prices of goods that need to be transported (nearly everything at this stage) and hence will mean that computer infrastructure (which has generally been declining at a rate consistent with Moore's Law) will likely start getting more expensive again relatively speaking (though it may end up spurring additional investment into local hardware production infrastructure such as chip fabrication plants).

The cost will also accelerate the decade long trend of distributing the corporations into islands of loosely connected individuals who may actually work half a world away from where they live. If your workers are spending four hours a day on the road and are forced to pay an increasing premium for the privilege, they will go elsewhere where they can telecommute instead.
No (immediate) bloodbath in IT. Corporations are going to be considerably more loathe to trim their IT staff, in part because that staff has only begun to recover from the last bloodletting, and in part because in many cases existing infrastructure has been pushed for about as long as it can be, so the imperative for improving core infrastructure will likely remain strong even in the face of slowing sales. On the other hand, its unlikely that you will see a lot of companies do anything beyond low cost pilot projects for R&D, especially after the first quarter of next year. As the recession spreads to other industries, I think that IT could be vulnerable on an industry by industry basis, but some of that may be mitigated by investment from foreign investors seeking firesale prices on companies.
Shakeout of vulnerable companies. There are a fair number of mid-sized IT companies in certain sectors that managed to survive the Tech Winter, but that are increasingly becoming vulnerable. Even a year ago, these companies would likely have been bought up by larger or more successful companies, but as the available moneys for buyouts dry up, what will more likely happen is that the less successful of these will be forced into bankruptcy instead, without the "Get Out of Jail" card that seems to be the privilege of companies in other sectors.

The irony is that a lot of new startups may actually end up surprising relatively intact. While there is some fluff in the startup space, it never reached the level of 1999, and a lot of the companies that started in the last five to six years are generally very lean (and fairly heavily open source-based, from all indications) and for the most part are considerably more pragmatic about the market. On the other hand, there will be relatively few IPOs for the next several years out of this sector (though this has been a trend since around 2000).

In the face of all of this, what can the astute developer do? Having gone through this a few times now, I've found the following strategies can help.

Save money when you can, and reduce spending on extravagances. If you currently have any debts (especially credit card debt) pay it off and get out of those cards, highest interest first. Debt ties you down and reduces your cash-flow, which will be threatened at the best of times.
Stay as liquid as possible, and if possible diversify your funds out of US dollars, at least for the next year or so. If you can land one or more clients that are out of the US, even better, though try to negotiate in funds out of US dollars if at all possible. It's likely that 2008 won't be quite the rout for the dollar that 2007 has turned out to be, but it is also likely that you still will likely turn out better in foreign funds, even with conversion rate charges.
On a similar note, for the short term, if you are invested in equities, resource and energy stocks will likely be the most solid for the foreseeable future, though you are looking there less for significant gains but to minimize losses.
Diversify your client list as much as possible - and be fairly ruthless in negotiating. They will be. Assume the likelihood of a default by one or more clients is at least a possibility, and plan accordingly ... its possible that all of your clients may go belly-up at once, but in most instance, if you have three or four clients, you should be able to maintain steady work. On the other hand, work hard to deliver as promised ... the ones that do survive generally are ones you want to keep.
If you do find yourself between clients, spend that time getting caught up on skills that you didn't have the chance to work on while you were too busy with clients. Unless you are fairly new to the programming field, you're best served by increasing the depth of your skill rather than the breadth, unless the shift to the new field is part of a larger move into that technology as a base. Be thinking a year ahead in terms of where tech will be.
In boom-times, programmers generally find recruiters to be pests. In bust-times, recruiters find programmers to be pests. Find a few good recruiters at various companies and get to know them, even refer people to them when you're offered a job you can't or do not want to take. They'll be more inclined to use you when times get tough.
Recessions generally happen for reasons that you have little or no control over, but it is easy to become stressed, bitter, depressed or angry during those times, often making it harder for you to get motivated to work or look for work if there isn't any. It's worth remembering that programmers in general usually tend to have very desirable skill sets. If you have the time, join and contribute to open source projects - not only will you keep your skills up to date and work on interesting (and occasionally even useful) projects, but you stand a better chance of making contacts, developing your interpersonal network and gaining early proficiency in new technology by doing so. It will also let you combat the tendency to want to crawl up in a ball and whimper all day. Indeed, staying connected in general is a good idea.
Keep alert and informed. In general, diversify your news sources (especially economic and political) and verify what's going on with one from another. Assume that no one knows the exact timing of when the "recession" or "recovery" first hits until after it has happened.

This was not meant to be a doom and gloom post. The coming recession is much like a Class 5 hurricane - it's coming, it will almost leave quite a trail of damage in its wake, but it will also have cleared the air enough for some major reforms to get instituted, and life will go on.

-- Kurt Cagle

Move to Metaphorical Web.org

2007-09-23T14:42:00.000-07:00

Trying to stay current on the web can prove a challenge for anyone, but if you publish a blog, you're likely to find making sure all the links that you have out there are in fact pointing to the right place one of the biggest challenges. I've recently moved Metaphorical Web to a new Drupal-based site:

www.metaphoricalweb.org

both because I needed the ability to customize the site to display code samples and the ability to set up and maintain web feeds, external links and a forum that I couldn't get from the Blogger environment. I hope that you join me there.

-- Kurt Cagle

Migrating Metaphorical Web

2004-12-22T17:09:00.000-08:00

I have started using a new blog software provide for The Metaphoric Web, incorporating it under M. David Peterson's UnderstandingX*L sites at http://www.understandingXML.com. Please join me there. I will be redirecting the Metaphorial Web URL (http://www.metaphoricalweb.com) to that address by the first of the year.

-- Kurt Cagle

Take 2: Chaos and XML

2004-12-21T21:53:00.000-08:00

When I wrote the last post, I didn't realize at the time that I was running about a 102 F temperature, and would end up spending the next three days in bed with alternating bouts of chills and sweating, talking in my sleep - it was not one of the more pleasant weekends I've ever spent. While the underlying concept of measuring complexity is sound, I erred on the definition of entropy. Entropy is not a measure of the number of states in a given system configuration, but rather a measure of the change in the number of states in a configuration over time - in a non-self organizing system, the potential number of states increases, consequently becoming more disorganized (or more properly, converting into heat).

The concept describing the log of the number of states itself is the multiplicity of a system. Multiplicity isn't of course the only measure of complexity, but its a pretty useful one. Think of the traditional model of chaos - the conversion of laminar flow to turbulance. Typically there the first state is simple laminar flow - each water molecule flows in a straight line. As the flow moves faster, interactions with the surrounding media become more prominant, causing first a split into two streams, then four, then eight, until eventually there are thousands of such substreams, and the water becomes turbulant (this is the typical model described by Lorentz equations, by the way).

I have to wonder, though, if in fact there is some underlying connectivity between language (whether human based or artificial, as in the case of XML) and chaos theory, by this same modelling. Specifically, as the multiplicity of a document schema increases, so does the turbulence induced by that schema. I'm not sure what that the interpretation of that turbulence might be, though informational noise would be an obvious candidate. A simple schema is unambiguous - the specific meaning of a given aspect (element content or attribute value) is clearly defined, within its own constraints. However, as the number of aspects increase, their potential for coupling increases, for two properties to both be dependent upon a third (potentially undeclared) property, or two elements may in fact describe the same property in different and potentially conflicting, ways. Ambiguity is a form of informational noise.

Keeping this short (for me) tonight. I will be migrating in the near future to a new website, though it should still be accessible via http://www.metaphoricalweb.com. Until next time ...

XML and Entropy

2004-12-19T22:40:00.000-08:00

Lately, I've been spending a lot of time reading outside my usual diet of programming books,in great part because I find that inspirations often strike when you can think about different endeavors and how problems were solved in those. One particular book that has set me to thinking has been Minds, Machines, and the Multiverse: The Quest for the Quantum Computer by Julian Brown, an intriguing discussion both of quantum computers and the multiverse interpretation of quantum probabilities.

In one section, Brown discusses the role of thermodynamics in information theory, and more specifically, the role of entropy. Entropy is one of those concepts that has gained, over the years, and almost mystical aura about it, the basis for all Murphy's Laws, but in point of fact it is actually a pretty simple concept to understand - and it has definite applications to one of the central problems that I see with how business people utilize XML technologies.

Entropy, in its information theory form, is a measure of the total number of states that a given system can be in. For instance, think about two bytes ... this can hold up to 65,536 possible states. Typically entropy is measured as the logarithm (and in the case of information sets, the logarithm base 2, or log₂) so that the total entropy of that system of bits would be log₂(65,536) or 16, which is, not coincidentally, the number of bits in two bytes.

One of the challenges faced by IT professionals working with XML is trying to figure out which tools work best for the scope of XML you're going to be working with. DOM or SAX manipulation typically does not handle complex XML well, XQuery is perhaps better for slightly more complex XML but lacks the recursive templating structure that works best for documents. The difficulty comes in determing at what point an XML resource shifts from one area of complexity to another.

Every XML document has a schema that describes the structure. If you make a few basic assumptions, you can in fact get an idea about the number of states that the schema allows:

multiplicities of an identical structure count only once if the upper limit of such multiplicities is "unbounded",
PCDATA data is immaterial, whether as text or as attributes. However, an attribute with multiple NMTokens will be treated as having one state for each enumeration.
alternatives within the schema will each be treated as separate trees for determining the total count of states.
If a given element can contain another element of that same name, then the count stops at that second element. This avoids infinite recursion.

By this measure, a schema with no variability would have an entropy of log₂(1) = 0, a schema with one element of variability would have an entropy of log₂(2) = 1, two elements of variability would be log₂(3) = 1.585, and so forth. By this measure, most business documents (invoices and so forth) would likely have entropies in the neighborhood of 0 to 4, XML processes (such as an XSLT transformation) might be in the neighborhood of 10-12, and literary documents might have entropies in the neighborhood of 15-20. Keep in mind that these are logarithmic values - an entropy of 20 would correspond to 2 to the 20th states, or roughly 1,000,000 possible schema instances.

Entropy is important because it can better clarify the domain at which it is best to work with a given document. XQuery I think provides a good case in point here. XQuery supports XPath, and so it has some of the advantages that XSLT has, but it's not really all that useful for dealing with documents -- converting a DocBook document into WordML or vice versa would be impossible in XQuery, but for many business schemas with comparatively low entropies, XSLT is definitely overkill.

Lately, its become somewhat fashionable, especially among the business set, to deprecate XSLT in favor of XQuery for enterprise level applications. I see nothing wrong with this. XSLT is not always easy to work with, requires a different way of looking at a set of problems, and is probably not worth the effort or overhead for many types of transformations. That does not mean that XSLT does not have a very important place in the ecosystem, and its my own personal opinion that thinking that you have to choose either/or will limit you when you do have to deal with high entropy documents, as is the case with most document management applications.

The Business Case for XSLT 2.0

2004-12-16T21:37:00.000-08:00

by Kurt Cagle

In my previous posting (Imports and Tunnelling in XSLT2) I started down a path that I've been planning on covering for a while: presenting a solid business case for migrating to XSLT2. When I first encountered XSLT, after an initial period of attempting to understand the paradigm, I found myself both impressed and disappointed. XSLT is an often underrated technology, in great part because it doesn't fit cleanly into the Algol-based model that is most commonly used today (C, C++, C#, Java, etc.).

I consider XSLT something of a jujitsu language - it is most effective when used sparingly, letting the XML itself do the heavy lifting with the XSLT providing just enough of the pivotal support to do incredible things. That's why it has quietly become the silent partner on any number of different platforms as XML becomes more pervasive on those platforms. It is used within any number of Java and .NET applications, just another piece of the infrastructure, though one that does a disproportionate amount of the real work in applications that are increasingly driven by dynamic GUIs and web services.

Yet what disappointed me about XSLT, especially the more that I had a chance to play with it, was the fact that it was deliberately shackled by its conception as an XML translator to HTML. You couldn't manipulate non-XML text with it, could only do a fairly limited number of strong manipulations (in a language that was, fundamentally, parsing text), you couldn't create intermediate nodes for processing, and things that should have been fundamental - the ability to create an indexed for-loop, for instance - necessitated some very ugly recursion that added considerable complexity to the language without a lot of reward.

I wasn't the only one who found this to be the case, by the way. Indeed, many developers have come to XSLT for its potential capabilities but found themselves so bogged down with the verbosity and complexity of XPath manipulations that they would soon beg to find some other, easier solution. This has, in turn, created something of a backlash to the language, and more than a few projects built around XSLT have consequently become management nightmares, because few developers wanted to develop the expertise to debug seemingly incomprehensible stylesheets, especially given that it fell into the "declarative ghetto" where salaries were often lower than for procedural programmers because of the bias to see XML expertise (and consequently XSLT) as being simply an extension of HTML expertise.

This motivated me to follow the development of the "next generation" of XSLT, with the hope that it might prove an improvement over what currently existed. XSLT 1.0 was not so much broken as incomplete, though there were some fundamental changes that needed to be made to the data model in irder to accomodate the additional changes. Thus began an arduous trek following the development of XSLT 2.0.

By the time that XSLT 1.0 came out, James Clark, the iconoclastic genius who created XSLT in the first place, had shifted his attention away from transformations and into schemas, eventually laying the groundwork for Relax NG. Meanwhile, Michael Kay, the author of the authoritative XSLT books for Wrox and the creator of the Saxon XSLT processor, took over the editorship of the XSLT working group, working in conjunction with people such as Jeni Tennison and Dimitre Novatchev to establish both a set of extensions to XSLT 1.0 under the EXSLT.org banner and ultimately a proposed XSLT 1.1 Working Draft by mid 2002.

However, a number of realizations about the depth of the problem with the data model (and consequently of XPath, which relies heavily upon this model) forced the withdrawal of the XSLT 1.1 Working Draft from the W3C and the formal establishiment of an XSLT 2.0 working group. The goal of this group was simple -- to do XSLT right, to fix some of the biggest problems of XSLT that came from being based upon certain intrinsic assumptions and to revise XPath so that it would be robust enough to handle a much wider range of problems.

Not Your Father's XSLT

The language that is emerging bears a number of broad similarities with XSLT 1.0, but underneath it is a considerably more sophisticated vehicle. Perhaps the biggest change has come in the introduction of sequences. A sequence is a linear list - a set of XML objects, numbers, text strings, and other objects that can be ordered and otherwise manipulated. In XSLT 1.0 (and more specifically within XPath 1.0), you could only work with node lists, and even though such lists could hold text content (what are called text nodes) these were still containers for content rather than the content itself. By generalizing the node-set into a sequence, several things were made to happen:

Sequences could hold references to multiple distinct XML trees, something which was included as function in XSLT 1.0 (the document() function) but not in XPath 1.0.

You could create temporary trees of XML from other XML operations, using this intermediate XML as a way to perform other transformations. Most XSLT implementations had implemented a "fiat" way of doing this (a node-set() function, typically) but the implementations varied and the underlying model was incompatible with this.

Sequences made it much easier to eliminate duplicates and perform other logical operations on XML data, such as grouping (something that can be fiendishly difficult with XSLT 1.0).

Sequences made it possible to create iterative loops (something analogous to for(i=0;i!=n;i++){do something with i;}). The to operator in XPath 2.0 let's you create constructs such as (1 to 5), which generates the sequence (1,2,3,4,5).

Sequences also lay at the heart of another critical requirement for XSLT - the ability to parse strings into constituent pieces.

Once this foundational step was laid, the next stage in the process was to build up from that, adding new capabilities while still trying to retain as much of the power of the old standard as possible. This came about through the introduction of other innovations into the XSLT and XPath standards:

Regular Expressions. Regexes (as they are often called) provide a powerful tool for both searching and manipulating patterns within text. XPath 2 now incorporates the Perl regular expression engine (with some minor additions), making it possible to convert text files into sequences and from there into XML documents. This key capability makes XSLT a particularly noteworthy candidate for handling compiling, something that will be discussed later.

User Defined XPath Functions. The XPath Working Group established a formal mechanism for binding external functions into XPath, providing a clean, consisten means to build new functions XPath that could be written in C++, C#, Javascript, Java, Perl ... and XSLT. This dramatically reduces the amount of code necessary to invoke XSLT named templates (often by an order of magnitude or more) and also makes it possible to migrate XSLT from a Java based system to a C# based one without needing to change any XSLT - you'd just rewrite the external functions but keep the same function signatures.

Conditional Expressions. With XPath 2.0, you can now write if/then/else and for() statements within XPath, making it possible to create much richer logic into the language. Not only does this also reduce the amount of verbosity in the language significantly, it also makes it possible to solve what was typically not even possible in XPath - such as adding taxes and discounts into item costs in an invoice before finding a total.

Date/Time Manipulation. Date and time manipulation was something of a nightmare in XSLT 1.0, yet because of the importance of such information in transformations, there was a thriving industry in building work-arounds. Now such capability, including finding the difference between two dates or times, is built into the language.

Complex Grouping. The data model in XSLT 1.0 made it very difficult to handle certain kinds of groupings, such as mapping the relatively flat structure of HTML to the group and section model of XSL-FO or DocBook. With sequences and regular expressions, generating such groups is now possible, especially in conjunction with certain additional XSLT 2.0 elements.

Multiple Outputs. XSLT 1.0 was assymetric -- it was possible to pass in multiple XML documents through parameters, but it was not possible to produce more than one formal output. That's changed with XSLT 2.0. Now, you can write transformations that will generate any number of XML or text output formats, either to be saved to local storage or to be sent to external web addresses, depending upon security restrictions.

Type Awareness. Perhaps one of the most contoversial aspects of XSLT 2.0 and XPath 2.0 is the introduction of schema-aware tranformations which were capable of validating and manipulating typed XML content from external XML objects. This is not an intrinsic part of the specification, however, so it is less likely that all XSLT 2.0 processors will be schema aware.

This combination of features fills most of the holes left from the XSLT 1.0 implementation and makes it possible to start thinking about XSLT sitting not just along the periphery of your operation, but right in the middle handling the processing of business logic.

XSLT For Businessmen

In a typical business, you buy or implement business software in response to your changing business needs. Much of this software is anticipatory in nature - the designers of the applications attempt to model ahead of time the scenarios that are most likely to occur in your business, and then build the business logic for these scenarios into the code itself.

Anticipatory design has a number of side-effects, few of them positive. For starters, the applicability of the software becomes a measure of the degree to which the application designers were successfully able to model the business processes that occur. When the modelling is close, the application integrates well into the work flow of the company. When the modelling isn't so close, the company is all too often forced to adapt to the workflow of the software, which introduces inefficiencies.

Moreover, over time, a company's business requirements change as the business itself changes. However, the software has almost certainly been written by someone who is no longer writing that particular piece of software -- best case scenario is that they are working on some other part, and consequently have to stop what they're doing to change the code. Worst case scenario is that your lead programmer is India (unless you are in India, in which case your lead programmer is in London), has long since left the company, and likely didn't document the code terribly well. Thus, over time, the software decays, until it is no longer useful to the company, forcing another massive expenditure into the whole in your business called IT Expenditures.

Finally, many such solutions are intimately tied not just to a particular operating system but a particular machine, and should something happen to that machine, your company could be left with a major problem.

All of these situations point out the limitations of anticipatory design, but such design is still the most prevalent because it 1) keeps software vendors in business, 2) keeps consultants in business, and 3) ultimately forces hardware upgrades, keeping hardware vendors in business. Of course, unless your business is specifically dedicated to keeping these three groups in business, such design often becomes a hidden tax on computer usage, a constant drain on expenditures that becomes very easy to accept as unavoidable. However, that cost really isn't as necessary as it may seem.

One of the great benefits of XML is the fact that it's use tends to encourage adaptive rather than anticipatory design. With adaptive design, the business logic of a company can be readily encoded in an easy to manipulate bundle of information which can work across any platform. Your code can generate your user interfaces in response to changes in data requirements, passing that information into transformations that can readily encode the business logic. Moreover, even the transformations themselves can be transformed, can be designed to change as business parameters change. In short, such systems, adapt to the changing requirements of the business.

XSLT 1.0 was an interesting first step in this process, but all of the points mentioned above - the complexity of the language, the verbosity of the code, and the often counterintuitive techniques necessary to handle frequent operations made it less than idea for this particular process. However, XSLT 2.0 is considerably simpler to follow, write, and maintain, can more reliably integrate with external processes and objects, and is able to handle multiple possible input and output forms at once.

As tools such as XForms (or some equivalent XML centric forms technology) becomes more prevalent, this also means that interface tools (and not necessarily just "web tools") will increasingly generate and send XML content directly rather than the more limited name/value pairs of HTTP (in essence what SOAP does via the agency of web services), and in general XSLT is a better tool for manipulating and extracting information form XML sources than DOM tools are ... if that extracted information is itself in XML format. In that respect, DOM can be thought of as being a binding mechanism that connects XML with other object representations (that is, other programming language data structures).

This use of XSLT within XML millieus is an important concept, with very broad implications. XSLT is not sexy. There are no marketing teams out there who put out multimillion dollar ad campaigns featuring well-coifed executives staring raptly at XSLT code on their laptops. Instead, XSLT is an infrastructure sort of thing, found deep within (a surprising number of) applications, increasingly taking over the task of document and object conversions that for years had been the domain of heavily sequestered filter writers. The application I'm writing this on right now, an HTML editor which I wrote, uses XSLT to convert between an editor component's internal representation and one of several XML formats -- including docBook, XHTML, Microsoft Word 2003's XML format and others. Yet without knowing that, you'd never even be aware of how critical that technology is, because it does exist so quietly.

Code Building Code

XSLT 2.0 will likely become much more pervasive, because it's domain of applicability is so much broader and because much of the design of the second version of the language is deliberately built around the manipulation of any textually represented object -- including other programming languages. Most programming languages have a very well-defined programming structure, independent of the commands themselves -- packages, classes, constructors, destructors, methods, properties, event handlers -- in most cases there are relatively few variations off of these basic entities, in great part because programming languages are process descriptions (at least imperative languages are).

XML in turn is a good language for the description of descriptions, and as a consequence, it can very readily incorporate larger functional blocks of code in a descriptive matrix. Once in that format, generating code in other languages becomes much easier using a language such as XSLT2, especially with the addition of regular expression tokenizing parsers. On the flip side, XSLT2 is also remarkably good at the inverse process -- parsing program language blocks and converting them into an XML representation. In short, XSLT2 could find itself becoming what amounts to a universal compiler/decompiler, at first into intermediate forms such as Java or C#, and then with increasing frequency, directly into bytecode or intermediate language (IL) generators (this is especially significant of C#, which already maps many of its languages into a common IL format).

From an application standpoint, this raises the real possibility that this next generation of XSLT2 could in fact not only handle the processing of business logic, but actually generate just-in-time compiled code to more quickly execute such logic, and could perforce route it to the locations where such JIT code would be needed. User interfaces could be built on the fly, something especially critical in business applications where a significant portion of the programming work (and hence cost) that takes place is oriented to developing the screens whereby people interact with the data. The combination of XSLT2 and web services can also abstract connections into SQL and LDAP repositories, meaning both that such data sources become more interchangeable and that the way of accessing something as different as a relational database and an LDAP directory becomes irrelevant.

Finally, XSLT2 simplifies the way that information moves within your enterprise, ironically by moving away from what had been the cornerstone of programming in the 1990s - the object-oriented-programming paradigm. One of the difficulties that has emerged from OOP principles has been in determining the decomposition of a problem space into distinct classes. Programmers new to OOP (and in all too many cases not so new) have a tendency to want to model everything as a class, and as a consequence their application code begins to resemble the Harry Potter series -- full of wonder and magic, but with entirely too many pages for what is, fundamentally, a children's story. The problem with this is that each class has to be written and tested, not only in isolation but also in tandem, and a seemingly trivial change in a base class can have profound consequences for other classes built upon it.

XSLT 2.0, on the other hand, shifts the approach taken from building up this complex zoo of class critters and pushes it back towards an approach which is coming back into vogue with the advent of Linux: streams and pipes. A stream can be thought of as data moving between two processes, whether those processes be a file reader, a web server, a program, a database, a web client, or the interface to some piece of hardware. A pipe on the other hand, is the conduit which carries the stream. The OOP revolution placed a huge amount of significance on the node where pipes met, and tended to relegate the pipes and streams to secondary status, at best.

However, XML and the web is changing this. One effect of XML web services is to envision programs as being the transmission of streams of data to distinct end-points, URLs, without necessarily caring about what happens within that end-point. An object orientation gives many more points of access into an object, but typically at a cost of dealing with that object's specific data restrictions. With a web service, I can send a stream of information to a URL, and the process at that end will choose (if it is well designed) to either determine that it is valid and usable (there are processes that are designed to work with that stream at that node), that it is valid but not immediately usable (it is sent off to a different process which will attempt to rectify it into something of significance to the first process) or that it is invalid (whereupon notification of this lack of validity is sent back to the sender).

XSLT2 can handle all three potential conditions (though the case where the data is not well-formed XML gets a little complicated). Well formed XML has an associated namespace, and this namespace can actually be used by the XSLT itself to determine the proper handling of XML content, perhaps with the concept that such an XSLT could pass on the parameters acting upon the transformation into part of a SOAP message and then routing that message to the appropriate final transformation. in a purely asynchronous model (one where each node can act both as a transmitted of XML and a receiver of XML under separate processes), the routing XSLT does not have to be the XSLT that handles the final processing of the associated data -- or the one that communicates back to the client. While this model doesn't quite work in the fairly strongly synchronous browser model that most people connected to the web currently use, contemporary web clients are in fact shifting to an asynchronous model where it will work just fine.

An Open Letter

If XSLT2 is such a superior technology, why has it raised the ire of a few of the largest software vendors (most notably Microsoft)? This is in fact a case of the question providing its own answer. XSLT 2.0 provides an answer to many (perhaps most) of the complaints of XSLT 1.0, including the most damning ... that it is too conceptually difficult and verbose for programmers to learn. XSLT2 is more functionally oriented than XSLT1, making it easier for programmers more comfortable with languages such as Java or C++ to use.

XSLT2 also binds to external classes much more transparently, making it much easier to communicate with external processes within the environment, regardless of what that environment is (or what kind of box that environment is running on). It doesn't require an expensive suite of tools, compilers, and libraries of objects to work with it, and it is fundamentally oriented to manipulating XML data (though not exclusively) without the strong-typing limitations that come with Algol based languages.

XSLT is also considerably more secure, based upon what I'd call the potency argument. In essence, most binary objects contain not only their own state but also the mechanisms by which that state gets expressed to the outside world. In essence, these objects are potent - they have the ability to create side-effects that may not be obvious from their interfaces, as they have to have a certain level of interaction with the system in order to be able to function. In effect, the level of trust that these objects require of the system simply in order to operate is too high, forcing the creation of human trust authorities.

With XSLT, on the other hand, the streams of XML information coming in are pre-potent. They provide a description of state, but are reliant upon the XSLT that is resident within the server to handle the manipulation of that state, and correspondingly to specifically provide exceptions for handling things outside of the boundaries of safe behavior. It is consequently incumbent upon the maintainer of the system to choose the interpreters of that data, rather than placing the security demands upon the (non-technical) users of the applications.

Given all this, XSLT2 enabled systems could serve to signficantly erode the requirements for complex components that sit at the heart of most operating systems, a potential boon to open source systems such as Linux but one that could dramatically impede the ability to effectively sell subsequent implementations of Windows (or so at least one line of reasoning goes that I've seen bandied about). It makes code generation a just-in-time process and so effectively blurs the distinction between data and code, a distinction that Microsoft still makes even as it defines its own XML user interface language (XAML, which requires a healthy dose of C# "code-behind" in order to do more than trivial applications).

Microsoft has chosen to include XQuery 1.0 (a data-binding language that builds on XPath) but not XSLT 2.0 in Longhorn, citing everything from lack of customer interest to complexity in implementation to insufficient maturity on the part of the specification. They have even gone so far as to try to develop an alternative language, C Omega, which is supposed to provide a C# oriented approach to manipulating XML.

I've played with C Omega some - it is a reasonably good way to avoid some of the tedium of working with the W3C DOM, and it is certainly possible to use it for some of the same purposes that you'd use XSLT2, though it lacks the powerful recursive templating capability that I think gives XSLT most of its power. It presupposes that the appetite for XQuery will be strong enough that they can essentially build a hybrid language around it, though after having written two books on XQuery that have between them garnered less than the production costs for the books in question I'm much less inclined to agree, especially as XPath2/XSLT2 becomes much more functionally oriented.

At the last Sells Brothers XML Conference (which I would heartily recommend, by the way) I gave a talk on Saxon.NET, an open source project in which M. David Peterson has converted Michael Kay's superb Saxon 8.0 XSLT 2 implementation over to .NET, with Dr. Kay's approval. I'm using it now for a content management system, and it has performed far better than I had even hoped. At any rate, when the Microsoft representatives at the conference later asked at the crowd whether they would rather have work on XQuery or XSLT2, the number of people (in many cases customers of Microsoft) who wanted to see a new XSLT outnumbered those of XQuery by a considerable margin.

While I strongly support Mr. Peterson's efforts, I also would like to make a plea to Microsoft to reconsider your stance on this. I believe that the demand for a more powerful version of XSLT is out there, and that it is being driven by application developers who are building applications for Windows as well as elsewhere. It will become the de-facto standard within your competitors' business productivity suites, web clients, home entertainment applications and operating systems, because if you choose not to develop such a processor, others will provide .NET versions that will be used in place of your own offerings. You will have already done most of the hard work in implementing it, as the major portion of the changes that occur in XSLT 2 is due to the revision of XPath 2.0, which you are already developing to support XQuery.

To business decision makers reading this, chances are really good that you will likely never actually have to sit and look at a screen of XSLT 2. However, as with XML six years ago, XSLT 2 is a technology that will likely end up shouldering much of the day to day processing within your organizations over the course of the next five years -- it is a natural complement to XML, which has, like kudzu, pretty much taken over the data infrastructure of most applications it comes in contact with.

There's another factor that comes into play here from a business perspective. In 1993-4, an independent consultant could earn $250 an hour writing HTML content. Today, HTML is considered a basic skill that every graphic designer needs to learn as part of his or her doing their job, and HTML generation is mostly handled via automation mechanisms. XSLT serves many of the same purposes that tools such PHP, ASP.NET, Perl, and JSP serve today, but as the world continues its adoption of XML as the standard for conveying structured content, XSLT is becoming something of a lingua franca - a common tongue - that developers in this space are learning, and are finding in that learning that by intelligent application of that XSLT that particular skill is cleanly transferable from a Linux box running Apache using PHP and a Windows box running IIS and ASP.NET.

XSLT 2 is not a new language - it is XSLT cleaned up to handle the areas it should have been capable of handling before, with much less verbosity, more integration, more power and a considerably easier development and maintenance path. This means that the learning curve for those developers going from XSLT to XSLT 2 will be much less extreme than having to learn another language in toto. This in turn means that within a space of a couple of years, if not less, XSLT2 will likely be just another core skill that a developer should have, yet one that helps them write platform and language neutral code for dealing with all of the XML that is even now circulating through your business. With skilled programmers in any area once again beginning to demand a premium, the coming ubiquity of XSLT2 skills should help keep your labor costs down not just in your web development department, but throughout your organization.

/EndTag

There's a tendency of late for writers in the XML space to want to play down some of the more recent standards from the W3C, a tendency to go "ho-hum, it's just the sequel". I think this attitude can blind people to what is actually happening. XSLT 1.0 was a uniquely different and powerful solution, and after having worked with it on an almost daily basis for nearly the last decade my respect for its innovation has only grown. However, even four years ago I felt that it wasn't powerful enough, and the amount of customization on the part of XSLT processor vendors over the last several years to me is testament to that. XSLT 2.0 is not profoundly different from XSLT 1.0 - it's in fact almost completely backwards compatible.

It does, however, rectify the shortcomings that emerged from the first iteration of the language, and does so in a way to make it an astonishingly powerful language. History is full of such standards, such as SQL, where it took one or two iterations to handle the inevitable discovery process that is part of any great human endeavor. This tool, XSLT, is already becoming one of the core work-horses in most contemporary applications, even given that it was never originally conceived to do what it is called for. To move forward to a version improved by half a decade of insight and exploration is not only logical, it's good business.

Imports and Tunnels in XSLT2

2004-12-12T23:34:00.000-08:00

Every so often, I discover that just when I think I've wrung everthing I can out of XSLT, it is still possible to do something subtle that changes my whole mindset about the language. To me, one of the hallmarks of greatness in a language is that very ability to surprise, to make even a somewhat jaded coder sit up and say "Hey, that's cool!".

I faced a challenge of that sort recently in an application I was working on. The issue that I was struggling with was that there are aspects to XSLT2 which are not quite polymorphic. Polymorphism, for those of you who might of slept through that particular aspect of OOP training, is the idea that you can have two distinct objects have a particular method which, when invoked, produce similar but not necessarily identical actions.

The canonical example of this comes with the Shape class, with the method draw(). If I have two child classes called Rect and Circle, each of which inherit the draw() interface from Shape, then I could create instances of these classes:

rect =new Rect(); rect.draw();

and
circle = new Circle(); circle.draw();

each of these objects will perform a draw operation relevant to its given class (drawing a rectangle or a circle, as appropriate). By creating such a set of conventions (and the notion of a consistent interface gained via inheritance) you can additionally work with a wide range of similar objects through the same interface without having to write special type handlers for each object.

The three pillars of OOP, inheritance, encapsulation, and polymorphism, can exist within XSLT, but only somewhat uneasily. Inheritance, for instance, can be gained by creating two stylesheets, the first containing a dominant set of templates, the second "replacement" templates that effectively replace the first if they have an equal or higher precedence match. The problem with this is that there is no real way of saying "utilize the replacement template, then invoke the original template", the analog to the super() function in languages such as Java, which invokes the parent's method when invoked the same method of a child object.

XSLT 2.0 does provide features to ameliorate this somewhat.The first involves a subtle shift (albeit one that can cause a lot of grief for XSLT 1.0 developers). In XSLT 1.0, imported stylesheets had a higher priority than importing ones. This made it possible to "subclass" templates easily, but provided no clear mechanism for being able to invoke the replaced templates.

However, in XSLT 2.0, the priorities have been switched, which means that the calling stylesheet's templates will have a higher priority than the called stylesheet's templates, unless the priority is explicitly set as an attribute as being different. This has the immediate effect of giving the illusion that imported templates don't work. That's not quite the case, however. Instead, such templates can now be invoked with the <xsl:apply-imports/> element. This causes the processor to only look at templates in the imported stylesheet when applying the rules, and it is this functionality that gives you "super()" like behavior.

For instance, suppose that you have a stylesheet that takes the value of an item, and if it is a number less than 0, renders that number in red (in the file accounting.xsl)

<xsl:template match="example">
<xsl:choose>
<xsl:when test="number(text()) != NaN">
<xsl:choose>
<xsl:when test="number(text()) ge = 0">
<span style="color:green"><xsl:value-of select="."/></span>
</xsl:when>
<xsl:otherwise>
<span style="color:red"><xsl:value-of select="."/></span>
</xsl:otherwise>
</xsl:choose>
</xsl:when> <xsl:otherwise> <xsl:value-of select="."/> </xsl:otherwise> </xsl:choose></xsl:template>

A second template for "example" in a calling document could handle placing this content into a table cell, using the <xsl:apply-imports> element, as follows:

<xsl:import href="accounting.xsl"/>
<xsl:template match="example">
<tr><td><xsl:apply-imports/></td></tr>
</xsl:template><xsl:template match="examples"><table> <xsl:apply-templates/></table></xsl:template>

If the source XML document had the form:<examples><example>125</example> <example>-150</example><example>Twelve</example></examples>

Then this will render as:<table><tr><td><span style="color:green">125</span></td></tr><tr><td><span style="color:red">-150</span></td></tr><tr><td>Twelve</td></tr></table>
125
-150
Twelve

Significantly, if no import document is specified, then this will act in the manner appropriate for <xs:apply-templates/> for processing child elements and text of the current context. In other words, without the imported stylesheet, this will render as:

<table> <tr><td>125</td></tr> <tr><td>-150</td></tr> <tr><td>Twelve</td></tr></table>

This fairly subtle change will have an enormous impact upon the way that you develop stylesheets, especially in conjunction with a number of the other features that are now in play, including the next bit of subtlety - tunnelling parameters.

Digging Those Tunnels

You have probably noticed that while its possible to talk about applying OOP techniques to XSLT, such concepts do not necessarily translate cleanly in a one to one manner to how XSLT itself works. One instance of this comes from the problems inherent with dealing with globals.

Global variables are bad in the way that cookies and sweets are bad ... one or two, taken sparingly, can improve your code without too many ill effects, but go overboard and you'll be seeing the results show up in bloated code (or at least a bloated waistline).

Normally, in OOP code, you define variables that are appropriate for a given class, and oany methods that are defined within that class have the variables available to them. You can also create public properties which appear like "public" variables that are available from outside the class, but in most languages, these "public" variables are being replaced by getter/setter methods that expose specific properties while at the same time providing a certain degree of insulation on these variables. Additionally, regardless of scope, variables defined in this manner may also be available to any derived class of the base class, through some variation of the friend keyword.

In XSLT, things work a little differently. You can define a global variable or parameter within the <xsl:stylesheet> element of a transformation, in which case the variables are available anywhere within the transformation. However, if you import a global variable from an external template into your transformation, that variable or parameter will be overridden by a variable in the calling document if they have the same name. In some cases, this may be desirable, but in others, the calling template may have defined these variables for handling some internal state within the transformation, at which point you're feeding gibberish into your transformations (and usually very hard to debug gibberish, at that, as the code is syntactically correct).

You could, of course, not use globals in the called routines (a good practice anyway) and instead choose only to define the parameters within the likely entry point of the called stylesheet and then pass them down from template to template. Unfortunately, this solution tends to be rather verbose, especially when dozens of such "internal variables" are needed. Especially if the called templates didn't require the parameters, this particular technique was really only worthwhile if you absolutely needed that data at some point.

In XSLT2, a different approach was taken. A new attribute, the tunnel attribute, is placed on parameters in matched templates (i.e., <xsl:template match="foo") elements, with tunnel = "yes" indicating that the parameter should participate in tunneling, and tunnel = "no" indicating that the parameter is not tunnelled. An apply-templates parameter in turn would require that the tunnel = "yes" attribute be added to the <xsl:with-parameter> element in order for that variable to tunnel.

So what exactly is this tunnelling? In essence, a tunnel indicates that if a template calls another template that does not have the parameter declared (and consequently does not use that parameter), and this template in turn calls a third template that does have tunnelled parameters, then the third (or fourth, etc.) template acts as if it was called with the parameters from the first template. This way, the only templates that receive the parameters are the ones that actually use them, and you are able to keep your code lean in mean in the process.

As an illustration, the following illustrates a "tunnelled" hello,world type transformation:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:template match="/">
<xsl:apply-templates select="foo">
<xsl:with-param name="username" select="'Aleria'" tunnel="yes"/>
</xsl:apply-templates>
</xsl:template>

<xsl:template match="foo">
<html>
<head>
<title>Hello, World!</title>
</head>
<body>
<xsl:apply-templates select="bar"/>
</body>
</html>
</xsl:template>

<xsl:template match="bar">
<xsl:param name="username" tunnel="yes"/>
<h1>Hello, <xsl:value-of select="$username"/></h1>
</xsl:template></xsl:stylesheet>

With the assumed input being:
<foo>
<bar/></foo>

In this particular case, the parameter "username" is invoked from the root template match for the "foo" template. However, foo has no need of the parameter, so it doesn't need to declare the parameter. The "bar" template, on the other hand, does need to use the parameter, so it creates a parameter declaration, including the critical tunnel = "yes". Without this, the XSLT processor would work on the assumption that the parameter was simply not invoked from the "foo" template, and as a consequence, the $username is assumed to be blank.

This form of parameterization can vastly simplify writing imported stylesheets. Typically such sheets will typically provide a single point of entry for subsequent transformations in the imported sheets(especially if the templates use the mode attribute), and rather than defining global parameters that could possibly be clobbered, the entry point template could define the parameters within its body as being tunnelled, making these parameters available to any member of that set of modal templates.

</wrapup>

h2>

Thus, in addition to making your code less verbose (and hence easier to both write and maintain) tunnelling can additionally serve to make your imported stylesheets more modular and classlike. In essence, this turns an imported stylesheet into something that has a number of the characteristics of classes within a more formal hierarchical language such as Java or C++, while at the same time retaining the templating architecture that makes XSLT such a powerful language in the first place.

Internet Time and the Long Now

2004-12-04T23:54:00.000-08:00

I wish to thank Edd Dumbill for syndicating the Metaphorical Web on his superb blog aggregator site http://planet.xmlhack.com. It's an honor and a privilege, and I hope to be able to produce commentary in keeping with the luminaries on this board.

I've been pondering for a while the state of the XML world. XML has now been in existence as a standard for nearly seven years. XSLT and XPath for six, SVG for four and change. By the scale of Internet time (how retrograde that expression seems now) these technologies are no longer bright and shiny new, but are actually approaching middle-aged. Heck, Microsoft would have gone through two world changing marketing campaigns by now, and would be gearing up on the third. So why does it seem that XML is still just getting started?

Oh, you could single out all the usual suspects, of course. The W3C moving at a snail's pace, vendor lock-in keeping these open standards from reaching their true pace, the lack of marketing to push standards into the forefront of computing and so forth (all of which are true to a certain extent). However, over the last year or so another idea has occurred to me, one that is perhaps heretical in the world of nanosecond processors, yet something that has nagged and tugged at my consciousness for a while.

Daniel Hillis, the founder of Thinking Machines and an Imagineer for Disney for a number of years, has of late spent a great deal of time concentrating on another project of his, The Clock of the Long Now. This particular clock is unusual in that it is intended to exist for 10,000 years. To put that into perspective, 10,000 years ago Europe was beginning to repopulate after the last of the continent covering glaciers had retreated, and the dominant language was an early Mesopotamian ur-language which would in time branch into the Indo-European languages.

Thus, to build such a clock, you have to design it to be simple, to be self-maintaining (who knows how many centuries may pass between people even knowing of it) and to be intuitively obvious regardless of what culture finds it. In many ways, the Clock is the ultimate exercise in user interface design. It is also, in its own way, a remarkable statement about the nature of time itself and man's interaction with it.

So what does such a clock have to do with XML? In any period of intense innovation, a pattern tends to emerge, regardless of the technology in question. Most of this should be familiar as a variant of the adoption curve:

An innovation is created, usually a variant of some existing technology but with a twist that moves the technology in a completely unexpected direction.
"Hobbyists" emerge, the earliest adopters who work with out while the technology is still in rough stages. They often end up becoming the evangelists for that technology later on.
The technology attracts the attention of early investors, who see this as a way to start companies that exploit the technology. Such investors seldom come from the hobbyists, but they are keyed in enough to what where the hobbyists are most excited.
The technology goes into its hype phase, where marketers begin promoting it for everything from improving efficiency to curing the common cold. This phase usually involves the appearance of semi-nude women seductively wrapped around the piece of hardware, working at a computer with the software, or otherwise doing improper things with a technological implementation.
The backlash occurs as people realize that it does not in fact cure the common cold, may actual decrease efficiency in certain ways, and tends to be a big turnoff to attractive young women at bars (who are usually after the early investors, not the schleppy users).
Meanwhile, those people who utilize the technology for a living figure out whether the technology actually does meet a specific need of theirs, and in the process will usually provide feedback to strengthen the technology in question.
If the technology manages to get past this stage into adoption, its development shifts from paradigm shifting innovation to more stable improvements, especially if the same technology inspires other implementations/variations. This competitive phase usually lasts until one particular implementation manages to gain a 90/10 mindshare. This phase also usually sees the emergence of some body of standards that define interconnectivity of that particular technology (for instance, standard gauge sizes in railroads, standard socket types, HTML). These standards usually reflect the implementation of the dominant player.
The technology then is able to sit with comparatively minor changes for a significant period of time - years or even decades, but because they are based upon these standards, the dominant player also has the greatest to lose and least to gain in changing those standards to reflects shifts in new technologies.Standards that were based upon a given implementation consequently freeze that implementation, until eventually, the standard is out of sync with additional innovations.
The dominant standard-bearer can suddenly find its position upended very quickly -- within a couple of years -- and find that it is now in a position with an aging infrastructure and a massive installed base. It either abandons that base and moves on, putting itself into a much more vulnerable position, or it sticks to that base even as the base itself moves on to new technology. In either case, the technology may remain for a while longer in a kind of white dwarf state, slowly cooling to a brown cinder. Periodically, the technology may be resurrected for a revival (think of all of the coin-op video games which now exist as software emulations) but such technology holds entertainment value only (restoring an 1890s railroad train engine, not because it is even remotely competitive, but because it has sentimental value).

Given this birth to death cycle for most technologies, why does XML feel so different? My suspicion is that XML is different -- it is a technology for the Long Now. XML of course has its roots in SGML, a technology that was relatively obscure for most people outside of corporate librarians and academics but that can in turn be traced back to the work of Charles Goldfarb in the 1960s. This means that it in fact predates another "long-now" technology: the Unix/C duality. SGML was not intended as a means to gain temporary market share; indeed, the high cost of creating SGML implementations meant that it was really commercially viable only for the largest of organizations.

SGML is not an implementation-driven technology -- it was from the first a vehicle that required consensus, because it's focus was at the heart of document communication, which means that from the start it was intended to model the way that people think, not what is necessarily the best way for machines to store artificially rigid class constructions. Because it was a meta-language, SGML by its very nature was intended to be implementation independent, long before Sun's "Write Once Run Anywhere" slogan came into play.

In addition to this, SGML is declarative. It's operant verb is BE, not DO. By doing this, it was able to provide a level of abstraction that bypassed the 1000 class monstrosities that emerged over the course of the next three decades. That abstraction lives on with XML, to the extent that it is dramatically affecting both the theory and practice of coding.

XML, in turn, has been a refinement of this notion of working with abstractions through an abstract interface, with the underlying assumption that so long as the expected behavior that's agreed upon is met, that specific implementation is irrelevant. One effect of this has been the increasing dominance of XML abstraction layers that in turn push the imperative code -- the C++ and Java and Perl of the world -- down the processing stack, away from the business logic and closer to providing a common low-level substrate. XUL, SVG, XAML, XFaces, XForms, etc. all provide manifestations of this principle to some degree. Create a binding layer such as XBL, and you can hide the API even more, with the consequence that you can increasingly reproduce sophisticated sets of object code with XML representations. The imperative code never goes away completely (nor should it) but it becomes much less significant in the scheme of things.

As a consequence of this, while certainly there have been companies that have ridden some aspect of XML through the technology business cycle described above, I think that XML as a technology is acting very much like the rise of mitochondrial RNA arising during the Cambrian era - providing a mechanism that serves as a substrate for a whole new kind of life (or in this case for a whole new kind of programming paradigm). The problem from the standpoint of those in what had been the dominant paradigm - the framework based OOP system - is that such XML provides few of the features that traditionally make money for vendors -- lock-in of file formats, opacity of data structures, variations in access formats that provide advantages to one or another vendor, arcane API implementations, reliance upon a specific language, and so forth.

Web services make it possible to do things that would have been impossible otherwise (the dominant of which seem to be less providing unique data feeds and more performing transformations between varying schemas). XML based GUIs (and even full application sets) are now becoming the standard way of building human faces to applications, cutting down on the reliance of specific language toolkits. XML is even being used for discovery of non-XML APIs, something which usually indicates a transient phase (if you can discover non-XML APIs, you are invoking an abstraction layer within the interface, which in turn makes it easier to decouple the implementations from an imperative basis.

Microsoft has gone through several core changes of technology (and more marketing "initiatives") in the last 30 years, yet curiously few of them have a staying power beyond about five years. This is not a disparagement of their products (which for all of my grumpiness about Microsoft even I will concede they do produce very usable and professional products), but I think is more indicative of the commercial software vendor market (I could replace Microsoft with Sun or Hewlett Packard in the above statement and be equally correct.) Their definition of Internet Time gets conflated with their definition of Business Time, where next quarter is more important than next year, and five years out is an eternity.

Ultimately, though, Internet Time to me is something that isn't measured in nanoseconds and months between deployment cycles. If, as I do, you believe that XML is becoming the neural circuitry of the Internet, then I have to wonder if perhaps the real Internet Time is measured in decades, and maybe even centuries.

Indeed, when Danny Hillis finally gets around to inscribing the operating instructions of the Long Now Clock on its stainless steel base, perhaps, just perhaps, he should write it ... in XML.

Tech (non-)Savvy?

2004-11-30T01:06:00.000-08:00

This morning as I was pulling into Starbucks for my pre-work latte, listening to The Beat on Seattle's KUOW (an NPR station), the topic of web security, Internet Explorer and Firefox came up. Beyond the moderator (Steve Schoerr, I believe) there were three other panelists - a web security consultant, a writer for the Seattle Times, and an analyst from Jupiter research.

The discussion was, for the most part, non-technical; at the level of defining what a virus or worm was and why they were so problematic, the kind of discussion you'd expect in a radio forum with a wide, but not necessarily technically savvy listenership. Even given that, I think there were several points to be taken from the show beyond what was mentioned directly.

Security is Killing Microsoft. Almost from the outset, these people worked upon the assumption that Microsoft's products are fundamentally insecure, and these insecurities are likely to force you to frequently reformat your hard drives, lose critical information on your computers, or have your credit card information stolen. Long-time readers know that I'm not generally a big Microsoft proponent, but the kind of public trashing that Microsoft received may very well do more to erode their market share than any hundred bad eWeek articles about the company. The idea that Microsoft is fundamentally insecure (which I don't agree with - properly managed, Microsoft is not really any less secure than properly managed Linux distributions) has now become established folk wisdom. This is devastating for Redmond, and if anything it necessitates that they not only start putting some serious PR dollars into stopping that hemorraging of public trust but that they also seriously re-examine what their role vis-a-vis the Internet should be.

Firefox as Browser. If the radio show hurt Microsoft, it served as an hour long infomercial on the benefits of not only Firefox but Mozilla and Open Source in general, and I have no doubt that there was probably an extraordinarily high concentration of downloads coming from the Puget Sound after that. Firefox has momentum, in a way that I've not really seen any technology in the last five years have momentum, and most people who try Firefox are not going back to Internet Explorer. I argued in my previous post that while its still too early for enterprises to be developing applications around it, there is enough solidity within the platform that enterprise developers and IT managers are beginning to examine it for fast tracking into less mission critical areas.

Tech (non-)Savvy. Working at the fore-front of the Internet revolution, it is often easy to forget that most people out there know how to use the technology but have little to no clue about exactly how the technology operates. Most of the callers were of the "my computer has lately become slower than it was, is there a virus at work?" variety, though it was rather heartening to hear that many of these same people did make the switch to Firefox and found immediate improvements, though whether this is due to anything more than switching from one already full cache (and tens of thousands of ghostly temp files that will now likely never be removed from their computers) to an empty cache.

This attitude of technology as (unstated) magic is worrisome because it underlies to me how easily it is to slip into an attitude where you assume that the computer is doing valid and legitimate things at all times (or, on the other hand, that there is some evil force of terrorists out there that are determined to turn their machines into some kind of mindless zombies [aren't they already?]). I'd like to teach people to think of their computers as being an organism within a complex ecosystem, and like most such organisms attracts viruses both benign and malicious (benign viruses being programs) that provide useful benefits with no adverse side effects, a category which can become remarkably slippery if you look at it too closely. However, the fear is that this particular metaphor only reinforces the notion that computers are self-contained (and self-aware) entities. I'm increasingly convinced that in today's culture, computer training should be a staple in elementary, secondary and collegiate education.

Whither SVG? Thither!!

The blogosphere has become, for me, an awful temptation, as there are any number of articles that get posted that I so want to refute. Perhaps one of the most recent came from a writer for whom I actually have a great deal of respect ... InfoWeek commentator Jon Udell. His article entitled Whatever Happened to SVG? has set the various SVG lists buzzing, as it questioned whether SVG had dissappeared off the face of the planet, one of those articles that makes you really despair about the state of things.

His contentions, that SVG was supposed to be the technology that would change the world and then failed to do any such thing is accurate as far as it goes. In 2000, SVG was wracked with contention because there was such strong disagreement between the principals, especially Microsoft, Macromedia and Adobe, that the SVG 1.2 specification lists neither company among the working group members. Patent issues surfaced the next year which caused the W3C to debate the viability of the Reasonable And Non-Discriminatory (RAND) patent which would have let companies charge royalties for the use of standards, which came to a head in August and September of 2001 with the W3C response servers being overloaded with thousands of comments from the web community including most of the major luminaries in the field, saying almost universally that RANDs were a bad way to go. This process in turn made all W3C standard patents Royalty Free only (though this is an area where second attempts are being made to push RAND back into the organization).

Since then SVG has been subject to other challenges ... it is an incredibly complex specification to implement, and to date there have been only a few successful implementations, most notably that produced by Adobe, though the last year has seen the emergence of nearly complete SVG 1.0/1.1 viewers by companies in the wireless space, SVG static editors, and a couple of nearly complete SVG dynamic editors. A year ago at this time, few applications exported to or imported from SVG, now Microsoft (Visio), Macromedia (Flash), Adobe (illustrator) and Corel (Draw) all have SVG as either an input or output vector, and in some cases both, and most vector editors (open source and proprietary) provide SVG as an alternative format (and in some cases as the primary format).

This last point may seem trivial, but in practice it's anything but. SVG differs from nearly all other graphical drawing formats out there in several key ways - it has an XML structure that can be readily parsed and has a verfiable schema, it can reference other distributed graphic and metadata content, it is non-proprietary and as such does not run the risk of another GIF debacle, it can work on any platform that supports an SVG viewer and it can maintain external namespaces within its body. Even before you get into the scripting features of SVG, these points are enough to make it an attractive vehicle for complex documents over the web.

SVG has been "almost there" for a while, and its easy enough after a bit to begin to think that SVG will never take off. I'll be the first person to say that I don't think it will take off either ... if, by take-off, you imply explosive, rocket-like growth. Rather, it is creeping in on little cat feet (to paraphrase Carl Sandburg), showing up in places that you'd never expect to find it. Graps showing disease vectors or population shifts, icons in an operating system, drop-down boxes on a web page, a block of text in a pre-press application. SVG doesn't have a multi-million dollar marketing budget, and the Rolling Stones won't be there to play when it's used as part of an operating system. It doesn't run ads on TV, doesn't take out multi-page spreads pointing out where it is in the supply chain.

In short, I fully anticipate that SVG will end up being the VW Beetle of the graphics/multimedia world -- a little comical perhaps, not taken all that seriously, certainly not for the class conscious (at least in its 1960s incarnation), but ultimately both ubiquitous and easy enough that a grade-school student can create simple graphics with it without needing to know anything about programming beyond a few simple rules, and without needing to buy expensive editing or animation programs. Okay, SVG'll also have an air-cooled engine (you need to know exactly how far you can extend a metaphor... ;-)

It seems to me that HTML went through this stage in 1994, when the pundits who caught the wave early enough were already beginning to write articles about how HTML was beginning to be old hat, and would be dead soon enough in the face of pressure from languages such as Visual Basic (and later Java). Java on the desktop is now seen as the bad joke, Visual Basic, while still widely used, is itself now rapidly going the way of the dodo ... helped by the less than spectacular adoption of Visual Basic.NET. Meanwhile HTML is still being written by hand by those grade schoolers. Will SVG take the same arc? I'm counting on it.

Firefox: Why Microsoft Should Be Worried

2004-11-28T13:39:00.000-08:00

I inadvertantly posted this earlier while it was still under development. Sorry for the confusion.

Firefox turned 1.0 last week, and in the process managed to hit 1,000,000 downloads in one day. Put that into perspective - Firefox is in the neighborhood of 5 MB, which means that the mozilla.org servers had something in the neighborhood of 5 terabytes of data streaming over their pipes. From a pure networking standpoint, that's pretty amazing, not to mention the indication about how major a release Firefox has become.

A few Netcraft statistics are perhaps just as revealing. Netcraft measures browser usage on the web, and according to its data, Firefox has managed to capture about 4% of the browser share just since October when the 1. 0 preliminary review was released. Given that the movement of any given browser usually tends to be in the neighborhoods of tenths of points from month to month, this jump was phenomenal. Firefox and Mozilla combined now occupy roughly 7% of the browser market, most of it at the expense of Microsoft's Internet Explorer. IE dropped below 90% or the market for the first time in several years.

I've seen a number of articles on the web asking whether Microsoft should be worried about the rise of Firefox, especially given their own market-share of around 90%. After all, ASP.NET is increasingly putting the orientation of web pages regardless of which server the browser is aimed at, a server-centric philosophy that seems to be consistent with the stance that the company took after moving from a rich client model in the late 1990s.

Personally, I would contend that Microsoft does need to worry ... a great deal. Internet Explorer is more than just a browser ... it is a critical piece of infrastructure that is used in any number of applications, including applications that don't necessarily talk to the web. The ability to create dynamic interfaces is not something to take lightly, as such systems are far easier to update, more readily customizable than precompiled binaries, and are often simpler to write applications around, for the vast majority of all such applications. Given that a significant proportion of such applications are written not for the home user but for the enterprise, Internet Explorer may in fact anchor Windows in businesses even more than Microsoft Office.

Given that, Firefox represents a significant threat to Microsoft. I have been working with Firefox and XUL for roughly three months now, building a number of tools for a content management system including a customized WYSIWYG XML editor and a versioning system monitor. With some work, I've managed to make these tools work across Windows, the Macintosh and Linux, using a combination of Mozilla's XUL, Javascript, and XSLT. The editor, as one example, overrides the Firefox menu, making it possible for me to actually piggyback on top of a user's version of Firefox to get the functionality that I need.

I am writing this blog using the editor I built on the Firefox XUL library and API, with the editor actually running as a tabbed pane within the browser itself. While it is certainly possible (and I'll discuss in more detail how it can be done) to set up such core functions as cut, copy and paste, text searching, undo and redo, and so forth through Javascript code, by building on top of the web browser itself I was able to effectively get all of this for free, leaving me with more time to implement functionality specific to the company's requirements. Perhaps the closest analogy I can think of as to the power of this would be as if you had access to the source code for Internet Explorer, could make changes to the interface using XML and Javascript, and could then run it on any platform without complex recompilation. The XAML model comes closest, but XAML is also still at least two years out, and it's unlikely that you'll actually get a chance to manipulate (or even see) the source code for the XAML rendering engine.

Yet for all this, perhaps the most intriguing aspect of Firefox is its ability to integrate multiple extensions. It's worth considering that most of Firefox is in fact an extension of some sort - some extensions are just bundled more tightly with the original package. Third party extensions exist to do everything from translate or speak selected text to showing the weather for the next few days. Some, such as the Web Developer's Toolkit, can actually work very nicely with an editor to show the dimensions and paths of images, boundaries of tables and divs, and activating and deactivating Javascript and Java components on the fly. These extensions can in fact be utilized in conjunction with your own applications as well -- I use a number of them with the editing suite I've developed, again letting me concentrate on the relevant business logic on my end rather than trying to reimplement everything from scratch. This capability will also increase considerably by mid-next year, when SVG and XForms are integrated into the mix - making it possible to generate rich, intelligent forms and interactive multimedia using SVG, XBL bindings and data-aware form components.

The Anatomy of Mozilla

I've been rather blithely throwing around terms and acronyms here that may be familiar to the XUL developers among you but may otherwise be somewhat mysterious to the rest of you. Consequently, digging into the innards of Mozilla may both end up explaining some of this and giving you a better understanding of what exactly applications such as Firefox can do.

Conceptually, Mozilla (and by extension Firefox and Thunderbird) can be broken down two ways: Gecko and SeaMonkey. Gecko is a set of core objects, written primarily in C++, that handle the detailed rendering and memory management of web-based applications. Gecko is perhaps the oldest part of the Mozilla project, started from scratch to better perform the drawing of web pages than the older Netscape browsers did. You can thank Gecko for Firefox's surprisingly fast speed in rendering. Gecko serves as the interface between tbe application and the native graphical rendering system (such as GDI on Windows or XFree86 on Linux and Unix based systems), freeing up developers from having to explicitly access this layer directly.

Gecko, however, is a largely invisible layer from the application developer standpoint. If you're writing an application, you are much more likely to be interfacing with it through SeaMonkey (you can probably begin to detect the direction of the Mozilla Foundation's code name strategy at work here). SeaMonkey provides the code interface layer that makes it possible for us ordinary mortals to write applications, and even to take over the Mozilla browser in order to create our own. SeaMonkey exposes an XML language called the XML User-interface Language (or XUL) that provides a set of building blocks that control various components - textboxes, formatting boxes, status bars, menus, lists, trees, and so forth, along with abstractions for creating key bindings, event observers and referential commands. This set is fairly rich (there are more than one hundred such tags), but it can also be extended with the HTML element set (useful for creating formatted markup within applications) and will further be augmented with the SVG tag-set by March 2005, and XForms by early 2006.

It is possible to put together applications with nothing but XUL, but they are generally trivial applications at best. As with any other application framework, the structural elements usually need to be bound together with some code of procedural code. SeaMonkey borrowed a page from Internet Explorer here (as well as .NET) - rather than building one language inextricably into the interface, SeaMonkey breaks the process up into two distinct technologies - XPCOM and XPConnect. XPCOM performs the same role for Mozilla that COM does for pre-.NET windows applications - it queries and binds object interfaces and makes them available for other coding applications to utilize. This cuts down on the requirement of maintaining a static-API, and provides a vehicle for writing binary extensions as XPCOM objects. While the two layers are not identical, there is enough similarity between XPCOM and COM that an ActiveX container for Mozilla should soon be supported, making it possible for Firefox applications to run ActiveX controls while at the same time providing a layer of security that prevents them from being the threat they've become under Internet Explorer.

To get around coding a specific language to SeaMonkey, XPCOM is designed to be accessed through XPConnect, a binding layer that maps XPCOM to a specific language's interfaces. Currently the primary such language is Javascript 1.5, though plans are in the work to incorporate Javascript 2.0 once that language goes through its final development phase and is approved by the ECMA (a body, incidentally, that has quietly become the de facto holder in trust of programming languages in general). I've covered some of the features of Javascript 1.5 before, including the use of setters and getters, robust regular expression support, the use of constants, and multiple try-catch statement support. However, bindings for other languages, including Python and Perl, are available, and a much more complete Java binder is also under development. Because of the open nature of Mozilla, I would not be at all surprised to see a C# implementation in the near future as well.

The list of XPCOM objects is quite impressive. A partial list includes the following functionality:

Core Functionality (See below)
Accessibility Components
Address Book Support
Clipboard and Selection
Content and Layout Managers
Cookies
HTML and XML DOM Support
HTML Editors
File and Stream Interfaces
Graphics Creation and Manipulation
Interprocess Communication (IPC)
LDAP
Localization
Mail Support
Network Support (Sockets, et al)
News Support
Preferences Objects
Security
Web Browser control
Web Services (SOAP/WSDL based)
Window Management
XML Support (Schema, XSLT, XPath)
XUL

The Core functionality provides a number of useful data structures (including dictionaries, arrays, property bags and enumerations) and language type support,along with threading libraries (and pools), timers, event resources, and exception management. While some of these are not necessarily that useful in Javascript, they do have definite utility in other languages such as C++. The graphics library includes interfaces for actually drawing on surfaces within the various objects, though accessing these services can be a little convoluted. The mail, LDAP and news support point out a subtle but important fact about Firefox and Thunderbird - they are simply applications that both sit on the same API - meaning that you could in fact build integrated mail services directly into Firefox if you wanted to.

XPCOM exposes these services and objects via a contract ID, something analogous to the classid used by Microsoft tools. The following, for instance, illustrates how you could create a new local File object:

var file = Components.classes["@mozilla.org/file/local;1"].

    createInstance(Components.interfaces.nsILocalFile);

The first part of the expression,

Components.classes["@mozilla.org/file/local;1"]

creates a reference to the local file class defined by the contract ID, "@mozilla.org/file/local;1". This is a class reference, not an instance reference (it points to a particular class definition, rather than one specific instance of the class). The createInstance() function in turn creates an instance of this object, using the Components.interfaces.nsILocalFile interface to expose that particular interface on the instance. A given object may conceivably have more than one interface; this code makes it easier (and more cost efficient in terms of computing) to get the specific interface properties. Once this object is retrieved, you can use its properties and methods in exactly the same manner you would do so in any other language.

The final piece of the SeaMonkey language is the XML Binding Language (or XBL). This XML-based language provides a transformation mechanism that will take user-defined tags written in XUL files and convert them into an internal XUL representation, complete with properties, methods, and event hooks. XBL provides a way of creating more sophisticated elements, and is in fact used within XUL itself for the definition of things such as tab-browsers, which combine tab boxes and browsers into a single component.

A very simple XBL file, one that builds a box with OK and Cancel buttons, might look something like this:


XUL (example.xul):



<?xml version="1.0"?>

<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>

<?xml-stylesheet href="chrome://example/skin/example.css" type="text/css"?>



<window

   xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">

 <box class="okcancelbuttons">

</window>



CSS (example.css):



box.okcancelbuttons {

   -moz-binding: url('chrome://example/skin/example.xml#okcancel');

}



XBL (example.xml):



<?xml version="1.0"?>

<bindings xmlns="http://www.mozilla.org/xbl"

         xmlns:xul="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">

 <binding id="okcancel">

   <content>

     <xul:button label="OK">

     <xul:button label="Cancel">

   </content>

 </binding>

</bindings>

The XUL file creates a reference to a CSS file, while in turn uses CSS selector and rule syntax for defining the bindings between a given class (okcancelbuttons) and an XBL file and the associated "okcancel" binding item. Real XBL can become much more complex than this, of course, but this is a topic for a different article.

As expected, SeaMonkey also handles the bindings between CSS and the XUL applications, with XUL heavily utilizing CSS not just for simple "styling" but for the actual creation of complex components through XBL. The CSS support that exists as a consequence is VERY impressive, including certain features that have been floating for a while, such as support for multiple flow columns.

The final piece of any XUL application is the use of overlays. An overlay is a XUL file that changes the XUL (and associated scripts) of a given application. By overwriting or extending (as a form of inheritance) you can do such things as create overlays on Firefox or Thunderbird itself. I do this myself to override the default load and save menu items and replace them with my own, making it possible for me to save to a custom XML schema and load from that schema later.

Firefox is an example of all of these principles in action, by the way. If you have Firefox running on your system (and maintain the Java SDK on your system), create a copy of the browser.jar file located in the chrome directory of your Firefox distribution somewhere outside of the firefox application folder. You can use the Jar file extractor from the SDK to convert this into a directory:

jar xvf browser.jar

This will create a folder called content, which in turn will hold the various XUL, XML, and CSS files for the Firefox browser. It is worth spending the time looking at these closely. One of the things you should realize pretty quickly on is that almost all of Firefox is contained within these XUL files, not in some form of C++ application, and that a significant portion of the coding for Firefox is handled by Javascript.

Firefox is remarkable to me in that it is one of a new breed of applications, built around an XML interface and scripting yet fully capable of handling some of the most serious challenges that any "formal" application written in C++ or even Java can handle. It is also eminently accessible, in a way that a lot of other applications aren't. It is this model, as much as any widgets or features of the Firefox application, that is the real story here.

Implications

I'm looking at the November 22, 2004 issue of eWeek on my desk at an article entitled Browser wars back on. I think that sums it up pretty well. Firefox is not just a shot across the bow to Microsoft -- stealing 5% of market share is much like taking out the yard-arm on your ship with that cannon-shot. A major portion of Microsoft's control over that market share has come from the fact that you could access other components from within it, turning the Internet Explorer shell into a general purpose shell for hosting any kind of application. No other browser out there has really managed to pull it off and still be able to maintain its quality as a browser.

Firefox opens up that possibility. It's not that much of a stretch to envision Open Office creating wrappers around their UNO wrapper class to make them work within Firefox, and as the editor application I've written myself illustrates, you can actually go a long way toward building commercially viable enterprise-level applications just using the core components from Firefox. The addition of SVG support and XForms provides another point of attack against both Power Point and InfoPath, and its not hard to envision data-access tools appearing in the next year (perhaps powered by XQuery?) that will give Access a run for its money. Such applications could run on multiple platforms with little or no modification, would be a menu-item away from normal browsing (and could easily run in one browser tab while you maintain your mail in a second tab and surf the web in a third).

None of this will happen overnight, of course, but its easy enough to see the general trendline. Already it is prompting Microsoft to come back with a number of new extensions and innovations on its own browser, though in most cases these extensions still rely upon the existing ActiveX architecture. The biggest danger that Microsoft faces from this comes in its tendency to pick and choose which standards it chooses to comply to; a truly standards compliant development system is likely to be far more politically attractive than one that is closed and proprietary, especially where it counts -- not in the big enterprise settings where the adoption of any new technology usually takes place only after such a technology has become very settled but in the spare bedrooms and coffeehouses and garage work-stations of the individual developers who are the ones who are learning (and in many cases developing) the technology of the future. For them, Firefox and the Mozilla Application Suite represents a huge step forward, one that will have reverberations for the next decade and beyond.

2004-11-28T12:17:00.000-08:00

This was sent to me by a friend, and I thought it would help put into perspective just exactly how far we have come in fifty years. No word yet on the laptop model.

Jazz

2004-11-18T23:07:00.000-08:00

Most of us, the people for whom the Metaphorical Web makes any sense whatsoever, are geeks. Not so long ago such a term was used with the greatest of derision, an open reference to carnival sideshow geeks - people who were bizarre or did bizarre things. I was a geek, the high school kid that would sit in the computer room (and former storage closet) writing programs on the Apple IIe there while everyone else was at lunch or chilling out during recess. I was the kid that would inevitably be picked at the tail end of whatever sports team match we were mandated to play, because everyone knew I had neither skill nor interest in team sports.

Somewhere along the line, being nerdy gained its fifteen minutes of fame, in large part because one of the most stereotypically nerdy people ever had also managed to become the wealthiest man in the world, and for a while, being the odd man out was in. Then the Technical Nuclear Winter hit, and the football team captains and the marketing types were once again in the driver's seat, laughing at how these stupid tech guys had been taken with stock options that were as empty as vaporware and SCO legal threats.

For the younger kids (and even for many of us who are beginning to see gray in our beards) the viciousness of this turnaround was stunning - going from being able to afford expensive houses to being considered one of the lucky ones if you had parents who could put you up in their spare bedroom proved a powerful blow to a lot of people, shaking them up and making them realize how truly random such wealth could be -- and how duplicitous other people could be, if they thought that they could use greed to control you.

I've been through this cycle a couple of times before - I entered into the programming marketplace in the mid-1980s, when career counselors were advising people not to go into computers, there was no future in it, and again in the multimedia bust of 1994, before the Internet really began to heat up. The tech field's like that ... sometimes you ride the big waves, and sometimes it's better just to take your board out of the water and spend the time waxing and repairing it, because the surf is about as flat as it can be.

Curiously enough, though, the real innovations that occur in the field don't occur when the economy is red-hot and the potential for making money is strongest - in fact that's usually the worst time. Your judgement becomes clouded because you're not asking the question "does this solve a real need?" but rather "can this make me rich?". No, the real breakthroughs come when you're sitting at home, chatting with friends via IRC (IM for you newbies) or email lists, playing with ideas or flaming away the dross, putting together something just to see if it can be done. There's a new toolkit I want to play with, there's an idea I saw that I think could work here as well, we need to figure out where the holes are in this specification, because they're causing real interoperability problems.

A surprising number of programmers are also musicians, though not necessarily world class ones. Part of the music/programming association, I suspect, has to do with the analytical nature of music, but a bigger part is that a musician is in his or her own way also a technician, someone who is interested less in the money than in making their tools do something really cool. The process of innovation has reminded me more than once of an extended jam session, continuous improvisation off a theme. Any musician knows that not all such jam sessions produce great music -- often what they produce is just noise, and the musicians just shake their heads and agree to meet again next week. Sometimes, though, everything clicks, everyone finds themselves in the groove, and before you know it the music ends in the wee hours of the morning with the participants exhausted but happy (and I"m deliberately avoiding another obvious metaphor here).

Lately I find that the meaningful work that I'm doing is not coming from the 9 to 5 grind, the thing that keeps the roof over my family's heads. It comes despite it, in the interstices, through the improvisational conversations that we all seem to be engaged in. There are some profound things shaping up in the software field right now ... things that have occurred not because a CEO somewhere had a grand initiative to add another billion dollars to the bottom line but because, in coffeehouses and pizza parlors and IRC chats and e-mails, people have been playing the music of innovation and inspiration, of trying to build something because it needs to be built, profit be damned.

The cycle is shifting yet again, the momentum building, the ideas exchanged at two in the morning at Starbuks going to make the next BIG THING. No doubt the former jocks and marketing types will begin to circle soon, sensing the potential for making money off these things that they did not create, were not a part of. I suspect they may discover that smart people can be fooled once, but that smart people by definition also learn very, very quickly. Yet I also pity these people, the ones that once reviled us by calling us geeks, for I suspect that deep down they have no music in their souls, that they will never know the real joy of creation.

No code today, though I promise some tasty morsels soon. Until then ... enjoy!

They're Movin' and Shakin' at the W3C

2004-11-11T16:52:00.000-08:00

<metaphoricalweb>

Thanks for the many responses I've had to my post yesterday concerning Canada - some very good thoughts and suggestions, including the primary things one needs to remember when in Canada:

"Moving to canada is no big deal, as long as you learn that re is better than er and eh is a verb, noun, and adjective all in one."

Thanks, Ronan.

The political statement out of the way, its time I got back to why I write this column in the first place ... GOSSIP! Er..., no, XML. Same difference.

The XML Bernie Botts

Watching the W3C at work is very much like shaking out Bertie Bott's Jelly Beans (the kind from Harry Potter that my 11 year old daughter so loves torturing her old father with). Sometimes the specs released by the W3C are very good - a vanilla or taffy flavored jelly bean that's surprisingly delicious. Sometimes the specs are more like "vomit" or "dirt", though with annotations.

The last couple of weeks I'd have to say there were a delightfully high number of blueberry delight and banana split and the worst to come out was still on the order of "grass" - edible, just not all that exciting. We're past the OWL specifications (there's another Harry Potter column coming on if you're not very careful) which quite frankly read like Noam Chomsky had just engaged in a serious argument with Richard Feynman. You had a drought where for a while all that was brewing were small specs like Accessibility, important in their own right but like listening to a lecture about going on dates from your spinster great aunt. Just when I was beginning to despair that the entire friggin' box was filled with "library paste" jelly beans, along came a bunch of very tasty treats indeed.

9 November 2004: XML Binary Characterization Use Cases
9 November 2004: xml:id Version 1.0 - Last Call Ends 13 December 2004
8 November 2004: Semantic Interpretation for Speech Recognition - Last Call Ends 5 December 2004
5 November 2004: XSL Transformations (XSLT) Version 2.0
2 November 2004: Assigning Media Types to Binary Data in XML - Last Call Ends 24 November 2004
1 November 2004: Timed Text (TT) Authoring Format 1.0 – Distribution Format Exchange Profile (DFXP)
29 October 2004: Pronunciation Lexicon Specification (PLS) Version 1.0 Requirements
29 October 2004: XQuery 1.0 and XPath 2.0 Data Model
29 October 2004: XQuery 1.0 and XPath 2.0 Functions and Operators
29 October 2004: XML Path Language (XPath) 2.0
29 October 2004: XQuery 1.0: An XML Query Language
29 October 2004: XSLT 2.0 and XQuery 1.0 Serialization
27 October 2004: Scalable Vector Graphics (SVG) 1.2 - Last Call Ends 24 November 2004

This was just picked up from the Recently Published Working Draft Section for the last two weeks. To quote my four year old daughter "This is AWESOME!"

Even more is hidden in the details here. For instance, I note that in the XSLT Serialization specification, there's not one editor but three: Michael Kay (Saxonica), Norman Walsh (Sun), Henry Zongaro (IBM). A lot of my job entails reading behind the lines and trying to understand the significance, but to me this is a pretty strong indication that both Sun and IBM are watching XSLT2 VERY carefully; you don't put a powerhouse hitter like Norman Walsh (the author of the DocBook specification) on a specification unless you think it'll be important.

That you'd need a specification just for the serialization aspect of XSLT2 may seem a little odd as well, until you understand that one of the key new features in that specification is the <result-document> element. This makes it possible for an XSLT2 document to generate more than one output. The most obvious uses of this capability is to use XSLT2 to generate secondary XML documents, but much like input and output streams this is just one of many areas. Generation of SOAP messages, creation of non-XML source code from XML documentation files (especially with many of the other new features in that specification), and rerouting of messages to databases all fall within the province of this new feature. Combine that with the elimination of the tree fragment such that intermediate XML can be created within a transformation and manipulated directly (no more node-set() function!), regular expression support, and the ability to import text, and XSLT2 is beginning to look pretty damn brawny.

Serialization gives you the ability to do things such as to set the content encoding of a document, making it much easier to handle Unicode UTF-16 encoding. The specification also introduces character mappings, which gives you a way of avoiding the need to incorporate DTDs within source code (or the transformations themselves) in order to define entities. While there are some features of DTDs which continue to be useful, XML is slowly losing its reliance upon them.

The XPath 2.0 specification is similarly an important document, and the news on this one was perhaps not quite as sweet as I'd hoped. XPath is a major specification because it underlies not only XSLT but XQuery, XForms, and SVG's xXBL may have some dependencies in its next draft (see below). The last specification was labelled Last Call, which means that it is usually considered one step before a Proposed Draft -- it is an opportunity to put the specification in front of people and get feedback. There was a LOT of feedback this time around, enough that after some consideration, XPath has been taken out of Last Call Status and is once again strictly a working draft.

The issues that have necessitated this are fairly broad, due in part to the need to more properly specify the character model (part of the serialization above) and in part to rectify some of the thornier issues with type conversions. Another common feeling (one that I personally agree with) is the huge number of date-time functions and the feeling that many of these are redundant or readily derivable, especially given the extension mechanisms that exist in both XQuery and XSLT. There are also some indication that one of the original goals of XPath 2 - providing data aware type objects, has receded considerably as people have thought more about what specifically the specification was intended to accomplish. Finally, collations, which affect string order, have been put on the table as things which need to be more clearly defined before the specification can be published.

Regardless, this means that most of the specifications involved will likely be in limbo for at least another six months because of this delay. The one positive aspect about this is the fact that the specifications are not likely to change considerably, so that so long as you stay clear of the more problematic functionality, you can actually use the beta XSLT2 processors that are beginning to surface with some reliability.

SVG 1.2 Goes Into Last Call

After the last paragraph, I'll be cautious about saying too much about this one, though because it is for me a little closer to home I've been watching this particular battle rage for the last couple of weeks. SVG 1.2 is, to put it bluntly, what SVG should have been four years ago:

SVG1.2 defines a mechanism for dealing with wrapping text in paragraphs, and does so in a spectacular fashion by making it possible to flow text through irregular shapes, and from one shape to another.
SVG1.2 defines video and audio tags for creating robust multimedia.
SVG1.2 includes a binding mechanism called sXBL (SVG XML Binding Language) that makes it possible to use XML to build complex components.
SVG1.2 includes an editable attribute to make it possible to change text on the fly.
SVG1.2 incorporates a mechanism to create filter vector effects.
SVG1.2 includes sockets and HTTP support for building distributed applications.

All in all, it puts together enough substance to make it into a serious graphics substrate layer, with the one lone difference between it and a language such as XAML being that it does not yet address 3D. Of course, given that the next version of SVG will be version 1."3", I'd say this isn't that big of a problem.

The SVG 1.1. specification was revisionary - it clarified a few features that were not completely specified in 1.0. SVG 1.2 on the other hand is meant more as an addendum, shuffling new chapters into the thing that we call SVG. It is also a very controversial specification, as SVG is now beginning to seriously encroach into other areas such as HTML and XSL-FO. This is not so much true in terms of the specifications - HTML is a logical description of a web page while SVG is a lower level graphics description language - but when you combine SVG with sXBL you have what amounts to a mechanism by which you can use SVG to render that HTML in a pretty fair representation of a web page. In this way what SVG serves to do is replace not so much the specifications, but at least SOME functionality of the browsers themselves.

X.Org Does SVG

This has some very interesting implications, especially in conjunction with what is happening in Linux. A year ago, the XFree86 organization, the group responsible for the X graphical system on Unix and later Linux, split on the basis of changes desired by certain members to make the license non-GPL compliant. The GPL faction formed a new organization, X.org, and immediately set out to solve another problem with XFree86 - the glacially slow development pace of XFree. XFree86 is ancient technology - the first X implementation was created in 1984, twenty years ago, and although the framework was remarkably flexible there were several places in the underlying graphics model that were proving remarkably constrictive.

X.org used this break as a chance to rebuild much of the more troublesome aspects of X. While this effort is still ongoing, already there are a number of new features to X which are making developers salivate:

Moving from a 24 bit to a 32 bit color space - the RGBA space - to provide 8 bits (256 levels) of alpha channel support. This makes it possible to push transparency directly into the chips, whereas before transformation had to be done much higher up the stack at the application level, resulting in far poorer performance.
Establishing much finer invalid region support into the graphics layer itself, resulting in much faster screen refreshes - an absolute must for animation.
More tightly integrating the 2D and 3D engines used by unix based systems, a critical feature for the Java Glass window initiative (and X3D).
Creating more robust events that can be captured at the X layer, rather than needing to be placed up in the application stack.
Incorporating hooks for SVG integration (yeah!!).

One implication of this is that SVG is bidding to become the equivalent to GDI under Microsoft Windows - in a role with Linux that'll compete directly with the XAML graphical layer under .NET. As SVG 1.2 will likely be a full specification about the time that most of the new X.org begins making its way into commercial applications, this implies the possibility that SVG 1.2 could become pervasive under Linux.

New Compound Document Formats Group Formed
One of the more intriguing ideas behind many of the presentation oriented specifications is the concept that you can embed SVG and MathML code within XHTML, possibly combining it with XForms content or other related specifications. Some browsers are already doing this, but when theory hits reality, inconsistencies between models can cause more headaches than expected.

In mid-October, the W3C announced the formation of the Compound Document Formats Working Group (CDF) specifically to provide ways to smooth that integration. The formation of this group sends a signal that the W3C is shifting from the development of base technologies specifications and to the interchange between the various specs, something which has needed to happen for a while. I suspect that interoperability will be the name of the game for the next couple of years, especially once the big specifications like XSLT2 and XPath2 finally go golden.

There has been a tendency in the past for specifications to occasionally develop potentially competing and certainly conflicting aspects with other specifications issued by the W3C, meaning that if you wished to use on technology you might potentially be locked out of using another, even though both are conformant with the W3C process. By getting this resolved, user agents of the future will be considerably more integrated with all of the W3C specifications, and the notion that all of these objects can effectively play within the same DOM-space is particularly exciting (your HTML controls the MathML, which in turn controls the SVG, populates the XForm data, and passes the results up to a pipe using the XML DOM).

Wrap-up
I've been busy on other projects and am beginning to wrap those up, freeing up more time for Metaphorical Web columns - so expect the activity to pick up again here. Is there something that your company or organization is doing in the XML space that you think is worth shouting about? If there is, contact me at kurt@kurtcagle.net with the inside scoop. Until next time, enjoy!

</metaphoricalweb>

Canada Bound?

2004-11-10T10:50:00.000-08:00

The last couple of weeks have been a time of exhiliration and heartbreak, and have forced me to seriously examine what my feelings are about the US and my relationship to it. I've started this same blog four different times now, each time trying to frame what I wanted to say. In the interests of getting back to the task at hand - discussing the world of XML, it seems best to make only a statement on what has happened.

On November 2, 2004, the two party system in the United States effectively collapsed. One party, a party that it notoriously hostile to civil rights, remarkably ignorant about technology, inclined to view such technology principally as a means either to wage war or control the populace, now controls all of the political organs of our state - the executive branch, the legislative branch, the legal branch, the military and intelligence services, and an overwhelming majority of all governorships, and to a great extent the media.

This party is not the GOP - though they were the agents through which this takeover was affected. The people who wield the power within this organization are students of a particularly virulent brand of nationalism that was last adopted by the German National Socialists, the Nazis ... as they came to be known. It is a philosophy of empire, of hegemony, of conquest for the sake of power and control, a philosophy that has learned to cloak itself in Madison Avenue friendly soundbites and wrap itself in the mantles of the cross and the flag, but it is friendly neither to people strong in their faith nor strong in their patriotism.

I voted for Kerry, not because I found that his ideas would have radically improved things, but because the party that Bush represents is making things far worse. Many of you, no doubt, will disagree with me ... and I welcome such disagreements - that is the spirit of honest debate. However, there are many indications that such honest debate will become much rarer, as will the spirit of innovation which is so critical to this field, as we become a nation under surveillance, so fixated on the enemy without that we do not see the enemy within.

A democracy needs two things to survive: a system of checks and balances and free and fair elections. We no longer have the first - the people in power can make pronouncements and judgements without the threat of what would in the academic community be called peer review - the ability to provide a second set of eyes to determine whether a given law is really good or whether it simply represents the desires of the person who makes the law for personal gain or aggrandizement.

In programming circles, the equivalent concept is code review. The Open Source Software community understands the value of this - in many ways OSS is essentially code review write large. No company worth its salt should ever accept software that's passed directly from the developer to the company at large (or their customers/clients) without at least two more sets of eyes looking at it; companies that fail to do this will not stay in business for long. The analogy holds true in government as well.

Were the elections free and fair? I personally do not believe so -- there are simply too many abberations at all levels, some due to human intervention, some due to machines either badly or maliciously programmed, and an overwhelming number of them favor the Republican party. Again, in statistics, a sufficiently large number of random errors will be distributed along a Gaussian Bell curve, such that errors usually tend to cancel out, not heavily favor one candidate or another. This will not change who gets inaugurated in January, but it does bode ill for the faith that one can place in the election process, not just now, but for the foreseeable future. Given that this faith is the cornerstone of democracy, the belief that each person within the society has a vote of equal value to all other people in that society, to compromise it to this degree begs the question of whether the United States is in fact a democracy anymore.

Many people in IT tend to be libertarian, and are either passionate about politics or supremely indifferent about it -- there is no middle ground. Thus to talk about it in a programming forum is often considered to be in bad taste. However, to those who see no relationship between the two, consider that with this election, several things are now set in motion that will have an impact upon both IT managers and programmers:

Enforcement of such legislation as the Digital Millennium Copyright Act will be tied much more closely with federal agencies, with the strong potential to restrict the free flow of all information (throttling the Internet) in the name of chasing pirates.
A concerted effort will likely be made on the part of several corporations to declare the Gnu Public License (the GPL) invalid, possibly along either DMCA or anti-competitive means. Patent cases under this administration have been settled overwhelmingly in favor of the largest busines interest ... expect this trend to continue.
There is already something of a brain drain going on as academics and high end developers leave the country, fearful of what they see already coming down the road and frustrated as universities become simply extensions of corporations because federal funding is drying up. Additionally, as corporations in the US have become leaner and meaner, they have typically slashed or eliminated outright their key R&D centers, expecting that they can buy the innovation on the open market. I see this assumption as being facile and short-sighted.
There is an increasing reluctance on the part of world standards bodies to want to deal with American corporations in light of political and military actions of late. I've been to a number of standards conferences in the last year or so, some in the US, some outside of it, and find that there are fewer and fewer Americans drawn into them. The heighened military posture of the US has also meant that certain American companies, such as Microsoft, are increasingly being seen as proxies for American intervention, and are losing business in Europe, Asia, and South America because of it. I personally don't think this belief is necessarily valid, but there is definitely a backlash building against US products because of that perception, with software being a major part of that.
I'm keeping my focus here strictly on what I see as the most conservative impacts that the new administration will have solely upon the IT sector. I think there will be many bigger effects, including a fairly severe recession by mid-to-late 2005 which will also affect the industry (though not as badly as the 2000-2003 Nuclear Winter did), but there are too many unknowns to make such guesses with any certainty.

Those of you who are long time readers of Metaphorical Web may remember my rhapsody on themes Canadian after the 2003 SVG Open conference. While the decision is yet to be made, I am leaning strongly toward emigrating there, to get involved with what I see as the vibrant open standards scene in Vancouver. The city has become a magnet for XML technologies, drawn by such luminaries as Tim Bray, Philip Mansfield, Paul Prescod, Ron DeSerrano and many others. It's also becoming a hub for software entertainment, a field that to date has been only peripherally influenced by XML, but which is probably due to be the next frontier for that particular set of standards.

Moving one's family to a new country (even one only a few hours drive away) is always a momentous decision; moving because your own country is undergoing what could be a dark and dangerous transformation is heartbreaking, because you have to struggle with the question of whether staying would make a difference. It is a decision I am still weighing, and would gladly welcome comments from others on both sides of the divide.

Conferences and Google

2004-10-24T23:16:00.000-07:00

The joy of being a consultant is that frequently you end up being very, very busy. The downside is that you frequently end up very, very busy, and the other things that you are working on tend to get short-shrift. I have been crazy busy, good for the pocket-book, but bad for things such as Metaphorical Web. I'm taking a brief break here though to catch up on the web and post what I've been up to the last couple of weeks.

Chris Sells' Applied XML Development Conference

I was invited to Chris Sells' Applied XML Development Conference, considered by many to be one of the best conferences for covering advanced XML techniques (the other being the Extreme XML conferences), to present a paper on XSLT2. Others will have blogged this conference much better than myself, but I do have a few notes to present as well.

Geek Dinner

Chris Sells put on a good conference, something that isn't always easy (as a habitual conference speaker, something analogous to a drug addict but with presentation tools as the opiate of choice, I should know). It was definitely a Microsoft-oriented event, and I do have to admit that I found the Microsoft propaganda at the show tedious and heavy handed, but at the same time I knew going in that it was likely not to be an agnostic crowd.

The accomodations, the Skamania Lodge in Stevenson, Washington, were remote (about 40 minutes from Portland) but otherwise quite spectacular, showing off southern Washington State at its best with scenic vistas of mountains and the Columbia Gorge. The lodge was the epitome of rustic - I keep wondering whether there is a "rustic" design specialty out there, with classes in how to properly design fireplaces and log walls. I drove down with M. David Peterson, the developer of the Saxon.NET XSLT2 processor and my co-conspirator in crime in putting together the presentation, and we spent the three hour trip down riffing on the nature of meta-data and models.

We didn't go directly to the lodge, however, instead making our way to downtown Portland for the Geek's Dinner. This has apparently become a time honored process at the Sells Cons, in which many of the attendees showed up at a dinner hosted by Don Demsiak (DonXML) at an appropriate venue - in this case the food court at Lloyd Center Mall. While most of the Microsoft big-players didn't show up, just about everyone else did. It was fun, we got a chance to just chat and make jokes before going into the more formal venue of the lodge itself.

Once we got to the lodge and settled ourselves, I went down to the bar and had the privilege of having a beer with Tim Bray, the main architect of XML and the driving force right now of the Atom specification for syndication. While I've met Tim before, this was the first time that I had a chance to really talk with him, and we covered everything from SVG and XUL to the best micro-breweries on the West Coast (he recommends the Yaletown Brewing Company, just down the way from his offices in the Yaletown district of Vancouver). I had a lot of respect for Mr. Bray before I met him, and after this conference my respect has gone up immeasurably.

Blue Pill, Red Pill, Purple Pill

The conference itself got underway around nine with Tim giving the keynote address. Not unexpectedly, it was about blogs and blogging (and the exponential growth of same) but he did manage to get in more than a few digs about a number of standards that were in the works, spread his frustrations with Microsoft and the W3C evenly, and also indicated that the world was not just Microsoft. This was a challenge to subsequent speakers, many of whom were talking the MS line.

Chris Anderson, a key Avalon developer at Microsoft, spoke next about Everyone Hates XML, a play no doubt on "Everyone Loves Raymond" but that oddly continued a theme that was both a little depressing and fairly antagonistic throughout the conference. Anderson's talk actually focused more on XAML and the design decisions used in its development. I'll have to confess that some of the examples that he used, such as

<button id="foo" value="Button">
<button.background>
<solidcolor value="Blue">
</solidcolor>
</button.background>
</button>

(note, may not be completely valid, as I don't have the example handy)

rub me very much the wrong way, as it effectively places stylistic attributes into a position where they effectively create a potentially huge combinatorial set of elements; its not validatable, and as a consequence requires having a very complex runtime in the background to handle. This very quickly jumped into a free-for-all with the audience asking about what exactly was a tag, and I noticed after a while that it was only the Microsoft regulars who were trying to push it into a non-issue. A prediction - the lack of XML validation will prove to be the undoing of XAML in the long run. The talk was othewise quite interesting, however.

Patrick Cauldwell and Scott Hanselman presented Bringing Strongly Typed Business Objects to Legacy Financial Systems with XML Schema, a mouthful which nonetheless proved to be a useful talk. One of the central ideas in their work was the use of XML Schemas to generate Word documents describing a financial system, something which could in turn be used as the basis of legal documents. This is a very interesting idea, one that I've actually pushed before myself. To whit, a legal document serves two purposes -- defining the language (services) upon which two or more parties agree to specify the terms of contract, and the penalties for failure to complete these services on either side. A schema in that regard can effectively perform at least one part of that process, stipulating the terms of the agreement. As a consequence, I suspect that over time, schema documents will begin to take on a quasi-legal status.

Don Box and I have sparred in the past, though I also took advantage of what he thought was a "safe" venue to make some observations about the WS-* initiatives, then in their infancy, that he felt (with some legitimacy) to be unfair. Don's grown more confident in his own role at Microsoft, and while I'm not completely wild still about the WS-* concepts I will concede that they have a certain utility. His talk WS-Why? went over many of these specifications, using a (somewhat strained) island metaphor to go over the latest Microsoft initiatives in this area. My biggest gripe came from his tendency to talk around the purposes of some of the initiatives without providing a little better explanation of what they are intended to be used for - I think even many who were ardent Microsoft proponents in the audience felt a little lost on this. I did note with amusement that UDDI had been relegated to the status of an "oops" kind of idea, one that had some lofty goals but bad implementation, citing many of the same reasons that I've had for disliking that standard (the overbearing use of "business metaphors" for instance). For all that, the talk was a good one, and while I didn't get a chance to talk with him in detail, I do hope to do so in more favorable circumstances in the future.

Then there was the absolute, no question greatest talk of the entire conference: Using XML for Navy Missile Systems, by Whit Kemmey of the Department of Defense. Nuclear missiles. Many submarine shots. Discussions about coding practices aboard ultra-secure environments. The Hunt for Red October could only wish it had been this cool. Whit was very definitely a military programmer, very clean cut, the only one in the entire auditorium wearing a suit, but for all of that he had a wry and deadpan sense of humor that caught some of the brightest programming minds on the planet off-guard more than once. Some favorite quotes:

Im a trenches guy. Im not a vendor. This stuff isnt for sale.
Our software has never been used for its intended purpose, and hopefully never will be.
Localization is not a big problem for us.

The Navy apparently uses a customized Unix solution on hardware that is running about five years behind the curve, largely because of the need to stress-test everything to insure that there are no unexpected gotchas (I'll leave it to you, gentle reader, to figure out the consequences of a system crash on a boatload of nuclear weapons). XML was used here to handle the generation and implementation of Standard Operating Procedures (SOPs) using LibXML as the central processor. There was a certain amount of astonishment at the way that it was being used by many of the Windows-oriented type, though the descriptions for actions used made sense in terms of needing to specify in great detail every single step of a process with such incredible consequences of something going wrong.

I did find it a little irritating at the tone used by more than a few of the commercial XML developers, a kind of mocking astonishment about the process of software development in the military, but having been in the Navy myself in the 1980s, I have to admit that more than a few of these people could have afforded to spend some time in the service to understand what writing mission critical software is REALLY about. When lives could be lost due to a software error, conscientiousness in coding is not only a nice to have but a necessity, a lesson that a few of those writing operating systems would be wise to consider.

Sam Ruby is an unheralded genius. Working for IBM, he also serves helping to put together the Atom specification, and is involved with both the KDE and Gnome groups. He was also easily one of the most depressing speakers there, chastising the crowd for the dangers inherent in many of the problems associated with web practices, from Unicode violations and errors to URL mis-encryptions in his talk XML is an Attractive Nuisance. These were all things that he's had to deal with when coding for Atom, but in the process of giving the talk he kept pointing out fundamental problems about the Internet itself. He did try to use the Matrix metaphor, though the relative disasters of the latter two movies ment that his demos didn't have quite the oomph I think he would have liked. I enjoyed the talk, and managed to identify many (though not all) of the erroneous examples he provided, but given his Matrix references I think more than a few of use were wondering not whether we should take the blue pill or the red pill, but who had a stash of The Purple Pill (Paxil, a commonly prescribed anti-depressant).

Daniel Cuzzolino (evangelist for Schematron) not surprisingly did a talk All About Schematron. For those of you unfamiliar with it, Schematron is a schema validation tool that uses XSLT and XPath to provide more sophisticated validation than can be handled with XML Schema Definition (XSD). Significantly, he brought up a number of features with Schematron that he wanted to be able to incorporate, but couldn't such as the use of regular expressions in his validation. After I gave my talk, we personally discussed Schematron in much more detail with an eye toward using XSLT2 for handling the next generation of the application.

My talk was next, and I will be discussing it in much more detail in the next issue of this blog. I'd put it at a middling success - I had too much material and being immediately before dinner, the attendees were more than a little distracted. If you can avoid it, try not to compete with food - empty stomachs make for wandering minds.

The Case Against XSD

After dinner, the speakers all headed up to the front of the dining hall for a round-table, and almost out of the chute the topic was the inadequacy of the XML Schema Definition Language (XSD).

A little background is in order here. While most of the W3C standards have had a certain degree of controversy associated with them, far and away the most controversial of the specifications has been XSD. It was begun about the same time that the XML standard itself was in its earliest stages, but even given that it took more than five years of very contentious meetings to finally agree to it, with the most vocal proponents being the database vendors of Microsoft, Oracle, and IBM. The difficulty is not in the simple type definitions - how one defines primitives such as integers, floating point numbers, and so forth. Instead, the challenges have come in defining more complex types, such as elements with subordinate children in them. There is a lot of ambiguity in the specification, especially when it comes to these complex data types, and what makes matters worse is that the underlying data model of XSD favors a one tag = one object model that is increasingly being shown to be inaccurate in describing data models.

A subtle revolution is going on, and parts of it emerged in this conference. While there are many (especially those vendors) who want to declare the matter closed and have effectively turned a deaf ear to the plea for re-examining schemas, the most promising alternative candidate, Relax-NG (also given as RNG) has quietly been showing up in all sorts of interesting places. For instance, last year the SVG 1.2 specification was published using RNG and not XSD as the schema. Tools such as Oxygen are making RNG available as their primary validation scheme, and content management engineers, who have known for sometime the deficiency inherent in XSD, have been switching over DTDs to RNG and bypassing the XSD spec altogether.

Consequently, it was interesting to hear the number of people who began by saying that they didn't like XSD but would support it because its already baked into so many other specifications. In the .NET world this is probably true - Microsoft has taken to schemas with a vengeance, perhaps too much so in some respects, even though there are certain core technologies such as XAML that would be much more accurately described (with less ambiguity) in RNG. The Relax-NG schema works at a little lower level than schema does, and is more descriptive of contents; right now it is facing a VHS vs. Beta type struggle, but I don't think it's necessarily a good idea to write it off at this stage.

The rancor which this topic brought up raises a more subtle spectre that vendors especially should be mindful of - is it possible that people are beginning to react to XML technologies that they feel were forced down their throat by bypassing those XML structures in favor of more ad-hoc ones? There is little in the way of true empirical evidence, but there is a lot of anecdotal evidence that suggests this. SOAP has definitely achieved success in the hard business sphere, but is being adopted much less readily by many companies that are dealing with document content. WSDL (and the implied RPC model that it brings to the table) has likewise been successful in a much smaller niche than the proponents of these specs had hoped.

A word to such companies -- your customers are looking for the best solution to their dilemmas, not yours. While it is necessary to place a stake in the ground over a particular technology periodically, making the argument that it's too late to make changes will only make the foundation that these technologies are building just that much more fragile. It also may make your technologies out of synch with the rest of the world, and in a world that is becoming increasingly heterogeneous, this can result in some serious discontinuities to the bottom line when a critical mass of users of the alternative standard is reached.

Is RNG that much better than XSD? I'm still trying to ascertain that myself, though earlier indications are that it is a better schema language. RNG was designed from the bottom up to be a compelling language for defining schemas, XSD was designed from the top down as being a wish-list for vendor support of certain features. Having written a couple of books on XSD over the years, I think I can speak with some authority on this -- it has some REAL problems.

I think that right now there IS a window of opportunity to make changes in the underlying schema language. SOAP and Web Services has not taken off as explosively as the pundits would have liked, in part because of the schema issue, and even the WS-* standards being promulgated by Microsoft are still largely in a prototype and deployment stage.

To me, the best solution, admittedly one of the more complex, is to separate schema from such things as the WS-* standards, making the specific mechanism for validating dependent upon a user defined schema attribute. This approach is already being used with a lot of success in the XSLT space (where transformations beyond XSLT 1.0 can be used for processing just by changing an appropriate selectionNamespace attribute. Schemas, which in general should be much lighter weight than transformations, could easily adopt this approach.

There's a storm brewing there, and because it is something that is so fundamentally - the language to be used for defining things - all parties should spend some serious time asking themselves whether the disinclination to change the default schema lays as much in a level of intellectual laziness as it does worrying about cost. I would rather my children have the best schema language possible when their Internet comes along, rather than something that's a dark horse candidate proposed because everyone hates it equally.

Microsoft Propaganda

I enjoyed the controversies of the first day, even if they were more than a little contentious. That's what healthy scientific debate is all about, and at the level of this conference, I think the word scientifically can be legitimately used. There were multiple viewpoints, and the most refreshing ones were the ones that weren't peddling the next API or service pack upgrade.

I think that the conference to me slid into propaganda on the second day. Doug Purdy presented a piece on versioning with web services that was intriguing, even though I have to admit that it was a little too cut and paste-ish for my tastes. If this had been all of his talk, I think I would have enjoyed it immensely.

What I didn't enjoy was the ten minute "movie" thrown at us promoting Microsoft Development Services, based upon a take-off of the TV show Queer Eye for the Straight Guy, and portraying Linux and Java developers as being idiots who would be far more productive if they would only drink the Microsoft koolaid. To those of the faithful in the crowd, it was a funny piece, but to those of us who tend to straddle both worlds (and more, have moved away from Microsoft technology because of some of its inherent problems) the ad was insulting, and more, its placement within the Sells conference announced like nothing else that this was no longer a technology neutral forum but was instead a wholly owned subsidiary of Microsoft. Until that point, I had been enjoying the conference a great deal, but the movie soured it for me. Can the PR, Microsoft! The people who went to this conference wanted ideas and the give and take of ideas for fixing problems within XML, not captive advertising.

Vindication

On the other hand, one of my personal victories at this conference occurred when Neetu Rajpal, the Program Manager for Microsoft's XML program, was covering changes that would be introduced into VS.NET 2005 in the XML space. Much of it was welcome and overdue - better XML editing, a semi-decent XSLT editor (I still prefer OxygenXML or Stylus Studio, but that they had was a fair sight better than what currently passed for editors for these technologies in Visual Studio). Inferring schemas is also a nice feature, though again one that has made its way into most commerical XML editors.

What was of course NOT in there was any mention of XSLT 2.0. Microsoft has chosen not to support XSLT 2.0, not now at least, and when asked when by a member of the audience (not me, I promise) Neetu said that it wasn't planned for development any time soon. My talk, on XSLT 2.0, apparently did affect more people than I'd thought, because another member of the audience stood up and asked for a show of hands: who there wanted to see XSLT 2.0 and who wanted to see XQuery 1.0. Half of the audience raised their hand for XSLT, maybe 35% raised it for XQuery. That there are a lot more people using the technology than Microsoft wanted to acknowledge may make the bizDev people think twice about not wanting to adopt it.

Amazing Amazon APIs

Jeff Barr, of Amazon, regaled the audience with the full range of Amazon web services APIs that are currently being rolled out for business developers; some very interesting things going on in this space. Through these APIs (currently available freely to anyone who signs up with the Amazon development program) it is possible to query Amazon listings and customer contents, and to provide certain post-back information for tracking purposes with sales, in both their publishing and general retail divisions. They've also managed to partner with Google in creating a generalized search framework that is similarly scriptable through web services, making it possible to build very elaborate web queries that can then be tied into applications, not just web pages. Finally, they now provide a way to create interface bindings on their pages to build branded store presences, something that I predict will be VERY big.

I have to admit, after looking at this, I felt a strong urge to start creating my own store-fronts (and actually will likely do that with Metaphorical Web Publishing, a venture I will discuss in a subsequent blog). It's cool technology.

Conference Wrap Up

I needed to get an early start back, so missed a couple of the sessions. There were a few other sessions that I have heard from other participants proved to be just as provative as the ones I was able to attend, and if I can I will try to attend the conference again next year. I'd prefer to see the Microsoft aspect downplayed -- yes Chris Sells is now a Microsoft employee, but I think that this conference would have more credibility if it was less hostile to alternative viewpoints; additional sponsorship by Novell, IBM, Oracle, or the Linux center (which was located, ironically, less than an hour's drive from the conference) would also serve to make it more balanced and appealing to all XML developers, and may even lure a few of those hardy souls off in Mono land back into Microsoft's embrace.

Chris himself is a gentleman and a scholar, and he has a truly wonderful family of an age with my own. I wish him luck next year with the next Applied XML Developer's Conference, though this year will be hard to beat.

Google in Kirkland

I just learned this week that Google will be setting up a new development center in Kirkland, Washington, within a couple of miles where I live. The idea of having a Google output in the area makes me giddy -- in addition to possibly applying for a position there myself (okay, admit it, who wouldn't want to work at Google right now?) I get to see the fireworks as yet another major Internet player moves into the Puget Sound.

There's an interesting confluence going on right now. With Google in the neighborhood, the Silicon Forest of the Pacific Northwest is beginning to take on a presence not unlike that of San Francisco in the 1990s. Portland, once a major hub for hardware manufacturers, has quietly been evolving into a Linux powerhouse, with Linus Torvald's move there a few months ago adding the crown jewel for a region that is boasting a renewed IBM and Novell presence as well as bunches of Linux start-ups. Microsoft's influence is still strong in Seattle, but Amazon and now Google are going to change that dynamic considerably, with a shift away from desktop systems to the next generation Internet. Vancouver is becoming a mecca not only for games and movie production but also for UI technologies - several SVG companies (many also sporting XAML arms) are located between Yaletown and Belltown in Vancouver.

Makes me proud of my adopted home. Rain is good for developers.

Signing Off

Don Demsiak has called me the master of the long blog (thanks .. I think) but to keep this from turning into war and peace, I'm going to wrap it up here. Expect for things to pick up on Metaphorical Web, as both this conference and my latest development efforts have provided LOTS of fodder for me for the next few weeks. Until then, enjoy!

-- Kurt Cagle

Picking a Winner

2004-09-30T13:26:00.000-07:00

One of the things that has often fascinated me is the way in which given kinds of technologies (and fads in general) spread throughout a population, a metric that I have often used as a way of gauging which technologies have potentially long staying power. This holds more than academic interest for me - as a writer of books, I often am trying to find those ideas, those memes, that will likely end up being popular enough that I can sell many copies of a book by the time it gets to market. Even if you are a writer of fiction, being able to spot these trends gives you a chance to get a book to market with a topic that's getting "hot", and also helps you to determine when a technology is "oversold" and consequently may be in danger of sitting on the shelves.

To that extent, I thought I'd dedicate my column today to a loose set of "rules" (observations really) that I use in looking into my crystal ball. Admittedly, what works for me not work for you, but I think most of these are pretty much common sense.

Open Standards Trump Closed Ones. Standards provide more than "playground principles" for getting people to play in a consistent manner - they also provide a way for smaller players to both avoid a lot of the conceptual research necessary to best describe a given technology, as well as insure that their product will have a market for what it produces. As standards typically form the basis for the way a given set of intellectual property evolves, open standards can also head off potential legal battles down the road.
Dead Tech Reincarnates in Software. Remember the drubbings that push channels, network computers, Hypercard, animated agents, and so forth received as they were pummelled into oblivion. Push lives on in spirit as Atom (nee RSS), network computers have morphed into cell phones and Blackberries, andHypercard was one of the big driving force behind the World Wide Web. If I was a betting man (and I am) I would wager that animated agents will come back into play very shortly as well via XML driven intelligent interfaces.

One of the things that causes a technology to fail is that the technology becomes used first as a vehicle for promoting advertisements. Often the ideas themselves were pretty good (channels for instance), but because channels very quickly became associated with companies piping their ads to your inwilling desktop, push marketing became a term of loathing for most people. However, when people went back and looked at what made the technology possible (in this case the syndication offered by RSS) they often find novel uses for it, such as Weblogs. I keep an eye on any number of failed ideas and try to figure out whether the failure was due to the concept or to its execution, and if it looks like the latter, I'll put out a watch on the web and elsewhere to see if the core tech is showing back up.
Watch the Novice Developers. In the early 1990s, Microsoft came up with an exceptionally innovative product called Visual Basic. It combined an easy to use computer language with the ability to create native applications quickly in its core Windows operating system. Up until that time, Windows was not quite a slam dunk - it was fragile, was competing with other graphical interfaces (such as the Macintosh) and had a relatively small developer base made up of mid-level programmers working within C++. With VB, however, a whole new cohort of power users joined the ranks of "programmers" and there was a huge surge in the development community. Microsoft benefitted by this because it meant that they had, seemingly overnight, a whole forest of applications being developed for it, which of course strengthened the attraction to using this particular OS.

However, starting in about 1997, several things changed. Microsoft shifted their focus away from the development community and toward the higher level IT managers, most of whom did not interact with the systems day to day. In the short term, this strategy paid off, as it sold more higher end server boxes. In the long term, though, Microsoft may have eaten its seed corn. Linux and the BSDs began to jump into prominence, and Java (and later XML) made it easier for developers to migrate off of the Microsoft ship. Suddenly, people were porting away from Microsoft in record numbers, driven mostly by the next cadre of young developers who cut their teeth on these new quasi-Unix systems.

Young developers create software, and the most innovative software usually does not start in the boardroom but in the garage office. To see what things will be hot tomorrow, look at what the most common apps in beta are at Sourceforge or Fresh Meat. Chances are that the applications being development there by themselves will not be the ones that are hot, but applications like those probably will be.
Ignore the Natterings of the Press. I've worked for a number of tech magazines. They live and die off advertising. The mainstream tech reporters know that market well, and expend huge amounts of ink on the doings of Microsoft, IBM, Sun, Hewlet-Packard, Oracle, and so forth. These companies strip whole forests bare to get press releases out to the magazines, and the magazines oblige by writing good reviews about products in hopes of getting advertising money. Some are more objective about the news than others, but given tight deadlines and constrained budgets, the jump from press release to article is usually not far. As press releases are essentially advertisements indicating that a given product has reached (or is nearing) completion, it means that someone there must have been at least a year ahead of you at that stage.

By the time it hits eWeek, information about trends is stale -- more or less. If you are in investor looking at getting into a given technology in the market, the magazines can give you a pretty good idea about what is beginning to bubble to the surface, but if you are a developer looking to catch the next big wave, or a writer looking to get his book pulblished in time for the market to take off, then you're probably late for the party once it makes the headlines.
Solutions in Search of Problems. One of the central questions I always have about a given technology is whether it is fufills an obvious need or whether it is a solution in search of a problem. "I need to make lots of money by selling this software" is not a "problem," though I am sometimes astonished at how many otherwise sane people believe otherwise. Visual Basic made it possible for people who otherwise didn't know the first thing about programming to write serviceable programs. In more contemporary terms, SVG is beginning to take off not because it has any immediate demonstrable impact in presentation graphics (ala Flash or Powerpoint) but because it makes intelligent graphics such as self-aware maps possible, and as people become more conversant with it, they are adapting it for other things.

When evaluating a technology, the first thing you must ask is "How will this help me?" If, after some time spent looking at it you cannot formulate several different one paragraph answers, then the technology is not worth you investing your time on it. I will typically role play different potential users when looking at something new, and I figure if I can't see any obvious advantages in that role play session, then most other people won't either.
Coolness is a factor. There have been a few technologies that I've looked at and immediately responded, "That's cool!" This isn't necessarily due to eye candy - if it makes it possible for you to conceive of things that you absolutely want to do with it right now, even if it isn't quite ready for prime time yet, then you've stumbled on something that will likely be huge down the road. Additionally, this does not necesssarily have to be within the realm of computer technology; I read science and technology magazines extensively, looking not necessarily for what the latest gadgets or ideas are, but instead looking at what things a given innovation (say Bayesian computing) could open up down the road.
Know Your Own Limitations. I've worked with web technologies for eleven years, and XML technologies for eight. I know these areas very well, especially in the context of user interfaces. I know less about databases, though I have worked with them sporadically over the last several years, and I specifically disavow domain knowledge there. This is not to say that you should not try to learn as much about technologies that touch on your own area of expertise as possible (that's the way you learn more, after all) but you should put a confidence factor into any assessment you make based upon how far it is from your core competencies.
Watch the Edges. The most significant innovations do not occur well within the established body of a given area of knowledge. They occur at the boundaries between two otherwise disparate fields. Most new technologies come from people realizing that an often well established solution or methodology in one field can be adapted for use in a different one. Often that technology is a "bridge" a means of translating core concepts back and forth between the two domains -- such as the use of Analog to Digital to Analog subsytems used in modern car engines that originally had their foundation in sampling music. To that extent, seeing the metaphorical similarities between two systems can often let you apply expensively funded innovation in one area to work much less expensively in others.
Think Systemically. Progress is not linear. It branches and weaves, ebbs and flows, sometimes appearing to stagnate and then get a sudden burst of energy to seemingly come out of nowhere. When one particular technology (or company) dominates, it usually ends up creating back-pressure specific to that technology. In the absence of other factors, these usually balance out, but eventually some change in the technological environment will cause the two dominant technologies to shift with dizzying speeds. As a realpolitik example of this, several countries in the former Warsaw pact have undergone significant improvements over the years, making them competitive in a manner well outside of their relative population side. They've leaped ahead of much larger economies in areas such as telecommunication because they effectively found it more cost effective to scrap decades old infrastructure and start from scratch, gaining the advantage of new technologies without having to spend the time and money needed to develop the intervening stages. Look for factors like this when deciding where a break-out company may come from.
Take Time to Think. In our go-go world, there is a tendency to feel that if you are not at your desk ten hours a day, six days a week, then you are failing (and may even be considered a thief for taking money for work during those unoccupied moments.). Yet I find that by setting aside a period of an hour or so a day for "research", finding out about new technologies, working through patterns, doing something besides sitting and coding, you will generally be able to create better code with more applicability, will be able to better foresee the requirements that you will have, and maybe even be able to see ways around particularly thorny problems. Most of my best insights came to me while I was "taking a break" in this manner.

I've long suspected that many of the problems that plague us today stem from a lack of reflection on the part of others in the past. Sometimes the best thing you can do is just turn off the TV, shut down the radio, and go take a walk.

I'd be interested in hearing about things that you do to help you "see the future". Until next time, enjoy!

-- Kurt Cagle

How "Widgety" should SVG Get?

2004-09-27T10:23:00.000-07:00

Widgets have an interesting pedigree. Originally used as a humorous term for a small machine of some sort in a manufacturing process, widgets have made their way into the programming lexicon as a short-hand term for components used in software application development. These components are generally lightweight and are very seldom stand-alone, serving instead as some kind of snap-together Legos* to make application development go faster.

HTML has long had a staple of seven or so such widgets for building web forms, components that are fairly primitive and limited in their capability, but that serve to acquire enough information to make them both easily implementable on about any platform and basic enough to minimize cross-browser differences.

XUL, covered in my previous post, is a larger (albeit still finite) collection of widgets that provide enough flexibility to build what we consider traditional desktop applications. What differentiates XUL from HTML, however, is the presence of another layer, the XML Binding Language (XBL) that provides a set of interfaces for defining new Widgets using a largely XML based framework. XBL can both extend the functionality of existing XUL widgets (and HTML widgets, for that matter) and can define new ones from scratch.

This extensibility mechanism is something that developers should think hard about. Limited component architectures can only carry you so far. Eventually, you come across a need for a component that can't readily be built within the existing widget framework, or that needs to be skinned in a different way than the current set can handle. XBL provides a way of building not only structure but data structres, attaching (or removing) events, and metadata into new items introduced into any given namespace. In simple terms, this means that you can keep your core presentation language (be it XHTML, XUL, or SVG) as focused on its task as necessary, while adding in a reusable set of intermediate components that push into the application layer.

SVG is already an extraordinarily large and complex namespace, one that is proving difficult for many vendors to implement in its entirety. To add into this the requirement for additional core widgets puts two forces against one another that should be cooperating - the vendor community trying to put together decent workable implentations and customers/developers who want to see more functionality at the core level.

When talking about XML technologies, I've often differentiated between foundational schemas - ones that effectively define the toolkits that people use to make the web - and application schemas (the languages that use these primitives to build something useful). SVG is a foundational schema, perhaps even more so than XHTML is. In theory, you could replace XHTML with SVG (though it would entail a huge amount of SVG to do it), you couldn't replaces SVG with XHTML, however. XUL isn't foundational, though it isn't quite at the application level, either - it falls somewhere in between. Again, however, it is possible to create XUL with SVG (not necessarily practical, but possible) while the opposite isn't true.

And this is consequently where the crux of the debate about SVG widgets comes from. It is possible to create an editable text box in SVG (1.2), even if there wasn't an editable attribute in the spec itself. The attribute exists because while it is possible to build this kind of editor with SVG, the overhead of doing this in script is prohibitive (I speak from personal experience here). Beyond this one particular component, however, it is safe to say that most widgets could readily be created by some combination of SVG elements. This is the prevalent view that is spurring the recent developments in sXBL. Rather than establishing this additional set of widgets (an approach taken by the now defunct Corel SVG efforts in creating dSVG), sXBL takes the approach that components can be built using an XML language that would be far more flexible than any limited widget set.

XBL is far from perfect. I am dismayed by the overreliance in the original Mozilla implementation of XBL of RDF, which, while suitable for some applications, adds a layer of complexity to building apps that many developers are uncomfortable taking. I find RDF confusing, and I've worked with it on and off for the last four years. I'd also prefer to see XPath rather than CSS being used as the selector language, for reasons I've outlined before. However, all of this pales in comparison to the obvious benefits that an XBL of any sort provides.

One of the other facets that I think will become a much bigger factor within two-to-three years is the idea of incorporating SVG as an equal partner within the DOM. Adobe's SVG viewer is a powerful application, but so long as it is limited to being "in the box" it's utility will always be more for stand-alone applications rather than components, for memory reasons if nothing else. ASV used as a behavior for defining SVG code directly within a web page has much more potential for making SVG a foundation for Widget development, especially once you have an XBL-like mechanism to create SVG as shadow-entities of other XML constructs. I've played with this some in the Mozilla SVG builds, and there's more than a little power in being able to reference an SVG group in exactly the same way that you do an HTML element.

Right now, we tend to pre-impose what we know of established functionality on our ideas of what can be done with such technologies, but realistically, once you can break out of the box (one of the biggest advantages that SVG promises, when you get down to it) there will be a whole generation of web designers who will see existing web applications as being staid and dull. We need to get to that point, first, but it's coming.

-- Kurt Cagle

2004-09-25T10:25:00.000-07:00

XUL Dialog sample from Who ya gonna call? XUL!