The Master Plan: Fragmenting and Extending HTML The W3C has been trapped in something of a dilemma since it first started its work. While the W3C exists "to lead the World Wide Web to its full potential by developing common protocols that promote its evolution and ensure its interoperability," a large part of its appeal has rested on its authority to codify an "official" flavor of HTML. Web developers looked to the W3C to produce standards they could hold browser vendors to, while the W3C saw itself largely as a research lab pushing the envelope on experimental technologies. The fact that the same browser vendors who were making users crazy were the primary participants and funders of the W3C probably hasn't helped, either. At the same time that the W3C wants to maintain its role and its reputation as the sole home of HTML, lots of proposals both inside and outside of the W3C have been itching to add new features to HTML. In some cases, such as Microsoft's addition of an xml element in Internet Explorer 5, they simply have moved forward with their own plans. In other cases – such as the W3C's hopes of adding Synchronized Multimedia Integration Language (SMIL), MathML, or Scalable Vector Graphics (SVG) to browser vocabularies – they've remained mostly stuck, unable to convince vendors to build these standards into their browsers.

Some of the problems lie in the politics of standards creation and adoption, which can range from the pleasant to the poisonous. It isn't clear that Microsoft, currently dominating the browser market, has much to gain from fully supporting the W3C specifications that it helped to create. Meanwhile, Netscape – trying to recover from a few years of free-fall – has embraced the W3C's standards (some of them at least) emphatically in Mozilla. While the competitive nightmares of the browser wars create incompatibilities, the incentives for accepting and implementing open standards tend to disappear when competition fades.

Escaping from this trap requires taking a different perspective on the Web and Web infrastructures. The business environment behind the Web isn't likely to change in a substantial way, so perhaps making relatively small changes in the technical infrastructure can create new openings for enhanced capabilities. Plug-ins and similar infrastructures (ActiveX controls, Java applets, and so on) have been adjuncts to HTML so far – only used in a limited number of situations and typically acting on information that comes from outside the HTML document itself.

XHTML modularization squares the circle, providing vendors with a standardized way of creating their own extension vocabularies while developing an opening in the infrastructure that makes it possible for those uninterested in building their own browsers to make substantial contributions. Modularizing XHTML doesn't guarantee that developers will follow up on W3C standards with modular browsers, but it might add incentives for doing so. At the very least, XHTML modularization provides a framework that developers can use on the server side for integrating XML and HTML vocabularies; and perhaps the client-side tools will catch up as well.

By embracing the diversity the W3C originally was hailed for quashing, and by enabling developers to add to the mix in a much more controlled way than was possible when vendors just threw in their own elements and attributes, the W3C hopes to open HTML up without polluting it. The W3C isn't defining any kind of program infrastructure to support modularization, but hopefully some kind of infrastructure will emerge that supports modularization in an unambiguous and widely useful way.

Didn't Namespaces Solve Everything? Another W3C Recommendation, Namespaces in XML, provides developers with a set of tools for identifying elements and attributes as HTML elements, SVG elements, MathML elements, or My elements. Browsers can home in on this information, and present the information in a way that reflects the element type correctly. Among other things, this means that users can drop html:img elements into XML documents for use with Internet Explorer 5 and Netscape 6 and actually get images to appear in the browser window.

As powerful and as useful as namespaces are, however, they don't address an enormous number of issues. Two critical problems are context and registration. Context issues arise when vocabularies are mixed, while registration problems emerge when you use namespaces in environments that don't have a way of dealing with them. XHTML modularization is designed to address the context issues. The registration issues remain for another set of tools described at the end of this article.

Context issues typically arise from the nested nature of XML documents. If all the pieces in a document can stand alone (as separate paragraphs, for instance), then an application shouldn't have any significant difficulties processing information as a series of standalone pieces. But if a fragment from one vocabulary is dropped into another vocabulary, it isn't clear how an application should handle it. For example, the following elements are derived from XHTML and SVG:

   <html xmlns="http://www.w3.org/1999/xhtml"  lang="en-US"
   xml:lang="en-US" xmlns:svg  = 'http://www.w3.org/2000/
   svg-20000303-stylable'>
   <head><title>random</title>
   <body>
   <p>This is an HTML paragraph  with a mysterious SVG path in it.
   <svg:path d="M 80 75 L 100  100 L 140 110 z"/></p>
 </body></html>

Perhaps the browser will draw the path within the paragraph block, or perhaps it will ignore it. If the browser draws the path, does it start at the end of the paragraph's text content? Or does the browser treat the paragraph block as its natural container and use that as its starting point? While the semantics of an HTML p element and an SVG path element are well understood in their "home" contexts, dropping them into other contexts strips them of some of their meaning. XHTML 1.0 already recognizes this to a significant extent, although it includes demonstrations of how mixing and matching vocabularies might work. The conformance rules for strictly conformant XHTML documents (in section 3.1.1) require the use of document type definitions and demand that the document must validate against the formal description that DTD provides. While the examples that immediately follow (in section 3.1.2) show a non-conformant document, the W3C makes clear that "Future work by W3C will address ways to specify conformance for documents involving multiple namespaces."

XHTML 1.1 is effectively that future work. The next few articles provide the details of how XHTML sorts out context and processing, describing how XHTML 1.1 fragments itself into smaller pieces and how you can create your own pieces to add to the mix. First, though, let's take a detour to explore some issues that haven't been resolved yet – but which will need fixing for these visions to succeed.

New Issues: Content Negotiation & Context Tangles Breaking XHTML down into parts makes it possible for applications to support both subsets and supersets of the HTML vocabulary. This allows the cell phones and PDAs to do less, and enables application developers on more powerful platforms to do more. At the same time, however, fragmentation introduces significant new complexities – some of which threaten the rough unity the Web has enjoyed up until now.

The first, and probably easier problem, involves creating ways for devices to identify which fragments of the HTML vocabulary they support. To this point, HTML has been identified using the MIME content type identifier text/html. This single identifier is no longer sufficient. In many ways, it hasn't been sufficient for a long while. Web developers have had to create a lot of workarounds for figuring out how to customize their content to different browser capabilities.

On the server side, it's common practice to "sniff out" browser versions using the User-Agent field of HTTP transactions. Once a server learns which kind of browser is on the receiving end of a request, it can generate browser-specific content or simply transfer the browser to a more appropriate view of the information. In extreme cases, users receive messages telling them to update their browsers or replace them with something entirely different.

Knowing which browser is in use isn't always enough, either. Users have significant control over their browsers, and can do things such as turn off (or never install) Java, ActiveX, JavaScript, and various plug-ins. You can use client-side code, typically JavaScript, to sniff for more detailed information within the browser, and either customize information there or send information back to the server. This approach is typically combined with the server-side sniffing because some browsers don't support JavaScript (such as Lynx) or don't permit JavaScript to inspect the browser environment (such as older versions of Internet Explorer for the Macintosh.)

These strategies only work well, however, when there is a relatively limited number of possibilities. If instead of hundreds of browser and version combinations there are thousands, the costs of looking up information rise accordingly. If instead of a few environment issues within browsers there are hundreds or thousands that need checking, and browsers don't all support checking through JavaScript, you may not be able to perform this checking efficiently. Some kind of new infrastructure must make efficient transfers between servers and clients possible.

This isn't an entirely new problem, although the potential scale of XHTML's use and the clear breakdown in older technologies for handling HTML makes it more ominous. The Internet Engineering Task Force (IETF) has discussed content negotiation issues for a long time. Other groups, such as the Wireless Application Protocol (WAP) Forum, have developed specialized solutions that meet their own needs. It's much easier to mandate content negotiation in a situation involving WAP, where a single organization controls the infrastructure design from end to end. Making content negotiation work on the XHTML and HTTP Web will be a long and slow process.

The W3C is developing a set of tools called Composite Capabilities/Preference Profiles (CC/PP), based on two Notes submitted in July 1999. As of this writing, the project is still developing requirements. However, the two Notes and the current requirement drafts at least present a rough picture of the work that lies ahead – and perhaps a vision of the Web as it will look in a few years. The W3C is working with both the WAP Forum and the IETF to create this architecture, which hopefully will ensure that lots of issues are resolved, although it may slow down the development process. Once the issues are resolved, of course, there will be a long period of integration with existing systems and gradual upgrades if CC/PP is widely accepted.

Tip Unlike the rest of the XHTML infrastructure, CC/PP is a project of the W3C's Mobile Access activity. For the latest on their work, see http://www.w3.org/Mobile/Activity. A public mailing list, "www-mobile", is also available for discussions of CC/PP and other Mobile Access work. You can find information on how to subscribe at http://www.w3.org/Mail/Request and you can view the archives at http://lists.w3.org/Archives/Public/www-mobile/.

CC/PP provides a framework that client devices can use to describe their capabilities and configuration to servers. (There isn't any similar framework that servers can use to describe their capabilities to clients.) CC/PP uses an XML syntax and Resource Description Framework (RDF) structures to create property lists describing a particular client's capabilities and identifying more generic features such as memory capacity. The requirements document emphasizes three key aspects of CC/PP: flexibility, extensibility, and distribution.

Note You can access the CC/PP Requirements and Architecture draft at http://www.w3.org/TR/CCPP-ra/. This discussion is based on the 28 February 2000 draft of that document. CC/PP is purely about descriptions. Those descriptions must be flexible enough for use with a wide variety of different devices and types of devices, extensible enough to support future needs, and capable of being distributed across networks rather than centralized on servers. (This last point represents a major change from the server-side sniffing techniques just described.) CC/PP is not a protocol – it is only a container, a document format, which can function in a variety of protocols. It includes some features, such as partial descriptions to indicate changed configurations, which need significant support by protocols, however.

The examples in the CC/PP Note (http://www.w3.org/TR/NOTE-CCPP) describe hardware and software capabilities, but they don't have anything to do with XHTML. Part of making CC/PP useful for identifying client capabilities with XHTML will be the development of an RDF vocabulary describing different XHTML modules and indicating whether a given client supports them. This shouldn't be difficult for modules built into XHTML 1.1, where the W3C controls the naming conventions and rules, but it may prove more complex for extension modules built outside of the W3C process.

The simplest use for CC/PP is negotiation between a client and a server. CC/PP documents give clients a way of describing formally to servers what information they can handle. If a client only supports XHTML Basic (http://www.w3.org/TR/xhtml-basic), there isn't much point in a server sending that client documents marked up for framesets. On the other end, servers may be capable of sending application-specific information that goes beyond XHTML if they know that a given client can actually process it. Instead of guessing about different types of "browsers," servers can customize their presentations for particular application instances.

CC/PP descriptions are useful in a variety of different circumstances. Because the descriptions emanate from each client, may be customized as necessary, and may pass through multiple systems on the way from client to server, it should be pos-sible to create chains of processors that support different capabilities. While proxy servers primarily function for caching and security today, CC/PP descriptions should make it possible to create new kinds of proxy servers that manage and transform information flows among clients and servers with different capabilities.

A cellular provider might, for instance, set up a proxy server that accepts CC/PP information from its customers, and then customize the information it retrieves over the Web to meet the needs and capabilities of customer phones. This proxy approach would give customers a choice of which phones to buy. It also would enable them to turn features on and off, while still ensuring that they weren't wasting expensive bandwidth and connection time on content that their phones couldn't process. The Web servers hosting the information might never see the cell phone CC/PP profiles, and instead receive a profile for the proxy server (or no profile at all, if the infrastructure does not exist.)

Alternatively, proxy servers could add information to incoming material if clients had special-purpose tools they could recognize. Perhaps a company distributes news feeds to its customers, but annotates them with secure messages regarding corporate relationships. Static Web pages might get transformed into forms designed to enable key people to redistribute information across a network. Proxy servers might generate extra information, like that described by XML Document Navigation Language (XDNL – http://www.w3.org/TR/xdnl/), which makes it easier for users of devices with small screens or limited bandwidth to handle large documents.

Making this work will require a lot more infrastructure than just the descriptions. It's already clear that CC/PP won't work with the older HTTP 1.0 protocol that many older browsers and Web servers still use, and making CC/PP work with the HTTP 1.1 protocol will require some design and implementation work on building extensions. The W3C Note, "CC/PP exchange protocol based on HTTP Extension Framework" (http://www.w3.org/TR/NOTE-CCPPexchange) outlines one possibility for making this work, but it may take further work on HTTP before it becomes reality. The W3C and the IETF are working on a generic extension framework (described in the experimental RFC 2774 at ftp://ftp.isi.edu/in-notes/rfc2774.txt). This work probably will have to be completed and implemented before further integration of CC/PP with existing Web architectures is possible.

Tip If you want to explore the HTTP Extension Framework and start building your own tools for testing CC/PP, you may want to try out the W3C's own Jigsaw server (http://www.w3.org/Jigsaw/). Written in Java (and open source at that), Jigsaw is a Web server with support for the HTTP Extension Framework. You can build your own extensions to Jigsaw to explore the possibilities. It also provides proxying capabilities and Java Servlet support.

Even if the W3C, WAP Forum, and IETF can settle the description issues and thoroughly integrate them with the Web infrastructure, modules still raise some thorny problems. Describing modules as atomic units makes it all sound easy, but modules can twist around one another and modify one another's contents in ways that make the interactions difficult to describe. Programs built to handle a given module may not be capable of handling the combination of that module with other modules. In many cases, this provides necessary functionality. For example, implementing frames (a task the W3C isn't taking on for XHTML 1.1) requires the addition of target attributes to a variety of link elements. While the frameset, frame, and noframes elements are critical to frame-based development, the target attribute is needed for complex frame-based interfaces to work properly. It isn't clear how software components built to handle a minimal XHTML module for linking will handle the additional information stored in target attributes. Integrating the software pieces is more difficult than declaring the extra attributes.

Similarly, developers who want to use the W3C's own XLink, with its attribute-based approach to describing hyperlinks, may need to add these attributes to XHTML element types. Software components built without any knowledge of XLink need to support some kind of dispatching to get the XLink information to a component that can handle it. Right now (although admittedly XLink remains a working draft), the W3C does not define any architecture for handling these kinds of tasks.

Attribute issues are perhaps an irritant, but element content models are another critical area where new problems can arise. Using Scalable Vector Graphics (SVG) within XHTML documents requires creating places within the XHTML where SVG may appear. It may be a while before all of these issues get straightened out. In the meantime, XHTML 1.1 is just getting started and is full of promise for a more powerful Web. If the W3C and its allies can navigate these complicated waters, the end results should make the Internet far more accessible and useful. In turn, the Web will move from being an important part of the computing world to being an important and very ordinary part of the world.