Web Structured, Schema’d & Searchable (Web3S) Frequently Asked Questions

DISCLAIMER: The follow reflects the views of Yaron Y. Goland and only Yaron Y. Goland. Nothing in this document should be construed as representing the views of the Windows Live Data team, the Dev Live Platform Group, the Windows Live division, the Microsoft Corporation or anyone else.

2.1 Why didn’t you just use ATOM?

We seriously considered ATOM as our base protocol but we ran into problems that eventually made us abandon that effort. But we have no particular objection to ATOM and if our concerns can be addressed we would be happy to reconsider. Web3S is a means, not an end and we will happily abandon it if we can find a consensus protocol that meets our needs.

In the case of ATOM we ran into three specific problems:

Replacement Semantics – Doing a PUT on an entry in a feed necessitates updating the entire entry. This is a problem for us from both a performance and versioning perspective. From a performance perspective it requires that updates always update all values in an entry even if only a few need to be changed. This puts a lot of unnecessary load on our system. From a versioning perspective replacement semantics means that every single time we add any values we will lose data when an old client updates an entry with new data. There are ways around this but they were judged to be too painful. For example, we could create a new feed URL every time we add a new type of value. Old feeds would only ‘see’ old value types and new feeds would see ‘newer’ value types. But this will create a URL explosion and turn into a management nightmare.

No Hierarchy – ATOM only supports a two level hierarchy, feed and entry. It is possible, of course, to create an entry that is really a pointer to another feed but that is both painful to handle at a protocol level and inefficient when one actually wants to retrieve an entire tree as one has to make many round trips to pull in all the values as one walks the feeds.

Too Verbose – ATOM likes to return meta data, everywhere. This is fine in a two level hierarchy but in a deeply nested hierarchy having to decorate everything, everywhere with wrappers to hold extra values that often won’t even be there is wasteful. E.g. having to wrap everything in <entry> wrappers and such just makes the protocol harder to read and the model more complex.

3 The Web3S Resource Infoset

3.1 Does the world really need another infoset?

We knew from day one that we have to support both XML and JSON and while we could try and shoe horn XML’s infoset into JSON the end result would completely destroy the simplicity that has made JSON so attractive.

Simultaneously, from a scalability perspective, the less we have to support the better. Every feature not only complicates the protocol but it complicates our implementation that needs to run successfully across hundreds of millions of users. Even small incremental savings, when spread over that large a user base, add up quickly.

The Web3S infoset is therefore the outcome of both of these issues. It provides us with the simplicity we need to serialize into/out of both JSON and XML while simultaneously giving us the simplicity of structure that leads to good performance and scaling.

But the good news is that our XML and JSON serializations are real serializations. All XML and JSON tools (well, o.k., XML tools) that can manipulate any arbitrary XML can manipulate a Web3S serialization into/out of XML. It is XML. It just has an unbelievably restrictive schema. That doesn’t violate XML though.

3.2 Why Not Just Use the XML Infoset?

The Web3S infoset was actually designed starting with the XML infoset and then removing information items we didn’t need.

In going through the infoset we threw out the processing instruction information item, the unexpanded entity reference information item, the document type declaration information item, the unparsed entity information item and the notation information items because these are all XMLisms that are of no relevance to anything we are trying to do.

We also threw out the comment information item because comments are not something we are going to make available in our infoset. They might be in the serializations but they aren't part of our processing model. E.g. we do not specify any semantics that directly or indirectly rely on the use of comments.

We also threw out namespace information items. The split between 'local names' and 'namespace names' in XML is an artifact of XML's history and not something that we would want in our infoset (see here for some background). Elements have globally unique names and that should just be that.

We decided we did not want to handle mixed content, that is, EIIs that can have a mix of EIIs and SIIs as children. The justification for this restriction is that we are only interested in self describing data and strings are not self describing (at least not to a machine). So any time a string is used we need it to be wrapped in an element in order to provide machine processable semantics. Since we are only worried about machine processing this seems a reasonable restriction.

A subject of more than a little debate around here has been the attribute information item. I was discussing this with Mark Nottingham from Yahoo and he made the point that attributes are great so long as no one but him is allowed to define how they are used. The point being that there are certainly cases where having an attribute available makes the data model simpler (think IDs, for example) but that in practice people abuse attributes. The fundamental problem with attributes is that they are not, strictly speaking, necessary (E.g. you can always encode in elements what you can express in attributes) and they are not extensible since they are just strings. After much discussion our opening position is that our infoset won't support attributes. On balance we think attributes cause more harm than good. We'll see in practice how well this holds up.

The document information item contains lots of interesting data that isn't relevant to us. It has things like PIIs, comments, DTDs, etc. It also has version information but this is version information for the serialization, not for the XML infoset. The two are not necessarily the same. In the end we decided we did not need to explicitly model a document information item.

So in the end we had only two information items left, element information items and string information items.

3.3 Why not just use the properties in the XML Infoset’s Element Information Item?

The element information item has a number of properties in the XML Infoset. Since our infoset gives each element a globally unique name we can throw away all the detritus of the XML namespace model. So the namespace name, local name, prefix, namespace attributes and in-scope namespaces properties can all be quickly discarded and replaced with a single name property that provides the globally unique name for the element.

For simplicity sake we will name our infoset items using DNS as a mechanism to both provide uniqueness and still make the result human readable. Specifically we will use the reverse DNS convention. E.g. a first name element could be named com.microsoft.live.schemas.firstName. We can then trivially map this to serializations like XML and JSON.

As we aren't supporting attributes we can get rid of the attributes property. The base URI property won't be supported because none of our early scenarios require it but I am confident that we will end up adding it back in as one of our first extensions. We will need the parent property although we can restrict it so that its only legal value is an element or empty in the case of a root.

This leaves the children property. In the XML infoset the contents of the children property are ordered. But in our infoset most data will be unordered. Order is a very big deal for markup languages because the human understandable semantics are embedded in the ordering (e.g. "putting off" does not mean the same thing as "off putting"). But in a machine focused language order is typically not all that relevant. If one thinks of say, a buddy list, the actual order of the buddies is usually a secondary consideration that deals more with human needs than machine ones. For example, if we are returning a buddy list the IM client receiving it could display the contents in any number of orders so the order that the list is sent in over the wire doesn't really matter.

There is however one major counter example, search order. When an expensive search operation is requested it is common to also specify the order in which the results are to be returned. Such an ordering can always be explicitly encoded into the results (e.g. each result member could have an explicit order number associated with it) but this doesn't seem to happen in practice.

We have resolved this problem by accepting reality. The reality is that a message on the wire has an ordering because IP requires some serialization. So we recognize that messages can have ordered contents but we specify that by default storage does not. In other words when interacting with the Web3S infoset in storage all the elements are treated as unordered. But when interacting with a Web3S infoset as serialized on the wire an ordering may be present.

3.4 Why not just use the properties in the XML Infoset’s Character Information Item?

The XML Infoset character information item has three properties, character code, element content whitespace and parent. In our case we will use a string, not a single character and we will allow all unicode characters without restriction. We view it as a serialization problem to escape the unicode characters in order to fit in any particular serialization format. We will address such serialization issues on a serialization by serialization basis (e.g. our solution for JSON will be different than XML because they have different reserved characters). All unicode characters are relevant in our infoset so we don't have the concept of whitespace handling, vis a vis XML. But when we specify a XML serialization we will have to address the whitespace issue.

3.5 Why is the Web3S Infoset hierarchical? Shouldn’t you handle graphs?

There seems to be a fairly obvious pattern in the growth of data formats over time. First, there was ASCII (ANSI, EBCDIC, etc.) which is linear. Then there was HTML (SGML, XML, etc.) which is hierarchical. So the next step would seem to be something graph based. Yes, I know, RDF will save us all. But until we can truly welcome our new RDF overlords my suspicion is that we will just have to make do with hierarchical rather than graph based data formats.

The real issue is programmer complexity. It takes no great effort to define a graph based infoset (we just have to remove the restrictions which require the infoset to be a single rooted acyclic tree) but to manipulate one seems to be beyond the current generation of programming tools. Once we see good programming constructs to manipulate graphs along with compelling use cases requiring their use then we will start to worry about graph support.

3.6 How is the infoset itself versioned?

So if this spec is V0.1 how do we transition to V1 to V2 of the infoset model?

One of the cardinal rules of system design is not to paint oneself into a corner. So it would seem natural to worry about versioning the infoset itself. But what would such versioning mean? Would we put into each message we serialize based on a specific infoset an identifier for that infoset? That would be our first example of tunneling. What would such an identifier achieve? The only time I can see such an identifier being important is if we implicitly rather than explicitly include semantics in a message that are derived from the infoset.

For example, let’s say that we extend the infoset to specify that all relative URLs in the infoset are relative to the root element. But when we serialize an infoset instance into say, JSON, we include relative URLs but we do not include any object or other identifier that explicitly states "relative URLs are relative to the root object". Someone trying to resolve our URLs in the message won't be able to unless they are using the same version of the infoset and so understand how to resolve the URL.

In a case such as the above it would make sense for us to include an identifier for which infoset we are using so that systems could understand when a message may have semantics they don't support.

But I actually think this approach would be a mistake as it essentially requires people to change the rules of their serializations. E.g. it’s not XML anymore, it’s Web3S XML. Therefore I would argue that any time Infoset semantics leak into a message (e.g. ordering, base URLs, whatever) those semantics need to be explicitly and individually marked in the message. So, for example, in the case of relative URLs we would either need to introduce our own JSON object saying "this message interprets relative URLs relative to the root" or, much better, we would work with the JSON community to come up with a community standard to handle relative URL resolution. But in no case should we put ourselves in a position where third parties need to know which version of our infoset a particular service is using in order to figure out how to use that service

So long as we stick to the rule that the infoset's semantics are explicitly made known in the message there is no need for us to explicitly version the infoset itself.

3.7 How does one record raw XML or raw HTML in this infoset?

The Web3S infoset does not contain all the concepts in the XML or HTML infosets. It is therefore impossible, in the general case, to encode generic XML or HTML into the Web3S infoset. We can certainly go in the opposite direction, that is, encoding Web3S into XML or HTML.

This means that if one needs to record generic XML or HTML as part of the infoset then the data will have to be recorded using a string information item. In practice we do not believe this is a problem because Web3S is only intended to be used with structured data that follows our infoset. Anything else is just a string.

3.8 How does one record binary objects in this infoset?

At the moment this can only be done by encoding the binary information into a string. But one imagines that having binary support would be interesting and this is certainly a reasonable candidate for an extension to the infoset.

3.9 Why aren’t IDs universally unique instead of only unique amongst siblings?

It was tempting to just specify the all IDs have to be GUIDs/UUIDs. We decided against this because:

Readability – Reading one of our urls already isn’t exactly the most pleasant experience in the world, but imagine if instead of:

com.example.whatever(234234)/com.example.something(134)/com.example.otherwise(2387283)

you had to instead try and read:

com.example.whatever(f81d4fae-9dec-11d0-a765-00a0c91e6bf6)/com.example.something(f81d4fae-7dec-11d0-a765-00a0c91e6bf6)/com.example.otherwise(f81d4fae-8dec-11d0-a765-00a0c91e6bf6)

Writability – If you can’t read the previous what chance does a programmer trying to debug a system have of writing it?

URL Size Limits – Certain browsers who shall remain nameless have a 2049 byte size limit on their URLs. So if we want this thing to work in places like JSON we have to keep the size down.

Server Controls – IDs are often artifacts of the underlying system so we want to leave servers with maximum flexibility to specify IDs that work for their system. Note however that this requirement only argues that we shouldn’t mandate GUID/UUIIDs as the means of achieving global uniqueness. We could still have the requirement and leave it to servers to fulfill it.

3.10 Shouldn’t I Encode Data into the IDs?

Please don’t.

Pretty please.

With sugar on top.

In fact, in the original design of Web3S we explicitly required that all IDs be integers in order to try our best to stop people from encoding data into IDs. The reason we introduced the integer requirement because within about 30 seconds of coming up with the protocol an implementation group came up with:

com.example.phonebook.phoneNumbers

com.example.phonebook.phoneNumber(Business)

com.example.phonebook.number

555-555-1212

This is a very dangerous way to use IDs. For one thing it completely kills extensibility of the data model. If I want to add some caveats, e.g.:

com.example.phonebook.Business

com.example.phonebook.MegaCorp

I can do so in a very obvious way. But in the model where the ID is assigned semantic meaning extensions aren’t possible. An ID is an ID and it can never change since the Web3S resource will now have told everyone “Oh the way you can find a business phone number is by using the business ID”.

This has all sorts of interesting side effects. For example, what happens when you want to have two phone numbers that are both of type business? Do you start with Business1, Business2, Business3, etc.?

One of the first rules you learn in the database business is – never give semantic meaning to index values! This is an old trick to save space and it always blows up because the fields change but the indexes can’t.

So we had wanted to explicitly try and stop people from designing bad schemas by making it harder for people to shove things into IDs, hence the original requirement that IDs be integers. Of course any variable value can be used to transmit data but by making the values into integers we made encoding data in the kind of ways people wanted to harder and so we hope to reduce the number of completely broken schemas in the world.

But in the end it really needs to be the server’s choice as to how it uses its IDs so we removed the integer requirement. Unfortunately this all but guarantees some really pathetically broken schemas will be introduced that use the ID in awful ways.

3.11 Why do you allow ID values to be reused?

The ultimate owner of the ID space for a particular resource is the resource itself. If it wants to behave badly that is its choice. It is always legal (although not great behavior) for the owner of a particular namespace to decide to renumber their system. That is, HTTP URLs are not guaranteed to point to the same resource for all time. In Web3S terms this means that a HTTP URL that pointed to a Web3S resource could suddenly point to a non-Web3S resource and it also means that the owners of Web3S resources have the right to change their IDs if they want to.

3.12 What about typing of data inside the SIIs?

From the perspective of the Web3S infoset all SIIs are just strings. But schemas on top of the infoset can restrict the values to be anything they can encode into Unicode.

3.13 Doesn’t ordering matter in storage?

The order of authors on books and articles is a very touchy area. People really care. If someone is ‘lead’ author on a book they want that know. So if a Web3S resource records information about authors those authors (especially the lead author) are going to want to make sure the ordering is correct. But in Web3S today there is no way to specify ordering at the infoset level. E.g. if you did a GET on a Web3S resource that records the authors of an article it can give you a different serialization on each response.

So:

com.example.books.authors

com.example.books.author(12)

com.example.books.name

“J. L. Nichols”

com.example.books.author(1)

com.example.books.name

“B. G. Jefferis”

And

com.example.books.authors

com.example.books.author

com.example.books.name(1)

“B. G. Jefferis”

com.example.books.author

com.example.books.name(12)

“J. L. Nichols”

Are both absolutely identical and both legal responses to a GET on the authors EII.

So if we wanted to track the ‘order’ of authorship you have to specify it as an explicit value:

com.example.books.authors

com.example.books.author(1)

com.example.books.name

“B. G. Jefferis”

com.example.books.authorOrder

“1”

com.example.books.author(12)

com.example.books.name

“J. L. Nichols”

com.example.books.authorOrder

“2”

And if we want to change order then we have to do PUTs such as:

<authors xmlns=”Web3SBase:com.example.books”

xmlns:web3s=”Web3S:”>

<web3s:id>1</web3s:id>

</author>

<web3s:id>12</web3s:id>

</author>

</authors>

And if that looks bad just imagine what fun it will be if we wanted to insert a new author whose ordering is between the existing two?!?!

So quite clearly not supporting ordering makes some scenarios really painful. But supporting ordering isn’t free. It complicates the heck out of everything. Suddenly our definitions for things like PUT and UPDATE get a lot more complicated. So we don’t want to support ordering unless we have to.

Our experience to date has been that we haven’t run into situations where an arbitrary numeric ordering (e.g. 1^st EII, 2^nd EII, etc.) really mattered. Most of the large ordered data sets we have (think phone books or blogs) are not arbitrarily ordered but ordered based on things like dates or names. In fact most of our “ordered” datasets are often ordered and re-ordered based on multiple fields and mostly the folks who care about that ordering are clients.

So to date it hasn’t been a big deal and if it stays that way then we will keep away from ordering. I suspect that if it turns out that we have a situation where an arbitrary numeric ordering really and truly matters we might have to invent a different protocol. We don’t want to complicate the heck out of Web3S for what seems like an edge case (albeit an important one).

3.14 Doesn’t not supporting mixed content make certain kinds of extensibility really hard?

Mixed content in the large sense means an arbitrary mixing of EIIs and SIIs. We don’t have any convincing scenarios where not supporting that matters.

But where I strongly think we made a mistake is not allowing for an EII to have zero or more EII children and zero or one SII children. The reason I’m convinced this was a mistake has to do with extensibility.

For example, let’s say someone has a book catalog and so they have an EII org.example.books.booktitle whose value is a SII. But later on the folks defining the schema realize that they would like to be able to record what language a booktitle is in. Their only option is to create a sibling to org.example.books.booktitle that records the language of the booktitle. But that’s bad design. You want shared values to hang out together. What should happen is that the language should end up as a child of booktitle. E.g. org.example.books.booktitle/org.example.language. But that’s illegal in the current infoset because SIIs can’t have siblings. I suspect we need to change that.

3.15 Why doesn’t the Web3S infoset support empty SIIs?

Because the semantics makes our heads hurt.

If we support empty SIIs then what is the meaning of:

Should this translate to:

com.example.anything.stuff

“”

In other words, does the XML translate to an EII with no children or to an EII with an empty SII child?

Trying to distinguish between these two at the infoset level is just a nightmare. Yes, there is xsi:nill but that has its own issues. For example, let’s say we have the following XML schema:

<xsd:element name="stuff" nillable="true"/>

<xs:complexType>

<xs:sequence>

<xs:element name="foo" minOccurs="0" />

</xs:sequence>

</xs:complexType>

</xsd:element>

In that case the following:

Means that the stuff element has no value.

But

Means that the stuff element has a null value.

In other words “null” in XML land has a specific semantic that is different than empty. Oh and neither of the situations has anything to do with empty strings!

Rather than jumping into this morass we have decided to require that all SIIs MUST consist of at least one character. When someone has a situation where they really need to distinguish between an EII with no children and an EII with one child that is a SII that is empty then we recommend they define the schema for the SII to be a quoted string and assign the semantics “empty string” if there is no content between the quotes.

4 Syntax for representing a Web3S Infoset Instance

4.1 Why not make this into a real serialization?

Doesn’t the world have enough serializations already? Can we really justify requiring yet another parser? Would we not be substantially better off just re-using existing parsers?

4.2 If you don’t want to introduce yet another serialization then why use this one for examples?

Because we need a way to specify and give examples of rules applied directly against the infoset itself and not against any particular serialization. So the rules around how to update an EII for example are the same no matter what serialization is used.

5 Addressing Web3S Information Items in HTTP

5.1 Why is it important that we use “/” to delimit elements, why not just have a single query after the “?”

RFC 3986 explicitly defines the path segment of a URI as hierarchical (section 3.3) and the query segment (e.g. the part after the “?”) as non-hierarchical (section 3.4). So the plain text of the RFC would require that we place the hierarchical addressing of Web3S EIIs in the hierarchical part of the URI.

Also if we placed the address of the EII into the query string then we would lose the ability to use relative URIs. Section 4.2 of RFC 3986 explicitly forbids apply relative URI logic to the query parts of a URI.

5.2 Why do you use () for IDs? Why not [] or {}?

The generic syntax for URIs is defined in RFC 3986. The path part of a URI is based on the pchar production and near as I can tell “[“ and “]” are explicitly illegal in a pchar (although they can appear elsewhere) and “{“ and “}” are not allowed to appear anywhere in a URL without URL Encoding. We felt that requiring URL encoding for the ID wrapper would make the wrapper much less readable so we went with parenthesis.

5.3 Why not use a single separator character and not a wrapper for the ID?

Instead of say com.example.foo(123) we could have com.example.foo:123. The real answer is that when we were thinking about the path addressing we were mostly thinking of XPATH style semantics so our original spec actually used brackets but when we discovered that brackets weren’t allowed we had to switch to parenthesis. But overall the parenthesis seem to do a better job of ‘calling out’ the ID as being a special part of the segment name than a colon does. But this is obviously a subjective call.

5.4 Why aren’t SIIs URL addressable?

The only benefit we can see to making SIIs URL addressable is that deleting a SII would be cleaner. Right now one has to PUT to the parent EII with no content as a way to delete a SII. If SIIs were URL addressable then it would be possible to issue a DELETE method against the SII directly. So the extra work of specifying a URL for the SII, just to cover this one case, didn’t seem to be really worthwhile.

However if we decide to allow EIIs to have a single SII sibling then we will most likely need to be able to address SIIs directly. One would imagine us using a reserved value for this. For example: com.example.foo(123)/SII(). As long as we use some character that can’t appear in a domain name we will avoid collisions with EIIs.

5.5 Why isn’t the whole server required to be a Web3S server?

This would have some potentially interesting consequences. For example, if the whole server is a Web3S server then the root of the server path would have to be a Web3S resource (e.g. http://foobar.example.com/ where the last “/” is the root). But if the root is a Web3S resources and since all children in the URL namespace of a Web3S resource must themselves be Web3S resources this would mean that the entire server constituted a single Web3S resource. In cases where resources represent disparate processes that do not need or want to communicate with each other this level of unification would be neither desirable nor practical.

Besides, the root EII of a Web3S resource has a name and that name is required to be in the path so it isn’t possible to have a root named “/”. Therefore, by consequence of syntax, the root of a HTTP URL can never be a Web3S resource.

So combining the “/” problem with the “huge resource” problem at a minimum one will not want to mandate that the root of the path be a Web3S resource. This means then that there is at least one step in the path that isn’t a Web3S resource and if there is one step then it’s just as easy to allow for many steps in the path. So we decided to err on the side of flexibility since it doesn’t put a substantial implementation burden on the clients.

Besides, it’s general bad practice to specify requirements that are going to be ignored. Folks will place their Web3S resources where they want no matter what the spec says.

5.6 Is it legal to mix Web3S and non-Web3S resources?

Strictly speaking, yes. There is no requirement that all the children in the HTTP URL segment space of a Web3S resource be Web3S resources. So in theory http://example.com/com.foo.bar could be a Web3S resource and http://example.com/com.foo.bar/blah could not be a Web3S resource. Of course if someone retrieved http://example.com/com.foo.bar the Web3S infoset serialization would not make any mention of blah.

Still, the fact that Web3S resources can have children that aren’t Web3S resources can lead to confusion. For example, what if one has http://example.com/com.foo.bar/com.foo.blah. A reader of this URL wouldn’t be out of place assuming that com.foo.blah is a Web3S resource. But there is no law that says that non-Web3S resources can’t have names that look like Web3S resources. So in practice the only way that a caller can figure out what are the URLs for http://example.com/com.foo.bar’s children is first to do a GET on http://example.com/com.foo.bar and see what values come back. In other words, just because you happen to know that a particular URL is a Web3S resource you can’t assume that any URLs you see that have that previous URL as the base and ‘look’ Web3S like are necessary Web3S resources.

5.7 Why all the long names, how come you don’t use namespaces or something in the URL to shorten the length of the URL?

We want to be able to pass pointers to resources to non-Web3S systems and for those non-Web3S systems to be able to figure out if two URLs point to the same resource. If we use namespaces or compression then comparison by outside systems may become impossible. E.g. how would any system that isn’t a Web3S system figure out that:

http://example.com/yack/a:bar/a:blah(3)/b:blah?ns:a=com.example&ns:b=com.example.ick

and

http://example.com/yack/ex:bar/ex:blah(3)/ick:blah?ns:ick=com.example.cik&ns:ex=com.example

are the same URI pointing to exactly the same resource?

Furthermore, given the experience with namespaces in XML, it would seem reasonable to expect that many systems will never properly implement something like the previous and instead will just hardcode specific prefixes and fail if those prefixes aren’t used. This will become an interoperability nightmare as each client (and server) has to figure out what namespaces the server (or client) they are talking to has hardcoded.

An alternative approach would be to not use prefixes but instead use a compression scheme. In other words:

http://example.com/yack/com.example.bar/com.example.blah(3)/com.example.ick.blah

would turn into:

http://example.com/yack/com.example.bar/blah(3)/ick.blah

If we have rules that say “if segment N shares a substring that ends in a period with segment N-1 then remove that substring from segment N”. This rule is homomorphic, for any given ‘fully qualified URL’ (e.g. without the compression trick) there is exactly one compressed URL that will match.

But then we have to ask – do we want to mandate this behavior? If not then it’s still possible for the same Web3S resource to have two different URLs. The compressed and uncompressed version.

Also, is requiring that every Web3S processor in the world has to write code to detect this situation really worth dealing with? Is the compression valuable enough to put this code burden on the whole planet? It means every client and server would have to be able to read the compressed form. This seems like a big stumbling block.

6 Web3S Content Types

6.1 Common Requirements for all MIME types

6.2 Application/Web3S+XML Mime Type

6.2.1 Why do we need an Application/Web3S+xml MIME type, why not just use application/xml?

The purpose of MIME types is to identify what processor is needed to properly understand a particular piece of content. In the case of Web3S our use of XML does have some additional semantics around things such as the ID element and the ignore rule as well as a number of restrictions such as no use of DTDs.

So under ideal circumstances one wants a dedicated Web3S processor to handle the content, not a generic XML processor. To make that happen we have to tell the MIME resolution software that it needs to prefer the use of a Web3S processor rather than an XML processor. To do that we need a new MIME type.

However the Web3S XML serialization is XML and can be successfully (if not fully) processed by a XML processor. In many cases if there is no Web3S processor available handing the content to a XML processor is a good idea. Therefore we want to make it clear that the content is XML. This is where the “+xml” extension comes from. It follows the RFC 3023 mechanism (+ xml) to indicate when a MIME type can fallback to a XML processor if the primary processor type is not available.

6.2.2 Since PIIs are specifically for application processing and as we don’t support them in Web3S why allow them in the XML file at all?

Mainly because the XML version declaration in a XML file is itself a PII. So if we ban PIIs then we ban the ability to specify what version of XML is being used. We could get slightly fancier and specify that the only legal PII is one that is a child of the document information item that contains the XML version PII but we decided not to be so fancy. Perhaps we should?

6.2.3 Why do we ban things like DTDs?

We try to ban things that we think will not ‘behave’ well with the web in general or Web3S in particular. DTDs, for example, have a number of issues. One, DTDs are too limited in general to provide meaningful schema definitions so a Web3S processor will always need more sophisticated schema logic, typically Turing complete logic. This doesn’t mean that schema processing languages need to be Turing complete but it does mean that requiring that a DTD sent with a message be enforced in addition to normal schema processing is wasteful. There are also security issues with DTDs since they can require information outside of the message to process. So we decided to just ban them.

6.2.4 Why don’t you just use xml:id for element IDs instead of introducing a new web3s:id element?

As defined in http://www.w3.org/TR/xml-id/, the XML:ID value is required to be “…unique within the XML document”. In other words, if we used XML:ID then in practice all IDs for all Web3S elements would have to be unique at least within the scope of the root EII.

But, as explained in the design rationale section of the Web3S infoset definition, we have explicitly chosen not to require IDs with uniqueness across the local scope.

Therefore we cannot use xml:ID.

6.2.5 What about xml:lang? Can that be used?

A flaw in the Web3S infoset design is that we do not specify how to define what language a string is defined in. Unfortunately due to issues such as Han Unification it is necessary in order to properly handle a string to know what language it came from. When we do tackle this problem one would hope we could leverage existing work on xml:lang and specify its use in Application/Web3S+XML.

6.3 Application/Web3S+XML with Merge Semantics

6.3.1 Shouldn’t you have an Application/Web3SMerge+XML Content Type?

The Application/Web3S+XML content type is used to define a Web3S infoset. Even in the merge case that is what Application/Web3S+XML is used for. What changes in the merge case is what is done with that infoset not what it defines. Think of the difference between PUTing an image and GETting an image. In both cases the image is the same, what changes is the semantics associated with it. In the PUT case the image is intended to update whatever was at the server where as in the second case the image represents what was stored on the server. But the image is still an image. So it is with Application/Web3S+XML. In all cases it represents a Web3S infoset. It’s just the semantics that change.

6.4 Application/Web3SDelta+XML Mime Type

6.4.1 Why do we need a separate MIME type for Application/Web3SDelta+xml?

Because it’s syntax and semantics are fundamentally different than application/Web3S+xml. Web3s+xml represents an infoset. Web3SDelta+xml represents a set of instructions on how to update an infoset. Web3SDelta+xml also includes an annotation command (delete) with mandatory semantics that are required to be enforced by the processor. This annotation command would be ignored by an application/Web3S+xml processor. So a new MIME type was needed.

6.5 Application/Web3SDefPage+XML Mime Type

6.5.1 Why can’t part of the content be returned along with the DeferredContent annotation command?

Because it leads to bad habits. For example, some server implementing some schema will always return a consistent set of ‘partial results’. Clients will get used to getting those partial results. But if those clients get used with a different server that implements the same schema but returns different partial results then all hell breaks loose. This kind of scenario is unfortunately depressingly common.

It also leads to plain screw ups. One can trivially imagine a server programmer getting lazy and deciding to write a process that generates a response skeleton filled with DeferredContent annotation commands but then lets the process run for a set time and retrieve and fill in as much data as it can in that time. This leaves the clients in a quandary. A response they got ‘looks’ complete but is it really? How can they know? Do we now mandate that when DeferredContent is used at least something in the infoset has to be missing?

Of course paging leads to all of these bad behaviors so the arguments made above do feel rather weak. One is again forced to wonder if we should just get rid of DeferredContent in favor of paging.

6.5.2 Shouldn’t we help the client programmer by returning the location URL in the DeferredContent annotation command?

One of the basic rules of good protocol design is never repeat information. There is never a situation where the URL needed to get the deferred content isn’t fully discernable from the response body. Therefore including the actual URL is redundant information and redundant information is just a good opportunity to introduce bugs.

6.5.3 Why don’t you use the range header for paging?

One could imagine us returning an annotation command that gave some range the caller could use to make subsequent calls using the HTTP range header. There are, however, a few practical problems with this approach.

First, the extension syntax for the range header was never well defined in RFC 2616. This has led to confusion in the implementation community where some implementations actually mandate that ranges must be integers. This means that if a server wants to use some kind of session token they will have to map it into a number which will cause a significant size increase. HTTP headers are not infinitely sized in practice so the size of these sessions IDs could become prohibitively large.

Second, there is an implicit assumption in the range header’s name that it is to be used for ‘ranges’. But in our case we are looking at individual pages, not ranges. In fact, in many cases we would explicitly not want to support ranges because this would require returning multiple pages at once. So the actual extension syntax would most likely involve only being able to specify a single page number, which isn’t exactly a range at all.

Third, if we used the range header we would have to deal with some fairly scary issues on the semantics of methods other than GET. Although the range header is explicitly defined for use on all methods there is still a lot of arguing going on about what that means in practice.

So all in all we decided to just stay away from the range header.

6.5.4 Doesn’t returning a URL as the body of the PagingContent XML element mean that it will never be possible to extend the PagingContent XML element in the future?

No. PagingContent is defined in XML, not in Web3S. Therefore PagingContent is not held to the Web3S infoset model. This mean, amongst other things, that PagingContent can hold mixed content. The only thing Web3S defines that affects PagingContent is requirement 3SAFK. This requires that if extension elements are placed in PagingContent then, if not recognized, they must be ignored.

6.5.5 Why doesn’t paging return information about how many pages are available and how to randomly access a page?

One could imagine paging results returning a standard URL format where we say “take the base URL for paging and append to the end a ? with the integer for the page you want, btw, there are X pages available.”

There are a couple of problems with this approach. First, in more than a few cases, the server doesn’t know how many pages are actually available. The cost of calculating that value is too high so the server doesn’t try to find out.

Second, especially in the case of search results, the server may not necessarily be able to page randomly between results. This is especially the case when the underlying data may change between page requests.

None of which is to argue that we couldn’t eventually add an extension to support returning information about how many pages are available and how to randomly access the pages. If it looks like that is compelling feature then we can obviously extend Web3S to provide it.

6.5.6 Isn’t there a possible loop condition with paging where the URL the paging element points to only return a paging element?

Yes, this is theoretically possible. One could get a PagingContent XML element that contains a URL whose response is the parent element of the PagingContent XML element and a single child, another PagingContent XML element. This would potentially lead to a loop.

Paged results explicitly allow one to return results that have been previously returned. So if at least one result has to be returned nothing stops that result from being the same result that was previously returned.

One can argue that we could put in some kind of upper limit of repeated elements but that isn’t easy to write in a meaningful way. Is the same result allowed to be returned once? Twice? 10 times? What about combinations of results? If the first page returned 10 results and subsequent pages return some random subset of 2 of those results when has the behavior become illegal?

In the case of deferred content there was a natural solution to looping, don’t allow a root to contain deferred content. But no equivalent natural ‘line’ seems to exist with paging.

6.5.7 Since you can clearly use paging to do anything deferred content can do why support both features?

The original motivation for the features was different.

Deferred content was meant to deal with the case where a Web3S infoset is very deep and the server only wants to return the children but not the rest of the progeny.

Paging was intended to deal with two different use cases.

The first use case was ‘the really big directory’. That is, a Web3S infoset that contains an enormous number of entries. The number of entries are so large that the server doesn’t want to tie up bandwidth and disk space retrieving them all at once. The server would rather force the requestor to break up the request into chunks so as to be able to provide network/cpu/disk access to other requesters in a manner that doesn’t require the server to maintain state. So the idea was that the server could return some subset of the results and then generate a paging URL that encodes what next subset to return.

One of the consequences of this approach is that there is no promise of consistency. That is, if between the initial request that returned a page and a subsequent request for the next page changes were made to the underlying infoset then those changes would be reflected in the paged results. In other words, the pages are not generated on an isolated data set.

The second use case was dynamically generated results. In order to save CPU, disk, bandwidth, etc. the server only wants to return a subset of the results and then give the requester a way to get the rest. But again, the results may not be consistent.

So the idea was that deferred content would be very simple. It wouldn’t involve any of the dynamic issues, consistency issues, etc. Instead it would be a very easy to implement flag that says “Do a GET on my parent to get its children’s values.” Paged content on the other hand would be more sophisticated. It would allow for dynamic results, inconsistent results, overlapping results, etc.

But in fact we could collapse both into one. We could get rid of deferred results all together and just support paging. If the URL in the paging element contains the same path as the current resource then the caller knows that this is a ‘deferred content’ situation and none of the inconsistency or other issues apply. The caller also knows that the URL is a Web3S URL, something that is not usually guaranteed with paging. The costs of this simplification seem to be twofold:

#1 – Writing a client is slightly more complex since one now needs to check the actual URL in the paging result to see if it is the same base URL as the current resource. Without that check the client doesn’t know if the URL is a web3s URL or not (at least not without an OPTIONS round trip). Although we could mitigate this damage by marking if the URL is Web3S.

#2 – In the case of deferred content the size of the resource would get larger as each and every paging element would contain a URL.

And before someone asks, no, we probably won’t just support relative URLs. Relative URL processing is a black art that is all but guaranteed to cause confusion so we try to stay away from it.

7 Web3S Resources & HTTP Methods

7.1 Common Requirements For All Web3S Methods

7.1.1 What, no COPY/MOVE?

So far we don’t have scenarios that would really benefit from COPY/MOVE. If we run into any we will reconsider supporting those methods.

7.1.2 Well Formed Request-URI Requirement

7.1.3 Optimistic Concurrency

7.1.3.1 Why use etags, why not just used Dates with if-modified-since and if-unmodified-since?

Because HTTP dates are only accurate to the second which is too low a resolution given the speed of modern data processing systems. So the possibility of a race condition where two changes are made to the same resource in the same second is just too high.

7.1.3.2 Why use strong etags?

Section 13.3.3 of RFC 2616 says “Clients MAY issue simple (non-subrange) GET requests with either weak validators or strong validators. Clients MUST NOT use weak validators in other forms of request.” In other words, it’s illegal to submit weak etags on any method but GET.

7.1.3.3 Wait, don’t we need the ability to return etag data within responses?

Yes, we do. With the current system an etag can only be returned at the level of the resource that the method was applied against. But this can cause inefficiencies.

For example, imagine the following Web3S infoset with the addition of an etag property in brackets that uniquely identifies the state of the resource and its children.

com.example.foo[XYZ]

com.example.bar[ABC]

“value”

com.example.blah[DEF]

The semantics are that the complete state of foo are identified by the etag XYZ, e.g. the presence of bar and blah along with the SII “value” under bar. The state of bar is identified by the etag ABC and that only identifies that bar has a SII child “value”. Finally the etag DEF identifies the state of blah which has no children.

If requester A executes a GET request for foo then they will get back XYZ in the etag header.

Now imagine that requester B comes along and alters bar’s value. The resulting infoset would look like:

com.example.foo[GHI]

com.example.bar[JKL]

“another value”

com.example.blah[DEF]

Because bar’s SII value was changed this changed bar’s etag to JKL and because bar is a child of foo this changed foo’s etag to GHI.

Now imagine that requester A comes back and they want to UPDATE blah’s value. Since requester A only got back the XYZ etag in the HTTP response header the only etag A can put on its request is XYZ.

If the semantics requester A was looking was “only update com.example.foo/com.example.blah if com.example.foo’s state is the same as identified in XYZ” then requester A got what it wanted.

But if requester A didn’t care about the global state and only wanted to make sure that the blah EII in particular hadn’t changed then requester A is out of luck. Their request will fail because XYZ is invalid and requester A will have to execut a GET directly on com.example.blah to get the DEF etag and then will have to do another UPDATE. In all requester A will be forced to make three method requests because they didn’t happen to have a good etag for blah.

But imagine if requester A’s initial GET on com.example.foo had returned:

HTTP/1.1 200 OK

Etag: XYZ

<bar>

Value

<web3s:etag>ABC</web3s:etag>

</bar>

<blah>

<web3s:etag>DEF</web3s:etag>

</blah>

</foo>

With the etag data provided inline requester A could see that the etag for blah is “DEF” and use that value in its UPDATE to blah thus avoiding two extra round trips.

Note btw that just because it is possible to place an etag on every value does not mean it’s required. For example, imagine a system that records contacts. It has a “change” indicator for the entire phone book as well as for individual contacts but not for data within contacts. It might return an infoset that looked like:

com.example.contacts[abc]

com.example.contact(1)[def]

com.example.name

“Joachim Seidler”

com.example.contact(23)[ghi]

com.example.name

“Ivaylo Glavchovski”

In the previous infoset with the given etag granularity if either name EII is changed then the etag for both its parent contact EII and the root EII will change. So if a requester wanted to update com.example.contacts/com.example.contact(23)/com.example.name and only wanted to make sure that the name hadn’t changed since they had last seen it then the best they could do is use the etag for the parent contact EII. In other words, if an EII doesn’t have its own etag then it inherits its parent’s. This leaves the server in complete control over the granularity with which it will track changes while telling the client how to figure out the most specific etag to use depending on the scope of change the client wants to protect against.

7.1.3.4 Don’t etags introduce starvation conditions?

Potentially, yes. If a system has a very low granularity (for example, an etag can only be supported at the root EII) and a very high change rate (e.g. many different requesters all pounding on members of the same root EII) then optimistic concurrency would become functionality impossible.

7.1.3.5 Why don’t you return the resource that an etag is rooted at?

One could imagine us introducing a new HTTP header that gave the URL for the resource at which the returned etag was rooted. This would address situations where the etag is rooted at a resource that is an ancestor of the resource that returned the etag. We haven’t come up with any compelling scenarios where this data is useful.

7.1.4 511 – Response Too Large

7.1.4.1 Shouldn’t this just be a 406 Not Acceptable?

Let’s take the example of GET. Web3S requires that resources must respond to GET with application/Web3S+xml. If that requirement wasn’t present then one could argue that a 406 is appropriate, the server simply chooses not to return application/Web3S+xml and may only support something like application/Web3SDefPage+xml. But the requirement to return application/Web3S+xml isn’t optional so if the server refuses to respond to the GET request exclusively because of a desire not to serialize a big response in application/Web3S+xml then the server has an error. In this case we allow the server to specify the nature of its error. Hence the 511.

7.2 GET

7.2.1 Where is the definition of HEAD?

This specification only defines behaviors for Web3S resources that go above and beyond RFC 2616. Nothing about how a Web3S resource would respond to a HEAD is different than how any other HTTP resource would handle that method so nothing extra needs to be said.

7.3 OPTIONS

7.3.1 Why support OPTIONS at all?

It can be useful to use an OPTIONS request to ping a resource in order to find out if it supports Web3S. Just because the resource’s name seems to follow the Web3S URL convention doesn’t mean the resource is a Web3S resource. With OPTIONS it’s possible to tell definitively if the resource supports Web3S.

7.3.2 Isn’t there a race condition where a resource supported Web3S when an OPTIONS request was made but doesn’t support it when subsequent method requests are made?

Yes, there is. In theory all HTTP resource types suffer from this same race condition. Nothing requires that a HTTP resource stay the same ‘resource’ for all time. In theory we could address this problem using the EXPECT header or some kind of etag but experience from protocols like WebDAV have shown that the ‘switching nature’ problem doesn’t seem to show up often enough to be worth addressing.

7.4 PUT/POST/DELETE/UPDATE Common Requirements

7.4.1 So PUT/POST/DELETE/UPDATE aren’t ACID, what are they, AC?

ACID stands for atomic, consistent, isolated and durable.

We don’t require atomic functionality, at least not in the usual sense, but we do require all or nothing semantics. This is intended to make Web3S easy to handle for client developers. Having to write a client that can deal with a situation where a request can leave the server in some half digested state is a nightmare. It is felt that this is much more of a nightmare than creating a server that can provide all or nothing semantics.

We avoid isolation because in practice implementing isolation is incredibly expensive. It is common for data that appears to be a single Web3S infoset instance is in fact a synthetic view generated over multiple tables/databases. In order to guarantee isolation in such a system we would need a distributed transaction manager (DTM). While DTM technology is certainly well understood it is equally well understood that using it is quite expensive and has a significant effect on performance. So while a Web3S system is free to implement isolation and can certainly use this as a competitive advantage the Web3S specification does not require the use of isolation.

Because we do not provide isolation it is extremely important that we guarantee consistency. This means that the publicly accessible resource must never be visible in an inconsistent state.

com.example.marchingOrders

com.example.commander

“Yael”

com.example.subcommander

“Ron”

Example 1 - Simple Web3S Infoset to explain consistency requirement

Imagine that the schema for the marchingOrders EII states that both the commander and subcommander elements are completely optional. Now imagine that someone does a POST to create a new marchingOrders with the content listed above. In that case the following timeline is possible:

Time 0 – Requester A asks to create a new marchingOrders EII with the commander and subcommander children.

Time 1 – The parent Web3S resource creates the marchingOrders EII and the commander EII but hasn’t yet created the subcommander EII.

Time 2 – Requester B executes a GET on the parent Web3S resource and gets back the new marchingOrders EII and the commander EII.

Time 3 – The parent Web3S resource encounters an error in creating the subcommander EII and deletes the marchingOrders EII and its commander EII child in order to ensure all or nothing semantics.

Under the current Web3S requirements the previous scenario is completely legal. Requesters can get results that represent only a partial application of the requested changes. The only requirement is that any externally visible changes must be consistent with the resource’s schema.

But an overly restrictive schema can have seriously performance implications. Imagine in the previous example that the schema was changed such that both the commander and subcommander EIIs must always be present under the marchingOrders EII. In that case the previous timeline is illegal because the state of the resource that Requester B would see at Time 2 (where there is a marchingOrders EII, a commander EII but no subcommander EII) violates the schema. Of course if the commander and subcommander values are actually recorded in separate tables then schema will mean that the implementation has little choice but to use some kind of DTM.

As for durability that is outside of the protocol’s preview. How hard a Web3S implementation works to make sure changes ‘stick’ is implementation specific.

7.4.2 What happens if someone updates a resource in the middle of another update?

Let’s continue with Example 1 but now imagine a different timeline:

Time 0 – Requeste