Web Structured, Schema’d & Searchable (Web3S) Frequently Asked Questions

DISCLAIMER: The follow reflects the views of Yaron Y. Goland and only Yaron Y. Goland. Nothing in this document should be construed as representing the views of the Windows Live Data team, the Dev Live Platform Group, the Windows Live division, the Microsoft Corporation or anyone else.

2.1 Why didn’t you just use ATOM?

We seriously considered ATOM as our base protocol but we ran into problems that eventually made us abandon that effort. But we have no particular objection to ATOM and if our concerns can be addressed we would be happy to reconsider. Web3S is a means, not an end and we will happily abandon it if we can find a consensus protocol that meets our needs.

In the case of ATOM we ran into three specific problems:

Replacement Semantics – Doing a PUT on an entry in a feed necessitates updating the entire entry. This is a problem for us from both a performance and versioning perspective. From a performance perspective it requires that updates always update all values in an entry even if only a few need to be changed. This puts a lot of unnecessary load on our system. From a versioning perspective replacement semantics means that every single time we add any values we will lose data when an old client updates an entry with new data. There are ways around this but they were judged to be too painful. For example, we could create a new feed URL every time we add a new type of value. Old feeds would only ‘see’ old value types and new feeds would see ‘newer’ value types. But this will create a URL explosion and turn into a management nightmare.

No Hierarchy – ATOM only supports a two level hierarchy, feed and entry. It is possible, of course, to create an entry that is really a pointer to another feed but that is both painful to handle at a protocol level and inefficient when one actually wants to retrieve an entire tree as one has to make many round trips to pull in all the values as one walks the feeds.

Too Verbose – ATOM likes to return meta data, everywhere. This is fine in a two level hierarchy but in a deeply nested hierarchy having to decorate everything, everywhere with wrappers to hold extra values that often won’t even be there is wasteful. E.g. having to wrap everything in <entry> wrappers and such just makes the protocol harder to read and the model more complex.

3 The Web3S Resource Infoset

3.1 Does the world really need another infoset?

We knew from day one that we have to support both XML and JSON and while we could try and shoe horn XML’s infoset into JSON the end result would completely destroy the simplicity that has made JSON so attractive.

Simultaneously, from a scalability perspective, the less we have to support the better. Every feature not only complicates the protocol but it complicates our implementation that needs to run successfully across hundreds of millions of users. Even small incremental savings, when spread over that large a user base, add up quickly.

The Web3S infoset is therefore the outcome of both of these issues. It provides us with the simplicity we need to serialize into/out of both JSON and XML while simultaneously giving us the simplicity of structure that leads to good performance and scaling.

But the good news is that our XML and JSON serializations are real serializations. All XML and JSON tools (well, o.k., XML tools) that can manipulate any arbitrary XML can manipulate a Web3S serialization into/out of XML. It is XML. It just has an unbelievably restrictive schema. That doesn’t violate XML though.

3.2 Why Not Just Use the XML Infoset?

The Web3S infoset was actually designed starting with the XML infoset and then removing information items we didn’t need.

In going through the infoset we threw out the processing instruction information item, the unexpanded entity reference information item, the document type declaration information item, the unparsed entity information item and the notation information items because these are all XMLisms that are of no relevance to anything we are trying to do.

We also threw out the comment information item because comments are not something we are going to make available in our infoset. They might be in the serializations but they aren't part of our processing model. E.g. we do not specify any semantics that directly or indirectly rely on the use of comments.

We also threw out namespace information items. The split between 'local names' and 'namespace names' in XML is an artifact of XML's history and not something that we would want in our infoset (see here for some background). Elements have globally unique names and that should just be that.

We decided we did not want to handle mixed content, that is, EIIs that can have a mix of EIIs and SIIs as children. The justification for this restriction is that we are only interested in self describing data and strings are not self describing (at least not to a machine). So any time a string is used we need it to be wrapped in an element in order to provide machine processable semantics. Since we are only worried about machine processing this seems a reasonable restriction.

A subject of more than a little debate around here has been the attribute information item. I was discussing this with Mark Nottingham from Yahoo and he made the point that attributes are great so long as no one but him is allowed to define how they are used. The point being that there are certainly cases where having an attribute available makes the data model simpler (think IDs, for example) but that in practice people abuse attributes. The fundamental problem with attributes is that they are not, strictly speaking, necessary (E.g. you can always encode in elements what you can express in attributes) and they are not extensible since they are just strings. After much discussion our opening position is that our infoset won't support attributes. On balance we think attributes cause more harm than good. We'll see in practice how well this holds up.

The document information item contains lots of interesting data that isn't relevant to us. It has things like PIIs, comments, DTDs, etc. It also has version information but this is version information for the serialization, not for the XML infoset. The two are not necessarily the same. In the end we decided we did not need to explicitly model a document information item.

So in the end we had only two information items left, element information items and string information items.

3.3 Why not just use the properties in the XML Infoset’s Element Information Item?

The element information item has a number of properties in the XML Infoset. Since our infoset gives each element a globally unique name we can throw away all the detritus of the XML namespace model. So the namespace name, local name, prefix, namespace attributes and in-scope namespaces properties can all be quickly discarded and replaced with a single name property that provides the globally unique name for the element.

For simplicity sake we will name our infoset items using DNS as a mechanism to both provide uniqueness and still make the result human readable. Specifically we will use the reverse DNS convention. E.g. a first name element could be named com.microsoft.live.schemas.firstName. We can then trivially map this to serializations like XML and JSON.

As we aren't supporting attributes we can get rid of the attributes property. The base URI property won't be supported because none of our early scenarios require it but I am confident that we will end up adding it back in as one of our first extensions. We will need the parent property although we can restrict it so that its only legal value is an element or empty in the case of a root.

This leaves the children property. In the XML infoset the contents of the children property are ordered. But in our infoset most data will be unordered. Order is a very big deal for markup languages because the human understandable semantics are embedded in the ordering (e.g. "putting off" does not mean the same thing as "off putting"). But in a machine focused language order is typically not all that relevant. If one thinks of say, a buddy list, the actual order of the buddies is usually a secondary consideration that deals more with human needs than machine ones. For example, if we are returning a buddy list the IM client receiving it could display the contents in any number of orders so the order that the list is sent in over the wire doesn't really matter.

There is however one major counter example, search order. When an expensive search operation is requested it is common to also specify the order in which the results are to be returned. Such an ordering can always be explicitly encoded into the results (e.g. each result member could have an explicit order number associated with it) but this doesn't seem to happen in practice.

We have resolved this problem by accepting reality. The reality is that a message on the wire has an ordering because IP requires some serialization. So we recognize that messages can have ordered contents but we specify that by default storage does not. In other words when interacting with the Web3S infoset in storage all the elements are treated as unordered. But when interacting with a Web3S infoset as serialized on the wire an ordering may be present.

3.4 Why not just use the properties in the XML Infoset’s Character Information Item?

The XML Infoset character information item has three properties, character code, element content whitespace and parent. In our case we will use a string, not a single character and we will allow all unicode characters without restriction. We view it as a serialization problem to escape the unicode characters in order to fit in any particular serialization format. We will address such serialization issues on a serialization by serialization basis (e.g. our solution for JSON will be different than XML because they have different reserved characters). All unicode characters are relevant in our infoset so we don't have the concept of whitespace handling, vis a vis XML. But when we specify a XML serialization we will have to address the whitespace issue.

3.5 Why is the Web3S Infoset hierarchical? Shouldn’t you handle graphs?

There seems to be a fairly obvious pattern in the growth of data formats over time. First, there was ASCII (ANSI, EBCDIC, etc.) which is linear. Then there was HTML (SGML, XML, etc.) which is hierarchical. So the next step would seem to be something graph based. Yes, I know, RDF will save us all. But until we can truly welcome our new RDF overlords my suspicion is that we will just have to make do with hierarchical rather than graph based data formats.

The real issue is programmer complexity. It takes no great effort to define a graph based infoset (we just have to remove the restrictions which require the infoset to be a single rooted acyclic tree) but to manipulate one seems to be beyond the current generation of programming tools. Once we see good programming constructs to manipulate graphs along with compelling use cases requiring their use then we will start to worry about graph support.

3.6 How is the infoset itself versioned?

So if this spec is V0.1 how do we transition to V1 to V2 of the infoset model?

One of the cardinal rules of system design is not to paint oneself into a corner. So it would seem natural to worry about versioning the infoset itself. But what would such versioning mean? Would we put into each message we serialize based on a specific infoset an identifier for that infoset? That would be our first example of tunneling. What would such an identifier achieve? The only time I can see such an identifier being important is if we implicitly rather than explicitly include semantics in a message that are derived from the infoset.

For example, let’s say that we extend the infoset to specify that all relative URLs in the infoset are relative to the root element. But when we serialize an infoset instance into say, JSON, we include relative URLs but we do not include any object or other identifier that explicitly states "relative URLs are relative to the root object". Someone trying to resolve our URLs in the message won't be able to unless they are using the same version of the infoset and so understand how to resolve the URL.

In a case such as the above it would make sense for us to include an identifier for which infoset we are using so that systems could understand when a message may have semantics they don't support.

But I actually think this approach would be a mistake as it essentially requires people to change the rules of their serializations. E.g. it’s not XML anymore, it’s Web3S XML. Therefore I would argue that any time Infoset semantics leak into a message (e.g. ordering, base URLs, whatever) those semantics need to be explicitly and individually marked in the message. So, for example, in the case of relative URLs we would either need to introduce our own JSON object saying "this message interprets relative URLs relative to the root" or, much better, we would work with the JSON community to come up with a community standard to handle relative URL resolution. But in no case should we put ourselves in a position where third parties need to know which version of our infoset a particular service is using in order to figure out how to use that service

So long as we stick to the rule that the infoset's semantics are explicitly made known in the message there is no need for us to explicitly version the infoset itself.

3.7 How does one record raw XML or raw HTML in this infoset?

The Web3S infoset does not contain all the concepts in the XML or HTML infosets. It is therefore impossible, in the general case, to encode generic XML or HTML into the Web3S infoset. We can certainly go in the opposite direction, that is, encoding Web3S into XML or HTML.

This means that if one needs to record generic XML or HTML as part of the infoset then the data will have to be recorded using a string information item. In practice we do not believe this is a problem because Web3S is only intended to be used with structured data that follows our infoset. Anything else is just a string.

3.8 How does one record binary objects in this infoset?

At the moment this can only be done by encoding the binary information into a string. But one imagines that having binary support would be interesting and this is certainly a reasonable candidate for an extension to the infoset.

3.9 Why aren’t IDs universally unique instead of only unique amongst siblings?

It was tempting to just specify the all IDs have to be GUIDs/UUIDs. We decided against this because:

Readability – Reading one of our urls already isn’t exactly the most pleasant experience in the world, but imagine if instead of:

com.example.whatever(234234)/com.example.something(134)/com.example.otherwise(2387283)

you had to instead try and read:

com.example.whatever(f81d4fae-9dec-11d0-a765-00a0c91e6bf6)/com.example.something(f81d4fae-7dec-11d0-a765-00a0c91e6bf6)/com.example.otherwise(f81d4fae-8dec-11d0-a765-00a0c91e6bf6)

Writability – If you can’t read the previous what chance does a programmer trying to debug a system have of writing it?

URL Size Limits – Certain browsers who shall remain nameless have a 2049 byte size limit on their URLs. So if we want this thing to work in places like JSON we have to keep the size down.

Server Controls – IDs are often artifacts of the underlying system so we want to leave servers with maximum flexibility to specify IDs that work for their system. Note however that this requirement only argues that we shouldn’t mandate GUID/UUIIDs as the means of achieving global uniqueness. We could still have the requirement and leave it to servers to fulfill it.

3.10 Shouldn’t I Encode Data into the IDs?

Please don’t.

Pretty please.

With sugar on top.

In fact, in the original design of Web3S we explicitly required that all IDs be integers in order to try our best to stop people from encoding data into IDs. The reason we introduced the integer requirement because within about 30 seconds of coming up with the protocol an implementation group came up with:

com.example.phonebook.phoneNumbers

com.example.phonebook.phoneNumber(Business)

com.example.phonebook.number

555-555-1212

This is a very dangerous way to use IDs. For one thing it completely kills extensibility of the data model. If I want to add some caveats, e.g.:

com.example.phonebook.Business

com.example.phonebook.MegaCorp

I can do so in a very obvious way. But in the model where the ID is assigned semantic meaning extensions aren’t possible. An ID is an ID and it can never change since the Web3S resource will now have told everyone “Oh the way you can find a business phone number is by using the business ID”.

This has all sorts of interesting side effects. For example, what happens when you want to have two phone numbers that are both of type business? Do you start with Business1, Business2, Business3, etc.?

One of the first rules you learn in the database business is – never give semantic meaning to index values! This is an old trick to save space and it always blows up because the fields change but the indexes can’t.

So we had wanted to explicitly try and stop people from designing bad schemas by making it harder for people to shove things into IDs, hence the original requirement that IDs be integers. Of course any variable value can be used to transmit data but by making the values into integers we made encoding data in the kind of ways people wanted to harder and so we hope to reduce the number of completely broken schemas in the world.

But in the end it really needs to be the server’s choice as to how it uses its IDs so we removed the integer requirement. Unfortunately this all but guarantees some really pathetically broken schemas will be introduced that use the ID in awful ways.

3.11 Why do you allow ID values to be reused?

The ultimate owner of the ID space for a particular resource is the resource itself. If it wants to behave badly that is its choice. It is always legal (although not great behavior) for the owner of a particular namespace to decide to renumber their system. That is, HTTP URLs are not guaranteed to point to the same resource for all time. In Web3S terms this means that a HTTP URL that pointed to a Web3S resource could suddenly point to a non-Web3S resource and it also means that the owners of Web3S resources have the right to change their IDs if they want to.

3.12 What about typing of data inside the SIIs?

From the perspective of the Web3S infoset all SIIs are just strings. But schemas on top of the infoset can restrict the values to be anything they can encode into Unicode.

3.13 Doesn’t ordering matter in storage?

The order of authors on books and articles is a very touchy area. People really care. If someone is ‘lead’ author on a book they want that know. So if a Web3S resource records information about authors those authors (especially the lead author) are going to want to make sure the ordering is correct. But in Web3S today there is no way to specify ordering at the infoset level. E.g. if you did a GET on a Web3S resource that records the authors of an article it can give you a different serialization on each response.

So:

com.example.books.authors

com.example.books.author(12)

com.example.books.name

“J. L. Nichols”

com.example.books.author(1)

com.example.books.name

“B. G. Jefferis”

And

com.example.books.authors

com.example.books.author

com.example.books.name(1)

“B. G. Jefferis”

com.example.books.author

com.example.books.name(12)

“J. L. Nichols”

Are both absolutely identical and both legal responses to a GET on the authors EII.

So if we wanted to track the ‘order’ of authorship you have to specify it as an explicit value:

com.example.books.authors

com.example.books.author(1)

com.example.books.name

“B. G. Jefferis”

com.example.books.authorOrder

“1”

com.example.books.author(12)

com.example.books.name

“J. L. Nichols”

com.example.books.authorOrder

“2”

And if we want to change order then we have to do PUTs such as:

<authors xmlns=”Web3SBase:com.example.books”

xmlns:web3s=”Web3S:”>

<web3s:id>1</web3s:id>

</author>

<web3s:id>12</web3s:id>

</author>

</authors>

And if that looks bad just imagine what fun it will be if we wanted to insert a new author whose ordering is between the existing two?!?!

So quite clearly not supporting ordering makes some scenarios really painful. But supporting ordering isn’t free. It complicates the heck out of everything. Suddenly our definitions for things like PUT and UPDATE get a lot more complicated. So we don’t want to support ordering unless we have to.

Our experience to date has been that we haven’t run into situations where an arbitrary numeric ordering (e.g. 1^st EII, 2^nd EII, etc.) really mattered. Most of the large ordered data sets we have (think phone books or blogs) are not arbitrarily ordered but ordered based on things like dates or names. In fact most of our “ordered” datasets are often ordered and re-ordered based on multiple fields and mostly the folks who care about that ordering are clients.

So to date it hasn’t been a big deal and if it stays that way then we will keep away from ordering. I suspect that if it turns out that we have a situation where an arbitrary numeric ordering really and truly matters we might have to invent a different protocol. We don’t want to complicate the heck out of Web3S for what seems like an edge case (albeit an important one).

3.14 Doesn’t not supporting mixed content make certain kinds of extensibility really hard?

Mixed content in the large sense means an arbitrary mixing of EIIs and SIIs. We don’t have any convincing scenarios where not supporting that matters.

But where I strongly think we made a mistake is not allowing for an EII to have zero or more EII children and zero or one SII children. The reason I’m convinced this was a mistake has to do with extensibility.

For example, let’s say someone has a book catalog and so they have an EII org.example.books.booktitle whose value is a SII. But later on the folks defining the schema realize that they would like to be able to record what language a booktitle is in. Their only option is to create a sibling to org.example.books.booktitle that records the language of the booktitle. But that’s bad design. You want shared values to hang out together. What should happen is that the language should end up as a child of booktitle. E.g. org.example.books.booktitle/org.example.language. But that’s illegal in the current infoset because SIIs can’t have siblings. I suspect we need to change that.

3.15 Why doesn’t the Web3S infoset support empty SIIs?

Because the semantics makes our heads hurt.

If we support empty SIIs then what is the meaning of:

Should this translate to:

com.example.anything.stuff

“”

In other words, does the XML translate to an EII with no children or to an EII with an empty SII child?

Trying to distinguish between these two at the infoset level is just a nightmare. Yes, there is xsi:nill but that has its own issues. For example, let’s say we have the following XML schema:

<xsd:element name="stuff" nillable="true"/>

<xs:complexType>

<xs:sequence>

<xs:element name="foo" minOccurs="0" />

</xs:sequence>

</xs:complexType>

</xsd:element>

In that case the following:

Means that the stuff element has no value.

But

Means that the stuff element has a null value.

In other words “null” in XML land has a specific semantic that is different than empty. Oh and neither of the situations has anything to do with empty strings!

Rather than jumping into this morass we have decided to require that all SIIs MUST consist of at least one character. When someone has a situation where they really need to distinguish between an EII with no children and an EII with one child that is a SII that is empty then we recommend they define the schema for the SII to be a quoted string and assign the semantics “empty string” if there is no content between the quotes.

4 Syntax for representing a Web3S Infoset Instance

4.1 Why not make this into a real serialization?

Doesn’t the world have enough serializations already? Can we really justify requiring yet another parser? Would we not be substantially better off just re-using existing parsers?

4.2 If you don’t want to introduce yet another serialization then why use this one for examples?

Because we need a way to specify and give examples of rules applied directly against the infoset itself and not against any particular serialization. So the rules around how to update an EII for example are the same no matter what serialization is used.

5 Addressing Web3S Information Items in HTTP

5.1 Why is it important that we use “/” to delimit elements, why not just have a single query after the “?”

RFC 3986 explicitly defines the path segment of a URI as hierarchical (section 3.3) and the query segment (e.g. the part after the “?”) as non-hierarchical (section 3.4). So the plain text of the RFC would require that we place the hierarchical addressing of Web3S EIIs in the hierarchical part of the URI.

Also if we placed the address of the EII into the query string then we would lose the ability to use relative URIs. Section 4.2 of RFC 3986 explicitly forbids apply relative URI logic to the query parts of a URI.

5.2 Why do you use () for IDs? Why not [] or {}?

The generic syntax for URIs is defined in RFC 3986. The path part of a URI is based on the pchar production and near as I can tell “[“ and “]” are explicitly illegal in a pchar (although they can appear elsewhere) and “{“ and “}” are not allowed to appear anywhere in a URL without URL Encoding. We felt that requiring URL encoding for the ID wrapper would make the wrapper much less readable so we went with parenthesis.

5.3 Why not use a single separator character and not a wrapper for the ID?

Instead of say com.example.foo(123) we could have com.example.foo:123. The real answer is that when we were thinking about the path addressing we were mostly thinking of XPATH style semantics so our original spec actually used brackets but when we discovered that brackets weren’t allowed we had to switch to parenthesis. But overall the parenthesis seem to do a better job of ‘calling out’ the ID as being a special part of the segment name than a colon does. But this is obviously a subjective call.

5.4 Why aren’t SIIs URL addressable?

The only benefit we can see to making SIIs URL addressable is that deleting a SII would be cleaner. Right now one has to PUT to the parent EII with no content as a way to delete a SII. If SIIs were URL addressable then it would be possible to issue a DELETE method against the SII directly. So the extra work of specifying a URL for the SII, just to cover this one case, didn’t seem to be really worthwhile.

However if we decide to allow EIIs to have a single SII sibling then we will most likely need to be able to address SIIs directly. One would imagine us using a reserved value for this. For example: com.example.foo(123)/SII(). As long as we use some character that can’t appear in a domain name we will avoid collisions with EIIs.

5.5 Why isn’t the whole server required to be a Web3S server?

This would have some potentially interesting consequences. For example, if the whole server is a Web3S server then the root of the server path would have to be a Web3S resource (e.g. http://foobar.example.com/ where the last “/” is the root). But if the root is a Web3S resources and since all children in the URL namespace of a Web3S resource must themselves be Web3S resources this would mean that the entire server constituted a single Web3S resource. In cases where resources represent disparate processes that do not need or want to communicate with each other this level of unification would be neither desirable nor practical.

Besides, the root EII of a Web3S resource has a name and that name is required to be in the path so it isn’t possible to have a root named “/”. Therefore, by consequence of syntax, the root of a HTTP URL can never be a Web3S resource.

So combining the “/” problem with the “huge resource” problem at a minimum one will not want to mandate that the root of the path be a Web3S resource. This means then that there is at least one step in the path that isn’t a Web3S resource and if there is one step then it’s just as easy to allow for many steps in the path. So we decided to err on the side of flexibility since it doesn’t put a substantial implementation burden on the clients.

Besides, it’s general bad practice to specify requirements that are going to be ignored. Folks will place their Web3S resources where they want no matter what the spec says.

5.6 Is it legal to mix Web3S and non-Web3S resources?

Strictly speaking, yes. There is no requirement that all the children in the HTTP URL segment space of a Web3S resource be Web3S resources. So in theory http://example.com/com.foo.bar could be a Web3S resource and http://example.com/com.foo.bar/blah could not be a Web3S resource. Of course if someone retrieved http://example.com/com.foo.bar the Web3S infoset serialization would not make any mention of blah.

Still, the fact that Web3S resources can have children that aren’t Web3S resources can lead to confusion. For example, what if one has http://example.com/com.foo.bar/com.foo.blah. A reader of this URL wouldn’t be out of place assuming that com.foo.blah is a Web3S resource. But there is no law that says that non-Web3S resources can’t have names that look like Web3S resources. So in practice the only way that a caller can figure out what are the URLs for http://example.com/com.foo.bar’s children is first to do a GET on http://example.com/com.foo.bar and see what values come back. In other words, just because you happen to know that a particular URL is a Web3S resource you can’t assume that any URLs you see that have that previous URL as the base and ‘look’ Web3S like are necessary Web3S resources.

5.7 Why all the long names, how come you don’t use namespaces or something in the URL to shorten the length of the URL?

We want to be able to pass pointers to resources to non-Web3S systems and for those non-Web3S systems to be able to figure out if two URLs point to the same resource. If we use namespaces or compression then comparison by outside systems may become impossible. E.g. how would any system that isn’t a Web3S system figure out that:

http://example.com/yack/a:bar/a:blah(3)/b:blah?ns:a=com.example&ns:b=com.example.ick

and

http://example.com/yack/ex:bar/ex:blah(3)/ick:blah?ns:ick=com.example.cik&ns:ex=com.example

are the same URI pointing to exactly the same resource?

Furthermore, given the experience with namespaces in XML, it would seem reasonable to expect that many systems will never properly implement something like the previous and instead will just hardcode specific prefixes and fail if those prefixes aren’t used. This will become an interoperability nightmare as each client (and server) has to figure out what namespaces the server (or client) they are talking to has hardcoded.

An alternative approach would be to not use prefixes but instead use a compression scheme. In other words:

http://example.com/yack/com.example.bar/com.example.blah(3)/com.example.ick.blah

would turn into:

http://example.com/yack/com.example.bar/blah(3)/ick.blah

If we have rules that say “if segment N shares a substring that ends in a period with segment N-1 then remove that substring from segment N”. This rule is homomorphic, for any given ‘fully qualified URL’ (e.g. without the compression trick) there is exactly one compressed URL that will match.

But then we have to ask – do we want to mandate this behavior? If not then it’s still possible for the same Web3S resource to have two different URLs. The compressed and uncompressed version.

Also, is requiring that every Web3S processor in the world has to write code to detect this situation really worth dealing with? Is the compression valuable enough to put this code burden on the whole planet? It means every client and server would have to be able to read the compressed form. This seems like a big stumbling block.

6 Web3S Content Types

6.1 Common Requirements for all MIME types

6.2 Application/Web3S+XML Mime Type

6.2.1 Why do we need an Application/Web3S+xml MIME type, why not just use application/xml?

The purpose of MIME types is to identify what processor is needed to properly understand a particular piece of content. In the case of Web3S our use of XML does have some additional semantics around things such as the ID element and the ignore rule as well as a number of restrictions such as no use of DTDs.

So under ideal circumstances one wants a dedicated Web3S processor to handle the content, not a generic XML processor. To make that happen we have to tell the MIME resolution software that it needs to prefer the use of a Web3S processor rather than an XML processor. To do that we need a new MIME type.

However the Web3S XML serialization is XML and can be successfully (if not fully) processed by a XML processor. In many cases if there is no Web3S processor available handing the content to a XML processor is a good idea. Therefore we want to make it clear that the content is XML. This is where the “+xml” extension comes from. It follows the RFC 3023 mechanism (+ xml) to indicate when a MIME type can fallback to a XML processor if the primary processor type is not available.

6.2.2 Since PIIs are specifically for application processing and as we don’t support them in Web3S why allow them in the XML file at all?

Mainly because the XML version declaration in a XML file is itself a PII. So if we ban PIIs then we ban the ability to specify what version of XML is being used. We could get slightly fancier and specify that the only legal PII is one that is a child of the document information item that contains the XML version PII but we decided not to be so fancy. Perhaps we should?

6.2.3 Why do we ban things like DTDs?

We try to ban things that we think will not ‘behave’ well with the web in general or Web3S in particular. DTDs, for example, have a number of issues. One, DTDs are too limited in general to provide meaningful schema definitions so a Web3S processor will always need more sophisticated schema logic, typically Turing complete logic. This doesn’t mean that schema processing languages need to be Turing complete but it does mean that requiring that a DTD sent with a message be enforced in addition to normal schema processing is wasteful. There are also security issues with DTDs since they can require information outside of the message to process. So we decided to just ban them.

6.2.4 Why don’t you just use xml:id for element IDs instead of introducing a new web3s:id element?

As defined in http://www.w3.org/TR/xml-id/, the XML:ID value is required to be “…unique within the XML document”. In other words, if we used XML:ID then in practice all IDs for all Web3S elements would have to be unique at least within the scope of the root EII.

But, as explained in the design rationale section of the Web3S infoset definition, we have explicitly chosen not to require IDs with uniqueness across the local scope.

Therefore we cannot use xml:ID.

6.2.5 What about xml:lang? Can that be used?

A flaw in the Web3S infoset design is that we do not specify how to define what language a string is defined in. Unfortunately due to issues such as Han Unification it is necessary in order to properly handle a string to know what language it came from. When we do tackle this problem one would hope we could leverage existing work on xml:lang and specify its use in Application/Web3S+XML.

6.3 Application/Web3S+XML with Merge Semantics

6.3.1 Shouldn’t you have an Application/Web3SMerge+XML Content Type?

The Application/Web3S+XML content type is used to define a Web3S infoset. Even in the merge case that is what Application/Web3S+XML is used for. What changes in the merge case is what is done with that infoset not what it defines. Think of the difference between PUTing an image and GETting an image. In both cases the image is the same, what changes is the semantics associated with it. In the PUT case the image is intended to update whatever was at the server where as in the second case the image represents what was stored on the server. But the image is still an image. So it is with Application/Web3S+XML. In all cases it represents a Web3S infoset. It’s just the semantics that change.

6.4 Application/Web3SDelta+XML Mime Type

6.4.1 Why do we need a separate MIME type for Application/Web3SDelta+xml?

Because it’s syntax and semantics are fundamentally different than application/Web3S+xml. Web3s+xml represents an infoset. Web3SDelta+xml represents a set of instructions on how to update an infoset. Web3SDelta+xml also includes an annotation command (delete) with mandatory semantics that are required to be enforced by the processor. This annotation command would be ignored by an application/Web3S+xml processor. So a new MIME type was needed.

6.5 Application/Web3SDefPage+XML Mime Type

6.5.1 Why can’t part of the content be returned along with the DeferredContent annotation command?

Because it leads to bad habits. For example, some server implementing some schema will always return a consistent set of ‘partial results’. Clients will get used to getting those partial results. But if those clients get used with a different server that implements the same schema but returns different partial results then all hell breaks loose. This kind of scenario is unfortunately depressingly common.

It also leads to plain screw ups. One can trivially imagine a server programmer getting lazy and deciding to write a process that generates a response skeleton filled with DeferredContent annotation commands but then lets the process run for a set time and retrieve and fill in as much data as it can in that time. This leaves the clients in a quandary. A response they got ‘looks’ complete but is it really? How can they know? Do we now mandate that when DeferredContent is used at least something in the infoset has to be missing?

Of course paging leads to all of these bad behaviors so the arguments made above do feel rather weak. One is again forced to wonder if we should just get rid of DeferredContent in favor of paging.

6.5.2 Shouldn’t we help the client programmer by returning the location URL in the DeferredContent annotation command?

One of the basic rules of good protocol design is never repeat information. There is never a situation where the URL needed to get the deferred content isn’t fully discernable from the response body. Therefore including the actual URL is redundant information and redundant information is just a good opportunity to introduce bugs.

6.5.3 Why don’t you use the range header for paging?

One could imagine us returning an annotation command that gave some range the caller could use to make subsequent calls using the HTTP range header. There are, however, a few practical problems with this approach.

First, the extension syntax for the range header was never well defined in RFC 2616. This has led to confusion in the implementation community where some implementations actually mandate that ranges must be integers. This means that if a server wants to use some kind of session token they will have to map it into a number which will cause a significant size increase. HTTP headers are not infinitely sized in practice so the size of these sessions IDs could become prohibitively large.

Second, there is an implicit assumption in the range header’s name that it is to be used for ‘ranges’. But in our case we are looking at individual pages, not ranges. In fact, in many cases we would explicitly not want to support ranges because this would require returning multiple pages at once. So the actual extension syntax would most likely involve only being able to specify a single page number, which isn’t exactly a range at all.

Third, if we used the range header we would have to deal with some fairly scary issues on the semantics of methods other than GET. Although the range header is explicitly defined for use on all methods there is still a lot of arguing going on about what that means in practice.

So all in all we decided to just stay away from the range header.

6.5.4 Doesn’t returning a URL as the body of the PagingContent XML element mean that it will never be possible to extend the PagingContent XML element in the future?

No. PagingContent is defined in XML, not in Web3S. Therefore PagingContent is not held to the Web3S infoset model. This mean, amongst other things, that PagingContent can hold mixed content. The only thing Web3S defines that affects PagingContent is requirement 3SAFK. This requires that if extension elements are placed in PagingContent then, if not recognized, they must be ignored.

6.5.5 Why doesn’t paging return information about how many pages are available and how to randomly access a page?

One could imagine paging results returning a standard URL format where we say “take the base URL for paging and append to the end a ? with the integer for the page you want, btw, there are X pages available.”

There are a couple of problems with this approach. First, in more than a few cases, the server doesn’t know how many pages are actually available. The cost of calculating that value is too high so the server doesn’t try to find out.

Second, especially in the case of search results, the server may not necessarily be able to page randomly between results. This is especially the case when the underlying data may change between page requests.

None of which is to argue that we couldn’t eventually add an extension to support returning information about how many pages are available and how to randomly access the pages. If it looks like that is compelling feature then we can obviously extend Web3S to provide it.

6.5.6 Isn’t there a possible loop condition with paging where the URL the paging element points to only return a paging element?

Yes, this is theoretically possible. One could get a PagingContent XML element that contains a URL whose response is the parent element of the PagingContent XML element and a single child, another PagingContent XML element. This would potentially lead to a loop.

Paged results explicitly allow one to return results that have been previously returned. So if at least one result has to be returned nothing stops that result from being the same result that was previously returned.

One can argue that we could put in some kind of upper limit of repeated elements but that isn’t easy to write in a meaningful way. Is the same result allowed to be returned once? Twice? 10 times? What about combinations of results? If the first page returned 10 results and subsequent pages return some random subset of 2 of those results when has the behavior become illegal?

In the case of deferred content there was a natural solution to looping, don’t allow a root to contain deferred content. But no equivalent natural ‘line’ seems to exist with paging.

6.5.7 Since you can clearly use paging to do anything deferred content can do why support both features?

The original motivation for the features was different.

Deferred content was meant to deal with the case where a Web3S infoset is very deep and the server only wants to return the children but not the rest of the progeny.

Paging was intended to deal with two different use cases.

The first use case was ‘the really big directory’. That is, a Web3S infoset that contains an enormous number of entries. The number of entries are so large that the server doesn’t want to tie up bandwidth and disk space retrieving them all at once. The server would rather force the requestor to break up the request into chunks so as to be able to provide network/cpu/disk access to other requesters in a manner that doesn’t require the server to maintain state. So the idea was that the server could return some subset of the results and then generate a paging URL that encodes what next subset to return.

One of the consequences of this approach is that there is no promise of consistency. That is, if between the initial request that returned a page and a subsequent request for the next page changes were made to the underlying infoset then those changes would be reflected in the paged results. In other words, the pages are not generated on an isolated data set.

The second use case was dynamically generated results. In order to save CPU, disk, bandwidth, etc. the server only wants to return a subset of the results and then give the requester a way to get the rest. But again, the results may not be consistent.

So the idea was that deferred content would be very simple. It wouldn’t involve any of the dynamic issues, consistency issues, etc. Instead it would be a very easy to implement flag that says “Do a GET on my parent to get its children’s values.” Paged content on the other hand would be more sophisticated. It would allow for dynamic results, inconsistent results, overlapping results, etc.

But in fact we could collapse both into one. We could get rid of deferred results all together and just support paging. If the URL in the paging element contains the same path as the current resource then the caller knows that this is a ‘deferred content’ situation and none of the inconsistency or other issues apply. The caller also knows that the URL is a Web3S URL, something that is not usually guaranteed with paging. The costs of this simplification seem to be twofold:

#1 – Writing a client is slightly more complex since one now needs to check the actual URL in the paging result to see if it is the same base URL as the current resource. Without that check the client doesn’t know if the URL is a web3s URL or not (at least not without an OPTIONS round trip). Although we could mitigate this damage by marking if the URL is Web3S.

#2 – In the case of deferred content the size of the resource would get larger as each and every paging element would contain a URL.

And before someone asks, no, we probably won’t just support relative URLs. Relative URL processing is a black art that is all but guaranteed to cause confusion so we try to stay away from it.

7 Web3S Resources & HTTP Methods

7.1 Common Requirements For All Web3S Methods

7.1.1 What, no COPY/MOVE?

So far we don’t have scenarios that would really benefit from COPY/MOVE. If we run into any we will reconsider supporting those methods.

7.1.2 Well Formed Request-URI Requirement

7.1.3 Optimistic Concurrency

7.1.3.1 Why use etags, why not just used Dates with if-modified-since and if-unmodified-since?

Because HTTP dates are only accurate to the second which is too low a resolution given the speed of modern data processing systems. So the possibility of a race condition where two changes are made to the same resource in the same second is just too high.

7.1.3.2 Why use strong etags?

Section 13.3.3 of RFC 2616 says “Clients MAY issue simple (non-subrange) GET requests with either weak validators or strong validators. Clients MUST NOT use weak validators in other forms of request.” In other words, it’s illegal to submit weak etags on any method but GET.

7.1.3.3 Wait, don’t we need the ability to return etag data within responses?

Yes, we do. With the current system an etag can only be returned at the level of the resource that the method was applied against. But this can cause inefficiencies.

For example, imagine the following Web3S infoset with the addition of an etag property in brackets that uniquely identifies the state of the resource and its children.

com.example.foo[XYZ]

com.example.bar[ABC]

“value”

com.example.blah[DEF]

The semantics are that the complete state of foo are identified by the etag XYZ, e.g. the presence of bar and blah along with the SII “value” under bar. The state of bar is identified by the etag ABC and that only identifies that bar has a SII child “value”. Finally the etag DEF identifies the state of blah which has no children.

If requester A executes a GET request for foo then they will get back XYZ in the etag header.

Now imagine that requester B comes along and alters bar’s value. The resulting infoset would look like:

com.example.foo[GHI]

com.example.bar[JKL]

“another value”

com.example.blah[DEF]

Because bar’s SII value was changed this changed bar’s etag to JKL and because bar is a child of foo this changed foo’s etag to GHI.

Now imagine that requester A comes back and they want to UPDATE blah’s value. Since requester A only got back the XYZ etag in the HTTP response header the only etag A can put on its request is XYZ.

If the semantics requester A was looking was “only update com.example.foo/com.example.blah if com.example.foo’s state is the same as identified in XYZ” then requester A got what it wanted.

But if requester A didn’t care about the global state and only wanted to make sure that the blah EII in particular hadn’t changed then requester A is out of luck. Their request will fail because XYZ is invalid and requester A will have to execut a GET directly on com.example.blah to get the DEF etag and then will have to do another UPDATE. In all requester A will be forced to make three method requests because they didn’t happen to have a good etag for blah.

But imagine if requester A’s initial GET on com.example.foo had returned:

HTTP/1.1 200 OK

Etag: XYZ

<bar>

Value

<web3s:etag>ABC</web3s:etag>

</bar>

<blah>

<web3s:etag>DEF</web3s:etag>

</blah>

</foo>

With the etag data provided inline requester A could see that the etag for blah is “DEF” and use that value in its UPDATE to blah thus avoiding two extra round trips.

Note btw that just because it is possible to place an etag on every value does not mean it’s required. For example, imagine a system that records contacts. It has a “change” indicator for the entire phone book as well as for individual contacts but not for data within contacts. It might return an infoset that looked like:

com.example.contacts[abc]

com.example.contact(1)[def]

com.example.name

“Joachim Seidler”

com.example.contact(23)[ghi]

com.example.name

“Ivaylo Glavchovski”

In the previous infoset with the given etag granularity if either name EII is changed then the etag for both its parent contact EII and the root EII will change. So if a requester wanted to update com.example.contacts/com.example.contact(23)/com.example.name and only wanted to make sure that the name hadn’t changed since they had last seen it then the best they could do is use the etag for the parent contact EII. In other words, if an EII doesn’t have its own etag then it inherits its parent’s. This leaves the server in complete control over the granularity with which it will track changes while telling the client how to figure out the most specific etag to use depending on the scope of change the client wants to protect against.

7.1.3.4 Don’t etags introduce starvation conditions?

Potentially, yes. If a system has a very low granularity (for example, an etag can only be supported at the root EII) and a very high change rate (e.g. many different requesters all pounding on members of the same root EII) then optimistic concurrency would become functionality impossible.

7.1.3.5 Why don’t you return the resource that an etag is rooted at?

One could imagine us introducing a new HTTP header that gave the URL for the resource at which the returned etag was rooted. This would address situations where the etag is rooted at a resource that is an ancestor of the resource that returned the etag. We haven’t come up with any compelling scenarios where this data is useful.

7.1.4 511 – Response Too Large

7.1.4.1 Shouldn’t this just be a 406 Not Acceptable?

Let’s take the example of GET. Web3S requires that resources must respond to GET with application/Web3S+xml. If that requirement wasn’t present then one could argue that a 406 is appropriate, the server simply chooses not to return application/Web3S+xml and may only support something like application/Web3SDefPage+xml. But the requirement to return application/Web3S+xml isn’t optional so if the server refuses to respond to the GET request exclusively because of a desire not to serialize a big response in application/Web3S+xml then the server has an error. In this case we allow the server to specify the nature of its error. Hence the 511.

7.2 GET

7.2.1 Where is the definition of HEAD?

This specification only defines behaviors for Web3S resources that go above and beyond RFC 2616. Nothing about how a Web3S resource would respond to a HEAD is different than how any other HTTP resource would handle that method so nothing extra needs to be said.

7.3 OPTIONS

7.3.1 Why support OPTIONS at all?

It can be useful to use an OPTIONS request to ping a resource in order to find out if it supports Web3S. Just because the resource’s name seems to follow the Web3S URL convention doesn’t mean the resource is a Web3S resource. With OPTIONS it’s possible to tell definitively if the resource supports Web3S.

7.3.2 Isn’t there a race condition where a resource supported Web3S when an OPTIONS request was made but doesn’t support it when subsequent method requests are made?

Yes, there is. In theory all HTTP resource types suffer from this same race condition. Nothing requires that a HTTP resource stay the same ‘resource’ for all time. In theory we could address this problem using the EXPECT header or some kind of etag but experience from protocols like WebDAV have shown that the ‘switching nature’ problem doesn’t seem to show up often enough to be worth addressing.

7.4 PUT/POST/DELETE/UPDATE Common Requirements

7.4.1 So PUT/POST/DELETE/UPDATE aren’t ACID, what are they, AC?

ACID stands for atomic, consistent, isolated and durable.

We don’t require atomic functionality, at least not in the usual sense, but we do require all or nothing semantics. This is intended to make Web3S easy to handle for client developers. Having to write a client that can deal with a situation where a request can leave the server in some half digested state is a nightmare. It is felt that this is much more of a nightmare than creating a server that can provide all or nothing semantics.

We avoid isolation because in practice implementing isolation is incredibly expensive. It is common for data that appears to be a single Web3S infoset instance is in fact a synthetic view generated over multiple tables/databases. In order to guarantee isolation in such a system we would need a distributed transaction manager (DTM). While DTM technology is certainly well understood it is equally well understood that using it is quite expensive and has a significant effect on performance. So while a Web3S system is free to implement isolation and can certainly use this as a competitive advantage the Web3S specification does not require the use of isolation.

Because we do not provide isolation it is extremely important that we guarantee consistency. This means that the publicly accessible resource must never be visible in an inconsistent state.

com.example.marchingOrders

com.example.commander

“Yael”

com.example.subcommander

“Ron”

Example 1 - Simple Web3S Infoset to explain consistency requirement

Imagine that the schema for the marchingOrders EII states that both the commander and subcommander elements are completely optional. Now imagine that someone does a POST to create a new marchingOrders with the content listed above. In that case the following timeline is possible:

Time 0 – Requester A asks to create a new marchingOrders EII with the commander and subcommander children.

Time 1 – The parent Web3S resource creates the marchingOrders EII and the commander EII but hasn’t yet created the subcommander EII.

Time 2 – Requester B executes a GET on the parent Web3S resource and gets back the new marchingOrders EII and the commander EII.

Time 3 – The parent Web3S resource encounters an error in creating the subcommander EII and deletes the marchingOrders EII and its commander EII child in order to ensure all or nothing semantics.

Under the current Web3S requirements the previous scenario is completely legal. Requesters can get results that represent only a partial application of the requested changes. The only requirement is that any externally visible changes must be consistent with the resource’s schema.

But an overly restrictive schema can have seriously performance implications. Imagine in the previous example that the schema was changed such that both the commander and subcommander EIIs must always be present under the marchingOrders EII. In that case the previous timeline is illegal because the state of the resource that Requester B would see at Time 2 (where there is a marchingOrders EII, a commander EII but no subcommander EII) violates the schema. Of course if the commander and subcommander values are actually recorded in separate tables then schema will mean that the implementation has little choice but to use some kind of DTM.

As for durability that is outside of the protocol’s preview. How hard a Web3S implementation works to make sure changes ‘stick’ is implementation specific.

7.4.2 What happens if someone updates a resource in the middle of another update?

Let’s continue with Example 1 but now imagine a different timeline:

Time 0 – Requester A asks to create a new marchingOrders EII with the commander and subcommander children.

Time 1 – The parent Web3S resource creates the marchingOrders EII and the commander EII but hasn’t yet created the subcommander EII.

Time 2 – Requester B executes a GET on the parent Web3S resource and gets back the new marchingOrders EII and the commander EII.

Time 3 – Requester B executes a POST on the new marchingOrders EII to add a subcommander EII.

Time 4 – The parent Web3S resource encounters an error in creating the subcommander EII and deletes the marchingOrders EII and its commander EII child in order to ensure all or nothing semantics.

The previous timeline is legal under the current Web3S specification. A variant timeline that is also legal would end:

Time 4 – The parent Web3S resource changes the new subcommander EII created by Requester B to set it to the value specified by Requester A.

The previous result is legal because in theory Requester B could not tell the difference between the scenario described above and a situation where Requester A submitted a request with marchingOrders and commander at time 0 and then made a separate request at time 4 to change subcommander.

The following is also legal:

Time 4 – The parent Web3S resource, seeing that a subcommander EII has been created, decides to keep Requester B’s value but returns a result body that presents the marchingOrders EII as containing the subcommander value specified by requester A.

The previous result is legal because there is no way for Requester A to tell the difference between someone having decided to update the resource at Time 5 (e.g. after the previous timeline is done) and having done it at Time 3. Even if Requester A is sneaky and checks the resource state at time 3.5 it’s still possible that someone else could come in and change subcommander to the originally requested value at time 3.6 and then change it back to the Requester B value at time 4.1. Yes, this is all rather convoluted reasoning but the point is that there exists a legal set of state changes that could lead to the result of having commander with Requester A’s value and subcommander with Request B’s value even if Requester A’s original request had happened without interruption.

Just to really make things confusing the following is also legal:

Time 4 – The parent Web3S resource, seeing that a subcommander EII has been created, decides to keep Requester B’s value and returns that value in the response to requester A.

This is legal because Web3S explicitly says that resources may apply ‘side effects’ to a change and one could argue that changing subcommander to some value other than the one requested by request A is a ‘side effect’. Of course this is an incredibly rude side effect and one probably wouldn’t want to do business with this resource in the future but it is legal.

Also note that all the previous timelines assume that the values for commander and subcommander are completely independent. If there was some kind of cross dependency between them then the various renaming tricks might violate the resource’s schema even if they don’t violate the Web3S spec.

And yes, this is all rather confusing. One suspects that Web3S implementations that do not at least provide some kind of write locking (e.g. so that the resource won’t change in mid-update) won’t be terribly popular.

But we feel that we need to provide this level of flexibility because there are situations where write locks just aren’t economically feasible and where the possibility of mid-update collisions is small enough that it’s better to let those resources be exposed via Web3S with the possibility of collisions then ban them from Web3S all together.

7.4.3 What happens if someone uploads an extension element that is supposed to be multivalue but the Web3S resource doesn’t know this and the upload is marked as single value?

<web3s:id/>

Rudyard Kipling

</author>

</authors>

<title>The Jungle Book</title>

</book>

Example 2 – Body of a POST request to a Web3S resource

In this example the schema for author specifies that it is multivalued so even though this instance only contains a single author the POST is still required to include an empty ID. But let’s say that the poster made a mistake and didn’t include the web3s:id value in the author EII. Let’s further say that the Web3S resource supports an open schema that allows anything to be posted and the resource has no clue that author is supposed to be multi-valued and so doesn’t detect the requester’s error.

In that case all the resource can do is just assume that author is single valued and present it that way.

However if another requester shows up and POSTs another author element to the same book EII that includes an empty ID then it’s up to the Web3S resource to decide what to do. Since the Web3S resource in this example doesn’t know the author schema there is a conundrum:

Does the Web3S resource trust the first poster who said (by not including the ID element) that author is single valued and reject the second requester’s POST as violating the schema by trying to create two sibling instances of the same single valued EII?

Does the Web3S resource trust the second poster and assume the first poster made a mistake and change the author EII to be multivalued? If so then anyone who was given the URL net.example.stuff.book/net.example.stuff.authors/net.example.stuff.author is going to get an error the next time they try to access the resource since the server will now treat author as multivalued and not accept requests for it that don’t include an ID.

There is a potential middle solution which is to trust the second poster but somehow keep net.example.stuff.book/net.example.stuff.authors/net.example.stuff.author (e.g. with no ID) pointing to the original author EII. But that requires ordering (or at least tagging) and will almost certainly make someone’s head explode when they see a URL that clearly violates the schema.

Web3S does not mandate how the Web3S resource should resolve this situation but a general rule of thumb would be to assume that if someone explicitly says that an EII is multi-valued then that’s probably the safest assumption. It’s easier to forget an ID but only marginally harder to insert an ID that shouldn’t be there.

Keep in mind however that all of the previous only applies in the case that the Web3S resource has a schema that explicitly allows it to store ‘any’ EIIs (E.g. the schema is *) and it doesn’t know the schema of the content being POSTed.

7.5 DELETE

7.5.1 Does it make sense to return 200 when deleting a resource that couldn’t even theoretically exist?

Imagine we have a Web3S resource that consists of a single EII com.example.foo.one. Now imagine that someone issues a DELETE for com.example.foo.one/com.example.foo.two/com.example.foo.three. Because the Web3S resource only consists of com.example.foo.one it isn’t even theoretically possible for com.example.foo.three to exist because com.example.foo.two doesn’t exist.

A related scenario is that the schema for com.example.foo.one states that its only legal child value is a SII. In that case even if someone submitted a DELETE for com.example.foo.two, even that request could never be even theoretically correct since it violates the schema.

Yet in both of these cases we would return a 200 OK. The general theory is that deleting an already deleted resource is not a failure, it’s a success. The goal of the request was “delete this resource” and sure enough the resource isn’t there. The fact that the delete could never have theoretically found a resource there seems immaterial.

7.6 POST

7.6.1 What if I’m creating an EII that has data that refers to itself by ID, how can I create such an EII if I can’t specify ID values in a POST?

net.example.lists

net.example.list()

net.example.listname

“Stuff to do at home”

net.example.relatedlists

“net.example.lists/net.example.list(?)”

net.example.listentry

net.example.priority

“1”

net.example.listtext

“Clean up house after finishing work to dos listed at

<href a=”net.example.lists/net.example.list(?)”

>work</href>.”

net.example.list()

net.example.listname

“Stuff to do at work”

net.example.listentry

net.example.priority

“1”

net.example.listtext

“Finish Web3S spec!”

The previous is the contents of a hypothetical POST to some parent that enables the creation of a new ‘lists’ container that contains two lists. The first list refers to the second list in two different ways. First, it has an optional EII relatedlists that provides a pointer to other lists that have content that relates to this list. Second, in the listtext EII there is an SII that is actually HTML (not that Web3S cares, it’s just text to Web3S) that contains a link that needs to resolve to the list.

In order to make this POST work the client needs to have some way to submit IDs for the second list so that it can be pointed to by the two values in the first list. But the situation is made more complex by the fact that it is always and without exception up to the server to decide what IDs to use for EIIs. This requirement is necessary because many server back ends have their own restrictions about IDs that they need to enforce.

So our problem is really two fold – the client needs a way to specify IDs so entries in the same EII value that is being POSTed can cross-reference each other and the server needs to be able to recognize those values in all the places they are used (E.g. in SIIs) so that if the ID that is actually assigned is not the same ID that the client submitted the appropriate updates can be done.

All of this can be done of course. We can allow for the specification of ID values with the explicit proviso that the server is free to change them and we can even define some kind of flag in our serializations to say “Hey, you see this string value right here, it’s an ID referenced in this POST, so if you change the ID of the EII this is pointing to then change this value here.” E.g. in XML we could do something like:

<list>

<w3s:IDTemp>1</w3s:IDTemp>

<listname>Stuff to do at home</listname>

net.example.lists/net.example.list(<w3s:IDSub>2</w3s:IDsub>)

</relatedLists>

<![CDATA[Clean up house after finishing work to dos listed

at <href a=”net.example.lists/net.example.list(]]>

<w3s:IDSub>2</w3s:IDsub>

<![CDATA[)”>work</href>.]]>

</listtext>

</listentry>

</list>

<list>

<w3s:IDTemp>2</w3s:IDTemp>

<listname>Stuff to do at work</listname>

<listtext>Finish Web3S spec!</listtext>

</listentry>

The Web3S server would understand w3s:IDTemp as meaning “I, the client, want to create a new multivalue element, but I need to refer to that multivalue element in this request so I am going to assign my own temporary ID and use it within this request but I expect the server to change the iD.”

The Web3S server would then understand w3s:IDSub as meaning “In the XML request I needed to refer to an ID I specified using w3s:IDTemp. I am wrapping that reference in w3s:IDTemp so that when the server chooses the actual ID value to use it can remove <w3s:IDSub>…</w3s:IDSub> and replace it with the actual value.”

At the moment it isn’t clear if this scenario happens often enough that it’s worth mandating a solution.

7.6.2 Rather than returning the entire resource serialization in the POST response can’t we just return the new IDs?

In theory. But is it really worth coming up with a whole new data format or profiling application/Web3S+xml and making developers support it to save a bunch of bytes? Right now we are choosing simplicity which we believe mandates verbosity.

7.6.3 Why isn’t there a more sophisticated error response mechanism?

Currently if someone submits a bad request, e.g. bad syntax or semantics, then some HTTP error code is returned but we don’t have a defined format to return errors in the body.

We have been considering more sophisticated error handling but it isn’t clear how much it would be used. It has been the general experience with protocol clients that in all but a handful of cases (e.g authorization) there are only two responses to a request – it worked or it didn’t. The client code rarely differentiates beyond that. It’s true that for debugging purposes clients like more information and we encourage servers to provide it but that in itself doesn’t justify creating a more sophisticated error reporting mechanism. For simple debugging a human readable error description will work wonders.

Should it turn out that there are real use cases for machine readable rich error reporting we have been noodling around an approach that is similar to XML fragments.

Imagine we have a request where the schema requires that an author’s name be broken into firstname and lastname EIIs but that wasn’t what was submitted. In that case we could return an error like:

HTTP/1.1 400 Bad Request

Content-Type: Application/Web3SError+XML

Content-Length: xxx

<web3s:id/>

<web3s:error>

<web3s:SIINotAllowed/>

</web3s:error>

<web3s:error>

<web3s:mandatoryElementsNotIncluded>

<web3s:missingElements>

</web3s:missingElements>

</web3s:mandatoryElementsNotIncluded>

</web3s:error>

</author>

</authors>

</book>

Example 3 - A hypothetical rich error format for Web3S error responses

We provide a structure that mimics the schema structure (e.g. book/authors/author) and then annotate it with web3s:error elements to provide information on what went wrong. In this case there was a SII where there shouldn’t have been one (the text version of the author’s name) and two mandatory elements (firstName and lastName) were missing.

Defining this kind of rich error formatting is not trivial and implementing it isn’t easy. So if we are going to do this work we need to be very sure there is a good justification for it. E.g. we need proof that clients actually require rich, machine process-able error formats.

7.6.4 Doesn’t the requirement to leave ID values out of POST request bodies violate the application/Web3S+xml syntax?

It depends really. Requirement 3SABI requires that the ID element be present and set to its value in the Web3S infoset. But one can reasonably argue that in the case of the post the infoset for the Web3S EII serialized in the post request doesn’t contain any values for its IDs. Still, this is a bit stretched in terms of reasoning but the general feeling was that it wasn’t worth introducing yet another content type to handle this specific situation.

7.7 PUT/UPDATE Common Requirements

7.7.1 Why don’t you require that the response contain the serialization of the updated resource?

Imagine an UPDATE to the root of a Web3S resource tree that only applies to a few EIIs far down in the tree. If we require that the response contain the entire updated resource we would have to return the entire tree which would be very expensive in both processing and bandwidth terms.

Still, if there are side effects or if an etag is returned on the response then the client needs to know what the new resource state is in order to understand the full consequences of its actions. This requires returning state.

204s are particularly dangerous. Imagine a caller executes a PUT with an if header specifying some etag. The request succeeds and a 204 is returned with a new etag. What is the caller to think? Should they think that the only difference between the state described in the original etag and the new etag is the changes in the PUT? In other words, if they get a 204 is the caller allowed to assume their call had no side effects (at least within the scope of the request-URI)? Or do they have to assume that there could be side effects in which case the etag is mostly useless since they don’t really know what state the tag represents. E.g. they know the ‘new’ state at least includes their PUT but it could also contain additional content.

7.7.2 Why do you ban specifying the ID of the root EII in the request body of a PUT/UPDATE and doesn’t that violate the application/Web3S*+xml syntax?

For both PUT and UPDATE the root XML element is required to be the same as the last segment of the request-URI. E.g. if someone does a PUT on http://example.com/foo.bar.com(23) then the root EII has to be foo.bar.com.

But it is illegal to put the ID element under the root EII in these cases. E.g. the body will be:

…

</com>

And not

<Web3s:ID xmlns:Web3s=”Web3S:”>23</Web3S>

…

</com>

The reason for the restriction is that repeating information in protocol requests is a bad idea. It just introduces a new way to cause confusion. What should a server do if the ID in the request-URI doesn’t match the ID in the body?

So we resolve this problem by banning the use of the ID in the body. The ID in the request-URI states the situation.

But this can be reasonably argued to cause a violation of the application/Web3S+xml and application/Web3SDelta+xml serialization rules which both require that multi-value EIIs have IDs.

For now we are just fudging it but perhaps we should revisit this decision.

7.8 PUT

7.8.1 Why can’t PUT just delete whatever is in the existing EII and replace its contents?

Imagine we have a schema that consists of a systemDescription EII that has three children – name, status and statusCheckDate. This is a description of some piece of hardware where the name is the name of the hardware, status indicates a ‘quality’ flag representing the equipment’s current status and statusCheckDate specifies when that indicator was last checked. Because of the relationship between status and statusCheckDate the schema requires both elements to always be present and requires that any updates involving both fields have to be atomic. Again, this is a requirement of the schema not the Web3S protocol.

Requester A makes a request to create a new systemDescription with the following content:

POST /stuff/com.example.foo.systems HTTP/1.1

…

<web3s:ID/>

<name>Main Router</name>

</contact>

The response to the POST identifies the ID for com.example.foo.systemDescription as being 1234.

It turns out however that there is an extension to this schema that is supported by the Web3S resource, the lastFullServiceDate EII. This indicates when the identified hardware was last visited by a service tech and given a full service. Older clients, such as the one used by requester A don’t know about the lastFullServiceDate EII. But requester B’s client does know about lastFullServiceDate EII and so sets it for this systemDescription.

POST /stuff/com.example.foo.systems/com.example.foo.systemDescription(1234)/com.example.foo.status HTTP/1.1

…

1/15/2007

</lastFullServiceDate>

So at this point the state of the Web3S resource is:

com.example.foo.systemDescription(1234)

com.example.foo.name

“Main Router”

com.example.foo.status

com.example.foo.green

com.example.foo.statusCheckDate

“2/2/2007”

com.example.foo.lastFullServiceDate

“1/15/2007”

A little bit later however requester A is back. They have talked to the router and have new status information. So they now want to update the Web3S resource that represents the router. Knowing that status and statusCheckDate should be updated together they issue the following request:

PUT /stuff/com.example.foo.systems/com.example.foo.systemDescription(1234)/com.example.foo.status HTTP/1.1

…

<name>Main Router</name>

</systemDescription>

And now things get interesting.

Historically HTTP servers have been file systems. So the traditional semantics for PUT have been “delete + create”. E.g. delete the existing file and create a new file to replace it with the contents of the PUT. Keep in mind that we do not mean a literal delete, e.g. the ‘file creation’ time usually doesn’t change because of a PUT, only the ‘last changed’ time. We only mean “delete + create” in the sense that the contents of the byte stream representing the file are typically deleted and then replaced with the contents of the byte stream in the PUT request.

Requester A is clearly assuming that the traditional “delete + create” semantics apply which is why Request A submitted the name EII even though they only wanted to update the status and statusCheckDate. After all, if request A only submitted status with statusCheckDate and “delete + create” semantics applied then the name EII would be deleted. So in effect “delete + create” semantics forces requester A to re-upload the complete contents of systemDescription. Requester A could have tried to use two separate PUTs to update status and statusCheckDate separately but the schema requires these values to be updated simultaneously in order to keep the resource state consistent. So the only choice is to update the entire systemDescription EII.

One could argue that PUT isn’t a super flexible tool and requiring a full upload is unfortunate but workable. However there is a side effect here. What happened to lastFullServiceDate?

Requester A didn’t know about lastFullServiceDate so they didn’t include it in the PUT. Therefore a side effect of updating status and statusCheckDate is that lastFullServiceDate got deleted even though the change RequesterA made had nothing to do with lastFullServiceDate.

One could argue that before doing the PUT requesterA should first have done a GET and then made sure to upload all the content, including EIIs it didn’t recognize like lastFullServiceDate. The problem with this approach is that methods like PUT are not ‘safe’. That is, when a requester sends up a non-safe method they are saying “I know what’s in this request, I know what effects it will have and I take responsibility for those effects.” But if Web3S’s design mandates that clients send up requests containing content they don’t understand and can’t vouch for then ‘safe’ and ‘non-safe’ methods become meaningless. It seems like fundamentally bad design to require clients to make requests they have no hope of understanding.

A similar problem is what if a newly added field is read only? Requester A downloads the field, realizes that it doesn’t recognize the field and then uploads the field as part of a PUT in order not to destroy state it doesn’t recognize. Except the request will be rejected because the new data is read only. So now Requester A can never update data on the server because it’s requests will always fail due to the fact that it’s trying to be a good citizen!

In addition to violating good protocol design the GET/PUT strategy also violates good protocol efficiency. The crux of protocol design is reducing latency. If latency didn’t matter we would probably all just use RPCs. But latency is crucial and we need to reduce it. Requiring two round trips for each update (e.g. a GET to get the current state including unrecognized values and a PUT, including unrecognized values, to make a change) is therefore going in the wrong direction.

There is also a practical issue around bandwidth. Imagine that systemDescription was a full SNMP MIB with tons and tons of data. Having to upload the whole bloody thing to just update two values seems nuts and will waste enormous amounts of bandwidth and bandwidth (contrary to popular belief) isn’t free.

One possibility is to just ban PUT. We can declare that its semantics are just too dangerous. A blind “delete + create” makes versioning impossible since ‘innocent bystanders’ will be destroyed unintentionally. And a GET/PUT approach is wrong from an architectural, latency and bandwidth perspective.

But instead we decided to see what it would take to make PUT work in a reasonable way and that has meant using merge semantics.

7.8.2 Couldn’t PUT use “database” semantics where only named EIIs are updated but those values use “delete + create” semantics?

In a database update semantics typically consist of pointing to a record and then specifying fields in that record whose values are to be changed. If a field isn’t mentioned in the update then its value isn’t altered. If we used the ‘update’ definition of PUT then requester A’s method to change the status and statusCheckDate values (from the previous question) would turn into:

PUT /stuff/com.example.foo.systems/com.example.foo.systemDescription(1234)/com.example.foo.status HTTP/1.1

…

</systemDescription>

Because neither the name EII (which requester A knew about) nor the lastFullServiceData EII (which requester A didn’t know about) are specified their values would be unaltered. Instead only the status and statusCheckDate values would change.

Unfortunately PUT as “update” is still an incomplete solution. For example, imagine we extended the schema so that it includes a new child for status called statusCheckPeriod. The semantics of this EII are that the status is supposed to be refreshed on the interval specified in statusCheckPeriod. Requester A doesn’t know about this extension but Request B does. So if the current state of the resource is:

com.example.foo.systemDescription(1234)

com.example.foo.name

“Main Router”

com.example.foo.status

com.example.foo.green

com.example.foo.statusCheckPeriod

com.example.foo.days

“2”

com.example.foo.statusCheckDate

“2/2/2007”

com.example.foo.lastFullServiceDate

“1/15/2007”

And the PUT request:

PUT /stuff/com.example.foo.systems/com.example.foo.systemDescription(1234)/com.example.foo.status HTTP/1.1

…

</systemDescription>

Is submitted then what is the new value for status?

Does including an EII in the PUT then apply “delete + create” semantics such that the current status EII is deleted and replaced with the value in the PUT? In other words, after the PUT is the value of the status EII:

com.example.foo.status

com.example.foo.yellow

Notice that if we use “delete + create” semantics (but only for named EIIs, e.g. only the status and statusCheckDate EIIs are affected because they are the only EIIs explicitly named in the PUT) then we are right back where we started. The extended data (in this case statusCheckDate) has been accidently deleted as a consequence of an operation that had nothing to do with it.

7.8.3 Why doesn’t PUT support application/Web3SDelta+xml? Or at least support the delete annotation command?

As discussed in the question Why introduce UPDATE, why not just use PUT? UPDATE requires non-idempotent semantics.

But we still could have put in support for the delete annotation command since its semantics are idempotent. The reason we choose not to is that it violates some people’s version of HTTP philosophy and the issue didn’t seem to be important enough to pick a fight over.

PUT is traditionally defined as “Take this data and use it to change the state of the resource.” This means that the body of the PUT request is just plain data. But if we put command annotations into the body then suddenly the PUT method’s definition changes to be “Take this request body and apply the commands it contains and then apply the data it contains to the resource.” This doesn’t quite jive with the usual way of viewing PUT.

Personally I’m o.k. with putting delete annotations into PUT but this fight just didn’t seem worth having so we didn’t allow it.

7.8.4 Is supporting PUT with merge semantics a violation of RFC 2616?

It depends on who you ask. The spec itself is more than a little vague on the subject.

PUT has certainly traditionally been a replace verb. E.g. “take the contents of the request body and use it to replace the contents of the resource.” And there is a whole school of HTTP thought that argues that PUT is, in effect, the inverse of GET. That is, PUT should be used to specify the exact, byte for byte, data stream you will get back when you do a GET. To this school of thought using merge semantics with PUT is certainly a sin.

But then there is another school of thought that says things like “wait a minute, servers have always been ‘fixing up’ content that was PUT to them and almost nobody ever really guaranteed that the exact byte stream that was specified in a PUT is what the server would return in a GET.” The HTTP spec itself has always been (intentionally, some would argue) vague on the exact semantics of PUT for the exact purpose of allowing servers to choose exactly how to apply a PUT. And in fact there does not seem to be any language in RFC 2616 that settles the matter one way or another.

So the best one can answer to this question is that while there is nothing in RFC 2616 that prevents one from using merge semantics with PUT.

7.8.5 But couldn’t you just apply the usual Database “update” functionality recursively? E.g. keep digging down the PUT contents until you hit some atomic quantity like a SII and then update that?

We tried to define PUT as update semantics recursively but it lead to some bizarre consequences.

PUT /stuff/com.example.foo.systems/com.example.foo.systemDescription(1234)/com.example.foo.status HTTP/1.1

…

</systemDescription>

Above is the same PUT request as used in the previous question but let’s look at it from the perspective of recursive “update”. The logic would be something like:

1. There is a status EII in the PUT request so the status EII in the resource will be updated.

2. There is a yellow EII in the status EII in the PUT request so the yellow EII will be updated.

3. There is nothing in the yellow EII in the status EII in the PUT request so….?

The logic recursed from systemDescription to status to yellow and then it got stuck.

The problem is that the purpose of requester A’s request was actually to delete the green EII and replace it with the yellow EII. But how is the Web3S resource supposed to figure that out?

Having to guess at caller’s intentions is obviously not good design. What’s missing is a way for the caller to say “well, actually, what I meant was, replace <green/> with <yellow/>”. So maybe the request should be:

PUT /stuff/com.example.foo.systems/com.example.foo.systemDescription(1234)/com.example.foo.status HTTP/1.1

…

<web3s:replace>

<web3s:currentValue>

</web3s:currentValue>

<web3s:newValue>

</web3s:newValue>

</web3s:replace>

</status>

</systemDescription>

Of course replace is really just an amalgam of two different commands – a delete followed by a create (sound familiar?). So we could make things more explicit:

PUT /stuff/com.example.foo.systems/com.example.foo.systemDescription(1234)/com.example.foo.status HTTP/1.1

…

<web3s:delete>

</web3s:delete>

</status>

</systemDescription>

In this case the logic would be:

· If the EII is inside of a web3s:delete wrapper then delete the EII with the same name (and ID for multivalued elements).

· If an EII is not inside of a web3s:delete wrapper and there is an existing EII on the resource with the same name (and ID…) then apply the “update” semantics recursively

· If an EII is on its own and there is not a matching EII on the resource then create the EII on the resource.

7.8.6 Can’t we just use DELETE/POST and get rid of PUT?

Sure, as long as there doesn’t exist any schema anywhere that requires two or more values to be changed as a unit. In a very real sense PUT is DELETE + POST + All or Nothing semantics.

Another approach is to require that all updates occur by first deleting the content to be updated and then POSTing a new version. In other words taking the “delete + create” semantics discussed previously literally. But this will bring up a number of issues. First, in large infoset instances with many cross correlations it would mean having to delete and upload large amounts of content. Second, the semantics of DELETE usually have side effects that may not be appropriate (e.g. “someone just deleted the router, I guess it must not be part of our inventory anymore”). Third, it introduces naming issues since people may have been recording the URLs for the deleted values and there is no guarantee that the new entries will have the same IDs as the old entries.

So in general DELETE and POST are not a substitute for PUT.

7.8.7 Is this definition of PUT idempotent?

We believe so but must admit that we haven’t done the math to prove it.

Also note that the ability of Web3S resources to change the state of the resource from what was PUT before returning the result does not necessarily mean that PUT isn’t idempotent. So long as the Web3S resource is consistent in its changes then PUT would be idempotent. How consistent Web3S resources will be in this matter remains to be seen.

But one should remember that idempotent doesn’t mean “byte for byte identical”. It’s really about having a consistent server state. So it is possible for Web3S resources to alter the state of multiple identical PUT responses in different ways and still be idempotent in its full sense if not necessarily at the level of individual bytes.

7.9 UPDATE

7.9.1 Why introduce UPDATE, why not just use PUT?

In order to allow for schema consistent transitions it is necessary to enable UPDATE to create elements that the caller knows do not current exist. This covers the simplest case where a schema has two possible but mutually exclusive children. A schema consistent transition requires the ability to delete one of the elements and create the other. But as soon as create/append style functionality is added then the method is no longer idempotent. Since RFC 2616 explicitly defines PUT as being idempotent it would have been impossible to both solve the problem of schema consistent transitions and providing an idempotent method.

Imagine a schema to record visits to customers by service personnel. A service record is schematically only allowed to be in one of two mutually exclusive states.

org.example.visit()

org.example.scheduledDate

org.example.servicePerson+

org.example.visit()

org.example.dateVisited

org.example.servicePerson+

A service visit is scheduled and so the following EII is created:

org.example.visit(834)

org.example.scheduledDate

“12/1/2009”

org.example.servicePerson(123)

“Gilead Almosnino”

After the visit the record entry is to be updated to show its resolution. The intended update value is:

org.example.visit(834)

org.example.dateVisited

“1/2/2010”

org.example.servicePerson(456)

“Henrik Larsen”

org.example.servicePerson(789)

“Dariusz Korzun”

In other words the visit was originally scheduled for 12/1/2009 by Gilead Almosnino but in the end the visit actually occurred on 1/2/2010 and was serviced by Henrik Larsen and Dariusz Korzun.

The problem is that there is no sequence of DELETE/POST/PUTs that can translate the infoset state from the start state to the end state without requiring the infoset be placed into an intermediate state that is not valid according to the schema.

For example, if the scheduledDate element is DELETE’d the visit EII is now schema invalid since its schema requires that there either be a scheduledDate or dateVisited element. If instead the dateVisited element is POST’d or PUT into the infoset then the instance becomes schema invalid because there would now be a scheduledDate and dateVisited EII in the infoset when the two EIIs are mutually exclusive.

The PUT method, if it could express delete semantics, could handle the previous situation. But things get more tricky when we deal with the issue that not only is the servicePerson who was scheduled not the person who showed up but in fact two servicePerson showed up. We could argue that PUT could be used to update from, say, Gilead Almosnino to Henrik Larsen. But how exactly do we get the second servicePerson value inserted?

We could first update the resource’s state from:

org.example.visit(834)

org.example.scheduledDate

“12/1/2009”

org.example.servicePerson(123)

“Gilead Almosnino”

org.example.visit(834)

org.example.dateVisited

“1/2/2010”

org.example.servicePerson(123)

“Henrik Larsen”

Using the putative PUT with delete semantics mentioned above. But we would then have to issue an additional POST in order to add the Dariusz Korzun entry. This means that there would be a period of time when the resource’s state is false. That is, it may be schema legal, but it does not accurately reflect what happened, which is that two service people showed up. To prevent this problem we need the ability to not only update existing information and delete existing information but we also need the ability to append new information. But we can’t use PUT in that case because append is not idempotent.

Also note that using PUT to change org.example.servicePerson(123) from Gilead Almosnino to Henrik Larsen may not necessarily be a great idea. But that’s another story.

To resolve these issues we need a way to issue, effectively a PUT/POST/DELETE in a single request. Without that ability we would require transitions through schema illegal or at least schematically incorrect states. So that is where UPDATE comes in.

7.9.2 Couldn’t you just cheat and remove the ability to specify EIIs without IDs and so make UPDATE idempotent?

The trick would be to ban EIIs in the UPDATE with empty IDs. If a caller did need the equivalent of an UPDATE with an EII with an empty ID then they would just make up some value to stick in the ID, count on the value not existing and thus count on the Web3S resource creating an EII with the specified ID. This would still be idempotent since issuing the UPDATE multiple times would still result in the same EII with the same ID existing on the resource.

However this leaves open the possibility of collision. If it turns out that there actually is an EII with the ‘randomly’ chosen ID then the semantics of the action would be other than what the requestor intended.

Also Web3S resources have the final say on IDs. If an EII with an ID in an UPDATE request doesn’t match an existing EII on the resource then the Web3S resource can still choose to reject the request because it doesn’t wish to create a new EII with the specified ID even though it would have been happy to access the EII if it had an empty ID so the resource could pick its own ID. So removing the empty ID functionality would make UPDATE methods very risky.

7.9.3 If the server can’t find an ordering that will allow it to execute an UPDATE shouldn’t that be a 5xx error not a 4xx error?

It is the client’s responsibility to submit requests that the server can handle. Yes, that’s a wimp out.

7.9.4 If the EII in the UPDATE is empty and the matching EII on the resource has a SII then the SII is deleted, but if the matching EII on the resource has EII children then nothing happens, why?

UPDATE uses, well, update semantics. So its job is to ‘walk’ the EII tree until it finds a value it can update. In the case of a SII the SII can be updated. But in the case of an EII what is to be updated in the EII on the resource if the EII in the UPDATE is empty? We could just delete the contents of the EII on the resource but we think that is too likely to lead to errors. See the question “Why can’t PUT just delete whatever is in the existing EII and replace its contents?” for more details.

7.9.5 Why are DELETE command annotations processed from the root on down?

DELETE /net.example.bar HTTP/1.1

…

<web3s:delete>

</web3s:delete>

<blah>

<web3s:id>4</web3s:id>

<web3s:delete>

</web3s:delete>

</blah>

</bar>

Example 4 - A DELETE Request

Let us imagine that the net.example.blah EII is defined such that when it is created the Web3S resource will automatically create a net.example.foobar EII with some resource specified contents.

In the previous example if the deletes were not executed from the root on down but instead from the leafs on up (or some other order) then the semantics would be “delete the foobar EII from the existing net.example.blah(4), delete net.example.blah(4) and then create net.example.blah(4).”

But the desired semantics are actually “delete net.example.blah(4), create net.example.blah(4), delete the automatically created net.example.foobar EII.”

Admittedly this is an edge case but it is the motivation for requiring that deletes be processed from the root on down.

7.9.6 Why aren’t you using a XML Diff format instead of inventing your own Diff format?

Our Diff format doesn’t actually diff of the XML infoset. It diffs off the much simpler Web3S infoset. This allows us to have a radically simpler syntax and to ignore XML issues like attributes and ordering that have no relevance to Web3S.

More importantly, using a Web3S infoset specific diff format allows us to directly express the changes that are to be made in the Web3S infoset. If we used a XML specific diff format (ala XML Patch out of the SIMPLE working group in the IETF) then we would be specifying changes indirectly. E.g. the diff format would say “take this piece of XML and make it look like this other pieces of XML” then we would process the XML into a Web3S infoset and that would be the final state. This kind of indirect two stage alteration just begs for translation errors.

So all in all we felt it was best to just stick with our own Web3S infoset specific diff format.

8 Extended Options

8.1 Location Information

8.1.1 What’s the point of having location information?

Oddly enough people seem to really like to point at things, typically using URLs. We figured having a standard way to do that in Web3S was a good idea. It’s not mandatory. It’s just a convention.

And yes, we thought of naming it HREF but since we aren’t actually pointing at hypertext it didn’t seem a good name to use.

8.2 Web3S Location Information

8.2.1 What use is the Web3S Location flag?

It lets the processor know that in so far as the person who specified the URL is aware the URL points to a Web3S resource. This is useful for linking between Web3S resources.

8.3 Sort Query Segment Extension

8.3.1 Shouldn’t this be a HTTP header? Not part of the URL?

The query segment of a URL is well defined in RFC 3986. So we are just using it for its intended purpose – to control the data that is returned. In this case we are controlling the sort order. So we believe our use of the URL is appropriate.

8.3.2 But since sort is optional anyway won’t it hurt caching?

If sort were a header and wasn’t included in vary on the response then a proxy could record the sorted response and return it to people looking for the non-sorted response. The only problem with this approach is that if someone is explicitly looking for a sorted response and submitted a sort header that wasn’t discussed in vary then the cache would return the cached response which might have been sorted differently than the client wants. So this means that anyone making sort requests will have to know to put in cache-control: no-cache, which isn’t necessarily the end of the world but is annoying and is likely to lead to unexpected errors.

On the other hand if we put the sort value into a header then there is no way for one person to just hand a URL to another person that will return sorted results. Since sorting is more a function of massaging the returned data than changing its semantic or even syntax we suspect sort is best specified in the URL.

8.3.3 Why don’t you support publishing the sort tokens in-band?

Just having a list of the sort tokens isn’t terribly useful. What’s needed is an explanation of what those tokens will actually do. But even that list isn’t terribly useful without a good understanding of the schema of the resource since the sorting token will apply to that schema. So what’s really needed is a way to discover a human readable representation of the resource’s schema and sort tokens.

There has been some thought that OPTIONS might be a good way to discover this data but we aren’t sure if that means that every single EII should return OPTIONS with full data or just the top resource or maybe use redirect or…

In any case we have decided to punt on this for now.

8.3.4 Why require the sort tokens to be reverse DNS names?

Mostly to prevent accidental collisions. This is mostly just being ‘very careful’ since in theory someone should not be submitting sort tokens to resources that haven’t defined them. But in case someone does submit a token for one resource to another resource by using reverse DNS names we allow the resource to discover the problem.

8.4 XPATH Query Segment Extension

8.4.1 Why isn’t XPATH syntax specified in the spec?

We ran out of time. So much spec, so little time.

8.4.2 Can you really support full XPATH?

Probably not. There are parts of XPATH (such as functional call outs) that are insecure, parts that are irrelevant (attributes, ordering, etc.) and parts that have horrendous performance implications (union comes to mind). Our expectation is that we will need to profile XPATH down to something that is reasonably implementable.

8.5 SEARCH Method Support

8.5.1 Why isn’t SEARCH syntax specified in the spec?

We ran out of time. So much spec, so little time.

8.5.2 Why do we need a SEARCH method? Isn’t XPATH enough?

XPATH is a great selection language but it isn’t a full query language. A query language not only allows one to specify what data one wants but how that data should be returned. To specify this kind of complex query one really needs a request body which means one needs a method which therefore leads one to SEARCH.