Re: X-* header fields (Was: Getting 2822 to Draft)

[Bruce Lilly <[e-mail]>]

Pete Resnick wrote:

> X-* fields are perfectly legal in 2822 (they fall under 
> "optional-field"); they simply are not given the special treatment of 
> 822, that they won't ever be official published extensions to the 
> standard. The only difference then between 822 and 2822 in this 
> respect is that 822 gives publication guidelines for extensions where 
> 2822 does not. This has absolutely *no* effect on implementations of 
> the protocol.

There is a minor effect. A user-defined field (per 822 definition) can 
be recognized as such by examining the
first two octets of the field name, which is quite efficient.  While 
there exist efficient methods of identifying
fields (e.g. http://www.gnu.org/directory/gperf.html), they are not 
quite *as* efficient as a two-octet (case-
insensitive) comparison.  Moreover, "X-" serves to differentiate 
user-defined fields from non-defined
fields (i.e. those field names for which there is no IETF published 
definition or which the implementation
does not recognize).

> Now, let me use this opportunity to explain *why* I don't think X-* 
> fields should be in the standard in the first place:
>
> The idea behind X-* fields (what 822 called user-defined fields) was 
> so that folks could use new field names without worrying that they 
> would interfere with the standard fields. That way, you didn't have to 
> go through the trouble of publishing an RFC just to use a field that 
> was only going to be internal to your particular system. The downside 
> was that someone else could come along and potentially use the same 
> field name in a completely incompatible way, but that was the risk 
> that you took.
>
> But since then, two things have made it clear that such user-defined 
> fields are unnecessary and problematic:
>
> 1. Unnecessary: We now have IANA registration services. We could very 
> simply create an IANA registry of e-mail field names. Then, if you 
> wanted to create a new field, you just fill out the registration form 
> and you're not only guaranteed that you won't interfere with published 
> standard fields, you're also guaranteed that nobody else would use the 
> field you chose in an incompatible way. And you wouldn't have to have 
> a name that started with "X-".

Yes, but we don't yet have such a field name registry (and obviously we 
didn't have one in 2001 when
2822 was published).  In order to be at all useful to implementors, 
registration would have to be contingent
upon the existence of a stable, formal, public definition of the 
proposed field's syntax (with ABNF) and
semantics.  That differs from publication via RFC regarding review and 
comment; and I'm not convinced
that foregoing such review and comment would be a good thing.

> 2. Problematic: Some folks thought it was nice to have "X-*" fields 
> for completely private use. But history teaches us that inevitably 
> "X-*" fields *will* leak out onto the rest of the Internet. As several 
> people in this thread have pointed out, we now have some "X-*" fields 
> that are in widespread use. Sometimes, people use them very 
> interoperably and they have served a fabulous purpose. Sometimes, they 
> don't get used interoperably. It would be great if we could publish 
> some standards to make sure that they do get used interoperably. But 
> guess what: We can't! It is forbidden by RFC 822 to publish any 
> standard for a field name that beings with "X-". No matter how 
> widespread the use, no matter how useful the field is, it can not be 
> standardized *by definition*. That decreases interoperability on the 
> Internet.

It's difficult to see how such problems are either unique to X- or are 
more of a hindrance to
interoperability than for other fields. As an example, consider "Status" 
which was in "private" use
(I believe) by BSD "mailx" decades ago, and which is currently in use by 
several other MUAs,
and which *does* leak out; there is also a formal definition of a 
"Status" field -- incompatible
with the private usage -- defined as one of the delivery status 
notification fields (RFC 3464).
So neither leakage nor incompatibility seem to be unique to X- fields.  
Note that if BSD mailx'
author(s) had used X-Status for private use, there would be no conflict 
with the formal DSN
Status field.

> Now, personally, I would love the document to say, "There is no good 
> reason to use fields that begin with 'X-' as defined by RFC 822; any 
> name that does not interfere with a currently registered or published 
> field name is fine, and using an 'X-' field is discouraged because 
> they can't ever be standardized." But there was widespread 
> non-consensus in the DRUMS Working Group when this was discussed. 2822 
> does say "Extension header fields no longer specifically called out." 
> In 2822bis, I think it would be fine to clarify that and say 
> "User-defined header fields (those starting with 'X-') no longer 
> specifically discussed", and even have an informational reference to a 
> document (or 2) on why it was pulled. But I see absolutely no reason 
> that this should stop 2822bis from moving to full Standard and 
> obsoleting 822.

There is one very good reason to use X- for private or experimental use, 
viz. interoperability.
Use of X- as a field name prefix for private or experimental use 
guarantees that there will be
no conflict with a formal field name.   Use of other names can lead to 
conflicts, as in the case
of "Status", the damage in that case fortunately being limited by the 
fact that in that case, DSN
fields are significant only in one part of a specific type of MIME 
multipart message and the
private use occurs only in the top-level message header.

An issue not mentioned above is migration of X- fields to a formal 
definition following
successful experimental use.  That obviously entails a name change. 
However name changes
are not uncommon, e.g. refer to the MIXER RFCs, which define a number of 
fields whose
names have changed.  That merely means that parsers need to recognize 
one name as a
synonym for another.

I don't see the X- issue as any reason to keep 2822 from progressing; 
from a practical point of
view (as noted by others), parsers will need to be able to handle the X- 
fields already in use,
reading "old" messages still requires parsers to be able to handle RFC 
822 syntax -- and that
will continue to be the case even after 822 is eventually obsoleted 
regardless of what 2822
et succ. have to say about the matter, just as it is now necessary to 
support at least some
RFC 733 and 724 constructs to be able to read really "old" messages even 
though those RFCs
have been formally obsolete for decades.