Re: Last Call: SMTP Service Extension for Content Negotiation to Proposed Standard

[Keith Moore <[e-mail]>]

> At 01:53 AM 7/2/2002 -0400, Keith Moore wrote:
> >executive summary:  I recommend  against approval of this document
> >withut extensive changes.  It's not well thought-out
> 
> What additional thinking out are you looking for?

see below.
 
> >  (except for the rather narrow application of SMTP to a one-recipient fax 
> > gateway)
> 
> In fact the motivation for it is not necessarily a gateway, but rather any 
> recipient environment that conforms to this profile, including usage that 
> is wholly retained among Internet-capable participants.
> 
> For that matter, the mechanism applies on a per-recipient basis, so what 
> prompts the claim that it works only for a single recipient?

if different recipients have different capabilities, the sender has
to either send a single message with a format that is suitable for
all recipients, or it has to do a RSET and resend the RCPT commands
separately for each version of the message that it sends.  nowhere
in the document do I see any mention of this, and furthermore it
complicates error handling.  so I assume that it wasn't adequately 
considered.

> >and it seems likely to degrade SMTP interoperabliity if widely adopted.
> 
> How will that happen?  How is it likely?

because the proposal complicates the SMTP state machine (especially when 
you consider the multiple recipient case), it adds additional
failure modes, and the error handling is not well described.
 
> >1. layering problems
> >
> >this functionality is a poor fit for SMTP because SMTP is the
> >mail *transport* protocol, and as such should not be dealing
> >with content at all, which is a user agent function.
> 
> The construct of using SMTP to verify support of a higher level feature is 
> not all that different from DSN. 

now that's a bizarre statement.  DSN is a per-mta feature.  it is either
supported or not supported by an MTA.  it doesn't modify content.  
DSN doesn't reveal anything about a recipient's user agent. 

> >but what's to stop this mechanism from getting out-of-sync also?
> 
> The end-to-end asynchrony and latency for this mechanism certainly do 
> provide an opportunity for a recipient's capabilities to change, after it 
> has advertised a particular set.  However the key point to note is that 
> there is no mediating storage/directory service, so the advertisement has a 
> degree of directness not possible through any other mechanism.

Dave, the document itself recognizes the possibility of a directory
service, even listing a specific error code for this case, so your statement 
is facially false.  Furthermore it must be recognized that the vast majority
of MTAs do not interact directly with user agents at all, so the 
"degree of directness" you claim is completely bogus.

> For recipients concerned about such volatility, all they have to do is 
> refrain from advertising CONNEG support.  

in other words, what you are saying is that this extension has very
narrow applicabliity - it is not  usable on the vast majority of
message transfer agents, but only on those that are tightly coupled
with an MUA or gateway that applies to all recipients.  

so why are we complicating SMTP with an extension that only applies
to a very narrow case?  is support for fax really so important 
that we have to extend SMTP in such a way that it's only usable for
fax, and in such a way that fax-capable MTAs are a completely different
class than other MTAs?

> For senders concerned about such 
> volatility, they should use CONNEG=Required.

the volatility isn't really the issue I'm concerned about - the issue 
is the lack of a standard mechanism for communication of capability
sets from the UA (or the user) to the inbound MTA; and more generally, 
the difficulty of using the CONNEG mechanism to support users who
use multiple UAs with different capabilities.  at least, the latter
scenario doesn't seem to have been considered.

> >it seems like this extension has very limited applicability - it will
> >only work reliably when the SMTP server and user agent or gateway are
> >very closely tied and when the recipient always uses the same UA
> >or UAs of very similar capabilities.  (certainly not true of me!)
> 
> 1.  The specification already highlights that it is tailored for rather 
> 'direct' scenarios.

true, but it also says that indirect ones are possible.

> 2.  Each user can decide what to advertise.  

HOW?  there's no mechanism defined that allows them to do so.

> If they vary their UA they can 
> choose to advertise a safe and lesser set of capabilities.  In any event, 
> it is their choice.  The mechanism does not impose any problems that are 
> not inherent in use of multiple UAs.  However it DOES permit relatively 
> direct advertising of capabilities, which is currently not feasible.
> 
> 
> >but if we're going to extend SMTP to perform functions
> >on behalf of the user agent, it needs to be made very explicit that this
> >negotiation is being done on behalf of the user agent - and in particular
> >that the negotiation being requested represents the preferences of the
> >*user*
> 
> In what way does that affect the technical specification?

because if third parties make unauthorized modifications to messages
then the end-to-end integrity of the message is destroyed.

is that technical enough for you?

> That is, how is the current specification deficient, with respect to your 
> concern?
> 
> 
> >in particular there seems to be no
> >ability to apply content-transformation on a per-recipient basis,
> >to handle the case where different transformations are acceptable to
> >differnet recipients.
> 
> We must be reading different specifications.  Choice of enforcement of this 
> option is per-recipient.  Each recipient may generate a capabilities 
> response if they wish a different form of the content.
> 
> What, exactly, are you looking for that is missing?

reread my earlier message, and see above.

> >of course it's not easy to get SMTP to handle different client-side
> >transformations for different recipients, since SMTP is designed to
> >transfer the exact same message body (modulo received fields) to each
> >of a set of recipients.
> 
> It sounds as if you are assuming that later transmission of content that is 
> revised according to capabilities information is required (or expected) to 
> be sent using that same address list.

well, by the time the sender knows what the recipients' capabilities are,
the receiver already thinks it has the recipient list, and it is expecting
a single messgae to be sent to all of those recipients.

maybe you need to read the draft you claim to have co-written?

> 
> That is like requiring that all BCC copies are sent along with the TO and 
> CC copies.
> 
> In other words, there is not such requirement.  It is OK to do it and it is 
> OK not to.
> 
> Yes, the current CONNECT option is intended to send the same content to all 
> recipients, since that is the style of SMTP.  There is nothing preventing 
> later sending an alternative content form to selected recipients.

so you're suggesting that multiple copies of the message be sent to those
recipients?  

> Interestingly, this would not actually create any additional round-trips or 
> additional content transfers.

that's about the most bizarre thing I've ever seen you write...

> 
> >  this is a different model than used by
> >phone-based facsimile protocols which expect there to be only one
> >recipient of any given transmission,
> 
> You might want to take a look at fax broadcasting services.  The last-hop 
> telephone portion behaves as you describe, but only after a 'generic' 
> posting process.

perhaps, and I'm not familiar with these.  but my guess is that they don't
do per-recipient negotiation on earlier hops.

> >  so it's not surprising that the fax-style negotiation doesn't fit well 
> > into SMTP.
> 
> Actually, the proposed scheme permits content transmission tailoring that 
> is considerably more flexible than is typical in the real fax world.
> 
> 
> >in order to make client-side negotiation a clean fit into SMTP, it is
> >necessary for the client to first query the server for the capabilities
> >of each intended recipient, then to deduce how many total transformations
> >are necessary, then to send one or more transformed versions of the message.
> 
> Ahh.  I see.  You want to insist on an interaction scenario that is known 
> to be unacceptable, since it is guaranteed to introduce too much delay 
> before there is any utility at all.

PIPELINING would make the delay negligible.  and it's far preferable to 
overloading the RCPT command.
 
> >   overloading the RCPT
> >command to return capabilities breaks the RCPT command and forces
> >the client to do RSET in order to send different versions to
> >different recipients.
> 
> If that is what the client chooses to do, then yes.  However there are 
> other courses of action available.  At the least, note that the content is 
> not sent until the client has CONNEG responses about all of the recipients.

perhaps you'd like to describe those "other courses" which allow the
client to meet the requirement that it deliver the highest quality 
content to each recipient?  or for that matter, that allow the client
to satisfy mutually conflicting needs of recipients?  (e.g. one recipient
will only take gif while another will only take jpeg)

> >putting content negotiation in the transport protocol is likely
> >to result in a significant increase in anomolous behaviors
> >from the mail system which are difficult to diagnose and fix -
> >and the proposal is being introduced at a time when the mail system
> >is already suffering from serious reliabliity problems (often due
> >to ill-conceived measures for blocking spam).
> 
> How does use of the CONNEG mechanism affect reliability or accountability?

it complicates error handling.

> >   at this time,
> >*any* proposal for adding new functionality to SMTP should be viewed
> >with strong reservations, unless it somehow offers an improvement
> >in reliability.
> 
> I was not aware that we had any problems being attributed to SMTP 
> options.  Please provide some citations, so we can compare their details to 
> the current proposal.

the problems are with SMTP in general.  if you want citations, go read
delivery failure notices from a few large mailing lists.

> 
> >4. naming and applicability
> >
> >the protocol claims to be a general-purpose content-negotiation scheme
> >but actually has "fax" assumptions embedded very deeply.
> 
> The fax world is the basis for the idea, yes.  However the specification 
> has been done so as to be independent of fax, thereby using an approximate 
> model that is known to be useful from the fax world and then applying it 
> for general enhancement of email.

It's evident that not very much thought has been invested in making
the non-fax case work.

> >first of all, it assumes that there is a linear ordering relationship
> >for document suitability to the recipient.
> 
> Ok.  I give up.  I have no idea what "linear ordering relationship for 
> document suitability" means.

it means that the proposal assumes there is a notion of "higher quality" 
vs "lower quality" versions of a message that can be determined for all 
messgaes and that is obvious to an implementor.

> >a.  I strongly recommend that you redesign this extension so that, instead
> >of overloading the RCPT command, you define a separate command to
> >request a recipient's capabilities.  e.g.
> 
> In spite of the proffered benefits of a separate command, I fail to see how 
> it produces any real benefits.
> 
> It certainly does not make anything simpler, and it certainly vastly 
> increases relay delay, due to nearly doubling the round-trip exchanges 
> during an SMTP session that uses the option.

it gets you out of the "oops I don't really want to send this message to
that recipient because that recipient needs a different format of the
message" trap.

as far as round trips, RCAP RCAP RCAP RCPT DATA RCPT DATA RCPT DATA
is slightly better than RCPT RCPT RCPT RSET RCPT DATA RCPT DATA RCPT DATA

but you really ought to pipeline these things anyway.  
 
 
> >one nice side-benefit is that it doesn't change the error handling
> >model,
> 
> How does the current proposal change the error handling "model"?

by introducing new error conditions that require different handling
than existing errors.

> >b. the idea that the sender should send the "best" or "highest
> >quality" version needs to be relaxed or generalized,
> 
> It already has been relaxed.  Note the SHOULD rather than MUST.

it's still a conditional requirement of the specification that
is so vaguely defined as to be unimplementable except perhaps
for the narrow fax case.  different implementors could easily interpret
the SHOULD in different ways even absent any conditions that would allow
them to disregard the SHOULD.  in other words, it doesn't produce 
predictable behavior.

> >  since there's currently no clear notion of what this means.
> 
> Yet we have lived with that notion in multipart/alternative for some years.

in retrospect we might consider mp/alternative a bug.
I don't think citing it adds weight to your argument.

> 
> >c. this extension needs to be made less fax-specific.
> 
> What, specifically, makes it fax specific, and what changes are you 
> specifically seeking that will be less fax-specific?
> 
> 
> >it needs to be examined with some realistic non-fax-specific
> >test cases, and it needs non-fax conneg examples also.
> 
> Test cases are always a good thing.  Feel free to offer some that satisfy 
> your non-faxiness goal.

hey, you're the one making the proposal, seems like it's your job to
make sure it really has the applicability that you claim it does.  

I don't really think the proposal provides enough benefit to make it
worthwhile even if it's done right. but rather than rejecting it out 
of hand (and recognizing that the fax community wants this badly)
I'm trying to offer some constructive criticism that would make it 
less objectionable.

> >d. there needs to be a good way for the recipient to specify
> >capabilities to the SMTP server, at least for any SMTP server
> >that can serve arbitrary UA recipients.  probably this means another
> >extension along with SMTP AUTH.
> 
> You want to require that that mechanism be in place before the current 
> specification can be advanced?  I think the phrase "chicken and egg" works 
> here.  You create a deadly embrace for reaching any utility.

I think it's called 'having a complete specification'.   I didn't
say anything about having the mechanism deployed, but I do think it
needs to be defined.

> >still, this is a big can of worms, because there's never been
> >any ability of a UA to specify something to its inbound SMTP
> >servers before - there's presently not even a good way for
> >a UA to know which server(s?) should be given this information...
> 
> Think hard.  What is likely to drive creation of such a 
> mechanism?  Requiring that it be built in the abstract or looking for it to 
> be built in reaction to actual demand?

no argument there - but my point is that this is breaking new ground, 
and as such there are many issues that need to be considered that we
haven't thought of before.

> 
> >e. make it absolutely clear that such transformations are only to be
> >made with explicit authorization by the sender and/or recipient.
> 
> This sort of parental direction in IETF specifications probably does no 
> harm, but it is difficult to imagine that it does much benefit.

tell that to the OPES people who were trying to push for web 
proxies that would modify content without either the provider's
or consumer's consent.     the last thing I want is an ISP 
advertising content capabilities on my behalf without my authorization -
and I can think of some large ones that would do exactly that.

I used to think we could rely on each layer to maintain the appropriate
degree of transparency for that layer without our demanding that it be so.
history has proven me wrong.  it needs to be specified.
 
> >f. the document needs to consider the effect of message forwarding
> >on the returned string - if a recipient has his mail forwarded
> >elsewhere it's not clear what conneg string should be returned.
> 
> There are all sorts of implications in forwarding.  In general, forwarding 
> that is not simply SMTP relaying is outside of current IETF 
> specification.  Imposing a requirement about it here, without demonstrating 
> actual problems, is a tad extreme.

apparently you'd rather just deploy a new protocol extension without 
even bothering to think about how it affects the infrastructure.

let's just say I disagree.

> >g. the document alludes to the ability to work through intermediaries
> >but this isn't explained anywhere, and it needs to be explained.
> 
> All it says is that it works with SMTP relaying, just as one would expect 
> and SMTP extension tow work.

how does it work with SMTP relaying if there's no defined way to 
relay the capabilities?  either you leave it to the last hop,
which (while fine with me) makes the statement kind of trivial,
or you do something really ugly...  again, a bit of clarification
of this statement wouldn't hurt.

> > > 4.2 Client Action:
> >...
> >this incorrectly assumes that (a) there is only one target when
> >the client can easily specify multiple RCPT commands, and
> >(b) there is a clear notion of "highest level of capability"
> >for the target.
> 
> It does not make the assumption you suggest.  In fact, I can't imagine how 
> you came to the assessment that it makes that assumption.

well, I can't imagine how you came to a number of the assessments you
are claiming.  perhaps we live in different universes.

> 
> > > 4.3 Server Action:
> > >
> > >      If the client specifies "CONNEG = REQUIRED" in the RCPT-TO,
> > >      but the target does not support the CONNEG parameter, the target
> > >      MUST reject the RCPT-TO command with a 504 reply.
> >...
> >(it may be that other extensions have similar wording, in which case
> >they're similarly flawed)
> 
> It is probably a good idea that we not try to fix SMTP (yet again, given 
> the protracted drums effort.)  And it certainly is not a good idea to 
> require that the current specification fix something this vague when no 
> SMTP-related specification is required to.

so we should always write new specifications as poorly as existing ones?

this wasn't so much of a problem when we didn't have so many extensions -
but the more extensions we have the more problems like this we introduce .

> > >      If the client specifies "CONNEG = OPTIONAL" in the RCPT-TO, the
> > >      target MUST process the address and message as if the requested
> > >      CONNEG capabilities had not been specified.
> >
> >this doesn't seem to make sense as written.  is "CONNEG = OPTIONAL"
> >really supposed to be interpreted the same as omission of the
> >CONNEG parameter?
> 
> I think that the text that is in the first paragraph of that section ("but 
> the target does not support the CONNEG parameter") got deleted during 
> revision.  In general, those first 3 paragraphs are dealing with lack of 
> support, whereas the following paragraphs are dealing with presence of support.
> 
> 
> >also, I don't think there should be spaces around the = sign -
> 
> ack.
> 
> 
> >I wonder here - assuming that this can of worms does not get closed
> >back up, and there might later be demand for other mechanisms
> >to request recipient capabilities (e.g. since this one is not very general),
> >might there be multiple kinds of strings emitted in the RCPT response?
> 
> Please indicate what capabilities are missing from this capabilities 
> mechanism that would make it more general and for which there is a 
> demonstrated need.

actually this one is the least of my concerns about the document.
if the other problems can be fixed this one is easy...
but the fix for this depends to some degree on how you fix the other
problems, so I'll defer making suggestions for now.

Keith