Re: [apps-discuss] Review of draft-melnikov-smtp-priority-14

Alexey Melnikov <alexey.melnikov@isode.com> Fri, 01 June 2012 19:59 UTC

Return-Path: <alexey.melnikov@isode.com>
X-Original-To: apps-discuss@ietfa.amsl.com
Delivered-To: apps-discuss@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B91A611E80CC; Fri, 1 Jun 2012 12:59:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.893
X-Spam-Level:
X-Spam-Status: No, score=-102.893 tagged_above=-999 required=5 tests=[AWL=-0.294, BAYES_00=-2.599, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0uVSED1ZPB+5; Fri, 1 Jun 2012 12:59:07 -0700 (PDT)
Received: from rufus.isode.com (cl-125.lon-03.gb.sixxs.net [IPv6:2a00:14f0:e000:7c::2]) by ietfa.amsl.com (Postfix) with ESMTP id 4FE5611E80A0; Fri, 1 Jun 2012 12:59:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1338580745; d=isode.com; s=selector; i=@isode.com; bh=ulLGL7Jt5UaIcnHerVyNu62UkX28P9GeRH+QegregtQ=; h=From:Sender:Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version: In-Reply-To:References:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description; b=L+y9W70Lgu4oWR8CnVihOzDW/jrcC7aNMWhWYCa+uG5DQ6LnMsjlMk+cSpYhGBOfSHOCtA zlhKwOF8vLLtvRhelIJqBR/TR9c8SbnLgLoPZXwWgLiMbMZNQoMWK76VtMYSLRedyxfaCd LP/lj/9EzdnJlf/J6FlysJMQSAvqgtA=;
Received: from [192.168.1.144] ((unknown) [62.3.217.253]) by rufus.isode.com (submission channel) via TCP with ESMTPSA id <T8kfCAAE42bU@rufus.isode.com>; Fri, 1 Jun 2012 20:59:05 +0100
X-SMTP-Protocol-Errors: NORDNS PIPELINING
Message-ID: <4FC91F09.7070701@isode.com>
Date: Fri, 01 Jun 2012 20:59:05 +0100
From: Alexey Melnikov <alexey.melnikov@isode.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
To: Ned Freed <ned.freed@mrochek.com>
References: <6.2.5.6.2.20120521130747.0c219ab0@elandnews.com> <4FBDF199.2050300@isode.com> <4FC7204B.8020403@dcrocker.net> <01OG663WXFQA0006TF@mauve.mrochek.com>
In-Reply-To: <01OG663WXFQA0006TF@mauve.mrochek.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"; format="flowed"
Content-Transfer-Encoding: 7bit
Cc: draft-melnikov-smtp-priority-13.all@tools.ietf.org, iesg@ietf.org, apps-discuss@ietf.org
Subject: Re: [apps-discuss] Review of draft-melnikov-smtp-priority-14
X-BeenThere: apps-discuss@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: General discussion of application-layer protocols <apps-discuss.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/apps-discuss>, <mailto:apps-discuss-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/apps-discuss>
List-Post: <mailto:apps-discuss@ietf.org>
List-Help: <mailto:apps-discuss-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/apps-discuss>, <mailto:apps-discuss-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Jun 2012 19:59:08 -0000

Hi Ned,

On 01/06/2012 17:18, Ned Freed wrote:
> A few comments are in order here.
>
>> >    particularly important during emergencies for first responders and
>> >    for environments such as military messaging, where messages have 
>> high
>> >    operational significance, and the consequences of extraneous delay
>> >    can be significant.
>> >
>> >    In order for an SMTP receiver to be able to send higher priority
>
>>     send -> relay
>
> Given the imprecise nature of this document overall, it's a stretch to 
> believe
> that being precise here is really necessary, but if it is, this fixes 
> only part
> of the problem. The message may have been received via SMTP, SUBMIT, or
> something else. And the higher *or* *lower* priority may apply to 
> whatever the
> next step is: SMTP, LMTP, or something else.
>
> If you want to be precise here, it needs to say something like:
>
>      In order for an MSA, MTA, or MDA to be able to relay, deliver, or
>      otherwise process higher priority messages first, ...
Yeah, I think this is less readable than what I have. I am sure readers 
understand the meaning of what is being said.
>
>> >    messages first, there needs to be a mechanism to communicate 
>> (during
>> >    both Message Submission [RFC6409] and Message Transfer 
>> [RFC5321]) the
>> >    priority of each message.  This specification defines this 
>> mechanism
>> >    by specification of an SMTP [RFC5321] extension.
>> >
>> >    Implementors of this document might also consider implementing
>> >    [PRIORITY-TUNNELING] which talks about tunneling of Message 
>> Transfer
>> >    Priority information through Message Transfer Agents (MTAs) that do
>> >    not support this extension, using a new message header field
>> >    [RFC5322].
>
>> Consider replacing above paragraph with:
>
>>     In order to permit end-to-end use of this extension across email
>> infrastructure that does not support it, a companion tunneling mechanism
>> is defined in [PRIORITY-TUNNELING] through use of a new message header
>> field [RFC5322].
>
> This is better. These specifications are not just for implementors.
Yes, already used in -15.
>> > 2.  Conventions Used in This Document
>> >
>> >    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>> >    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 
>> this
>> >    document are to be interpreted as described in [RFC2119] when they
>> >    appear in ALL CAPS.  These words also appear in this document in
>> >    lower case as plain English words, absent their normative meanings.
>
>> "when they appear in ALL CAPS".  sigh. In other words, this document is
>> modifying RFC2119.
>
> On the contrary, all this is doing is being specific about the way RFC 
> 2119 has
> actually been used ever since it was created. I can easily produce a 
> long list
> of RFCs containing instances of these words in lower case where, even 
> though it
> wasn't stated, the intent *clearly* was for lower case versions of 
> these words
> to have their regular meanings.
>
> And even if I were to accept the notion that this document specifies a 
> novel
> derivation of RFC 2119 (the notion that it *modifies* it being patently
> absurd), that ship has also sailed. I can also easily point to RFCs 
> that do
> things like specify ALLS CAPS versions of these words have one 
> meaning, Mixed
> Case another, and lower case yet another.
>
>> Does anyone seriously expect this new nuance of usage to be read and
>> remembered by average readers of this document?
>
> The better question is does anyone seriously expect anyone to assume, 
> given
> abundant evidence to the contrary, that the lower case versions of 
> these words
> will be taken to have RFC 2119 meaning?
>
>> FWIW, it took me about 3 minutes to make the relevant changes, below.
>> Case-sensitivity in semantics invites comprehension problems here.
>> Worse, there is absolutely no need or benefit in creating the problem
>> for this document.  At a minimum, please do not try to modify IETF
>> document writing policy on the fly.
>
> You're the one who is arguing for a modification to the *actual* IETF 
> document
> writing policy here.

Agreeing with that.

>> >    The formal syntax use the Augmented Backus-Naur Form (ABNF) 
>> [RFC5234]
>> >    notation including the core rules defined in Appendix B of RFC 5234
>> >    [RFC5234].
>> >
>> >    In examples, "C:" and "S:" indicate lines sent by the client and
>> >    server respectively.  Line breaks that do not start with a new "C:"
>> >    or "S:" exist for editorial reasons and are not a part of the
>> >    protocol.
>> >
>> >    This document uses the term "priority" specifically in relation to
>> >    the internal treatment of a message by the server: messages with
>> >    higher priorities may be given expedited handling, and those with
>
>>     may -> can
>
>
>> >    lower priorities may be handled only as resources become available.
>
>>     may -> can
>
> Again, given the loosey-goosey nature of this document overall, I 
> don't think
> precision is really required, but if it is, both of these changes are 
> somewhat
> objectionable.
>
> Like it or not, "may" and "can" are *not* synonyms. "Can" is more 
> assertive,
> and carries with it a flavor that's closer to "should", whereas "may" is
> expresses a possibilit with no preference.
>
> We're talking about prioritization methodology here, specifically 
> expedited
> handling and deferred processing. There can be overhead associated 
> with both of
> these, and we should not be encourage the use of these techniques when 
> it isn't
> absolutely necessary.
> And yes, I'm being very pedantic here. That's what you're going to get 
> when
> you elect to go down this path.
:-)
>
>> >
>> > 3.  Definition of the Priority SMTP Extension
>> >
>> >    The Priority SMTP service extension is defined as follows:
>> >
>> >
>> >
>> >
>> > Melnikov & Carlberg     Expires November 25, 2012               
>> [Page 3]
>> > 
>> > Internet-Draft  Message Transfer Priority SMTP Extension        May 
>> 2012
>> >
>> >
>> >    1.  The textual name of this extension is "Priority Message
>> >        Handling".
>> >
>> >    2.  The EHLO keyword value associated with this extension is "MT-
>> >        PRIORITY".
>> >
>> >    3.  The EHLO keyword has no parameters.  Parameters are reserved 
>> for
>> >        possible future extensions and MUST be ignored by clients that
>> >        don't understand them.
>> >
>> >    4.  No additional SMTP verbs are defined by this extension.
>> >
>> >    5.  One optional parameter ("MT-PRIORITY") is added to the MAIL 
>> FROM
>> >        command.  The value associated with this parameter is a decimal
>> >        integer number from -9 to 9 (inclusive) indicating the priority
>> >        of the email message.  The syntax of the MT-PRIORITY 
>> parameter is
>> >        described by the <priority-value> ABNF non-terminal defined in
>> >        Section 6.  Higher numbers mean higher priority.
>
>> As a minor point, the form of -9 to 9 seems more complicated than
>> useful.  Is there a need for 19 values?  I'd think that 0 to 9 would be
>> sufficient.
>
> Well, to be fair, the specification only requires support for six 
> distinct
> values:
>
>   A message priority is a decimal integer in the range from -9 to 9
>   (inclusive).  SMTP servers compliant with this specification are not
>   required to support all 19 distinct priority levels (i.e. to treat
>   each priority value as a separate priority), but they MUST implement
>   at least the following 6 distinct priority levels: -4, -2, 0, 2, 4,
>   9.  I.e. an implementation that only supports the 6 priority levels
>   will internally round up a syntactically valid level that isn't
>   supported to the next higher supported number.  For example, such an
>   implementation will treat priority values below and including -4 as
>   priority -4, priority -3 as priority -2, and all priorities starting
>   from 5 are treated as priority 9.  An SMTP server MAY support more
>   than 6 different priority levels.  Decision about which levels to
>   support in addition to the 6 mentioned above is a local matter.
>
> Now, I have no idea what it actually means to "support" six values, but I
> assume it means that if you only make a processing distinction between 
> five or
> fewer values total, if you support only one "lower than normal" 
> priority, or
> you support fewer than three "higher than normal" priorities, you cannot
> implement this specification. (Note I'm assuming 0 is "normal" here - see
> below.)
More or less, yes.
>
> Of course there's a trivial way around this: You simply state that, as 
> a matter
> of implementation policy, you only support X number and here's the 
> mapping.
> This would allow, say, a system that only does the old X.400 non-urgent,
> normal, urgent thing to claim support.

Right. And there is already such a mapping in Appendix B.

>> For that matter, why are many values needed?  What is the basis for
>> choosing to have a range of more than a few values?
>
> IMO this is one of the rare cases where X.400 was spot-on correct.
>
>> This appears to be the only place the provides semantics to the priority
>> number.  As such, it should be more thorough.  The critical issue in
>> assigning the number, I think, is trying to get independent originators
>> to assign numbers according roughly the same criteria.  For example, if
>> I label "average" messages I send as as 0 and you label them as 1, then
>> your messages get preferential treatment over mine, although that was
>> not your intent.  (Assuming you're not trying to game the service...)
>
>> This issue goes to the core of the usage model for the feature:  It
>> really needs an environment tailored to its use, with all participants
>> not only within the same trust system, but sharing the same priority
>> assignment model/policy.  It's probably good for this document to permit
>> a range of assignment models, but should note that an operational
>> requirement for use of the extension is that an assignment policy be
>> defined for all participants.
>
>> This document might, then, offer a candidate model, to be use by default
>> and absent one specific to an environment.
>
> Well, this goes to the problem I have with the entire document - I 
> have no real
> idea what the policy associated with these priorities is supposed to 
> be. We
> have a prioritization capability in our product and I can tell you 
> what it
> does, but s to whether or not it satisfies the intended target of this
> specification, I realy couldn't say.

I hope I clarified this in -16. Also look at Appendix A and Appendix B.

> More generally, in a modern, high performance MTA that's capable of 
> processing
> large numbers of messages simultaneously, there's a significant problem
> actually implementing mechanisms that are guaranteed to preferentially 
> process
> certain messages.
>
> For example, suppose I have several dozen threads/processes/whatever 
> connected
> up to a given system, all busy transferring large messages. A higher 
> priority
> message comes in. If I shove this message on the thread that is the 
> closest to
> being done, there may still be a significant delay. OTOH, if I start a 
> new
> connection to handle it, what if I hit the limit on the number of 
> connections
> the remote MTA will accept from me?
>
> Put another way, if you assume a fairly strict policy is needed, your 
> point
> about this only being applicable in an environment that's specifically 
> tailored
> for it is, if anything, an massive understatement. For this to provide 
> a real
> priority boost with high reliability it's actually necessary to closely
> coordinate operational policies across all involved ADMDs. And not to 
> put too
> fine a point on it, the level of technical acumen needed to actually 
> do this
> sort of thing is something I've observed to be the exception and not 
> the rule.
>
> All that said, I think this actually argues for putting in some 
> statements
> as to what this document does *not* do. It calls for best effort to 
> provide preferential processing, nothing more. It does *not* provide
> any sort of guarantees.

Fair enough, I will add some text about that. Best effort was certainly 
the intent.

> And if that's not the case, then this needs to be experimental.
>
>> >    6.  The maximum length of a MAIL FROM command line is increased 
>> by 15
>> >        octets by the possible addition of a space, the MT-PRIORITY
>> >        keyword and value.
>> >
>> >    7.  The MT-PRIORITY extension is valid for the submission service
>> >        [RFC6409] and LMTP [RFC2033].
>> >
>> > 4.  Handling of messages received via SMTP
>> >
>> >    This section describes how a conforming SMTP server should 
>> handle any
>> >    messages received via SMTP.
>> >
>> > 4.1.  Handling of the MT-PRIORITY parameter by the receiving SMTP 
>> server
>> >
>> >    The following rules apply to SMTP transactions in a server that
>> >    supports the MT-PRIORITY parameter:
>> >
>> >    1.  A conforming SMTP server MUST NOT refuse a MAIL FROM command
>> >        based on the absence of the MT-PRIORITY parameter.
>
>> hmmm.   For some operational environments, it will be essential to have
>> /all/ handling conform to the priority policy model in force for the
>> environment.  In those cases, the absence of the parameter would be a
>> violation of the spec.
>
> Interesting point. I'm not sure what, if anything, to do about it.
>
>> >    2.  If any of the associated <esmtp-value>s (as defined in Section
>> >        4.1.2 of [RFC5321]) are not syntactically valid, or if there is
>> >        more than one MT-PRIORITY parameter in a particular MAIL FROM
>> >        command, the server MUST return an error, for example "501 
>> syntax
>> >        error in parameter" (with 5.5.2 Enhanced Status Code 
>> [RFC2034]).
>> >
>> >    3.  When inserting a Received header field as specified in Section
>> >        4.4 of [RFC5321], the compliant MTA/MSA SHOULD include the
>> >        "PRIORITY" clause whose syntax is specified in Section 6.
>> >
>> >
>> >
>> > Melnikov & Carlberg     Expires November 25, 2012               
>> [Page 4]
>> > 
>> > Internet-Draft  Message Transfer Priority SMTP Extension        May 
>> 2012
>> >
>> >
>> >    4.  If the sending SMTP client specified the MT-PRIORITY 
>> parameter to
>> >        the MAIL FROM command, then the value of this parameter is the
>> >        message priority.
>> >
>> >    5.  If no priority has been determined by the above, the server may
>
>>     may -> can
>
>
>> >        use its normal policies to set the message's priority.  By
>> >        default, each message has priority 0.
>
>> ouch.  This chooses a particular model for meaning of the numbers, and
>> without having defined the model beforehand.
>
> I for one was assuming 0 was the normal, default priority. But now 
> that you
> mention it, at least this needs to be made explicit.

I believe the document already says that, but I will double check.

>> That is, I think the problem with this default is that it really is
>> assuming a particular policy for meaning of the values, namely that
>> average mail is mid-priority, assuming the numbers have linear 
>> semantics.
>
> That sure was what I was assuming.

Yes.

>> >    The SMTP server MUST NOT allow upgraded priorities from untrusted
>> >    (e.g. unauthenticated) or insufficiently trusted sources.  (One
>> >    example of an "insufficiently trusted source" might be an SMTP 
>> sender
>> >    which authenticated using SMTP AUTH, but which is not explicitly
>> >    whitelisted to use the SMTP MT-PRIORITY service.)  The server MAY,
>
>> It is good that this issue is raised, but it really requires an earlier,
>> separate and careful discussion of the need for this extension to be
>> used within an administered trust environment.  Given such an
>> introductory section, the above text would be sufficient.
>
> Agreed.

Ok.

>> In its absence, the reference to a whitelist is based on the very large
>> assumption that participants in the extension are whitelisted.  This is
>> not document elsewhere.
>
> True.

Hmm, Ok.

>> >    however, allow an untrusted source to lower its own message's
>> >    priorities -- consider, for example, an email marketer that
>> >    voluntarily sends its marketing messages at low priority.
>
>> To beat the point a bit deader:  this assumes a particular policy for
>> the meaning of the priority values.  Worse, it also appears to
>> contradict the earlier default of 0, but perhaps I've misunderstood 
>> that.
>
> I think that's a misunderstanding.

I already replied in another message that it is.

> I generally agree with the remainder of your comments, although I 
> think some
> of them become less important as long as it's clear this doesn't provide
> guaranteees, but they still should be addressed.

I believe I addressed most of them in -15 and -16. I might need to add a 
couple of clarifying sentences as per above.