[apps-discuss] RFC5336 and VRFY/EXPN (was: Re: [EAI] apps-team review of draft-ietf-eai-rfc5336bis)

[John C Klensin <[e-mail]>]

(writing as a personal opinion, not speaking for the WG.  Shawn
and others have already addressed parts of this, but, for
completeness...)

--On Wednesday, December 22, 2010 18:01 -0800 Ned Freed
<[e-mail]> wrote:

>...
> (1) 3.1 item 4 and 3.6.4.2 add an optional UTF8REPLY parameter
> to VRFY and     EXPN. They do this by redefining the syntax to
> be:
> 
>     vrfy = "VRFY" SP ( uLocal-part / uMailbox )
>                      [ SP "UTF8REPLY" ] CRLF
> 
>     expn = "EXPN" SP ( uLocal-part / uMailbox )
>                      [ SP "UTF8REPLY" ] CRLF
> 
>     The original syntax for VRFY and EXPN given in RFC 5321 is:
> 
>     vrfy = "VRFY" SP String CRLF
>     expn = "EXPN" SP String CRLF
>     String = Atom / Quoted-string
>     Atom = 1*atext
> 
>     It's not immediately obvious, but the new syntax is in
> fact a superset of     the old. (uLocal-part subsumes
> local-part, which in turn allows atoms     and quoted
> strings.) However, this change does much more than hang a
> parameter on the command; for the first time an entire address
> is     permitted as an EXPN/VRFY argument. But no semantics
> are given for     this significant syntax expansion, nor does
> this expansion appear to be     within scope for EAI.
> 
>     A more reasonable syntax would be:
> 
>     vrfy = "VRFY" SP uString [ SP "UTF8REPLY" ] CRLF
> 
>     expn = "EXPN" SP uString [ SP "UTF8REPLY" ] CRLF
> 
>     uString = uAtom / uQuoted-string

Yes.  I don't think the above mess was intentional.   In
fairness to the authors, the ABNF has been changed too many
times by too many people and I think the situation above is a
side-effect.   As you point out, the syntax changes may not be
adequate (see below).

>     But this now fails along a different dimension: When
> adding a parameter     capability to a command, it should be
> done in a general way, so that any     future extensions can
> build on a cleanly specified foundation. Like so:
> 
>     vrfy = "VRFY" SP uString [ SP vrfy-parameters ] CRLF
>     vrfy-parameters = esmtp-param *(SP esmtp-param)
> 
>     expn = "EXPN" SP uString [ SP expn-parameters ] CRLF
>     expn-parameters = esmtp-param *(SP esmtp-param)
> 
>     uString = uAtom / uQuoted-string
> 
>     Of course this will add 555 as a possible response to
> VRFY/EXPN, and that needs to be noted.

Right. See below.

In addition, one must have a session for which mail transactions
are permitted (i.e., EHLO must have been sent) in order to use
any parameters at all.  RFC 5321, Section 4.1.4, explicitly
permits sending VRFY or EXPN without first having EHLO in the
session.  So there is another implicit requirement in all of
this that should almost certainly be made explicit.   (I think
this is the topic about handshaking that Dave and Shawn have
been circling around, but I'm not sure).

 
> (2) In response to Dave's critique/query about why the
> parameter on VRFY and     EXPN is needed, it's because these
> commands can appear outside of a     transaction and thus
> there's not necessarily an indication that the client     is
> prepared to accept UTF-8 responses. 

Yes, exactly.

> However, this brings up
> another     issue, which is that a server that supports this
> extension needs a way     to tell a client that doesn't that
> it cannot return a proper response     to VRFY/EXPN without
> that parameter being specified. A new SMTP error     status is
> needed, but the document only defines an extended status
> for this purpose; it doesn't specify a regular status code to
> use.

If we go down this path, the WG clearly needs to reexamine
whether such a code is necessary.  I think you have made a
persuasive case, but I can't speak for the WG.

Indeed, this has all gotten sufficiently complex that I think
perhaps the WG should reexamine its decision to handle the "VRFY
and EXPN with possible non-ASCII responses" situation by
parameterizing those commands rather than introducing new
commands, say VRF8 and EXP8.  IIR, the WG did consider that
alternative (long ago) and conclude that adding a parameter was
the more straightforward option.  But, perhaps, given the
various complexities that have emerged, that was not the right
decision.  With other commands, such as MAIL and RCPT, we've
already got a history of parameters and the base (821/5321)
syntax is very clearly delimited.  By contrast, I certainly
remember seeing VRFY followed by unquoted strings with blanks in
them in the wild (e.g., arrangements like 
    VRFY Ned Freed
) and some servers tolerating them in the name of robustness or
in memory of 821, which had
    VRFY <SP> <string> <CRLF>
    <string> ::= <char> | <char> <string>
    <char> ::= <c> | "\" <x>
    <c> ::= any one of the 128 ASCII characters, 
        but not any <special> or <SP>
    <x> ::= any one of the 128 ASCII characters
       (no exceptions)
which makes
    VRFY Dave\ Crocker
perfectly valid.    I don't know if similar examples still occur
and with what frequency, but...

In addition, if we handle EAI-extended VRFY and EXPN with
parameters, we end up in the odd situation in which the
no-parameter versions can be issued without mail session
initialization (the EHLO command) but the extended,
parameterized ones require a previously-initialized session.
The MAIL and RCPT commands dn't have an issue because they
require session initialization anyway.  That restriction on
extended VRFY and EXPN certainly works technically, but isn't
going to be the easiest thing to explain and might cause some
"interesting" behavior if implementations screw it up or
clients, having not sent EHLO, assume that they are talking with
legacy 821 servers.   By contrast, a rule that requires session
initialization prior to using a new command is a no-brainer to
specify and implement cleanly, with no excuses about, e.g.,
ancient legacy environments.

I would welcome your thoughts on this tradeoff between adding
new commands (whose syntax would be identical to 5321 VRFY/EXPN
but expanded to UTF-8 input and/or output) and the current
proposal for a parameterized multiple-case situation, presumably
fixed up more or less as you have suggested.

    john