Re: [Emailcore] Ticket #5: G.5. Remove or deprecate the work-around from code 552 to 452?

[John C Klensin <[e-mail]>]

Hi.

This is another long note, so a semi-summary first although I
would encourage everyone to read all the way through.  I think
Victor's note exposes two key issues and a couple of things not
directly related to this ticket:

(1) Independent (at least for now) about what clients do when
they get a 552, have enough servers switched to sending 452
rather than 552 when "too many recipients" is intended that the
text about interpretation of the 552 code as a 4yz one can
actually be removed (or retained only in an historical note).

(2) As I read Victor's note, there were many places where my
response was "that is what the spec say to do already".  That
implies that either Victor and I are reading the same text
differently (in which case I think some clarification effort is
in order) or that he and I are having problems understanding
each other.  I don't yet have a strong opinion as to which it
is; probably his response to this note will help with that.

In addition...

* The requirement of 100 minimum recipients is a key part of
Victor's explanation.  We may need to look at that type of limit
as well as the "size" ones that are now part of ticket #14
(whether there or in a separate ticket).  However, if we want to
minimize changes, I believe that the "operational necessity"
principle (see prior note) is sufficient to cover that case.

* Victor has made the 100 pound Gorilla argument.  I think that
argument is a serious challenge for the WG and its decisions
about scope and goals.  I'll try to address that soon in a
separate note.

Again, please keep reading.
  

--On Thursday, March 11, 2021 21:29 -0200 Viktor Dukhovni
<[e-mail]> wrote:

>> On Mar 11, 2021, at 7:18 PM, John C Klensin
>> <[e-mail]> wrote:
>> 
>> That suggests that, if changes are needed, there are probably
>> three of them, the first two in the third paragraph of that
>> section ("If an SMTP server has..."), not in the first one:
>...
>> NEW/PROPOSED:
>> 	If an SMTP server has an implementation limit on the
>> 	number of RCPT commands and this limit is exhausted (See
>> 	Section 7.8),
 
> I am having trouble piecing together the fragments here, with
> context not present in this message.

Sorry.  Was trying to keep the message from getting even
longer... and to encourage people to read the entire section.

>  Therefore, I am posting
> a proposed update with commentary:
> 
> 4.5.3.1.10.  Too Many Recipients Code
> 
> <quote 1>
>    RFC 821 [1] incorrectly listed the error where an SMTP
> server    exhausts its implementation limit on the number of
> RCPT commands    ("too many recipients") as having reply code
> 552.  The correct reply    code for this condition is 452.
> </quote 1>
> 
> The above is correct and needs no changes.

Good.  We are on the same page so far.

> <quote 2>
>                                      Clients SHOULD treat a
> 552 code in    this case as a temporary, rather than
> permanent, failure so the logic    below works.
> </quote 2>
 
> The above is not followed by any of the major MTAs (Exim,
> Postfix, or Sendmail, nor any others reported).  It MUST be
> scrapped.  If the server responds with 552, the recipient in
> question will almost always be immediately bounced by the
> sending client.

For whatever it is worth, I don't know what it means to "bounce
a recipient".   The spec only talks about bouncing messages.
More important, I would hope that the client would accumulate
undeliverable addresses and include them in a single "bounce"
message, not send one bounce message per undeliverable recipient
for reasons I hope are obvious.  The former is certainly
consistent with behavior I've observed with other 5yz responses
to RCPT commands.   "Immediately" would imply one bounce message
per recipient.

And, of course, if the the SMTP sender getting the 552 message
from its receiver is operating in some sort of relay mode so it
can simply return the code in real time to whatever it got the
message from, that changes the story (and is not what we usually
think of as a "bounce".

But see below.

> Note, that the 552 in reply to "RCPT" will not be taken to mean
> anything about the message as a whole, earlier or subsequent
> recipients. If any recipients are ultimately not rejected the
> partially accepted message envelope will be relayed by the
> client unless the server rejects the "DATA" command.

See my questions to Jeremy, but, if that is what they are doing,
then they are conforming to the requirement to treat 552 as 442,
because that is exactly the behavior one would expect 442 to do.
Whether they (or you) think about that as conforming behavior
doesn't make a lot of difference.  If there is a difference it
is in your suggestion above about an immediate bounce.  

And that immediate bounce, or even returning the 552 code to an
earlier SMTP sender, raises another problem.  One of the most
important principles in SMTP-land, going back to 821, is that
one does not report a message as undeliverable and then deliver
it anyway.  "Delayed" messages are an odd case for which 821->
2821-> 5321 do not have any provision, but, if the message (or,
if you prefer, address) is bounced, that is the end of it -- the
only way that message is going to be delivered to that recipient
is if the originator creates and sends what, as far as SMTP is
concerned, is a new message.

> There are no clients that abort the MAIL transaction to start
> again with smaller recipient batches.  Some could take 552
> (when not too deeply pipelined) as a single to defer even
> recipients not yet mentioned, that'd be fine.  I don't know of
> any that do that.

Good.  Again, you are describing behavior consistent with the
client treating the 552 as equivalent to 452 (or some other 4yz
code).

> Instead clients typically limit the number of recipients per
> envelope to somewhere south of 100 (perhaps hand-tuned for mail
> destined to some of the 800lb gorilla providers), and just send
> all the RCPT commands they intended to send, with perhaps the
> tail of the list ending up deferred.

Again, consistent with treating the 552 as a 442, whether they
precisely follow the rest of the description in that section or
not.

> <quote 3>
>    When a conforming SMTP server encounters this condition, it
> has at    least 100 successful RCPT commands in its recipients
> buffer.  If the    server is able to accept the message, then
> at least these 100    addresses will be removed from the SMTP
> client's queue. </quote 3>
> 
> I wish that were true.  It would certainly aid
> interoperability. But the said 800lb gorilla providers have
> felt free to dictate arbitrary limits below the RFC
> requirements, and the rest of us have needed to make
> adjustments if we want to deliver list mail to these players.
> So long as the server is willing to defer (4XX) 100-N
> recipients or more without prejudicing any it already
> accepted, it will interoperate with clients that expect to be
> able to send up to 100, despite the lower limit.

But that is an issue with the current semi-hard limit of a
minimum or 100, not with the 452/552 requirements.  And, while
the text is not as precise as we might like it, clients (and,
for that matter servers) are allowed to pick lower limits under
the "Resistance to Attack" / Operational necessity principle.
So there is nothing in the above that is non-conforming, even to
the current text in 4.5.3.1.10.   And, fwiw, I note your
explanation above uses "defer" and "4XX" together, not 552.

> If a server both sets a lower limit, and prevents clients that
> overshoot that limit from sending even the initially accepted
> recipients, then the server is outside the interoperability
> regime defined by the specification.  Email to the server will
> only work if clients are willing to hand-tune their MTAs for
> the server in question.  Such hand-tuning is not always
> possible by server name or IP (in Postfix we can only hand
> tune envelope recipient counts by destination domain, before
> the domain's MX hosts are known).  Therefore, such servers are
> particularly likely to impede email delivery when acting as MX
> hosts for a large and varying set of email domains.

Yes.  And still consistent with what 5321bis says.  In
particular, note the comments in the first paragraph of Section
7.9.

> <quote 4>
> 
> When the    client attempts retransmission of those addresses
> that received 452    responses, at least 100 of these will be
> able to fit in the SMTP    server's recipients buffer.  Each
> retransmission attempt that is able    to deliver anything
> will be able to dispose of at least 100 of these    recipients.
> </quote 4>
 
> Largely correct, except that again with the 800lb crowd the
> actual number may be well south of 100.  And yet even the 800
> pounders should not feel quite so much at liberty to violate
> the specs. Don't know what other MTAs, but Postfix expects the
> server (by default) to accept 50 recipients without friction.

See above.  And a separate note to follow about the 800lb
argument.

> Another reason for fewer than 100 is when servers want have the
> envelope split by some server-specific classification of
> recipients, that may affect recipient-specific downstream
> content transformations. In such a case the server might 452
> every recipient in a class that is different from that of the
> first accepted recipient.  The remaining recipient classes
> would then need to be tried in a separate transaction.

Again,  5321 now allows clients and/or servers to pick a limit
lower than 100 and encourages / specifies orderly behavior when
that is done.  Modulo the recommendation to use 452 for clarity
rather than 552, that is what everyone you have described seems
to be doing in practice.

> Such a policy is risky when the number of distinct recipient
> classes is more than a "handful", as a message sent to a large
> list may then require more retries than a client is able to
> schedule before the message expires.

Yep.   If you think it is useful for either 5321bis or the A/S
to explicitly point that out and discuss it, please suggest how
and where to do that.  My personal guess (and so-far weak)
preference is that the A/S is headed for a discussion of the
practical issues and tradeoffs associated with the various cases
and options.  In particular, from the standpoint of 5321 "more
retries... before the message expires" is not 5231bis-level SMTP
problem.

> On the other hand, messages with a large number recipients are
> most often sent by mailing-list software that does automated
> bounce processign, and arranges for each recipient to be
> associated with a separate envelope sender address, so that
> bounces can be tracked back to the intended recipient.
> 
> Only large mailings by an occasional human to a large list of
> friends, romans, countrymen, ... 

Agreed.  See above.  Independent of where I stand personally, I
suggest that any attempt to further expand Section 3.9 to talk
about these issues would get considerable resistance from within
the WG, including from (but not limited to) those who have
suggested that 3.9 be removed entirely.   That is another
argument for taking the above discussion to the A/S.

> <quote 5>
>    If an SMTP server has an implementation limit on the number
> of RCPT    commands and this limit is exhausted, it MUST use a
> response code of    452 (but the client SHOULD also be
> prepared for a 552, as noted    above).
> </quote 5>

> Yes, the server MUST respond with 452 when the intent is to
> have the client "split the envelope", and send the remaining
> recipients that were not accepted in a separate batch.
> 
> If, on the other hande, based on the sequence of presented
> recipient addresses the server has somehow concluded that the
> client is a spammer, it can of course with full knowledge of
> the fact that the recipients will NOT be retried, send 552 or
> better yet at that point, 554.

Agreed.  And I think that is what the text says.  If you think
it needs clarification to distinguish between those two cases,
please say so and encourage opening of another ticket.

> <quote 6>
>            If the server has a configured site-policy
> limitation on the    number of RCPT commands, it MAY instead
> use a 5yz response code.  In    particular, if the intent is
> to prohibit messages with more than a    site-specified number
> of recipients, rather than merely limit the    number of
> recipients in a given mail transaction, it would be
> reasonable to return a 503 response to any DATA command
> received    subsequent to the 452 (or 552) code or to simply
> return the 503 after    DATA without returning any previous
> negative response. </quote 6>
 
> This is problematic and generally pointless.  It serves no
> purpose and breaks interoperability.  Clients can always send
> multiple copies in separate envelopes, so such absolute limits
> (below 100) damage interoperability and don't achieve the
> desired goal of actually stopping the same message arriving to
> more than a given number of recipients.

But that is precisely intended to allow for the 552/554 response
you are suggesting above.    So, "pointless", perhaps not.  More
confusing than it ought to be, maybe.  Suggestions welcome.

> The solution is quite simple delete the above text, the
> overall picture is much simpler.  Respond with 4XX to defer
> and 5XX to permanently reject a given RCPT.  There's no reason
> for the specification to dwell on all the creative ways in
> which a decision to reject might be made.

However, what I thought we heard Wednesday morning was that a
number of servers continue to send 552 where 452 is now
preferred ("preferred" may be a little weak).   As long as that
is the case, then the purpose of that extended section is to try
to explain ways to handle the situation, including both
receiving 452 and receiving 552 in a situation where it is
intended to be interpreted as 452.  If we are convinced that all
of the SMTP servers on the Internet (or enough that others don't
count -- and I'm counting gorillas who feel free to define their
own standards as one each, not by their user or daily message
counts) have followed the advice of 5321 so that no server is
any longer sending 552 when 452 (and breaking up the mail
transaction and resending) is intended, then the section should
be significantly reworked, possibly including a historical note
for servers that are still in existence but we cannot identify.  

> The key thing in this section is to specify that "452" MUST be
> used when the client should retry the recipient separately
> because too many were sent so far to process in a single batch
> on the server.  The number of "452" replies the server is
> willing to send without blocking message delivery or imposing
> punitive delays that'll prevent transaction completion needs
> to be at least 100 minus the number of initially accepted
> recipients.

And, modulo the limit of 100, which I continue to believe is
either a separate issue or no problem at all given the current
text in 5321, I believe that is exactly what that current text
says.

Again, suggested clarification welcome.

   john