Re: [Extra] draft-ietf-extra-imap4rev2-04 review

["Bron Gondwana" <[e-mail]>]

On Mon, Mar 25, 2019, at 23:05, Timo Sirainen wrote:
> I see this was already in RFC 3501, but this "MUST be greater" has commonly been violated by servers. I think it should say "MUST be different". This is mainly because RENAME commonly doesn't change UIDVALIDITY, so:
> 
> RENAME mailbox-with-uidvalidity2 mailbox-with-uidvalidity2b
> RENAME mailbox-with-uidvalidity1 mailbox-with-uidvalidity2
> 
> Would result in mailbox-with-uidvalidity2 shrinking UIDVALIDITY from 2 to 1.

So the risk here is that you need to be very careful that you don't THEN rename back to '2' again ever, or a client that's been away for a long enough time can come back and find a reused UID. This is why "must always be greater" is a safer than "must not be any value which has ever been seen by a client in the past". Note I say "seen by a client" rather than "used" since the guarantees are only per client :)

> > A server MUST NOT unilaterally close the connection without sending
> > an untagged BYE response that contains the reason for having done so.
> 
> Also already in RFC 3501, but this MUST NOT has to be violated sometimes. For example otherwise it would allow client to FETCH a large message and keep reading it very very slowly over days, and server wouldn't be allowed to disconnect it, because it can't send BYE in the middle of the large mail literal.

Yeah, plus middleware proxies. This is clearly bogus.

> > 4.5. NIL
> 
> This is pretty easy to get wrong. It would help to have some examples of NIL and "NIL" in astring and nstring and what they mean. Like:
> 
> * LIST () "/" NIL
> is equivalent to:
> * LIST () "/" "NIL"
> 
> but
> 
> * FETCH 1 (BODY[1] NIL)
> is not equivalent to:
> * FETCH 1 (BODY[1] "NIL")

Agree that clarifying NIL behaviour would be great. The magicness of NIL and lack of quoting requirement on atoms is one of the horror cases of IMAP parsing.

> > 5.1. Mailbox Naming
> 
> > servers MAY accept a de-normalized UTF-8 mailbox name and convert it to Net-Unicode prior to mailbox creation).
> and:
> > Some server implementations
> > are fully case-sensitive in ASCII range; others preserve case of a
> > newly-created name but otherwise are case-insensitive; and yet others
> > coerce names to a particular case.
> 
> This seems rather difficult from client's point of view. It CREATEs a mailbox, but the client can't find it anymore because it changed. Seems to me it would be less strange if the server simply failed the CREATE entirely.
> 
> If this is allowed for CREATE, is such conversion done for SELECT/EXAMINE/APPEND as well? What about LIST "" <non-normalized name> - would it return the normalized name?
> 
> > A.1. Mailbox International Naming Convention
> > 
> > By convention, international mailbox names in IMAP4rev2 are specified
> > using a modified version of the UTF-7 encoding described in [UTF-7].
> 
> This chapter needs removal or rewrite.
> 
> > 6.2.2. AUTHENTICATE Command
> > 
> 
> > Client and server
> > implementations SHOULD implement additional [SASL] mechanisms that
> > do not use plaintext passwords, such the GSSAPI mechanism
> > described in [SASL] and/or the [DIGEST-MD5] mechanism.
> 
> I don't think we should be recommending DIGEST-MD5 anymore. Maybe SCRAM-SHA-256 instead?
> 
> > 6.3.9. LIST Command
> > 
> 
> > Note: The interpretation of the reference argument is
> > implementation-defined. It depends upon whether the server
> > implementation has a concept of the "current working directory"
> > and leading "break out characters", which override the current
> > working directory.
> 
> 
> Is there really any point in having the reference argument anymore? We could just require clients to use "" for it and simplify the protocol a lot?

This I would also agree with :) The Cyrus server just concatenates the two, pretty much.

> > 6.3.10. LSUB Command
> > 
> > A special situation occurs when using LSUB with the % wildcard.
> > Consider what happens if "foo/bar" (with a hierarchy delimiter of
> > "/") is subscribed but "foo" is not. A "%" wildcard to LSUB must
> > return foo, not foo/bar, in the LSUB response, and it MUST be flagged
> > with the \Noselect attribute.
> > 
> > 6. Possibly fold in the following extensions/RFC: Base LIST-
> > EXTENDED syntax plus deprecate LSUB (replace it with LIST
> > \Subscribed) minus the requirement to support multiple list
> > patterns, STATUS-in-LIST, SEARCHRES, BINARY (only the FETCH
> > changes on leaf body part and make APPEND related ones optional.
> > See the mailing list discussion) - done.
> 
> I think at the very least we could remove the \Noselect kludge and return \Nonexistent instead.

Likewise this would simplify the Cyrus server... though again the problem is that if you want to remain backwards compatible is just increases the complexity of the server cases.

> > 6.3.11. NAMESPACE Command
> > 
> 
> > If a client is configured such that it is required to
> > create a certain mailbox, there can be circumstances where it is
> > unclear which Personal Namespaces it should create the mailbox in.
> > In these situations a client SHOULD let the user select which
> > namespaces to create the mailbox in.
> 
> Is any client ever going to actually implement this? Maybe just change it to say to use the first personal namespace?

There's a bug in Thunderbird where it creates in whatever namespace the mailbox name is a prefix of, which leads to it trying to create "Other Folders/Other" when the user types "Other". Ouch. Fricking namespaces.

On the other hand, SHOULD is still not a requirement, and most servers only have one personal namespace, so I'm pretty meh about this.

> > 6.4.1. CHECK Command
> 
> Time to drop this command?
> 
> > 6.4.5. SEARCH Command
> > 
> 
> > In all search keys that use strings, a message matches the key if the
> > string is a substring of the associated text.
> > 11. Allow word-based searching (as per Chris Newman)?

There's already FUZZY.

> Yes, I think we can't require substring matching anymore, since there's no practical way of implementing it efficiently for large installations. And many servers no longer support it.
> 
> > 6.4.6. FETCH Command
> > 
> 
> > ALL Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE ENVELOPE)
> > 
> > FAST Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE)
> > 
> > FULL Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE ENVELOPE
> > BODY)
> 
> Time to simplify and drop all macros?

I would be fine with dropping macros.

> > BINARY[<section-binary>]<<partial>>
> > BODY[<section>]<<partial>>
> > 
> > The \Seen flag is implicitly set; if this causes the flags to
> > change, they SHOULD be included as part of the FETCH responses.
> 
> Time to simplify and require .PEEK versions? The auto-\Seen flag has just caused unnecessary complications.

Particularly since it's the default - I'd love to ditch the auto-\Seen, because right now in Cyrus it's done with an exclusive lock regardless of whether it finds anything to change... for sure that's acutally a silly idea and I _should_ have implemented it as an up-front scan followed by only exclusively locking if there was something to change.

(I'm pretty sure the auto-foo is done first so the modseq bump is returned in the fetch response)

> > 6.4.10. UID Command
> > Responses: untagged responses: FETCH, ESEARCH
> 
> EXPUNGEs can also be returned by UID EXPUNGE.
> 
> > 7.4.2. FETCH Response
> > 
> 
> > RFC822 Functionally equivalent to BODY[], differing in the syntax of
> > the resulting untagged FETCH data (RFC822 is returned).
> > 
> > RFC822.HEADER Functionally equivalent to BODY.PEEK[HEADER],
> > differing in the syntax of the resulting untagged FETCH data
> > (RFC822.HEADER is returned).
> > 
> > RFC822.TEXT Functionally equivalent to BODY[TEXT], differing in the
> > syntax of the resulting untagged FETCH data (RFC822.TEXT is
> > returned).
> 
> Time to simplify and drop support for these?

I'd have to update all my client code :p But yeah, besides it's an old name...

> > CAPABILITY Followed by a list of capabilities. This can appear in
> > the initial OK or PREAUTH response to transmit an initial
> > capabilities list. This makes it unnecessary for a client to send
> > a separate CAPABILITY command if it recognizes this response.
> 
> Could also mention that it is often sent in LOGIN or AUTHENTICATE tagged reply.
> 
> > 6.3.11. NAMESPACE Command
> > 7.2.5. NAMESPACE Response
> 
> These are talking about "Namespace_Response_Extensions" but it's not explained anywhere. I've no idea what that is.
> 
> There are also some old imap-list discussions I had gathered in https://www.imapwiki.org/Specs/Rfc3501Problems - I think the relevant ones are:
> 
> * LIST reply should never mix namespace separators. Different namespaces can have different separators, but in that case LIST shouldn't be listing those namespaces without explicitly being asked to.
> * Except INBOX is special and can have a different separator from others (I wouldn't mind dropping this though)
> * Should FLAGS / PERMANENTFLAGS be sent after a new keyword is created within the session
> * SEARCH TEXT includes searching MIME headers, while SEARCH BODY doesn't


--
 Bron Gondwana, CEO, FastMail Pty Ltd
 [e-mail]