[nfsv4] 答复: Draft is updated following your comments, Thanks//答复: draft-mzhang-nfsv4-sequence-id-calibration-01

["yangjing (U)" <[e-mail]>]

Hi Rick,

I think we are talking about two issues,

1. When a client get NFS4ERR_SEQ_MISORDERED response, whether the client should resend this request.
I agree with you that a sequenceid being misordered implies that the client is trying to break "exactly once semantic", but denied by the server with NFS4ERR_SEQ_MISORDERED. 
And Next, client MUST NOT resend this request with the original session/slot/new sequence, or with the original session/new slot, or with a new session. If the client do so, it will break EOS.  
You are worrying about that if the misordered slot is recovered, the request (originally denied) will be resent to server (with the original session/slot/new sequence), right? 
In fact, client and user app can do this even if the slot is marked as broken, for example, it can resent the request with the original session/new slot.

2. When a client get NFS4ERR_SEQ_MISORDERED response, you prefer the following two methods on how to deal with the session/slot. I have different opioion.
Method1: Don't destroy the session, but mark the slot as broken. Excluding the impact of concurrence, we have to maintain slot status. 
If so, current data structures have to be changed. Is this acceptable? 
Method2: Don't destroy the session, get correct sequenceid of this slot by attempts with only SEQUENCE operation. In this way, Interactions between client and server are unknown. 
One request and one reply is the best case. However, with the expansion of operations to get correct sequenceid, one request and one reply is the only case.

> -----邮件原件-----
> 发件人: Rick Macklem [mailto:[e-mail]]
> 发送时间: 2023年4月21日 10:28 PM
> 收件人: yangjing (U) <[e-mail]>
> 抄送: [e-mail]
> 主题: Re: [nfsv4] Draft is updated following your comments, Thanks//答复:
> draft-mzhang-nfsv4-sequence-id-calibration-01
> 
> On Thu, Apr 20, 2023 at 8:08 PM yangjing (U) <[e-mail]>
> wrote:
> >
> > Hi Rick, Thanks for your comments and sorry for the late response ^_^
> >
> >
> > On Mon, Apr 10, 2023 at 2:20 PM Rick Macklem <[e-mail]>
> wrote:
> > >
> > > On Mon, Apr 10, 2023 at 1:58 PM Rick Macklem
> <[e-mail]> wrote:
> > > >
> > > > On Mon, Apr 10, 2023 at 1:15 PM Rick Macklem
> <[e-mail]> wrote:
> > > > >
> > > > > On Sun, Apr 9, 2023 at 11:39 PM yangjing (U)
> > > > > <yangjing8=[e-mail]> wrote:
> > > > > >
> > > > > > The draft is updated as
> > > > > > https://www.ietf.org/archive/id/draft-mzhang-nfsv4-sequence-id
> > > > > > -c
> > > > > > alibration-02.txt
> > > > > Here are a few generic comments about the draft (I know nothing
> > > > > about detailed formatting, etc).
> > > > >
> > > > > 1 - It is my understanding that new operations can only be added
> > > > > to minor version 2 and not
> > > > >      minor version 1.
> >
> > I agree with you,  If new operations can only be added to minor version 2.
> >
> > > > > 2 - I largely agree with what Tom Talpey said in his comments. I
> > > > > think you need to figure out
> > > > >      how/when the sequenceid is getting messed up.
> > > > >      Note that it is my understanding that the main purpose of
> > > > > sessions is to achieve
> > > > >      "exactly once RPC semantics" (the fact that the reply cache
> > > > > is bounded in size is a nice
> > > > >      added feature, but not the principal purpose).
> > > > >      As such, a sequenceid being misordered implies that
> > > > > "exactly once" semantics could
> > > > >      already have been broken by a software bug in the client.
> > > > >
> > > > >      You note that the misordered sequenceid could be caused by
> > > > > "network partitioning".
> > > > >      In a correct implementation that should not be the case.
> > > > > After a network partition heals,
> > > > >      the client should re-send all outstanding RPCs with the
> > > > > same Session/slot/sequenceid
> > > > >      as was used in the original RPC request message. These
> > > > > should succeed (no misordered
> > > > >      error reply) using cached replies, as required.
> > > > >      --> If a client's recovery from network partitioning is not
> > > > > doing this, then it needs to be
> > > > >            fixed.
> > > > >      --> As above, if a misordered sequenceid error reply is due
> > > > > to a client bug, then that
> > > > >            must be fixed, since "exactly once" semantics may already be
> broken.
> > > > > The common client bug that results in a misordered sequenceid
> > > > > error reply happens when an in-progress RPC gets interrupted
> > > > > before the RPC reply is processed.
> >
> > You are right. Thanks for your analysis on when misordered sequenceid
> happens. But this draft does not care how and when it happens, the same as
> the protocol specification.
> > We  focus on how to deal with this error.  If this error never
> > happens, we will not define it, right? We did find that misordered
> sequenceid happens and IO performance is affected because session is
> reestablished, but we are not sure why and how it happens, because no
> packages are being captured at that moment.
> >
> > > > > This is not a case
> > > > > where your suggested extension should be used, since the
> > > > > in-progress RPC may still be "in-progress". Although I am not
> > > > > sure there is a 100% correct way for a client to handle the need
> > > > > to interrupt an in-progress RPC, marking the session slot as
> > > > > broken and no longer using it is the best approach I am aware
> > > > > of. (I suspect you might be referring to the Linux client and I
> > > > > cannot comment on how it handles interruption via signals or
> > > > > timeouts for session slots, since I am not a Linux client
> > > > > implementor.)
> >
> > Yes, when we find IO performance is affected, we check Linux client code,
> find that client will drop the session, then we check the NFS protocol
> specification....
> > I don’t think marking the session slot as broken is the best way. What is next?
> > First: Leaving the slot broken. On one hand, with more and more slots
> > broken, concurrency of this  session will decline, On another hand, Is new
> mechanism needed to maintain and notify slot status between client and
> server?
> > Second: Recover the broken slot. That is what this draft is trying to do, I
> think.
> But you cannot resynchronize the broken slot safely unless the RPC that
> caused the sequuenceid to become non-synchronized is an idempotent one
> (one that can be done more than once on the server safely).
> This is difficult (maybe impossible) to determine when a reply to a subsequent
> RPC using the same session/slot is NFS4ERR_SEQ_MISORDERED.
> 
> To do so for all cases where a RPC reply of NFS4ERR_SEQ_MISORDERED is
> received could result in a non-idempotent RPC being executed multiple times
> on the server. This is a serious breakage and essentially defeats the main
> purpose for using sessions.
> 
> You are correct that, once a slot is marked bad, performance degradation is
> possible. However NFS4ERR_SEQ_MISORDERED failures should be rare (some
> might say that this error should never occur for correctly implemented clients)
> and, as such, having several slots on a session should also be a rare
> occurrence.
> --> After some percentage of the slots are marked bad, the client will
>      need to do a CreateSession and replace the session with the bad slots
>      with a new one.
>      What percentage is a design tradeoff. It sounds like Linux chose
>     "> 0%" (or first bad slot, if you prefer). For FreeBSD, I chose "100%"
>     (or all slots bad) if you prefer.
> >
> > > > >
> > > > > It sounds like migration might be an exception to the above, but
> > > > > it should be addressed specifically (as Tom Talpey notes).
> >
> > For migration scenario, we will analyze it later
> >
> > > > >
> > > > > 3 - If your extension were used by a client, how would the
> > > > > client know whether or not
> > > > >      there are still RPC(s) in-progress that have been assigned
> > > > > that session/slot/sequenceid?
> > > > An additional comment:
> > > > - If the client knows that the outstanding RPC request on the
> > > >   session/slot/sequenceid that got a misordered error reply
> > > >   is one where "exactly once" semantics is not required,
> > > >   then I can see that the client implementor might be able to
> > > >    safely resynchronize the sequenceid.
> > >Oops again. It would be the outstanding RPC request for which no reply has
> been received that causes the sequenceid to be misordered and that is the
> RPC that cannot require "exactly once" semantics.
> >
> > >rick
> > > >   To do this, I do not think your extensions are needed.
> > > >    If the client issues an RPC that consists of only the
> > > >    Sequence operation repeatedly, with the sequenceid incremented
> > > >    by one each attempt after waiting for a reply to the previous
> > > >    attempt, it should get NFS_OK within a few attempts.
> >
> > The misordered sequenceid is not known. If the cached sequenceid is N, any
> value beyond N+1 and N will be identified as misordered. The range of
> sequenceid is (1, 2^32 - 1).
> > So, I don’t think attempting repeatedly is a better choice.
> Well, here is the basic algorithm the client would use to maintain a slot's
> sequenceid. (Note there should never be more than one RPC on the fly for a
> given session/slot.)
> - When an RPC reply is received from the server with NFS_OK for the
>   Sequence operation, advance the sequenceid for the session/slot by 1.
> As such, I do not see how a sequenceid can ever be out by more than 1 unless
> there is a coding bug. For the case of a coding bug, fix the bug.
> 
> For example:
> - Lets assume that the client is configured (not a good situation, but
>   possibly unavoidable) so that a timeout or POSIX signal results in
>   the client not waiting "forever" for an RPC reply.
> - Lets also assume that client becomes network partitioned from the server.
> 
> When this is the case, it is possible that a few (even many) RPCs will be sent to
> the server with the same session/slot/sequenceid, one after another, each
> timing out before an RPC reply is received. But note that they all will have the
> same sequenceid.
> --> When this happens, it is possible that one of these RPC requests
>      will make it to the server, be processed, and advance the sequenceid
>      by one.
>      --> All subsequent RPCs with the same session/slot/sequenceid
>          processed at the server will get a NFS4ERR_SEQ_MISORDERED reply.
> 
> Once the network partition is healed, the client will see a
> NFS4ERR_SEQ_MISORDERED reply (plus possibly other delayed replies for the
> same session/slot/sequenceid), but the sequenceid will only be out by 1
> compared with what the server's sequenceid is for the same session/slot.
> --> This implies that the first attempt of a RPC that consists of only
>      the Sequence operation with sa_sequenceid set to N + 1
>      (where N is what the client is currently using for sequenceid for
>       the session/slot) will succeed.
>      --> Whether a subsequence attempt with sequenceid set to N + 2
>            is justified is debatable, but certainly doing more that that does
>            not seem reasonable to me.
> 
> If, due to some serious breakage, the above does not work. I would say a
> CreateSession to replace the now badly broken session, is in order.
> 
> Note again "this re-synchronization can only be safely done if the first RPC for
> which no reply was received is an idempotent one" and that is going to be
> difficult to determine when the NFS4ERR_SEQ_MISORDERED is received for a
> subsequent RPC that uses the same session/slot.
> --> If the client notes when it "gives up on waiting for an RPC reply"
> --> that the
>       slot is bad and it sees that the sa_cachethis argument in the Sequence
> operation
>       for the outstanding RPC request was set false (indicating it is an
> idempotent RPC)
>       then I believ it could safely do a re-synchronization of the slot at that point
> in time.
>       --> I chose to not do that and just mark the slot bad, but I can see an
> argument
>            for doing so, since the RPC is known to be idempotent.
> 
> rick
> 
> >
> > > >    Then the correct sequenceid to be used for the session/slot is known.
> > > Oops, "known" is probably too strong a word here. Since the cause of
> > > the misordered sequenceid is not known, I think there is a very low
> > > probability that an RPC using the session/slot is still in flight
> > > and will change the sequenceid when processed on the server, causing
> > > another misordered sequenceid error reply.
> > >
> >
> > I agree with you on "There may be still RPC(s) in-progress that have
> > been assigned that session/slot/sequenceid" when misorder reply getting
> the client, I will update the draft to specify that client should wait for replies
> for all flying requests, then query correct sequenceid.
> >
> >
> > > rick
> > >
> > > >
> > > > rick
> > > >
> > > > >
> > > > > rick
> > > > >
> > > > > > _______________________________________________
> > > > > > nfsv4 mailing list
> > > > > > [e-mail]
> > > > > > https://www.ietf.org/mailman/listinfo/nfsv4
> >
> >
> > Thank you Rick
> >
> > Best Regards
> > Jing
> >
> >
> >