[nfsv4] The nfs-ganesha server keeps responding with NFS4ERR_SEQ_MISORDERED, and the client's business is stuck and cannot be recovered.

[飞虎 郑 <[e-mail]>]

Issue：
The nfs-ganesha server keeps responding with NFS4ERR_SEQ_MISORDERED, and the client's business is stuck and cannot be recovered.


Background：
We use nfs-ganesha as the server and Linux kernel as the client.
We conducted the following test:
The NFS server is a three-node environment, and the client accesses the NFS server through a virtual IP. After the NFS client was mounted, files were read and written to the NFS server via vdbench.
On the NFS server node, we simulated memory sub-health through a script (memory continues to increase, and the server node performs a hard reboot after it exceeds the set critical value), and the virtual IP would migrate to another node after the node reboot. After the node reboot, the client's business was always stuck at 0, and accessing the mount point also caused it to hang. By capturing packets via Wireshark, we found that the client kept sending requests in the SEQUENCE, PUTFH, WRITE, GETATTR combination (or other types of requests) to the server, while the NFS server kept responding with NFS4ERR_SEQ_MISORDERED. We have conducted this kind of memory sub-health test many times, but this problem only occurred in this particular instance.
Client kernel version: 4.18.0-193.14.2.el8_2.x86_64 (mockbuild@10-75-9-128).


Analysis：
In order to verify whether the NFS server's reply of NFS4ERR_SEQ_MISORDERED would cause the client's business to drop to 0, I modified the code of ganesha: when the value of the sequence_id corresponding to the session_id and slot_id is set, I added 10 to the sequence_id cached on the server to simulate a request sequence mutation, which would only occur once (subsequent sequence_id values for other slot_ids would not mutate). After the request sequence mutation, the client's business kept dropping to 0 and could not recover. By dynamically enabling ganesha's logs and using wireshark to capture packets, it can be seen that ganesha replied NFS4ERR_SEQ_MISORDERED to the client. Afterwards, the client used a new slot_id to send a request with the main request of GETATTR(SEQUENCE, PUTFH, GETATTR), and the server processed it normally and replied to the client. In addition, the client kept sending combination requests with the main request of LOOKUP(SEQUENCE, PUTFH, LOOKUP, GETFH, GETATTR) using the old slot_id and corresponding request sequence number, and ganesha directly replied NFS4ERR_SEQ_MISORDERED after detecting the loss of the SEQUENCE request sequence number. From the results, it seems that the client kept using the request sequence number corresponding to the old slot_id to send some op requests to the server, and the server kept replying NFS4ERR_SEQ_MISORDERED, causing the client's business to drop to 0. I noticed that if there is a request sequence disorder, there is no interface to query the server's request sequence number. If there is a request sequence mutation, the client cannot obtain the request sequence number. It seems meaningless for the client to keep querying. In addition, I modified the ganesha code so that if the request sequence number is out of order for a period of time, ganesha replies NFS4ERR_BADSESSION or if the request sequence number returns to normal, the client's business can be restored and completed.

If NFS-Ganesha responds with NFS4ERR_BADSESSION after the request sequence becomes disordered, does it modify the protocol and I wonder if this is appropriate? I have already provided feedback on this issue to the nfs-ganesha community, please visit: https://github.com/nfs-ganesha/nfs-ganesha/issues/941.

获取Outlook for Android<https://aka.ms/AAb9ysg>