RE: FIN + RST behavior

["Lubashev, Igor" <[e-mail]>]

For the record, I was not trying to describe my understanding of the current stream limit, just my thinking of what it should be.


>> We are only protecting reassembly buffers here
> Isn't that what the connection flow control window and the stream flow control window is for?

You are correct that connection flow control window protects the aggregate amount of data committed to the reassembly buffers.  The max number of concurrent streams controls the number of reassembly buffers (supposedly, there could be significant fixed costs per reassembly buffer).


>> while there are any unacked sent bytes.

Yes, I withdraw that alternative and would go with something like "streams that have sent any bytes and (have unacked bytes or RST_STREAM or if there was no STREAM+FIN sent)".


The size or number of write buffers should not be controlled by some static pre-negotiated number.  This is likely to be something that the sender can (and will want to) control dynamically (just like write buffers are controlled dynamically by the kernel under memory pressure).


-          Igor


From: Subodh Iyengar [mailto:[e-mail]]
Sent: Sunday, March 26, 2017 3:27 PM
To: Lubashev, Igor <[e-mail]>; [e-mail]
Cc: [e-mail]
Subject: Re: FIN + RST behavior


> We are only protecting reassembly buffers here
Isn't that what the connection flow control window and the stream flow control window is for?



> must count the stream toward his concurrency limit while there are any unacked sent bytes. (By peer, I mean "an endpoint".)

If this is the correct interpretation of how concurrency limit is accounted for in the current draft, then it seems like it provides is no better protection than the flow control window.

The real value of a stream concurrency limit would be to be able to protect the stream state table and to allow senders to protect their write buffers. Sure, senders can slow themselves down, however putting that local limit on yourself is another complexity in addition to the other sender side limits like congestion control, and a tradeoff between throughput and buffer build-up.

It seems like the draft could benefit from is explicit signaling of the available concurrent limit similar to the flow control window which is explicitly signaled. Then we can actually have the peer advertise a concurrent limit whenever it feels like it's ready to receive more streams, which would make this way easier to account for resource usage in the state table and allow the receiver to expand this limit in the future.



Subodh

________________________________
From: Lubashev, Igor <[e-mail]<mailto:[e-mail]>>
Sent: Friday, March 24, 2017 5:56:41 AM
To: [e-mail]<mailto:[e-mail]>; Subodh Iyengar
Cc: [e-mail]<mailto:[e-mail]>
Subject: RE: FIN + RST behavior

I am not thinking about it from the perspective of the initiator of a stream. I am really thinking about a stream having two independent directions. It is the sender of the bytes (on a stream initiated by him or by the peer, does not matter) that must count the stream toward his concurrency limit while there are any unacked sent bytes. (By peer, I mean "an endpoint".)

The management of retransmit buffers is a local concern of the sender. If the buffers grow too large, the local app would not get a chance to write more into them. We are only protecting reassembly buffers here. If we also want to protect a table containing state for active streams, we would need to modify the definition for concurrent streams to "number of streams that this endpoint has sent any data on and still have unacked data or fin, or unacked rst".

- Igor

-----Original Message-----
From: Subodh Iyengar [[e-mail]]
Received: Friday, 24 Mar 2017, 12:28AM
To: Lubashev, Igor [[e-mail]]; Martin Thomson [[e-mail]]
CC: [e-mail]<mailto:[e-mail]> [[e-mail]]
Subject: Re: FIN + RST behavior

@Lubashev, Igor<mailto:[e-mail]>



>Concurrency limit is about the amount of local resources a remote peer can consume.



Let's say that for every stream there is an initiator (who is subject to stream limit) and a peer.
Aren't the retransmission buffers which the peer has to hold onto until the initiator ACKs them also a part of the local resources here?



An initiator who doesn't ACK any bytes from the server forces a peer to keep state for it. However it could send it's own FIN which is ACKed normally.



My question generally here is that with the current state machine seems that it's difficult for both peers to get a consistent view of what each side thinks that the

open peer streams have. The peer should only release the retransmission buffers when it knows all of the data + FIN has been ACKed, and I think that's when it should release

the stream limit. However the initiator needs to know when this happens so it would need to keep track of the ACKs for the ACKs it sent. Unidirectional streams seem to make this simpler.

>To be put precisely, the concurrency limit can be defined as "the max number of streams with non-zero unACKed bytes sent by that peer".

I'm not sure I understand your meaning here or maybe I'm confused by what you consider as the peer.
Can't the initiator just send a FIN and get an ACK for the FIN, however never ACK the peer's frames? In this case the initiator would have "zero UNACKed bytes" but the peer would not.


Subodh

________________________________
From: Lubashev, Igor <[e-mail]<mailto:[e-mail]>>
Sent: Thursday, March 23, 2017 11:27:29 AM
To: Subodh Iyengar; Martin Thomson
Cc: [e-mail]<mailto:[e-mail]>
Subject: RE: FIN + RST behavior

Concurrency limit is about the amount of local resources a remote peer can consume.  It is not about how much local resources can be consumed by the local application - these limits can be enforced without any help from the protocol.

So concurrency limit is really the number of concurrent in-progress streams that a peer is willing to receive.  This maps well with unidirectional streams, but this is not a requirement.  To be put precisely, the concurrency limit can be defined as "the max number of streams with non-zero unACKed bytes sent by that peer".  This means, peers can negotiate separate concurrency limits.


-          Igor

From: Subodh Iyengar [mailto:[e-mail]]
Sent: Thursday, March 23, 2017 1:09 PM
To: Martin Thomson <[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]>
Subject: Re: FIN + RST behavior


>  The client can consider its own resources to be available for use by
the server as soon as it receives the FIN and processes it.  If this
were a server resource, then you are right, it's not that simple
because the client can't assume that the server has released state
yet.

Ya in the case of the previous example, it is a server resource since it's a client initiated stream.
@Martin Thomson<mailto:[e-mail]> I'm starting to like your proposal of unidirectional streams more and more :)



Subodh



________________________________
From: QUIC <[e-mail]<mailto:[e-mail]>> on behalf of Martin Thomson <[e-mail]<mailto:[e-mail]>>
Sent: Thursday, March 23, 2017 4:04:46 AM
To: Subodh Iyengar
Cc: [e-mail]<mailto:[e-mail]>
Subject: Re: FIN + RST behavior

On 23 March 2017 at 15:10, Subodh Iyengar <[e-mail]<mailto:[e-mail]>> wrote:
> How does the client know when the server has released a slot in it's
> concurrency limit? The server can't immediately release this when it sends
> out the FIN because it still must keep state until it receives all the
> ACKs for the stream. I think the client need to keep track of the ACK of the
> ACKs of all the server sent bytes in the stream, which would be a bit sad.


The client can consider its own resources to be available for use by
the server as soon as it receives the FIN and processes it.  If this
were a server resource, then you are right, it's not that simple
because the client can't assume that the server has released state
yet.