Re: [Taps] Review of draft-trammell-taps-post-sockets-03

[Tommy Pauly <[e-mail]>]

> On Nov 11, 2017, at 2:09 PM, Michael Welzl <[e-mail]> wrote:
> 
> Hi,
> 
> In line:
> 
>> On Nov 11, 2017, at 11:58 AM, Tommy Pauly <[e-mail] <mailto:[e-mail]>> wrote:
>> 
>> 
>> 
>>> On Nov 11, 2017, at 10:36 AM, Michael Welzl <[e-mail] <mailto:[e-mail]>> wrote:
>>> 
>>> 
>>>> On Nov 11, 2017, at 10:06 AM, Tommy Pauly <[e-mail] <mailto:[e-mail]>> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> Just a couple initial notes that may help:
>>>> 
>>>> - The version diff you should look at is between -01 and -03. -02 is the same as -03, but had a typo.
>>>> 
>>>> - You've mentioned previously that you thought that Post requires both peers to use that API. That is absolutely not the case in any way. Having implemented Post myself, I am only communicating to "legacy" servers that know nothing about Post. I think this fundamental understanding needs to be cleared up before coming to any conclusions. We can discuss during the WG.
>>> 
>>> Oh?!  That would be fantastic!!  But I stumbled over a number of things that require a system on the other side, or they just couldn’t happen (like the “carrier forking” notification I quoted below - but there were more). If you say you can talk to a “legacy” server that knows nothing about Post, WHAT does that server speak? Legacy which protocol?  Maybe it’s just a matter of me thinking TCP, and you thinking TLS… dunno! Tell me   :)
>> 
>> So, since Post (and TAPS in general) isn't defining any new protocol features, we need to of course be compatible with all existing protocols. Certainly, some functionality is a bit degenerate in some cases.
>> 
>> Whatever the server speaks, the client needs to conform to. So, we use our Post implementation for raw TCP, raw UDP, TLS over TCP, HTTP/2 Streams over TLS over TCP, QUIC streams over UDP, etc. These are all standard server configurations, but Post offers a client application a single set of APIs to talk over any of these protocols.
> 
> Oh, great!  But then you can’t guarantee that a message arrives as a message in case of raw TCP, or you assume application-level framing. This is what I’ve been proposing, but the post-sockets draft reads entirely different from that.

Messages do indeed "always work", but they may be degenerate. There's no new protocols here, just ways of looking at them:

- UDP: One message per datagram
- TCP: Entire stream is one big message that only ever ends when that half of the stream closes
- TLS: Generally viewed like TLS, but can view one record as one message 
- HTTP/QUIC: One message per request/response (so, framing)
- Length-value framing on streams, as used by some protocols (like IKEv2 over TCP): one message per L-V frame
Etc, etc.

That is what the draft does say in Section 2.2. Note how a TCP stream is the message in some cases.

A Message is the unit of communication between applications.
   Messages can represent relatively small structures, such as requests
   in a request/response protocol such as HTTP; relatively large
   structures, such as files of arbitrary size in a filesystem; and
   structures of indeterminate length, such as a stream of bytes in a
   protocol like TCP.


> 
> 
>> In the case of "Carrier Forking", that's essentially what you say to the protocol stack when you want to open a new stream to the same place. So, when I call "fork Carrier", it turns into:
>> - For raw UDP/TCP or TLS (or anything that is mono-streamed), open a new five-tuple to the same remote endpoint
>> - For HTTP/2 and QUIC, open a new stream on the same connection
> 
> Sure, that’s what I thought it would. Now, with SCTP, you wouldn’t get a “fork request” equivalent on the other side: the client would just start using a new stream, and that’s it.  (“request” also indicates the possibility to say no, which also doesn’t exist in SCTP AFAIK, at least not in the API - this would be a “stream reset”).  Now, in QUIC, I don’t know exactly how that is, but I would have thought it’s the same - you just go ahead and use a new stream. No “fork request” on the other side - just a new stream (in post, carrier) being used.

The term "fork-request" is misleading, yes. It really should just say: "forking a carrier creates a new flow to the same Remote, which may be a new stream on a multiplexing protocol. The protocol will handle any required signaling to the remote to open a new stream, based on the protocol".

> 
> 
>>>> - The point of the API is to give the shape of the application interaction, which is why you find it general. I believe that many of the protocol-specific options like you mention (disabling Nagle, retransmissions, etc) don't belong to this main abstract API draft, but to a different document that goes into how to configure options that can be used for setups like TCP/UDP. Again, in my implementation of Post, all of these options exist as part of the Configuration object mentioned in the draft. However, as I'll discuss in the WG, I believe that the general shape of the API needs to be defined to be more-forward looking than just what we've specified with the minset for TCP/UDP/SCTP. Essentially, these become a set of Configuration options that can be used, but as transport protocols continue to evolve (and when we need more protocol-specific options), we need to be able to expand. Certain aspects of the minset, like the connection state management, are of course general and common enough to be part of the highest level of API description.
>>> 
>>> It’s clear to me that we want a higher abstraction level than what the list from minset has - e.g., rather than a DSCP value, it would be better to specify general requirements (low latency or such) for a carrier. Rather than saying “disable Nagle”, we could also say “ low latency, even if it comes at the cost of some overhead” - we do such things in the NEAT API too. You’re focused on the interaction with the application (e.g. callback-based instead of traditional socket-style) - which is fine, but I think doesn’t have much to do with the actual protocol choice.
>>> 
>>> As for being more forward-looking, I wonder what the new transport features are that we’re missing out on (things that apps really see). I follow QUIC from a slightly too large distance (I simply run out of cycles there :(  ), but so far I’m not sure there’s anything we’d be missing  (but that’s maybe also because I’m not an HTTP/2 expert either). Except security of course, but there the argument is perhaps that it’s enough to consider falling back to TLS?
>> 
>> QUIC is using TLS 1.3, so it can give equivalent security properties to HTTP/2. Doing fallback between these is definitely an important feature. I think this flexibility is why the API needs to allow per-protocol configs.
> 
> Sure, but I think (and that was my point) this doesn’t change anything about the stuff above (security is in your separate document) - so what exactly are the “more forward looking” requirements except for these security needs?

I'm trying to distinguish certain aspects of the minset that are:
a) fundamental to all transports, that we have reason to believe will be the case for all future transports in some fashion
	CONNECT, SEND, RECEIVE, CLOSE, etc
b) derived from the protocols we happened to survey or based on current semantics (sockets), based on what exists
	Things I see in the NEAT draft like SET_MIN_CHECKSUM_COVERAGE, SET_TTL, SET_LOW_WATERMARK. Important to have, but often the values the application needs to set relate to which protocol might be used, and may have different requirements for future protocols.

So, there's a set of things in category (a) that should be stable as the main API go forward for quite a while, while things in category (b) may belong as configuration properties, in a list that may shrink or expand over time as new developments are made.

> 
> 
>>> As for how you represent Nagle etc. in your code, that sounds good to me, but I see an issue in that so much transport functionality is simply not covered by the draft. E.g., the packet sizes / fragmentation thing is important too … yes that’s lower level stuff, but UDP-based applications will only get inefficient if they don’t get this information exposed, which risks having app programmers move to the old ways again IMO.
>> 
>> A Post API definitely allows access to IP-packet level configuration and metadata. However, you're right that the draft doesn't specify this in detail now. I still think that may be a separate doc, but yeah.
> 
> I wouldn’t mind that (though it’s not up to me to decide), but —
> 
> 
>>> So, the general design approach of post-sockets is IMO fine, but what I’m missing is a long section with the nitty-gritty details on how such a system would really be implemented (not the happy eyeballing bit, this is fine to be elsewhere - but “how would this work with current protocols”). If I was to implement it from the current draft, I’d have a ton of question marks - e.g. the draft convinced me that there must be a system on both sides (this issue is also not discussed in the draft).
>> 
>> Getting feedback on how to clarify this for the implementer will be really important, so thanks for this feedback! I'd also like to discuss with the WG the scope of various documents that we want to produce. If you have one document that does the abstract connection-management and I/O API and all the protocol options and the racing/fallback strategy, that would be a very long and complex doc.
> 
> Great - I’m happy that this is constructive!  Regarding the requests I’m making about post sockets, a reason I think more of these implementation level considerations need to be in the draft is that this gave me so many hiccups of the sort “but then you need it on both sides”, “but then you need to add a lot more protocol functionality”, “but do you think you could do that?” … etc.
> I guess no matter if there are multiple docs or one, any document for charter item 3 should leave its reader with the warm feeling of understanding how this could be implemented.
> 
> To figure out these details, I’d like to mention that the minset is here to help  :-)    (that’s what it’s meant for). Yes it’s too low-level to be a good API proposal in its own right, but that’s also not the point - it tells you what can and can’t be done. Every decision in it can be traced back to specific reasoning based on the original RFCs; even if you end up disagreeing with reasoning X about a mechanism Y from a RFC Z, at least you know precisely what you’re doing in relation to the TCP, MPTCP, SCTP, UDP, UDP-Lite and LEDBAT specs).
> 
> Cheers,
> Michael