Re: [rmcat] 5 tuples and rmcat-cc-requirements-01

[Dave Taht <[e-mail]>]

On Sat, Jan 4, 2014 at 5:28 AM, Michael Welzl <[e-mail]> wrote:
> Hi,
>
> You answer both Harald and me in this email; I can't respond to the parts where you address Harald, and so I'll just cut some stuff away below.
>
>
> On 4. jan. 2014, at 05:18, Dave Taht <[e-mail]> wrote:
>
>> On Fri, Jan 3, 2014 at 3:25 AM, Michael Welzl <[e-mail]> wrote:
>>> On 3. jan. 2014, at 01:19, Harald Alvestrand <[e-mail]> wrote:
>>>
>
> [snip]
>
>
>>> Not so much to me. Dave's gut seems to argue for separate treatment of audio
>>> and video in the network - what for? To protect them from each other,
>>> because the sending application doesn't know what it's doing?
>>
>> 0) audio behaves way differently than video, and should be differentiated.
>> I personally care not a whit if my video goes to hell once in a while. I care
>> a lot if my audio does. Videoconferences do contain a lot of visual information
>> but audio, despite being truly tiny relatively, is critical.
>
> Tell this to an owner of website that offers WebRTC-supported striptease shows  :-)
> My point being: I agree that you describe a common requirement, but this decision should really be up to the application designer (via prioritiziation, via the JS API).
>
>
>> 1) To differentiate them from other flows on the network. The sending
>> application
>> can't know what those flows are doing.
>
> I don't think we need this.

I think you are wrong.

> Case 1: consider two RMCAT applications (browsers doing WebRTC) talking to each other across one bottleneck link. If you multiplex flows, irrespective of what they are, onto a UDP 5-tuple, you shouldn't need any help from the network: you're in complete control over priorities on the sender side, and you have only one bottleneck, i.e. something that acts like a single congestion controller should give you the best behavior. Remember that we're envisioning a delay-based cc. here that minimizes queuing delay anyway. In such a setting, I can't see how it would help to use different tuples for audio and video so that e.g. fq_codel could separate them.

Who is we? I'm not envisioning a delay based cc, but only a partially
based one. Delay based cc tcps don't work vs loss based. next
question.

> Case 2: consider the same scenario, but with other flows on the bottleneck that recklessly push up the queue. Then, if you have only one common tuple for video and audio, e.g. sfq is going to give this one tuple one bin, and one queue, and we're back at what I described for case 1 -- separating them would only help to protect the browser's own audio from the browser's own video by putting them in different bins.
>
> You mentioned that you'd like to do both, help from the network side *and* from within the application by coupling the congestion controls. Why, how? You might end up with the sender prioritizing flows in one way and the network working against that with sfq. I'd say that we should rather try to prevent sfq from doing its job on the sub-flows of an rtcweb application.

I think that all on one tuple should be allowed, and on different
tuples allowed also, and let the application designer choose.

>This is not at all to say that fq_codel is a bad thing! It's great if it protects rtcweb traffic from everything else - but I think that letting it operate on rtcweb sub-flows could give us a behavior that we don't want, for no real gain. The fairness of e.g. sfq may be inappropriate for individual rtcweb flows (it may go against what the person writing the JS app wanted), and the potential delay benefit should pretty much disappear if the common congestion control is designed to minimize queuing delay anyway.

I don't see how you can get sufficient insight into the underlying TCP
stack to have a common congestion control picture across that and
webrtc.

> You keep pointing at the value of real-life tests. Do note that if you run tests with Chromium today, fq_codel may help you: as you know, google's congestion control was found to have issues, and there is no congestion control coupling happening yet. But we're trying to develop something new here, so measuring what's out there right now won't help.

What was measured in this series of experiments was chromium vs cubic
tcp, under a variety of dsl, cable, wifi emulations that seem to be
pretty
accurate.

The whole point of the experiment was to get a grip on what happened
today and get insight as to be what could be done tomorrow.


>> 2) To make analysis easier from tools like wireshark (my immediate need).
>
> Agreed, this is a point in favor of separating. Then again, the multiplexing-or-not debate is old and I would guess that such things have already been debated?

I was too busy fixing routers to pay attention. Cites to relevant
arguments in the past on other mailing lists desired.

>Anyway there are many more pro's and con's to this whole thing than what we're discussing here (a lot of it is related to middlebox behavior).

I agree middlebox behavior is a serious problem. I had qualified this
conversation to be talking about two tuples rather than one for voice
and video, and also pointed out the non natted and ipv6 cases as being
the ones I was looking at primarily. I don't see a problem keeping two
ports open rather than one, even with middleboxes, actually, but am
open to opposing arguments.

>
>
>> What my gut says and what experiment shows is often very different. I
>> can see some problems with splitting tuples like this, like adding an
>> extra system call every 10ms or so, and having to explicitly be aware of
>> joint time stamps from two streams, but I think they are small problems
>> relative to the potential benefits.
>>
>> 3) To take advantage, where possible, of different scheduling and classification
>> techniques in the network. A strict drop tail or AQM system won't be
>> affected by adding in this support, although I foresee better mixing on the
>> endpoints as being a benefit across any network technology. I'd like
>> to be fiddling with ECN as well.
>
> See above, I don't think that you'd have a benefit, but potentially a disadvantage.

I basically already demonstrated a benefit in the captures I took, and some
additional simulations of isochronous traffic vs variable.

I'll take time out to write a paper after I get more code working. Maybe.

Further experiments are good.

>
>>> +1.
>>> Separating tuples can help to let the network protect a flow against *any*
>>> other flow, but for prioritization between flows coming from the same
>>> application, you should be much better off with joint congestion management,
>>> where you have 100% control over priorities and get other benefits too. My
>>> argument is: "send stuff into the network the right way rather than hoping
>>> for the network to correct your mistakes."
>>
>> I think it is best to do both.
>
> - discussed above.
>
>
>> I think it is incorrect to use the word prioritization when it comes
>> to the effects of packet scheduling. When I think of prioritization
>> I think of using diffserv or dpi to classify packets into bins. When
>> I think of scheduling, fair queuing, flow queueing, etc.,
>> I think of identifying flows and doing better mixing between them;
>> of turning packets back into packets again.
>
> http://tools.ietf.org/html/draft-ietf-rtcweb-use-cases-and-requirements-12
>
> "   F34     The browser must support prioritization of
>            streams and data."
>
> That's my basis for talking about prioritization. AFAIK, the fact that it's listed that way in this document means that it's going to be explicitly supported by the JS API.

Thank you for the steer to the relevant document. I will read.

>
>> So yes, I strongly support good mixing from the application level
>> and for it to be as self aware as possible of its total impact on the
>> network.
>>
>> But: although web browsers are complex and for many users
>> their entire OS (sigh), there is not enough information about
>> the hosts's other applications, the network and all the devices
>> held there to do all the job, thus giving the network more
>> information about flows in general is a good idea.
>
> I disagree because of what I wrote above. When a browser can bundle flows together, it means that it can 100% control their impact of these flows on each other

Agree and there is much fat to be sliced in the codebases I'm looking
at in favor of better mixing.

>, and then the network can't do things any better than the browser could do by itself.

disagree.

>
>
>> And I also strongly encourage experimentation with this stuff.
>
>> Me being me, I designed a bunch of experiments, emulated
>> every technology I could (DSL, cable, wifi, ethernet),
>> got a lot of data, started looking at captures long before
>> I started looking at source code and started raising questions
>> here on rmcat. I figure that's backwards from how most
>> people operate.
>
> Personally, I would be cautious about drawing conclusions from experiments with things that we know to be moving targets. For example, I can't see that you would be able to get an answer to the questions we're discussing here from experimenting with Chromium today.

I long ago drew a conclusion that current networks have way too much
buffering for any form of congestion control we know of to work right
for videoconferencing vs deployed tcps on cable, wifi, 3g and LTE.
Some hope on DSL & gpon. DSL because most that I have tried already
have some form of aqm or packet scheduling, gpon as it seems pretty
good on uploads, and not horrible on downloads.

As for experimentation, certainly answers can be obtained from the
moving targets!

Chromium is open source, so anybody can hack on it. I got a build done
last night. Adding in the bits of code and packet handling I'm
discussing experimenting with is fairly trivial. (finding where to add
it and working through the overlarge set of c++ abstractions is thus
far proving difficult.)

I am interested in the following experiments:

finding the "right" rate quickly
dealing with non-congestive (e.g. wifi) losses
improving recovery after a congestive loss
better analysis via rtp on separate tuples
looking at the effects of various proposed classification schemes, and
how to detect if they are preserved end to end
what ecn could be used for (and testing for end to end preservation also)
looking at the interactions of what network types exist today
(dsl,cable, wifi, 3g) in general
looking at aqm and packet scheduling vs drop tail behaviors.

Most of the core code we need is already written elsewhere. For
example being able to pay attention to classification, preservation
thereof, and
ECN is handled by these bits of code in mosh. (side note, TOS setting
is wrong for ipv6 in chrome & mosh, you need to use IPPROTO_IPV6 with
IPV6_TCLASS instead of IPPROTO_IP & IP_TOS, easy fix pending)

  /* request explicit congestion notification on received datagrams */
#ifdef HAVE_IP_RECVTOS
  int tosflag = true;
  socklen_t tosoptlen = sizeof( tosflag );
  if ( setsockopt( _fd, IPPROTO_IP, IP_RECVTOS, &tosflag, tosoptlen ) < 0 ) {
    perror( "setsockopt( IP_RECVTOS )" );
  }
#endif
}

string Connection::recv_one( int sock_to_recv, bool nonblocking )
{
  /* receive source address, ECN, and payload in msghdr structure */
  struct sockaddr_in packet_remote_addr;
  struct msghdr header;
  struct iovec msg_iovec;

  char msg_payload[ Session::RECEIVE_MTU ];
  char msg_control[ Session::RECEIVE_MTU ];

  /* receive source address */
  header.msg_name = &packet_remote_addr;
  header.msg_namelen = sizeof( packet_remote_addr );

  /* receive payload */
  msg_iovec.iov_base = msg_payload;
  msg_iovec.iov_len = Session::RECEIVE_MTU;
  header.msg_iov = &msg_iovec;
  header.msg_iovlen = 1;

  /* receive explicit congestion notification */
  header.msg_control = msg_control;
  header.msg_controllen = Session::RECEIVE_MTU;

  /* receive flags */
  header.msg_flags = 0;

  ssize_t received_len = recvmsg( sock_to_recv, &header, nonblocking ?
MSG_DONTWAIT : 0 );

  if ( received_len < 0 ) {
    throw NetworkException( "recvmsg", errno );
  }

  if ( header.msg_flags & MSG_TRUNC ) {
    throw NetworkException( "Received oversize datagram", errno );
  }

  /* receive ECN */
  bool congestion_experienced = false;

  struct cmsghdr *ecn_hdr = CMSG_FIRSTHDR( &header );
  if ( ecn_hdr
       && (ecn_hdr->cmsg_level == IPPROTO_IP)
       && (ecn_hdr->cmsg_type == IP_TOS) ) {
    /* got one */
    uint8_t *ecn_octet_p = (uint8_t *)CMSG_DATA( ecn_hdr );
    assert( ecn_octet_p );

    if ( (*ecn_octet_p & 0x03) == 0x03 ) {
      congestion_experienced = true;
    }
  }


-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

>
> Cheers,
> Michael
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html