Re: [rtcweb] Open data channel issues

Justin Uberti <> Mon, 03 March 2014 13:29 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id 5D96D1A0132 for <>; Mon, 3 Mar 2014 05:29:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.925
X-Spam-Status: No, score=-1.925 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.547, SPF_PASS=-0.001] autolearn=ham
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id HF2ctRTPg245 for <>; Mon, 3 Mar 2014 05:29:17 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:400c:c03::22c]) by (Postfix) with ESMTP id 4A1451A0123 for <>; Mon, 3 Mar 2014 05:29:17 -0800 (PST)
Received: by with SMTP id lf12so3556457vcb.31 for <>; Mon, 03 Mar 2014 05:29:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=52gZCBKmwyCNRAcejFGBTzQpxMxp39S77uIuQWdo6TA=; b=RPhcZ+b1qQeeNd0bJpUU0GlmFeJQWLCGUm8hCS5du34fwIkYmqC/0vAMk3zQKaF/Ff YrxH45FtZeOIuZOOsY7/5hBtVlL4+dp3oGs/ejl0Pb06dpjilUQsHrYs+hJwUAuAbR0m j8feS4vQABW0m3BS4h2iMXKz08aTol565z1lHO/mKJd/OedeOudMmQCke/B4b3O2gGDy K5rUYG778k74eIPJnrHCXLnLOeBjiOQN3gzgemWqtEfTnE2B4Q88yN0hVT+D+krT7U+x zzOpFf7p7CeTs5JifccITSk/yIWJ1kJ3kiPvAOEc3NME7eFeTCdkHdKSa0w3oN9iLlN5 Uf1w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=52gZCBKmwyCNRAcejFGBTzQpxMxp39S77uIuQWdo6TA=; b=ltKuVoDRmYW1z2/FLMv31FC79CpkX1ZJmd6zBdC9Br+oDXsjCg8iweGpH3JqoxQCk4 P6bUJkI8WFdT4v38dBRoRaPNJQoUKHU6DDsmpMJtoGk6UWi0Ztd7n3gAI9KpPvs5MwH2 o83WOzpghR2hP97xD/Crgasu+ZxchaYq2j92/21ngksudj9wwrsdwPdwgmj37GT/z8Ag Ja0Lv8kQvuK70r/i7YxClME9ngdaGi9cb9RPKJDrfqlpgsERgGYVpMXbikjvJIUtiQSG 4TPb71RlRHDEN8Fe8jhzy75jVjtPRS20jkwCkeLtppFphUUu/CVhEwcKbuPtn1OPtHiO xsPg==
X-Gm-Message-State: ALoCoQkGnWUxKxyFlzUgynW2Y2Ytrnx2lkZ6TvDcnFoc+QqJTJwBV6kSxaaZHrLH3mRzXcG1kYaXa0qJHhEsgl9Uu8n8YvKCmNcUegOhns7YlYxwHYqH08xmLSiZVUdsSTZNubJdf2+XIabio/9t+T4kpD0Cu6Nwkt0d/d8iNjw8fqc0ZFLPYojgx4YRF71mAhLxGHBfTxVF
X-Received: by with SMTP id w3mr17228619vej.22.1393853354138; Mon, 03 Mar 2014 05:29:14 -0800 (PST)
MIME-Version: 1.0
Received: by with HTTP; Mon, 3 Mar 2014 05:28:54 -0800 (PST)
In-Reply-To: <>
References: <> <> <> <> <> <> <>
From: Justin Uberti <>
Date: Mon, 03 Mar 2014 05:28:54 -0800
Message-ID: <>
To: Randell Jesup <>
Content-Type: multipart/alternative; boundary="089e013a0d420af76e04f3b3c611"
Cc: "" <>
Subject: Re: [rtcweb] Open data channel issues
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 03 Mar 2014 13:29:20 -0000

On Mon, Mar 3, 2014 at 3:44 AM, Randell Jesup <>wrote:

>  On 3/3/2014 6:18 AM, Justin Uberti wrote:
> On Sun, Mar 2, 2014 at 6:42 PM, Randell Jesup <>wrote:
>>  On 3/2/2014 4:27 PM, Justin Uberti wrote:
>> I'm not sure I understand the actual problem here. We *are* going to
>> handle arbitrary-sized messages once ndata gets done. The max-msg-size
>> stuff is just a solution until ndata is deployed. (To cite your options,
>> this is A followed by C).
>>  So, here's where I think there may be a disconnect (and if I'm wrong,
>> great):
>> ndata solves the monopolization issue between streams, allowing packets
>> for different streams to be interleaved on the wire.  It does not (so far
>> as I know) relax the usrsctp library's returning EMSGSIZE if the sendto()
>> is larger than available buffer space, and the alternative (EOR mode)
>> doesn't allow for interleaving of messages when sending at all (and I'm not
>> sure it allows interleaving on reception either, but I didn't check right
>> now).
>> Now, this is an implementation/API issue which could be fixed with some
>> work, but so far as I know no work has been done on it.  So ndata will
>> allow us to expand the max-sending size to min(SCTP_BUFFER_SIZE - N,
>> other-sides-max-receive-size).  It does not allow us to expand the size to
>> any useful Blob size.
>  Based on the discussion of ndata/eor in followup messages, I think this
> problem will be addressed in the near future.
> Hopefully so, though I hate forcing apps to include a bunch of code for
> backwards compat for a year or two (and many won't, or won't test it,
> etc).  That hurts the feature.  And today no one has ndata.
>>   That said, Send(blob) seems like a bit of a footgun to me anyway; I
>> think apps are going to typically avoid it, a) to avoid memory bloat, and
>> b) to get progress indications. The only way out of that situation is to
>> provide stream semantics, which seems like a lot to take on given that
>> WebSockets hasn't gone that route. I also agree that we shouldn't try to go
>> the (B) route that you mention.
>>  Well, Streams would be a good thing once they're done. Progress
>> indication could be added at the W3 level without much problem.  The memory
>> issue should be resolvable in a number of ways, per my discussion with
>> Sicking.  Note that application chunking has a serious memory issue today
>> in that the File Writer API hasn't been agreed to; I think there's progress
>> towards eventually adopting a version of the Filesystem API (with changes),
>> but that will be a while.  Again, Stream should help - and note that the
>> semantics for Blob reception allow it to be written to disk as it's
>> received and not held entirely in memory when it's handed to the
>> application, which is NOT possible today for application chunking.
>  Agree Streams will be useful when done, but that's probably post-1.0 (as
> Martin says, we should first let it be applied to WebSockets). However I
> don't think app chunking will have any memory problems - the chunks will be
> small enough that they can be easily spooled to disk as they come in.
> Agree on Streams (post 1.0).  Chunking does have memory problems unless
> you have a File Writer API (we don't) or a Filesystem API (we don't, though
> my understanding is that we believe that with some changes this can be
> adoptable in the future).  You have two choices today for app chunking (and
> I assume using blob slices for sending chunks):
> 1) create a blob from each received chunk then then create a new blob from
> the concatenation of all the sub-blobs.  This requires a large memory hit,
> though I suppose in theory maybe it could be done without a hit
> (painfully).  Even if the memory hit is avoided there will be a large disk
> IO hit to merge all those files and write a new file from them.
> 2) hold all the parts in memory, then write the entire thing as a blob
> after the last part is received.  Large memory and disk IO hit on reception
> of final part.
>>  So I still think manual chunking seems like a reasonable thing to
>> support. I don't quite get the congestion control concerns; just because
>> there is a max chunk size doesn't mean the impl can't buffer multiple
>> chunks in bufferedAmount; the app could let that fill up to a degree to
>> avoid having to poll constantly to prevent underrun.
>>  On a slow link this will work if the browser isn't janky.  On a fast
>> link GC pauses and other things may cause the buffer to drain out and go
>> idle for significant periods.
>  Since the amount buffered is up to the app, the app can just increase
> the buffer size to provide enough data to cover the next second or so of
> transfer.
> How does the app increase the size of the internal SCTP buffers?  (And
> that buffer is shared by all channels, of course).

What I meant is the app can make multiple calls to send() at a time, each
with a chunk, which will be buffered in the api layer and increase

>   Note that app chunking will clearly be needed in some of the more
> interesting cases we want to support, such as where the transfer is not
> 1:1. For swarm-style downloading, we need to make sure app chunking works
> well (and I think it does).
> Agreed, we need good solutions for application chunking (and thus my
> interest in resolving the Filesystem API issues in the W3).  My problem is
> that we don't have those today for the reception side - but this (for apps
> that *need* chunking) is largely a W3 issue.
> --
> Randell Jesup -- rjesup a t mozilla d o t com
> _______________________________________________
> rtcweb mailing list