Re: [rtcweb] Open data channel issues

[Randell Jesup <[e-mail]>]

On 3/3/2014 6:18 AM, Justin Uberti wrote:
>
>
> On Sun, Mar 2, 2014 at 6:42 PM, Randell Jesup <[e-mail] 
> <mailto:[e-mail]>> wrote:
>
>     On 3/2/2014 4:27 PM, Justin Uberti wrote:
>>     I'm not sure I understand the actual problem here. We *are* going
>>     to handle arbitrary-sized messages once ndata gets done. The
>>     max-msg-size stuff is just a solution until ndata is deployed.
>>     (To cite your options, this is A followed by C).
>
>     So, here's where I think there may be a disconnect (and if I'm
>     wrong, great):
>
>     ndata solves the monopolization issue between streams, allowing
>     packets for different streams to be interleaved on the wire.  It
>     does not (so far as I know) relax the usrsctp library's returning
>     EMSGSIZE if the sendto() is larger than available buffer space,
>     and the alternative (EOR mode) doesn't allow for interleaving of
>     messages when sending at all (and I'm not sure it allows
>     interleaving on reception either, but I didn't check right now).
>
>     Now, this is an implementation/API issue which could be fixed with
>     some work, but so far as I know no work has been done on it.  So
>     ndata will allow us to expand the max-sending size to
>     min(SCTP_BUFFER_SIZE - N, other-sides-max-receive-size).  It does
>     not allow us to expand the size to any useful Blob size.
>
>
> Based on the discussion of ndata/eor in followup messages, I think 
> this problem will be addressed in the near future.

Hopefully so, though I hate forcing apps to include a bunch of code for 
backwards compat for a year or two (and many won't, or won't test it, 
etc).  That hurts the feature.  And today no one has ndata.

>
>>     That said, Send(blob) seems like a bit of a footgun to me anyway;
>>     I think apps are going to typically avoid it, a) to avoid memory
>>     bloat, and b) to get progress indications. The only way out of
>>     that situation is to provide stream semantics, which seems like a
>>     lot to take on given that WebSockets hasn't gone that route. I
>>     also agree that we shouldn't try to go the (B) route that you
>>     mention.
>
>     Well, Streams would be a good thing once they're done. Progress
>     indication could be added at the W3 level without much problem. 
>     The memory issue should be resolvable in a number of ways, per my
>     discussion with Sicking.  Note that application chunking has a
>     serious memory issue today in that the File Writer API hasn't been
>     agreed to; I think there's progress towards eventually adopting a
>     version of the Filesystem API (with changes), but that will be a
>     while.  Again, Stream should help - and note that the semantics
>     for Blob reception allow it to be written to disk as it's received
>     and not held entirely in memory when it's handed to the
>     application, which is NOT possible today for application chunking.
>
>
> Agree Streams will be useful when done, but that's probably post-1.0 
> (as Martin says, we should first let it be applied to WebSockets). 
> However I don't think app chunking will have any memory problems - the 
> chunks will be small enough that they can be easily spooled to disk as 
> they come in.

Agree on Streams (post 1.0).  Chunking does have memory problems unless 
you have a File Writer API (we don't) or a Filesystem API (we don't, 
though my understanding is that we believe that with some changes this 
can be adoptable in the future).  You have two choices today for app 
chunking (and I assume using blob slices for sending chunks):

1) create a blob from each received chunk then then create a new blob 
from the concatenation of all the sub-blobs.  This requires a large 
memory hit, though I suppose in theory maybe it could be done without a 
hit (painfully).  Even if the memory hit is avoided there will be a 
large disk IO hit to merge all those files and write a new file from them.

2) hold all the parts in memory, then write the entire thing as a blob 
after the last part is received.  Large memory and disk IO hit on 
reception of final part.

>
>>
>>     So I still think manual chunking seems like a reasonable thing to
>>     support. I don't quite get the congestion control concerns; just
>>     because there is a max chunk size doesn't mean the impl can't
>>     buffer multiple chunks in bufferedAmount; the app could let that
>>     fill up to a degree to avoid having to poll constantly to prevent
>>     underrun.
>
>     On a slow link this will work if the browser isn't janky.  On a
>     fast link GC pauses and other things may cause the buffer to drain
>     out and go idle for significant periods.
>
>
> Since the amount buffered is up to the app, the app can just increase 
> the buffer size to provide enough data to cover the next second or so 
> of transfer.

How does the app increase the size of the internal SCTP buffers? (And 
that buffer is shared by all channels, of course).

> Note that app chunking will clearly be needed in some of the more 
> interesting cases we want to support, such as where the transfer is 
> not 1:1. For swarm-style downloading, we need to make sure app 
> chunking works well (and I think it does).

Agreed, we need good solutions for application chunking (and thus my 
interest in resolving the Filesystem API issues in the W3).  My problem 
is that we don't have those today for the reception side - but this (for 
apps that *need* chunking) is largely a W3 issue.


-- 
Randell Jesup -- rjesup a t mozilla d o t com