Re: [hybi] hum #3: Message

On Thu, Aug 5, 2010 at 1:33 PM, Julian Reschke <julian.reschke@gmx.de>wrote:

> On 05.08.2010 22:21, Maciej Stachowiak wrote:
>
>>
>> On Aug 5, 2010, at 12:52 PM, Julian Reschke wrote:
>>
>>  On 05.08.2010 21:31, Ian Hickson wrote:
>>>
>>>> On Thu, 5 Aug 2010, Maciej Stachowiak wrote:
>>>>
>>>>>
>>>>> That being said, I think you're misunderstanding Ian's point. He
>>>>> expressed concern about the overhead for small messages to having a
>>>>> large fixed length, not the overhead to large messages of a chunked
>>>>> encoding.
>>>>>
>>>>
>>>> I was trying to express both.
>>>>
>>>> Having to split a 2GB file into a bazilion pieces and write each one out
>>>> individually with frame headers, compared to just writing the whole
>>>> thing
>>>> at once, seems inefficient far beyond simply the concern about the
>>>> on-the-wire overhead.
>>>>
>>>
>>> Hum, where exactly do you see the additional overhead? It's not like you
>>> need to split it into inidividual *files* - you just interleave the data
>>> with the frame headers while writing to the stream...
>>>
>>
>> Sending a file from disk to the network without processing it in any way
>> is more efficient than cutting it into slices and interleaving with headers.
>> The IP layer is already going to cut it into slices, in a way that almost
>> certainly won't line up with WebSocket-level fragment boundaries.
>>
>
> There's no way to "send a file over the network without processing". You
> need to read it into memory, and then write it to the network. Of course
> adding frame headers *is* overhead, but it's really identical or close to
> the overhead on the wire, nothing more.

Well, that isn't entirely true. There are now kernel interfaces for doing
such things without userspace involvement. I doubt that we're at the level
of DMAing directly from the disk to the network device, so we're probably
hitting main-memory at some point, but there is certainly the potential for
higher thruput using these interfaces. My experience has been that this
theoretical increase isn't yet realized, but it *should* be soon.

-=R

>
>  It's also more efficient for the receiver if they know the length up
>> front. That way they can determine whether to allocate a large enough
>> buffer, or just start spooling to persistent storage immediately.
>>
>
> Absolutely. When the length is known (see video streaming and on-the-fly
> compression). Thus, it might be good to be able to announce the eppected
> size upfront.
>
> Best regards, Julian
>
> _______________________________________________
> hybi mailing list
> hybi@ietf.org
> https://www.ietf.org/mailman/listinfo/hybi
>