Re: [hybi] hum #3: Message

[Arman Djusupov <[e-mail]>]

Pieter Hintjens wrote:
> I don't think this is accurate. Framing is to allow the reading of messages
> by a layer that does not need to know their semantics. Also to allow
> preallocation and protection against overflows. Fragmentation is to allow
> long messages to be handled in smaller buffers. I think in all cases we can
> assume the sender knows the length upfront.

Preallocating resources for the entire message is not always needed in order to consume it (e.g. forward only XML parsers can consume/transform GBs of XML data using a buffer of a few KBs). The same applies to chunking which does not require a single frame to fit into the input buffer since the implementation can read frames of arbitrary size using whatever input buffer is available. So the size of the frame/message is not always related to the size of the buffer.

We cannot assume that the sender necessarily knows the length of the data beforehand because there are cases when responses are getting generated on the fly by the server side script/code. In such cases the server side cannot predict the final size of the message unless it already has the response cached or buffered. So it either spends time waiting for the full response to be generated or starts sending it right way by using the chunked transfer encoding thus saving time and memory.

Arman