Re: [tsvwg] Network congestion, dynamic lossless compression?

["David G. Pickett" <[e-mail]>]

Jonathan:
Already compressed: You may not have noticed that zip detects not compressible parts and just copies them into the archive.  Similarly, a packet stream compressor could send such blocks on verbatim in the stream, or leave the packets alone entirely.  There is a slight overhead for headers on no-compression data, so the protocol would need a threshold for deciding which path a packet takes.  Backlogs give the node time to detect such and make decisions.  You still get the potential for combining smaller packets into larger ones, gaining a reduction of overhead in many media with larger local link MTU and inter-packet lost media use.  There still needs to be a packet discard mechanism when the compressed flow falls too far behind.  What it gives us is a softer saturation.  And lost packets are not free, as the overhead of retransmission is a cost, as well as the pause in transmission hurting both latency and bandwidth.
Discussion is great: I wonder if the compression choice could also be to use a lower, faster mode when the backlog is light, and a slower, higher compression mode when things get worse?  In some cases, streaming small packets in big packets might increase link speed enough without any compression!
Demand grows: Parkins law does not justify no innovation for improvements.  Softer link saturation means less packet loss, both due to packet discard and overdue packet timeout.  While compression at first glance increases latency, when the link is saturated, the increased bandwidth translates into lower latency as the packet gets sent sooner.  And for momentary overload, the link would switch back to uncompressed packet by packet forwarding as soon as the bubble of load is disposed of, for lowest latency!

There is always room at the top!
Best,
David
-----Original Message-----
From: Jonathan Morton <[e-mail]>
To: David G. Pickett <[e-mail]>
Cc: tsvwg <[e-mail]>
Sent: Sat, Aug 3, 2019 2:34 pm
Subject: Re: [tsvwg] Network congestion, dynamic lossless compression?

> On 3 Aug, 2019, at 6:13 pm, David G. Pickett <dgpickett=[e-mail]> wrote:
> 
> Lossless compression could be applied without any effect on the transmitted data.

The chief problem with that idea is that most data transmitted these days is already compressed and/or encrypted, making it impossible to further compress transparently.  Even Web traffic, which used to be relatively compressible back when PPP options and in-modem compression were more relevant, is now encrypted more often than not (HTTPS replaces HTTP), and the encryption system itself applies compression to reduce exploitable entropy in the plaintext.

In the end, anyone can do better than your suggestion by "just" building a bigger pipe.  But that isn't a solution by itself, because demand grows to meet and exceed supply.  ("The bureaucracy is expanding, to meet the needs of the expanding bureaucracy.")  To actually improve quality of service in the long run means finding ways to reduce latency and packet loss, not to improve capacity.

 - Jonathan Morton