Re: [conex] byte vs packet counting

[Toby Moncaster <[e-mail]>]

On 5 Dec 2011, at 23:01, John Leslie wrote:

> T. Moncaster <[e-mail]> wrote:
>> 
>> I have been purposely not contributing to this thread over the past few 
>> weeks as I wasn't sure that more voices in the argument would actually
>> help things.
> 
>   IMHO, _any_ additional voices would help. (Of course, I may be biased:
> if nobody takes my ideas seriously it's easier to declare me "in the
> rough"...)
> 
>> However I feel things are now at a point where I can contribute
>> something (hopefully) useful.
> 
>   Toby has _always_ had something useful to contribute!!
> 
>> On Dec 2 2011, John Leslie wrote:
>>> Dirk Kutscher <[e-mail]> wrote:
>>>> [John Leslie <[e-mail]> wrote:]
>>>> 
>>>> Auditing is really a fundamental piece of the basic Conex mechanism,
>>> 
>>> I'd be happy to discuss that. I see no reason to bother auditing
>>> if the entire path is known to be free of congestion. Auditing might
>>> make sense if you expect traffic you send to encounter congestion
>>> later in the path, but only if you have some contractual arrangement
>>> calling for it. (Otherwise, you can't know what the effect of
>>> ConEx marks may be.)
>> 
>> I think there is a subtle difference between stating that auditing
>> is a fundamental part of ConEx and saying that auditing can never
>> be turned off. 
> 
>   Agreed... but it's not obvious which of these Dirk meant. :^(
> 
>> I would definitely agree that auditing is required in any network
>> where there are potential trust issues and where you can't guarantee
>> the whole-path network conditions.
> 
>   In other words, any real-world network…

indeed

> 
>> But I do think there are cases where it could be turned off (for
>> instance in private corporate networks using conex as a means to
>> control their monthly network bills).
> 
>   Is that really so different from real-world networks?

The main difference is that they may have an additional "big stick" that they can beat their users with if they are found to have cheated. In other words you can make the trust  separate issue.

> 
>   (An aside: I talked with Matt today, and we're making progress on
> understanding each other on what the WG should mean by "auditing".
> Stay tuned…)

That sounds like an interesting chat - looking forward to seeing the output of it.

> 
>>> Auditing every packet is plausible near the sender and receiver,
>>> but not in the backbone. I don't believe we have agreement among
>>> ourselves what a receiver's uplink will do if their auditing shows
>>> "abuse" of ConEx marking: so I don't see what auditing at the
>>> sender's uplink can do and be confident it's the right thing.
>> 
>> I agree, it seems hard to see how auditing something when you are
>> missing half the information is going to be hard.
> 
>   (I'm having trouble parsing that sentence.)

I think we are hitting the issue of "what is audit?" If audit is taken to mean accounting for ConEx marks, then that can happen anywhere. However I was taking auditing to mean accounting for the balance of ConEx marks with actual congestion. That function can only work somewhere where you can see both the congestion and the ConEx marks. The only way you can achieve that at the sender side is by inspecting the ACKs on the return path (which may be impossible).

> 
>> There are possible arguments along the lines of an ISP might make
>> a decision that all users of a given access node (B-RAS or whatever)
>> are likely to be experiencing roughly similar amounts of congestion.
> 
>   (Seems unlikely to me…)

Oh, patently unlikely, but it is the sort of thing I can (sadly) imagine some ISPs doing.

> 
>> So any customer that declares no congestion is probably not being
>> honest.
> 
>   This sort of assumption bothers me. What if that customer is using
> a particularly good LBE protocol?

Well quite - that is the argument against blanket rate limiting of all P2P, even if it is running on LBE transport.

> 
>>>> Having said that, it is certainly implementation-specific how
>>>> auditing really works -- there may be tradeoffs between efficiency
>>>> and effectiveness -- but for the sake of the discussion here, I'd
>>>> say it's best to assume that, yes, auditing evaluates every packet.
>>> 
>>> IMHO, it's _always_ best to assume otherwise.
>> 
>> My view is that we're arguing the wrong point - auditing every packet
>> seems trivial if we are assuming that most routers can mark a packet
>> (e.g. can do ECN). In that case they are already inspecting 2 bits of
>> the IP header, so asking them to inspect a 3rd bit seems reasonable.
> 
>   (Talking with Matt today, I am beginning to understand he may be
> assuming that "auditing" measures only ConEx marks versus ECN marks
> for a particular packet; and that the action taken on that basis
> will be simply and only dropping the packet where the imbalance is
> detected, keeping no state. That much could be implemented anywhere;
> but it falls far short of what we are led to believe is promised by
> his "auditing function". I'm hoping the next revision will assign
> more intuitive names to whatever his current "auditing function"
> proposes…)

That has been sort of my assumption, but with the proviso that you can only really tell the imbalance over a set of packets (see below)

> 
>> However, on a per-packet basis audit is meaningless.
> 
>   I think I agree with Toby.
> 
>> Audit only has any meaning over a period of time across a given flow
>> (or set of flows).
> 
>   There's a pretty big difference between auditing a particular
> flow versus auditing the particular set of flows represented by all
> traffic an ISP receives from a peer at a peering point... I'm not
> sure exactly what Toby refers to here.

Audit has two granularities - either you are auditing a given user to ensure his flow (or flows if he is using several) are correctly reporting the expected level of congestion, or you are auditing a whole peer network to ensure they aren't allowing excess congestion to enter your network without it being declared. These require (subtly?) different audit mechanisms

> 
>> There is a separate accounting mechanism where you can measure bulk
>> flows of congestion, but that has to be based on a belief in the
>> accuracy of the marking of the underlying individual flows.
> 
>   I don't see how that follows.

Measuring the bulk is very easy, but if all the flows making up the bulk are understating their congestion then the bulk will be understated as well. Bob always had in mind a clever mechanism to do spot checking at any border to identify persistent under (or over) declaration, but I never entirely understood how it was meant to work.

> 
>>> Your life will be easier if you don't depend on assumptions which
>>> are likely to be false.
>>> 
>>> We'd do better, when discussing this, to get specific about how
>>> auditing might improve the use-cases we're pushing.
>>> 
>>> Unless I misread draft-ietf-conex-concepts-uses, our principle
>>> use case is Informing Traffic Management. I have posted before that
>>> the most important feature of ConEx marking (IMHO) is informing
>>> any node along the way that the sender is knowingly sending packets
>>> despite having reason to doubt that congestion has cleared.
>>> 
>>> Auditing does nothing to help here. The abusive thing to do is
>>> to _clear_ ConEx marks -- and auditing can't detect that.
>> 
>> No. ConEx is about a balance of markings. To cheat you fail to send
>> forward into the network sufficient ConEx markings to cover the
>> congestion your flow is causing (or has caused).
> 
>   Not quite... an ISP can cheat by clearing marks which one sender
> placed in his outgoing traffic -- that is what I meant to refer to.

OK, yes. That is a separate requirement for audit. 

> 
>> So the function of auditing is to check that the two numbers
>> roughly match.
> 
>   (I note that Toby says "roughly match". Others seem to be assuming
> a much stricter match. I'm afraid we don't have a clear understanding
> here.)

Clearly, given the min 1RTT delay between getting congestion info and responding to it, there can never be a completely accurate match. I did have numerous debates with Bob about this topic. Basically you run the risk of a user just terminating his flow rather than making up his losses. You also have to be cautious that the actual process of dropping packets for audit doesn't simply distort the apparent congestion information (because the sender will respond by retransmitting and hopefully adding more ConEx marks)

> 
>> And yes, this is a very difficult problem to solve if you can't
>> assume ECN is being used.
> 
>   I absolutely agree with Toby here.
> 
>>> Auditing, to tell truth, may or may not prove useful after we
>>> have done enough experiments. IMHO, if it proves useful we'll find
>>> that it's useful when done statistically, rather than on every
>>> packet.
>> 
>> I think this may point to the heart of the issue. Yes, you can sample
>> to do audit, but only if you assume a relatively high density of
>> markings.
> 
>   I don't follow why that is true. If auditing is not strict, sampling
> should work for a wide range of density; if auditing is strict, it
> will fail pretty much regardless of the density…

OK, this comes down to the point below

> 
>> If ConEx info is unary encoded into packets then you can only measure
>> it by measuring long streams of the packets.
> 
>   I agree with what I think Toby intends to say: that the congestion
> metric of ConEx is a fraction, not a binary function per packet.

Which follows very naturally from ECN - ECN (RED) randomly decides whether or not to mark a packet, with the aim that the overall number of marks (very roughly) reflects the congestion level experienced at that queue. Because this is a random unary scheme, any ConEx mechanism based on ECN (and that is trying to audit against current network conditions) will need to track this over a few packets to see how many ConEx marks it should expect. OR the sender has to have sent sufficient credit up front.

Even in a scheme that is trying to identify either gaps in sequence space (dangerous - especially in an MPTCP world) or is trying to identify retransmissions (again, dangerous in MPTCP), then you have to have some sort of memory of what went before (a counter of congestion seen against ConEx credit seen for instance).

> 
>> If ConEx is a specific number carried in each packet then sparse
>> sampling may well give you enough to be able to achieve the aims of
>> audit.
> 
>   Carrying what I call "congestion-expected" as a multi-bit fraction
> is an interesting idea, in which I see potential benefits. I'd be
> happy to discuss it, but my impression is that we're not heading in
> that direction.

That was certainly how I understood a re-ECN like scheme working… Te whole core of this argument (should you account for bytes or packets) has always been a major bone of contention between me and Bob. Not least because it can be impossible for a sender to comply (it is very easy to construct pathological sequences of packets where the sender is forced to either over or under declare if they are doing per-byte auditing). I think this is a case where the IETF view has to diverge from the researcher's view. In the IETF we have to make prosaic engineering decisions. Personally I would like ConEx to account on a per-packet basis, BUT with an allowance for it to be more accurate if an operator so wishes (in other words MUST as a minimum account per packet, but MAY account per byte)

> 
>> Which brings me on to the aims of audit. To my mind there are two
>> possible aims.
>> 1) to guarantee that every single packet, flow, aggregation of traffic
>>   in the network has accurately declared their congestion; or
>> 2) to act as a "speed camera" that catches a sufficiently high
>>   proportion of infringements to ensure that senders don't risk
>>   under-declaring their congestion.
> 
>   I don't see auditing as either of these; but that may be more a
> function of my tendency to think of auditing as gathering information,
> not enforcing disciplinary measures based on this information.

Perhaps auditing is an unhelpful word to use. To my mind an audit implies an accountancy audit that tries to balance the books. I agree that carries no implication of policing or enforcement, but I think that a lot of this debate has assumed the two concepts are identical

> 
>   IMHO, "guaranteeing" much of anything isn't worth the effort over
> organizational boundaries. At significant expense, we could use
> cryptography to give arbitrarily high confidence to a particular
> "guarantee" having come from a particular entity, but without also
> signing the data "guaranteed", the effort is entirely wasted.
> 
>   (And processing all that cryptography in backbone routers seems
> a non-starter to me.)

Agreed, this has to be a lightweight mechanism. All the guarantees have to come from the idea that operators have to have some level of trust among one another (or at least, the realisation that an operator found to have cheated at the expense of others runs  risk of getting sat on heavily by the others),

> 
>> My own view is that the second is more feasible in the real world,
>> although the first might well be more desirable.
> 
>   I at least partly agree here. However, my approach to thinking
> about such problems simply stops descending into details when I reach
> a feasibility blockage.

And therein lies the lesson for ConEx. We HAVE to fix on a SIMPLE subset of ConEx that we can all agree on and that seems at least feasible in the real world. Arguments like the recent ones make ConEx appear to be still firmly stuck in the land of research (IRTF). Too many people (on all sides of all arguments) seem to want perfection and that is not sensible in the real world. That is just a recipe for ratholing ourselves to the point where the IESG declares our WG dysfunctional… 

Toby


> 
> --
> John Leslie <[e-mail]>