From nobody Sat Oct 16 16:38:32 2021 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 65BFB3A0D57 for ; Sat, 16 Oct 2021 16:38:30 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.1 X-Spam-Level: X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ygEc9dI-Xjs2 for ; Sat, 16 Oct 2021 16:38:25 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CD0EC3A0D56 for ; Sat, 16 Oct 2021 16:38:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=edWYgQQWrIwYDW+IF7yUnoMmYUtK7J21HxwE3NjjhB4=; b=jo249E+3dlNZf+mcrKEECOwnuX KNl0UKWbN0/FYOMKjndhm6Svp7VaxizeQzI+EEf4mMAYYZSLbz/AzoLxR2yGfRsJllsg7YG9v64cY LVdETX2ZXtjRoJtM8HCcdtsla5pNG2WJfRNwQ3E4knQWPTwT5n7qaX2u+wCaCWFvngFibQpIS7zBR DSIjpb2kTfejBxop5ZJ4Q8PahBkyNXGn6Sw0zdOk9dQ7o5OiWiDk4Tspryben7fugQtwaT89X75X+ s6wfHhBIKcncU44Mgn4SQxL/rb/jDl8Eq4Jiccnh1UsngFLybIebGev+eR5p8AMPM+d4nN6yfIP/i FuAtyOeg==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:37408 helo=[192.168.1.11]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1mbtG5-00B2Y4-NT; Sun, 17 Oct 2021 00:38:22 +0100 To: Luca Muscariello Cc: Sebastian Moeller , Gorry Fairhurst , "tsvwg@ietf.org" References: <7dd8896c-4cd8-9819-1f2a-e427b453d5f8@mti-systems.com> <360988450.1173982.1630607180962@mail.yahoo.com> <8415BA1F-2806-4224-9D9F-EB256DA7DF41@gmx.de> From: Bob Briscoe Message-ID: <24307985-3450-a4db-5ca0-8d51d4195155@bobbriscoe.net> Date: Sun, 17 Oct 2021 00:38:20 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------13350C00A36E81EA8A144C7C" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tsvwg] start of WGLC on L4S drafts X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Oct 2021 23:38:30 -0000 This is a multi-part message in MIME format. --------------13350C00A36E81EA8A144C7C Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Luca, see [BB] On 15/10/2021 18:02, Luca Muscariello wrote: > > > On Wed, Oct 13, 2021 at 12:03 AM Bob Briscoe > wrote: > > Sebastian, > > Sorry - just noticed I missed this email at the time. See [BB]... > > On 21/09/2021 21:54, Sebastian Moeller wrote: > > Hi Bob, > > > > > > > >> On Sep 20, 2021, at 18:40, Bob Briscoe > wrote: > >> > >> Alex, Gorry, [Sry, just discovered I didn't click end on this] > >> > >> In "all of a user's applications can shift" the word "can" is > important. > >> Similarly, in "The L4S architecture is intended to enable all > Internet applications to transition" the word "enable" is important. > >> > >> Neither statement was meant to imply that all applications > would move, or should move, just that they would all be _able_ to. > >> These statements are primarily intended as counterpoints to the > commonly held view that low latency comes from priority scheduling > (e.g. via differentiated service) so low latency can only be > provided for a subset of traffic, because if all packets requested > higher priority, there would be no difference to any packet's latency. > >       [SM] Given that at the core of the L4S reference AQM there > sits a (conditional) priority scheduler*, I fail to see how L4S' > low latency does not come from priority scheduling? Also in L4S > low latency can only be provided to a subset of traffic, namely > flows that are well behaved and ECT(1)-marked (given the shared > queue design, one bad apple in the L-queue can affect > latency/throughput for all flows). I am not sure whether L4S > really is contrary to the "commonly held view" here, sure the > details differ a bit, but at its core it seems rather similar to > traditional techniques for low latency. > > [BB] You say 'L4S low latency can *only* be provided to a subset' (my > emphasis), but the word 'only' is wrong. > That is the point. L4S low latency /can/ be provided to all traffic. > > Yes, L4S low latency /is/ only provided to flows that are > well-behaved > and ECT(1) marked, but all traffic /can/ behave smoothly without > losing > out, and all traffic /can/ use ECT(1) without losing out. > > The priority scheduler really is not at the core. > * Consider an L4S FQ scheduler like FQ-CoDel with the ce_threshold > parameter applied to ECT(1) traffic. There is no priority > scheduling but > all traffic can still have very low latency. > * Similarly, an L4S DualQ Coupled AQM can work with any scheduler > that > provides isolation, for instance the Time-Shifted FIFO. Scheduling > priority is not 'at the core', 'cos it's not even necessary. We only > ended up choosing a priority scheduler because it gave slightly > better > tail latency. > > The real core of L4S is scalable congestion control. That is the > difference that allows even capacity-seeking traffic to also maintain > consistently very low latency. It's easier to see in the L4S FQ > example, > that it's what keeps a flow's latency down. Not the scheduler. > > If TCP had been designed with a scalable congestion control from the > start, we would have had L4S with no need for a scheduler at all. The > scheduler is primarily a transition mechanism, to isolate from > Classic > traffic. But no traffic /needs/ to be Classic, it's just how "stuff > happened". > > > I do not think this is true in this specific case. > There is an advantage in terms of goodput for applications to use > loss-based congestion control and create bigger queues (in > absence of isolation). [BB] This may be true in the short term for non-latency sensitive bulk traffic, especially over variable capacity (e.g. radio) links where buffered packets can fill newly available capacity while the e2e congestion control catches up. Nonetheless, there is also the unscalable aspect of loss-based congestion control where, as flow rate scales, it takes longer and longer (minutes) to recover after each loss [RFC3649]. As was pointed out in footnote 6 of [Jac88], the noise sensitivity gets worse as flow rate scales. Meaning, even if there's a very low level of transmission losses or new flows arriving, a flow keeps getting knocked down by each noise event, while it's slowly trying to regain its previous position. It's possible that some new approach will get out of that unscalable rut, but it's been quite resistant to solutions so far. BBRv2 has already had to reinstate a capped dependence on loss probability, in order to remain backward compatible with the legacy of loss-based congestion controls. The setting of that cap will have to keep being reduced in order to scale flow rate, so it is trapped in the same noise sensitivity problem. I don't know whether bulk flows will prefer to continue to use the classical approach with more buffering, or the scalable approach with frequent control signals. I'm just saying it's not as clear-cut as you make out. [Jac88] Van Jacobson and Michael J. Karels. Congestion Avoidance and Control. Technical report, Lau- rence Berkeley Labs, November 1988. (a slightly modified version of the original published at SIG- COMM in Aug’88) > Unless these applications are policed one way or the other > it is hard to create incentives to move towards protocols that > generate little queuing. Isolation is one such policy as the reward > is immediate. [BB] When you have scalable congestion controls that fill capacity with consistently very low queueing, you can just use them (without policing) whether you need low latency, high throughput or both. As a fairly close analogy, can you imagine any individual or company gaming a PIE AQM to induce more latency on their own flow, in order to impose more latency on others in the same household? Policing hasn't even been necessary for 'TCP-friendliness' between application flows within a household, where there is bandwidth to gain. I don't know whether policing will be necessary for 'latency friendliness'. Maybe it will at first while there are people around who don't understand that there's nothing to gain. My prediction, FWIW: I suspect that some policing within some paths (e.g. in DOCSIS links) will be enough at first to keep globally deployed code disciplined, whether or not it's actually running over a path with policing. I doubt policing will be necessary longer term, but we will have it if we need it. So nothing is hanging on who predicts this correctly. Or are you saying that we should never allow anyone to even attempt to have low latency in shared queues in case it requires the complexity of per-flow policing? But instead we should only ever allow per-flow queuing for low latency, which is somewhat more complex than per-flow policing anyway. I hope you appreciate that avoiding the possible risk of a little complexity by requiring more complexity is broken logic. > There is also a major implication between priority or scheduling > as an isolation mechanism: for the latter there is no need to > allocate a bit in the network header to identify traffic types. [BB] I don't understand what you have in mind here. Pls explain more fully. Bob > > > > > > Bob > > > > > Regards > >       Sebastian > > > > > > *) > https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19 > : > > "In this case it would not be appropriate to call the queue an L4S > >     queue, because it is shared by L4S and non-L4S traffic.  > Instead it > >     will be called the low latency or L queue.  The L queue then > offers > >     two different treatments: > > > >     o  The L4S treatment, which is a combination of the L4S AQM > treatment > >        and a priority scheduling treatment; > > > >     o  The low latency treatment, which is solely the priority > scheduling > >        treatment, without ECN-marking by the AQM." > > > > > > > https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch-10 > : > > "The tension between prioritizing L4S and coupling the > >         marking from the Classic AQM results in approximate per-flow > >         fairness.  To protect against unresponsive traffic in > the L4S > >         queue taking advantage of the prioritization and > starving the > >         Classic queue, it is advisable not to use strict > priority, but > >         instead to use a weighted scheduler (see Appendix A of > [I-D.ietf-tsvwg-aqm-dualq-coupled])." > > > > > https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled > > > "Classic traffic needs to build a large queue to prevent under- > >     utilization.  Therefore a separate queue is provided for L4S > traffic, > >     and it is scheduled with priority over the Classic queue.  > Priority > >     is conditional to prevent starvation of Classic traffic." > > > > I ignore the yarn spun in those drafts, why this prioritization > would only result in latency but not longer-time-scale throughput > prioritization, because Pete's data show that for quite a number > of realistic traffic patterns DualQ fails to arbitrate capacity > equitably between the two queues. > > > > > > > >> I've made a note to myself to make it clear that this is not an > intention that all applications will transition, rather it is an > intention not to prevent any application from transitioning. > >> > >> > >> Bob > >> > >> On 02/09/2021 19:26, alex.burr@ealdwulf.org.uk > wrote: > >>> Hi Gorry, tsvwg; > >>> > >>> > >>> See inline [AB] > >>> > >>> > >>> > >>> On Thursday, August 12, 2021, 11:23:49 AM GMT+1, Gorry > Fairhurst> wrote: > >>> > >>> --- > >>> 1b) Issue: I’d also quibble with: “so that all of a user's > applications > >>> can shift to it when their stack is updated.“ - again, is the > word “all” > >>> really necessary or correct here? > >>> --- > >>> > >>> [AB] > >>> Not  answering your point, but related: > >>> > >>> L4s-arch has at the very start the following: “The L4S > architecture is intended to enable   _all_ Internet applications > to transition away from congestion control algorithms that cause > queuing delay, to a new class of congestion controls that induce > very little queuing, aided by  explicit congestion signaling from > the network. “ > >>> > >>> > >>>   There are two quite different end states here: > >>> > >>> * An internet in which there are, end-to-end, two classes of > traffic on an ongoing basis. > >>> > >>> * An internet in which nearly all traffic has transitioned to > L4S, remaining non-L4S traffic being supported on a > backwards-compatibility basis > >>> > >>> As I understand it, if L4S is deployed, either of these could > be possible end states. At present it is not clear that all use > cases could migrate (eg, long RTT traffic) > >>> > >>> > >>> It seems like it would be a good idea for the draft to make clear: > >>> * That either of these end states are possible > >>> * What the formal position of the WG would be if the drafts > are accepted, IE what decision has been made (or left for later) > regarding the desired end state. > >>> > >>> > >>> regards, > >>> > >>> Alex > >>> > >> -- > >> ________________________________________________________________ > >> Bob Briscoehttp://bobbriscoe.net/ > >> > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------13350C00A36E81EA8A144C7C Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Luca, see [BB]

On 15/10/2021 18:02, Luca Muscariello wrote:


On Wed, Oct 13, 2021 at 12:03 AM Bob Briscoe <in@bobbriscoe.net> wrote:
Sebastian,

Sorry - just noticed I missed this email at the time. See [BB]...

On 21/09/2021 21:54, Sebastian Moeller wrote:
> Hi Bob,
>
>
>
>> On Sep 20, 2021, at 18:40, Bob Briscoe <in@bobbriscoe.net> wrote:
>>
>> Alex, Gorry, [Sry, just discovered I didn't click end on this]
>>
>> In "all of a user's applications can shift" the word "can" is important.
>> Similarly, in "The L4S architecture is intended to enable all Internet applications to transition" the word "enable" is important.
>>
>> Neither statement was meant to imply that all applications would move, or should move, just that they would all be _able_ to.
>> These statements are primarily intended as counterpoints to the commonly held view that low latency comes from priority scheduling (e.g. via differentiated service) so low latency can only be provided for a subset of traffic, because if all packets requested higher priority, there would be no difference to any packet's latency.
>       [SM] Given that at the core of the L4S reference AQM there sits a (conditional) priority scheduler*, I fail to see how L4S' low latency does not come from priority scheduling? Also in L4S low latency can only be provided to a subset of traffic, namely flows that are well behaved and ECT(1)-marked (given the shared queue design, one bad apple in the L-queue can affect latency/throughput for all flows). I am not sure whether L4S really is contrary to the "commonly held view" here, sure the details differ a bit, but at its core it seems rather similar to traditional techniques for low latency.

[BB] You say 'L4S low latency can *only* be provided to a subset' (my
emphasis), but the word 'only' is wrong.
That is the point. L4S low latency /can/ be provided to all traffic.

Yes, L4S low latency /is/ only provided to flows that are well-behaved
and ECT(1) marked, but all traffic /can/ behave smoothly without losing
out, and all traffic /can/ use ECT(1) without losing out.

The priority scheduler really is not at the core.
* Consider an L4S FQ scheduler like FQ-CoDel with the ce_threshold
parameter applied to ECT(1) traffic. There is no priority scheduling but
all traffic can still have very low latency.
* Similarly, an L4S DualQ Coupled AQM can work with any scheduler that
provides isolation, for instance the Time-Shifted FIFO. Scheduling
priority is not 'at the core', 'cos it's not even necessary. We only
ended up choosing a priority scheduler because it gave slightly better
tail latency.

The real core of L4S is scalable congestion control. That is the
difference that allows even capacity-seeking traffic to also maintain
consistently very low latency. It's easier to see in the L4S FQ example,
that it's what keeps a flow's latency down. Not the scheduler.

If TCP had been designed with a scalable congestion control from the
start, we would have had L4S with no need for a scheduler at all. The
scheduler is primarily a transition mechanism, to isolate from Classic
traffic. But no traffic /needs/ to be Classic, it's just how "stuff
happened".

I do not think this is true in this specific case. 
There is an advantage in terms of goodput for applications to use 
loss-based congestion control and create bigger queues (in
absence of isolation).

[BB] This may be true in the short term for non-latency sensitive bulk traffic, especially over variable capacity (e.g. radio) links where buffered packets can fill newly available capacity while the e2e congestion control catches up.

Nonetheless, there is also the unscalable aspect of loss-based congestion control where, as flow rate scales, it takes longer and longer (minutes) to recover after each loss [RFC3649]. As was pointed out in footnote 6 of [Jac88], the noise sensitivity gets worse as flow rate scales. Meaning, even if there's a very low level of transmission losses or new flows arriving, a flow keeps getting knocked down by each noise event, while it's slowly trying to regain its previous position.

It's possible that some new approach will get out of that unscalable rut, but it's been quite resistant to solutions so far. BBRv2 has already had to reinstate a capped dependence on loss probability, in order to remain backward compatible with the legacy of loss-based congestion controls. The setting of that cap will have to keep being reduced in order to scale flow rate, so it is trapped in the same noise sensitivity problem.

I don't know whether bulk flows will prefer to continue to use the classical approach with more buffering, or the scalable approach with frequent control signals. I'm just saying it's not as clear-cut as you make out.

[Jac88] Van Jacobson and Michael J. Karels. Congestion
Avoidance and Control. Technical report, Lau-
rence Berkeley Labs, November 1988. (a slightly
modified version of the original published at SIG-
COMM in Aug’88)


Unless these applications are policed one way or the other 
it is hard to create incentives to move towards protocols that
generate little queuing. Isolation is one such policy as the reward
is immediate.

[BB] When you have scalable congestion controls that fill capacity with consistently very low queueing, you can just use them (without policing) whether you need low latency, high throughput or both.

As a fairly close analogy, can you imagine any individual or company gaming a PIE AQM to induce more latency on their own flow, in order to impose more latency on others in the same household?

Policing hasn't even been necessary for 'TCP-friendliness' between application flows within a household, where there is bandwidth to gain. I don't know whether policing will be necessary for 'latency friendliness'. Maybe it will at first while there are people around who don't understand that there's nothing to gain.

My prediction, FWIW: I suspect that some policing within some paths (e.g. in DOCSIS links) will be enough at first to keep globally deployed code disciplined, whether or not it's actually running over a path with policing. I doubt policing will be necessary longer term, but we will have it if we need it. So nothing is hanging on who predicts this correctly.

Or are you saying that we should never allow anyone to even attempt to have low latency in shared queues in case it requires the complexity of per-flow policing?
But instead we should only ever allow per-flow queuing for low latency, which is somewhat more complex than per-flow policing anyway.
I hope you appreciate that avoiding the possible risk of a little complexity by requiring more complexity is broken logic.

There is also a major implication between priority or scheduling 
as an isolation mechanism: for the latter there is no need to 
allocate a bit in the network header to identify traffic types.

[BB] I don't understand what you have in mind here. Pls explain more fully.



Bob




 



Bob

>
> Regards
>       Sebastian
>
>
> *) https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19:
> "In this case it would not be appropriate to call the queue an L4S
>     queue, because it is shared by L4S and non-L4S traffic.  Instead it
>     will be called the low latency or L queue.  The L queue then offers
>     two different treatments:
>
>     o  The L4S treatment, which is a combination of the L4S AQM treatment
>        and a priority scheduling treatment;
>
>     o  The low latency treatment, which is solely the priority scheduling
>        treatment, without ECN-marking by the AQM."
>
>
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch-10:
> "The tension between prioritizing L4S and coupling the
>         marking from the Classic AQM results in approximate per-flow
>         fairness.  To protect against unresponsive traffic in the L4S
>         queue taking advantage of the prioritization and starving the
>         Classic queue, it is advisable not to use strict priority, but
>         instead to use a weighted scheduler (see Appendix A of [I-D.ietf-tsvwg-aqm-dualq-coupled])."
>
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled
> "Classic traffic needs to build a large queue to prevent under-
>     utilization.  Therefore a separate queue is provided for L4S traffic,
>     and it is scheduled with priority over the Classic queue.  Priority
>     is conditional to prevent starvation of Classic traffic."
>
> I ignore the yarn spun in those drafts, why this prioritization would only result in latency but not longer-time-scale throughput prioritization, because Pete's data show that for quite a number of realistic traffic patterns DualQ fails to arbitrate capacity equitably between the two queues.
>
>
>
>> I've made a note to myself to make it clear that this is not an intention that all applications will transition, rather it is an intention not to prevent any application from transitioning.
>>
>>
>> Bob
>>
>> On 02/09/2021 19:26, alex.burr@ealdwulf.org.uk wrote:
>>> Hi Gorry, tsvwg;
>>>
>>>
>>> See inline [AB]
>>>
>>>
>>>
>>> On Thursday, August 12, 2021, 11:23:49 AM GMT+1, Gorry Fairhurst<gorry@erg.abdn.ac.uk>  wrote:
>>>
>>> ---
>>> 1b) Issue: I’d also quibble with: “so that all of a user's applications
>>> can shift to it when their stack is updated.“ - again, is the word “all”
>>> really necessary or correct here?
>>> ---
>>>
>>> [AB]
>>> Not  answering your point, but related:
>>>
>>> L4s-arch has at the very start the following: “The L4S architecture is intended to enable   _all_ Internet applications to transition away from congestion control algorithms that cause queuing delay, to a new class of  congestion controls that induce very little queuing, aided by  explicit congestion signaling from the network. “
>>>
>>>
>>>   There are two quite different end states here:
>>>
>>> * An internet in which there are, end-to-end, two classes of traffic on an ongoing basis.
>>>
>>> * An internet in which nearly all traffic has transitioned to L4S, remaining non-L4S traffic being supported on a backwards-compatibility basis
>>>
>>> As I understand it, if L4S is deployed, either of these could be possible end states. At present it is not clear that all use cases could migrate (eg, long RTT traffic)
>>>
>>>
>>> It seems like it would be a good idea for the draft to make clear:
>>> * That either of these end states are possible
>>> * What the formal position of the WG would be if the drafts are accepted, IE what decision has been made (or left for later) regarding the desired end state.
>>>
>>>
>>> regards,
>>>
>>> Alex
>>>
>> --
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>>

--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------13350C00A36E81EA8A144C7C--