Re: [mpls] [spring] Special purpose labels in draft-hegde-spring-traffic-accounting-for-sr-paths
Martin Horneffer <maho@nic.dtag.de> Wed, 22 November 2017 16:34 UTC
Return-Path: <maho@nic.dtag.de>
X-Original-To: mpls@ietfa.amsl.com
Delivered-To: mpls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 75A82129463; Wed, 22 Nov 2017 08:34:08 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Level:
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4PBginKrkdbm; Wed, 22 Nov 2017 08:34:04 -0800 (PST)
Received: from owl2.lab.dtag.de (Owl2.lab.DTAG.DE [194.25.1.238]) by ietfa.amsl.com (Postfix) with ESMTP id 87F7A126C0F; Wed, 22 Nov 2017 08:34:03 -0800 (PST)
Received: from localhost (localhost [127.0.0.1]) by owl2.lab.dtag.de (Postfix) with ESMTP id 03FB2C840E; Wed, 22 Nov 2017 17:34:02 +0100 (CET)
Received: from owl2.lab.dtag.de ([127.0.0.1]) by localhost (owl2.lab.dtag.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RGA4HN9zvmht; Wed, 22 Nov 2017 17:33:50 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by owl2.lab.dtag.de (Postfix) with ESMTP id 6AD29C850B; Wed, 22 Nov 2017 17:33:50 +0100 (CET)
Received: from TSUNAMI-Hippogryff-79.lab.DTAG.de (o-Hippogryff.lab.dtag.de [62.153.176.86]) by owl2.lab.dtag.de (Postfix) with ESMTPSA id 4E1A6C840E; Wed, 22 Nov 2017 17:33:50 +0100 (CET)
To: Robert Raszuk <robert@raszuk.net>, Adrian Farrel <adrian@olddog.co.uk>
Cc: mpls <mpls@ietf.org>, spring <spring@ietf.org>, "Zafar Ali (zali)" <zali@cisco.com>
References: <CA+RyBmUHAkuA3o-LpHhMwCbkh0k+emt9OZ3B8Njj2h=jaasTZw@mail.gmail.com> <3B1EE673-044F-4E47-9C56-6FF360905C58@cisco.com> <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE3047CEC9@NKGEML515-MBS.china.huawei.com> <CA+RyBmVC2OjEs-=1WsL13eBmycZtnYnM8ybSdmWhGPByLKNQfA@mail.gmail.com> <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE3047D106@NKGEML515-MBS.china.huawei.com> <EF064624-CF4D-4B88-823E-DAB9957B9336@cisco.com> <MWHPR05MB35512AD68B9CE96E8A5E7255C72E0@MWHPR05MB3551.namprd05.prod.outlook.com> <A9BFDECC-84A4-42E6-83CD-D09A2D48BA75@cisco.com> <189901d36076$aa76b4b0$ff641e10$@olddog.co.uk> <0EAD8CC9-8C65-4A78-896B-D96F42230020@cisco.com> <01de01d362f2$14766ef0$3d634cd0$@olddog.co.uk> <CA+b+ERmkHZe9BU-JZLWpOEcwAhPqx72Sw4-NyLgVjhzzfySdCQ@mail.gmail.com>
From: Martin Horneffer <maho@nic.dtag.de>
Message-ID: <a928b0a7-726c-a885-ba3d-3aedb255ca64@nic.dtag.de>
Date: Wed, 22 Nov 2017 17:33:49 +0100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <CA+b+ERmkHZe9BU-JZLWpOEcwAhPqx72Sw4-NyLgVjhzzfySdCQ@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------607330DFAAC0C711ECF2F008"
Archived-At: <https://mailarchive.ietf.org/arch/msg/mpls/UT-KObvaaa1Odaq8zD7YRCqAI1k>
Subject: Re: [mpls] [spring] Special purpose labels in draft-hegde-spring-traffic-accounting-for-sr-paths
X-BeenThere: mpls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Multi-Protocol Label Switching WG <mpls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/mpls>, <mailto:mpls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/mpls/>
List-Post: <mailto:mpls@ietf.org>
List-Help: <mailto:mpls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/mpls>, <mailto:mpls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 22 Nov 2017 16:34:08 -0000
+1 In other words, I confirm from an operators point of view that Robert got good network desing goals quite right. I also perfectly agree with the observations concerning the use of RSVP in the past. BR, Martin Am 21.11.17 um 19:34 schrieb Robert Raszuk: > Hi Adrian, > > I am not going to defend beauty of any architecture. I think there is > much bigger fundamental misunderstanding how in practice someone will > use SR-MPLS and I think this is the root cause for this little thread > and different perspectives of its participants. > > So SR-MPLS is not RSVP-TE and there is no EROs. The set of SIDs are no > more then hints on how to steer the packets within connection less > paradigm (same as IP tunnel so to say with less encap overhead) via > one or more of IGP segments. > * > * > *The less SIDs you add to the packet the better !* > > Yes that requires to be smart (or to have smart central controller) to > add only a very few labels/SIDs to accomplish the network traffic > distribution objectives. I clearly see folks thinking of SR-MPLS like > a RSVP-TE analogy with EROs, but this is IMO fundamentally wrong. Only > that you can do it (to build SR-MPLS paths all the way via your > domain) does not make it a good idea. No where in SR architecture I > see any pre-asssumption that last IGP segment will be connected to > domain egress node (with the exception of EPE but this is different app). > > In fact as some may recall we are 17 years after RSVP-TE shipping code > and only very few networks ever deployed it for all unicast traffic > end to end for many reasons. Most folks used it for FRR or for hot > spot bypass. > > Now as far as OAM sure it is great to have it both for IP networks and > MPLS-LDP networks and SR-MPLS networks. Especially iOAM is very > useful. But this is not really related to SR-MPLS architecture. > > With that I think the draft makes set of assumptions which are far > from how SR-MPLS should be deployed and this does make it rather > problematic. It is just like draft describing use of BGP for data > centers ... now everyone is using BGP for all data centers or even > other types of networks regardless if this is even applicable or best > choice for a given cluster scale they are building :). > > Counters are great, more counters are even better, but I fail to see > the value for yet again counting traffic arriving via specific IGP > segments when we are already counting packets arriving via given IGP > topology. My recommendation would be to solve it for MPLS-LDP in MPLS > WG first (which after all is one example where flooding domain wide > labels in IGP replaces) and then SR-MPLS will inherit the same solution. > > Cheers, > Robert. > > > On Tue, Nov 21, 2017 at 6:56 PM, Adrian Farrel <adrian@olddog.co.uk > <mailto:adrian@olddog.co.uk>> wrote: > > Hi, > > I understand that you doubt that this thread will yield anything > productive, but there are a couple of things you're raising that > need to be nailed down. > > Probably the most important of these is the concern that you > express that maintaining counters in the network goes against the > beauty of the SR architecture because it means holding state at > transit nodes. This seems to be a debate about the perfection of > an architecture versus the manageability of the network. Don't get > me wrong, I love a beautiful architecture, but only if the network > can be operated successfully. > > So, we should start at the top of the document and work our way > down. I assume that you don't have any issues with Section 1: it > seems to say what you are saying about the statelessness of SR. > Section 2 is probably where you start to be unhappy: it sets an > objective (to be able to count packets per flow) and sets some > requirements on any solution. > > That is, I think you believe that it is not necessary (or not > desirable?) to count packets in an SR network and assign those > counts to the SR paths that generated those packet flows. So the > challenge for you is to say whether the problem described in > Figure 1 is: > - not a concern in network management > - can be solved by other means without counting traffic at > transit nodes (Note Well that other ways of counting > traffic at transit nodes are still counting traffic at transit > nodes). > > But one other point I want to pick up on is your claim that "the > draft also talks about needs to break an SR Path into sub-paths". > Sub-paths that are achieved through an expansion of a Binding SID > are just part of the landscape and (of course) thy have to be > coped with. The draft doesn't introduce sub-paths, it just > observes that they exist. > > Lastly, the conversation on the number of labels as a multiplier > seems to have gotten out of hand. Why not just agree that you > original statement of "increased by up to 3x" was an exaggeration? > > Cheers, > Adrian > > > -----Original Message----- > > From: Zafar Ali (zali) [mailto:zali@cisco.com > <mailto:zali@cisco.com>] > > Sent: 20 November 2017 23:36 > > To: adrian@olddog.co.uk <mailto:adrian@olddog.co.uk> > > Cc: 'spring'; 'mpls' > > Subject: Re: [mpls] [spring] Special purpose labels in > draft-hegde-spring-traffic- > > accounting-for-sr-paths > > > > Hi Adrian, > > > > Some comments are provided in-line. > > > > Please note that, we all want to let this lingering tread die > and follow-up on the > > next steps noted during this email exchange. I will be happy to > have a webEx call > > and discuss it further, offline. > > > > Thanks > > > > Regards … Zafar > > > > On 11/18/17, 9:08 AM, "Adrian Farrel" <adrian@olddog.co.uk > <mailto:adrian@olddog.co.uk>> wrote: > > > > <snip> > > > > >>> procedure (in > draft-hegde-spring-traffic-accounting-for-sr-paths) that > > breaks SR > > >>> Architecture, highly unscalable and complicated to > implement. > > >> > > >> [JD] Do you have any evidence to justify any of your > assertions, above? > > > > > > Please note that in > draft-hegde-spring-traffic-accounting-for-sr-paths: > > > > > > • The transit node needs to be able to recognize the > special label, read > > > the SR Path Identification label and update the > counter against such > > > “states”. > > > > > Possibly worth noting that existing devices are capable of > maintaining many > > counters and updating them at line speed. > > > > > Several people have noted that ipfix is a process used for > accounting in > > networks. That approach may have to find the bottom of stack and > then match > > the packet that follows. > > > > > Other approaches (e.g., to ECMP) involve finding the bottom > of stack and > > hashing on the header of the payload. > > > > > Some hardware cannot perform either mechanism. This usually > results from a > > trade between low cost, high performance, and features. > Generally you can't > > have all three. > > > > The question is not about if the hardware is able to perform > such operations but > > regarding breaking the very beauty of SR – no states at the > transit/ egress nodes. > > In the context of label stack size explosion, the draft also > talks about needs to > > break an SR Path into sub-paths – thereby creating yet > additional states in the > > network for accounting reasons (see more detail on this in the > following). > > Furthermore, SR-MPLS is designed for SDN – the architecture > calls for > > simplification of the network not adding complexity in the > network fabric. Please > > also note that a network may have a large number of SR Path, > thereby creating > > another dimension for scaling limitations. > > > > The proposed procedure also does not work for node protection in > the network. > > The draft essentially calls for ALL nodes to implement procedure > proposed in the > > document; I am quoting from the draft. > > > > “When using extensions > > described in this document for traffic accounting and with node- > > protection enabled in the network, it is RECOMMENDED to make > sure all > > the nodes in the network support the extension.” > > > > <snip> > > > > > • The draft proposes to push (up to) 3 Labels for each > segment in the SR > > > Path. That means that label stack is increased up > to 3x times! This is a > > > serious a scaling issue. > > > > > John asked for evidence and you provided a misunderstanding > or misreading > > of our draft. > > > The document proposes adding 2 or 3 labels per SR Path > (noting as John did, > > that this is our own term). > > > That is not what you say, so perhaps you could retract or > provide a pointer to > > the text. > > > > > Thus, "increased up to 3x times" applies only with the > single case where the > > imposed label stack has exactly one label *and* the three label > option is applied. > > So, while what you say is true, it is clearly (and wilfully?) > exaggerating the > > severity of impact, and it is doubtful that 4-label stack is > actually a problem. > > > > There are many scenarios that will require SR-Path-Stats Labels > (up to 3 labels) to > > be present multiple times in the label stack. These scenarios > are not uncommon. > > The following scenarios as noted in the draft. > x`x> > > “The head-end node SHOULD insert the SR- > > Path-Stats Labels at a depth in the label stack such that the > nodes > > in the SR path can access the SR-Path-Identifier for > accounting. The > > SR-Path-Stats Labels may be present multiple times in the > label stack > > of a packet.” > > > > “It is possible to partially deploy this feature when not all the > > nodes in the network support the extensions defined in this > document. > > In such scenarios, the special labels MUST NOT get exposed on > the top > > of the label stack at a node that does not support the extensions > > defined in this document. This may require multiple blocks > of SR- > > Path-Stats Labels to be inserted in the packet header.” > > > > > • The controller needs to keep track of transit node > capability and > > > push the additional per-path labels, accordingly. > I.e., the controller > > > also needs to maintain such information for the > transit nodes. > > > > > In most cases, the controller/ingress only needs to care > about the capabilities > > of the egress nodes. That is, if the special purpose label > reaches the top of the > > stack it has to be able to handle it. > > > > > The only time when the transit node issue arises is when > there is a small RLD. > > That information may need to be known by the controller to > enable correct ECMP > > behavior, and it is distributed in the IGP. > > > If there is a desire to enable accounting at transit nodes > with a small RLD then > > the Path ID can be inserted higher up the stack and *that* means > that the > > controller has to be sensitive as to where in the network the > special purpose > > label will rise to the top of the stack. > > > > > It seems to me that: > > > - Controllers are not particularly resource constrained: > adding a flag per node > > > (or even per link!) would not break any scaling behavior. > > > - Adding another flag to the IGP alongside the RLD is not > significant scaling > > issue. > > > > The comment here was not so much related to scaling but was for > adding > > complexity to the controller/ ingress node. As you noted above > and in the draft, > > controller/ Ingress node needs to worry about the following > cases every time a > > path needs to be computed (quoting some of the cases from the > draft). > > > > “When the head-end node > > inserts the SR-Path-Stats labels in the label stack, the > place in the > > stack is decided based on whether the node where the special > label > > gets exposed is capable of popping those labels.” > > > > > > “While inserting the SR-Path-Stats labels, the head-end router MUST > > ensure that the labels are not exposed to the nodes that do not > > support them. “ > > > > “Because it is necessary that the SR-Path-Stats labels are removed > > when they are found at the top of the label stack, the node > imposing > > the label stack (the ingress) must know which nodes are > capable of > > stripping the labels.” > > > > In RLDC limitation cases, “To support traffic > > accounting in such cases it is necessary to insert the > SR-Path-Stats > > Labels within the Readable Label Stack Depth Capability > (RLDC) of the > > nodes in the SR path.” > > > > “The head-end node SHOULD insert the SR- > > Path-Stats Labels at a depth in the label stack such that the > nodes > > in the SR path can access the SR-Path-Identifier for accounting.” > > > > “The special labels MUST NOT get exposed on the top > > of the label stack at a node that does not support the extensions > > defined in this document.” > > > > “If the egress has not indicated that it is capable of removing the > > SR-Path-Stats Labels, then they MUST NOT be placed at the > bottom of > > the label stack. In this case the SR-Path-Stats Labels SHOULD be > > placed at a point in the label stack such that they will be > found at > > the top of stack by the latest node in the SR path that is > capable of > > removing them. “ > > > > “SR paths may require large label stacks. Some hardware > platforms do > > not support creating such large label stacks (i.e., imposing > a large > > number of labels at once). To overcome this limitation > sub-paths are > > created within the network, and Binding-SIDs are allocated to > these > > sub-paths.” … which means controller/ ingress software need > to also create/ > > install sub-paths. > > > > <snip> > > > > > > > _______________________________________________ > spring mailing list > spring@ietf.org <mailto:spring@ietf.org> > https://www.ietf.org/mailman/listinfo/spring > <https://www.ietf.org/mailman/listinfo/spring> > > > > > _______________________________________________ > spring mailing list > spring@ietf.org > https://www.ietf.org/mailman/listinfo/spring
- [mpls] Special purpose labels in draft-hegde-spri… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… ShaoWen Ma
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Shraddha Hegde
- Re: [mpls] [spring] Special purpose labels in dra… David Allan I
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Shah, Himanshu
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Chengli (IP Technology Research)
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] [spring] Special purpose labels in dra… Ruediger.Geib
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- [mpls] Whether both E2E and SPME performance meas… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] Whether both E2E and SPME performance … David Allan I
- Re: [mpls] Whether both E2E and SPME performance … Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] Whether both E2E and SPME performance … Ruediger.Geib
- Re: [mpls] [spring] Special purpose labels in dra… stephane.litkowski
- Re: [mpls] Whether both E2E and SPME performance … Robert Raszuk
- Re: [mpls] Whether both E2E and SPME performance … Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] Whether both E2E and SPME performance … John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] Whether both E2E and SPME performance … Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] [spring] Whether both E2E and SPME per… Alexander Vainshtein
- Re: [mpls] Whether both E2E and SPME performance … John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] Whether both E2E and SPME performance … John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Whether both E2E and SPME per… John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… Robert Raszuk
- Re: [mpls] [spring] Whether both E2E and SPME per… Alexander Vainshtein
- Re: [mpls] [spring] Whether both E2E and SPME per… Greg Mirsky
- Re: [mpls] [spring] Whether both E2E and SPME per… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Whether both E2E and SPME per… Greg Mirsky
- [mpls] Fwd: Re: [spring] Special purpose labels i… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] Fwd: Re: [spring] Special purpose labe… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] Fwd: Re: [spring] Special purpose labe… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] Fwd: Re: [spring] Special purpose labe… Uma Chunduri
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- [mpls] measurement requirements (was Re: [spring]… Martin Horneffer
- Re: [mpls] [spring] Special purpose labels in dra… Martin Horneffer
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] measurement requirements (was… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Adrian Farrel
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Adrian Farrel
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Martin Horneffer
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)