Re: [mpls] [spring] Special purpose labels in draft-hegde-spring-traffic-accounting-for-sr-paths
Greg Mirsky <gregimirsky@gmail.com> Tue, 21 November 2017 03:53 UTC
Return-Path: <gregimirsky@gmail.com>
X-Original-To: mpls@ietfa.amsl.com
Delivered-To: mpls@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3CB5512EB1F; Mon, 20 Nov 2017 19:53:27 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.699
X-Spam-Level:
X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vYPyTqZ-VnU5; Mon, 20 Nov 2017 19:53:24 -0800 (PST)
Received: from mail-lf0-x22a.google.com (mail-lf0-x22a.google.com [IPv6:2a00:1450:4010:c07::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6146012EB1A; Mon, 20 Nov 2017 19:53:23 -0800 (PST)
Received: by mail-lf0-x22a.google.com with SMTP id f134so12462332lfg.8; Mon, 20 Nov 2017 19:53:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=N1LrDwcr9IUgwesn7ZT7n+yI+gk50bjY+fdSMpoLvag=; b=p15Ka+h1TfW9sqMN4v0rWSwIfvCJGPqH43PaRJ8CMmBqrLCi9xDWaWJIB7v2nA6iQ/ tNgdOiiVDFomkyBjy31tX2S0VgGlOHaRaApGD1czI4F4uw1dbTrQ2yeqOX9pkEPIHNq9 GsgURkDO6iSVGYXMnLhBI6Wtf6dOc8bUxCRzxl8e38SyXzyVchgDlLIJTsN9drl+CrqK wAltNqtaDiuCiszOoZYuYXotb5WoSHns9r/FT2sthV9tTlSqaGBTLPtnVYSFWrwikVh0 vHSyWRnzUuorj3NWxgMdU5zfM4ZzFo1W+BN68kgV76xq8Ss+r9WREkDVGSMFjkCU4fXx gKgw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=N1LrDwcr9IUgwesn7ZT7n+yI+gk50bjY+fdSMpoLvag=; b=dkjkfQ1qHnEsYmkMIGWvuco3XoPd5RPzvI97yw4JIUF8tjyHHwnD/6x2NC42nhxBWh Y+NZOLOGnNnxtRDe0NHTlDEMNojptQbm6Qn7fx2yPjNYKOM9Ho9MBCCyTPgkkIiTnmT9 cr5X3kr990blECUnxgKaPvU8/eEqlQpaHpAEMVc05W2fGx2urdHBRpGOI6DYf/4rpt22 73WHGQwZ50kpZj65meY61X9mYMP75uL/cnkmlrnZaxMMmAJTGNvYiV8+TAw8M7fn7sxE Ani4ob9L4XpKHKa9/PUBWOv2pdwSZsVDiqBpfBrOzovCfO6h0CiEyQ9MSfC1XA60ZJfL So6Q==
X-Gm-Message-State: AJaThX6GH8tmikXzQo9mQQqOFIOY8750Pwf6cGtd2eoh3j/+9TgVfBNz NGsN4nnwzTD/3eUUUgMRgfSlq/v0JseX02YUsGQ=
X-Google-Smtp-Source: AGs4zMakm4L9MyhlFDAPectWAuYW5yPKUX9Vf/Jd7yG5yrm1wHKekcOKTC0QYMCZgQK3nLTHV16wddKTb+Z3leq4ElA=
X-Received: by 10.25.109.6 with SMTP id i6mr4543297lfc.73.1511236401480; Mon, 20 Nov 2017 19:53:21 -0800 (PST)
MIME-Version: 1.0
Received: by 10.46.32.136 with HTTP; Mon, 20 Nov 2017 19:53:20 -0800 (PST)
In-Reply-To: <F0C75C66-6CAB-4FAF-B0D1-4F9A3E8CE1AB@cisco.com>
References: <CA+RyBmUHAkuA3o-LpHhMwCbkh0k+emt9OZ3B8Njj2h=jaasTZw@mail.gmail.com> <3B1EE673-044F-4E47-9C56-6FF360905C58@cisco.com> <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE3047CEC9@NKGEML515-MBS.china.huawei.com> <CA+RyBmVC2OjEs-=1WsL13eBmycZtnYnM8ybSdmWhGPByLKNQfA@mail.gmail.com> <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE3047D106@NKGEML515-MBS.china.huawei.com> <EF064624-CF4D-4B88-823E-DAB9957B9336@cisco.com> <MWHPR05MB35512AD68B9CE96E8A5E7255C72E0@MWHPR05MB3551.namprd05.prod.outlook.com> <A9BFDECC-84A4-42E6-83CD-D09A2D48BA75@cisco.com> <189901d36076$aa76b4b0$ff641e10$@olddog.co.uk> <0EAD8CC9-8C65-4A78-896B-D96F42230020@cisco.com> <CA+RyBmXJev9x0MRh+XA9RrsBKizJPpCEMVPTEZc6aVVxAYJOPQ@mail.gmail.com> <F0C75C66-6CAB-4FAF-B0D1-4F9A3E8CE1AB@cisco.com>
From: Greg Mirsky <gregimirsky@gmail.com>
Date: Mon, 20 Nov 2017 19:53:20 -0800
Message-ID: <CA+RyBmWw4wuiJAs57+KaSLWTAkH4DVG+3RHPkq9aEVLAfjVGQA@mail.gmail.com>
To: "Zafar Ali (zali)" <zali@cisco.com>
Cc: "adrian@olddog.co.uk" <adrian@olddog.co.uk>, mpls <mpls@ietf.org>, spring <spring@ietf.org>
Content-Type: multipart/alternative; boundary="089e082ef6c4e1c90b055e76246d"
Archived-At: <https://mailarchive.ietf.org/arch/msg/mpls/Jv2zj4FdcfMiqDnmLZi2iwzXWwU>
Subject: Re: [mpls] [spring] Special purpose labels in draft-hegde-spring-traffic-accounting-for-sr-paths
X-BeenThere: mpls@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Multi-Protocol Label Switching WG <mpls.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/mpls>, <mailto:mpls-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/mpls/>
List-Post: <mailto:mpls@ietf.org>
List-Help: <mailto:mpls-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/mpls>, <mailto:mpls-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 21 Nov 2017 03:53:27 -0000
Hi Zafar, "overloading existing special label"? AFAIK, MPLS WG created Extended Special Label exactly for cases like we're discussing. Of course, alternative proposal for SR-MPLS use case will be most helpful. As description of GAL-ACH based proposal. Stay tuned. Regards, Greg On Mon, Nov 20, 2017 at 7:06 PM, Zafar Ali (zali) <zali@cisco.com> wrote: > Hi Greg, > > > > Overloading existing special label, optimizing stack overheads in > draft-hegde, etc. will not change the architectural direction of the > proposal. > > > > I feel additional discussion on this thread will be non-productive. Martin > Horneffer, Robert Raszuk, Stephane Litkowski and others have shared from > their operation experiences; their input cannot be ignored by the vendors. > They and others have outlined some alternatives. We need to follow-up on > additional documentation (on the alternatives including existing counters) > to help us converge. Please stay tuned. > > > > Thanks > > > > Regards … Zafar > > > > > > *From: *Greg Mirsky <gregimirsky@gmail.com> > *Date: *Monday, November 20, 2017 at 7:03 PM > *To: *"Zafar Ali (zali)" <zali@cisco.com> > *Cc: *"adrian@olddog.co.uk" <adrian@olddog.co.uk>, "mpls@ietf.org" < > mpls@ietf.org>, spring <spring@ietf.org> > *Subject: *Re: [mpls] [spring] Special purpose labels in > draft-hegde-spring-traffic-accounting-for-sr-paths > > > > Hi Zafar, > > I don't see how managing, using passive OAM, i.e. counting fly-by packets, > at transient SR-MPLS nodes can be equated to breaking "no-forwarding state > at transient LSR" model.I believe that practical is as important as > aesthetic and that network cannot be operated without comprehensive OAM > tool box. Of course, every tool in the box is optional and will not be > deployed unless the need arises. If we agree on that, perhaps we can > discuss new requirement to be added to the SR OAM Requirements document. > And then, then consider how to address such requirement. I believe that two > options were mentioned in the thread: > > - described in draft-hegde-spring-traffic-accounting-for-sr-paths > special purpose label (actually it may end up be extended special purpose > label) and up to two labels; > - use GAL and ACH (perhaps somewhat similar to approach used in RFC > 8169). > > Regards, > > Greg > > > > > > On Mon, Nov 20, 2017 at 3:36 PM, Zafar Ali (zali) <zali@cisco.com> wrote: > > Hi Adrian, > > Some comments are provided in-line. > > Please note that, we all want to let this lingering tread die and > follow-up on the next steps noted during this email exchange. I will be > happy to have a webEx call and discuss it further, offline. > > Thanks > > Regards … Zafar > > On 11/18/17, 9:08 AM, "Adrian Farrel" <adrian@olddog.co.uk> wrote: > > <snip> > > >>> procedure (in draft-hegde-spring-traffic-accounting-for-sr-paths) > that breaks SR > >>> Architecture, highly unscalable and complicated to implement. > >> > >> [JD] Do you have any evidence to justify any of your assertions, > above? > > > > Please note that in draft-hegde-spring-traffic- > accounting-for-sr-paths: > > > > • The transit node needs to be able to recognize the special > label, read > > the SR Path Identification label and update the counter > against such > > “states”. > > > Possibly worth noting that existing devices are capable of > maintaining many counters and updating them at line speed. > > > Several people have noted that ipfix is a process used for accounting > in networks. That approach may have to find the bottom of stack and then > match the packet that follows. > > > Other approaches (e.g., to ECMP) involve finding the bottom of stack > and hashing on the header of the payload. > > > Some hardware cannot perform either mechanism. This usually results > from a trade between low cost, high performance, and features. Generally > you can't have all three. > > The question is not about if the hardware is able to perform such > operations but regarding breaking the very beauty of SR – no states at the > transit/ egress nodes. In the context of label stack size explosion, the > draft also talks about needs to break an SR Path into sub-paths – thereby > creating yet additional states in the network for accounting reasons (see > more detail on this in the following). Furthermore, SR-MPLS is designed for > SDN – the architecture calls for simplification of the network not adding > complexity in the network fabric. Please also note that a network may have > a large number of SR Path, thereby creating another dimension for scaling > limitations. > > The proposed procedure also does not work for node protection in the > network. The draft essentially calls for ALL nodes to implement procedure > proposed in the document; I am quoting from the draft. > > “When using extensions > described in this document for traffic accounting and with node- > protection enabled in the network, it is RECOMMENDED to make sure all > the nodes in the network support the extension.” > > <snip> > > > • The draft proposes to push (up to) 3 Labels for each segment in > the SR > > Path. That means that label stack is increased up to 3x > times! This is a > > serious a scaling issue. > > > John asked for evidence and you provided a misunderstanding or > misreading of our draft. > > The document proposes adding 2 or 3 labels per SR Path (noting as > John did, that this is our own term). > > That is not what you say, so perhaps you could retract or provide a > pointer to the text. > > > Thus, "increased up to 3x times" applies only with the single case > where the imposed label stack has exactly one label *and* the three label > option is applied. So, while what you say is true, it is clearly (and > wilfully?) exaggerating the severity of impact, and it is doubtful that > 4-label stack is actually a problem. > > There are many scenarios that will require SR-Path-Stats Labels (up to 3 > labels) to be present multiple times in the label stack. These scenarios > are not uncommon. The following scenarios as noted in the draft. > > “The head-end node SHOULD insert the SR- > Path-Stats Labels at a depth in the label stack such that the nodes > in the SR path can access the SR-Path-Identifier for accounting. The > SR-Path-Stats Labels may be present multiple times in the label stack > of a packet.” > > “It is possible to partially deploy this feature when not all the > nodes in the network support the extensions defined in this document. > In such scenarios, the special labels MUST NOT get exposed on the top > of the label stack at a node that does not support the extensions > defined in this document. This may require multiple blocks of SR- > Path-Stats Labels to be inserted in the packet header.” > > > • The controller needs to keep track of transit node capability and > > push the additional per-path labels, accordingly. I.e., the > controller > > also needs to maintain such information for the transit nodes. > > > In most cases, the controller/ingress only needs to care about the > capabilities of the egress nodes. That is, if the special purpose label > reaches the top of the stack it has to be able to handle it. > > > The only time when the transit node issue arises is when there is a > small RLD. That information may need to be known by the controller to > enable correct ECMP behavior, and it is distributed in the IGP. > > If there is a desire to enable accounting at transit nodes with a > small RLD then the Path ID can be inserted higher up the stack and *that* > means that the controller has to be sensitive as to where in the network > the special purpose label will rise to the top of the stack. > > > It seems to me that: > > - Controllers are not particularly resource constrained: adding a > flag per node > > (or even per link!) would not break any scaling behavior. > > - Adding another flag to the IGP alongside the RLD is not significant > scaling issue. > > The comment here was not so much related to scaling but was for adding > complexity to the controller/ ingress node. As you noted above and in the > draft, controller/ Ingress node needs to worry about the following cases > every time a path needs to be computed (quoting some of the cases from the > draft). > > “When the head-end node > inserts the SR-Path-Stats labels in the label stack, the place in the > stack is decided based on whether the node where the special label > gets exposed is capable of popping those labels.” > > > “While inserting the SR-Path-Stats labels, the head-end router MUST > ensure that the labels are not exposed to the nodes that do not > support them. “ > > “Because it is necessary that the SR-Path-Stats labels are removed > when they are found at the top of the label stack, the node imposing > the label stack (the ingress) must know which nodes are capable of > stripping the labels.” > > In RLDC limitation cases, “To support traffic > accounting in such cases it is necessary to insert the SR-Path-Stats > Labels within the Readable Label Stack Depth Capability (RLDC) of the > nodes in the SR path.” > > “The head-end node SHOULD insert the SR- > Path-Stats Labels at a depth in the label stack such that the nodes > in the SR path can access the SR-Path-Identifier for accounting.” > > “The special labels MUST NOT get exposed on the top > of the label stack at a node that does not support the extensions > defined in this document.” > > “If the egress has not indicated that it is capable of removing the > SR-Path-Stats Labels, then they MUST NOT be placed at the bottom of > the label stack. In this case the SR-Path-Stats Labels SHOULD be > placed at a point in the label stack such that they will be found at > the top of stack by the latest node in the SR path that is capable of > removing them. “ > > “SR paths may require large label stacks. Some hardware platforms do > not support creating such large label stacks (i.e., imposing a large > number of labels at once). To overcome this limitation sub-paths are > created within the network, and Binding-SIDs are allocated to these > sub-paths.” … which means controller/ ingress software need to also > create/ install sub-paths. > > <snip> > > > > > _______________________________________________ > mpls mailing list > mpls@ietf.org > https://www.ietf.org/mailman/listinfo/mpls > > >
- [mpls] Special purpose labels in draft-hegde-spri… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… ShaoWen Ma
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Shraddha Hegde
- Re: [mpls] [spring] Special purpose labels in dra… David Allan I
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Shah, Himanshu
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Chengli (IP Technology Research)
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Xuxiaohu
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] [spring] Special purpose labels in dra… Ruediger.Geib
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- [mpls] Whether both E2E and SPME performance meas… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] Whether both E2E and SPME performance … David Allan I
- Re: [mpls] Whether both E2E and SPME performance … Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] Whether both E2E and SPME performance … Ruediger.Geib
- Re: [mpls] [spring] Special purpose labels in dra… stephane.litkowski
- Re: [mpls] Whether both E2E and SPME performance … Robert Raszuk
- Re: [mpls] Whether both E2E and SPME performance … Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] Whether both E2E and SPME performance … John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] Whether both E2E and SPME performance … Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Alexander Vainshtein
- Re: [mpls] [spring] Whether both E2E and SPME per… Alexander Vainshtein
- Re: [mpls] Whether both E2E and SPME performance … John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] Whether both E2E and SPME performance … John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] [spring] Whether both E2E and SPME per… John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… John E Drake
- Re: [mpls] [spring] Whether both E2E and SPME per… Robert Raszuk
- Re: [mpls] [spring] Whether both E2E and SPME per… Alexander Vainshtein
- Re: [mpls] [spring] Whether both E2E and SPME per… Greg Mirsky
- Re: [mpls] [spring] Whether both E2E and SPME per… Mach Chen
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Whether both E2E and SPME per… Greg Mirsky
- [mpls] Fwd: Re: [spring] Special purpose labels i… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] Fwd: Re: [spring] Special purpose labe… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Mach Chen
- Re: [mpls] Fwd: Re: [spring] Special purpose labe… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] Fwd: Re: [spring] Special purpose labe… Uma Chunduri
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- [mpls] measurement requirements (was Re: [spring]… Martin Horneffer
- Re: [mpls] [spring] Special purpose labels in dra… Martin Horneffer
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] measurement requirements (was… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Adrian Farrel
- Re: [mpls] [spring] Special purpose labels in dra… John E Drake
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Adrian Farrel
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Zafar Ali (zali)
- Re: [mpls] [spring] Special purpose labels in dra… Stewart Bryant
- Re: [mpls] [spring] Special purpose labels in dra… Martin Horneffer
- Re: [mpls] [spring] Special purpose labels in dra… Jeff Tantsura
- Re: [mpls] [spring] Special purpose labels in dra… Robert Raszuk
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)
- Re: [mpls] [spring] Special purpose labels in dra… Greg Mirsky
- Re: [mpls] [spring] Special purpose labels in dra… Carlos Pignataro (cpignata)