Re: draft-freed-sieve-in-xml status?
"Robert Burrell Donkin" <robertburrelldonkin@gmail.com> Fri, 19 December 2008 19:02 UTC
Return-Path: <owner-ietf-mta-filters@mail.imc.org>
X-Original-To: ietfarch-sieve-archive-Aet6aiqu@core3.amsl.com
Delivered-To: ietfarch-sieve-archive-Aet6aiqu@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5D6F33A69CC for <ietfarch-sieve-archive-Aet6aiqu@core3.amsl.com>; Fri, 19 Dec 2008 11:02:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.544
X-Spam-Level:
X-Spam-Status: No, score=-2.544 tagged_above=-999 required=5 tests=[AWL=0.055, BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x3IuqqR6BmU3 for <ietfarch-sieve-archive-Aet6aiqu@core3.amsl.com>; Fri, 19 Dec 2008 11:02:42 -0800 (PST)
Received: from balder-227.proper.com (properopus-pt.tunnel.tserv3.fmt2.ipv6.he.net [IPv6:2001:470:1f04:392::2]) by core3.amsl.com (Postfix) with ESMTP id 683093A6908 for <sieve-archive-Aet6aiqu@ietf.org>; Fri, 19 Dec 2008 11:02:41 -0800 (PST)
Received: from balder-227.proper.com (localhost [127.0.0.1]) by balder-227.proper.com (8.14.2/8.14.2) with ESMTP id mBJIsdpS054375 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 19 Dec 2008 11:54:39 -0700 (MST) (envelope-from owner-ietf-mta-filters@mail.imc.org)
Received: (from majordom@localhost) by balder-227.proper.com (8.14.2/8.13.5/Submit) id mBJIsdqF054374; Fri, 19 Dec 2008 11:54:39 -0700 (MST) (envelope-from owner-ietf-mta-filters@mail.imc.org)
X-Authentication-Warning: balder-227.proper.com: majordom set sender to owner-ietf-mta-filters@mail.imc.org using -f
Received: from mail-fx0-f20.google.com (mail-fx0-f20.google.com [209.85.220.20]) by balder-227.proper.com (8.14.2/8.14.2) with ESMTP id mBJIsQu6054352 for <ietf-mta-filters@imc.org>; Fri, 19 Dec 2008 11:54:37 -0700 (MST) (envelope-from robertburrelldonkin@gmail.com)
Received: by fxm13 with SMTP id 13so233224fxm.10 for <ietf-mta-filters@imc.org>; Fri, 19 Dec 2008 10:54:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=Xde2Zghi06Kbc7HuIEwQ9kDlLUs8AuBZx5gGXdw/AmY=; b=hvLu1m4wDfJn3C8gx/AoBVtDQyjBW7ky5GiCThkb65OFBo9918VdT6aIOYtVbUd+4X 1oILODFLZJpmaftDDVjScsiotMXd/R7QIpV639v9KSi7HbIVSSmBEW9IYJn3AY+A718Y XZg4fxAI+UwpwXFZzB2N/OICKi/25eUIz3Lbo=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=JYhST3rp5q/SqMW+22DYkCuTt6BsnZYbAOsE4dMYYrkNq7HNpZiHJQKO8A0az8GS3Z s2urqsFa2yKe9wkCY4LjNDkOxsqPrW9RUPkXy9Cj5JJBcjbrebd1UJIYA8a2Kw68WnMk t91OEvQsx4Fp5NPg6N8jmh/zNOUmkzKVoTFE0=
Received: by 10.180.252.8 with SMTP id z8mr1168849bkh.158.1229712865680; Fri, 19 Dec 2008 10:54:25 -0800 (PST)
Received: by 10.181.9.9 with HTTP; Fri, 19 Dec 2008 10:54:25 -0800 (PST)
Message-ID: <f470f68e0812191054l6669929bq6b9d430f93fe4bd3@mail.gmail.com>
Date: Fri, 19 Dec 2008 18:54:25 +0000
From: Robert Burrell Donkin <robertburrelldonkin@gmail.com>
To: Ned Freed <ned.freed@mrochek.com>
Subject: Re: draft-freed-sieve-in-xml status?
Cc: Alexey Melnikov <alexey.melnikov@isode.com>, ietf-mta-filters@imc.org
In-Reply-To: <01N33HK2005W00SE3A@mauve.mrochek.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <f470f68e0812041225x318bfdccg1bf9201b53ce8c2e@mail.gmail.com> <493E908E.70504@isode.com> <f470f68e0812090956j56c29f17s77fc554adaab1350@mail.gmail.com> <01N332VSGLBC00SE3A@mauve.mrochek.com> <f470f68e0812141424k40d5e528oc0ccc908288de442@mail.gmail.com> <01N33HK2005W00SE3A@mauve.mrochek.com>
Sender: owner-ietf-mta-filters@mail.imc.org
Precedence: bulk
List-Archive: <http://www.imc.org/ietf-mta-filters/mail-archive/>
List-ID: <ietf-mta-filters.imc.org>
List-Unsubscribe: <mailto:ietf-mta-filters-request@imc.org?body=unsubscribe>
On Mon, Dec 15, 2008 at 4:04 AM, Ned Freed <ned.freed@mrochek.com> wrote: >> On Sun, Dec 14, 2008 at 6:20 PM, Ned Freed <ned.freed@mrochek.com> wrote: >> >> is there a compliance test suite? >> > >> > Not as far as I know. I certainly have no plans to write one or subject our >> > implementation to any that someone else develops. I doubt if anyone else does >> > either. >> > >> > The IETF isn't big on compliance test suites as a rule - the overarching goal >> > here is interoperability, not compliance. > >> however, AFACT the charter of this group does include testing > > Unit tests != compliance test suites != interop testing. And the differences > really matter. intention != execution given the traditional problems that good interoperability and compliance testing have offered, there's a movement to take another look at these issues applying the lessons learnt from the experience of development-centered automated testing. >> > And this IMNSHO is a very good thing. in the past I've worked on software, >> > notably an X.400 implementation, governed by standards that emphasize >> > compliance testing. What I've observed is that after passing multiple very >> > extensive compliance test suites, the software still failed to interoperate >> > worth a damn and required all sorts of tweaks before it could. > >> there is no silver interoperability bullet :-) > > Actually, I sort of disagree. We've held interop test events in the past for > IMAP and SMTP that have been hugely successful in weeding out implementation > problems and making stuff place nice together. > > It may not be a silver bullet, but it's surprisingly close. The main problem is > participation, or rather lack of it - getting some of the major vendors to show > up has been tough. And when there are popular implementations out there that > makes no pretense of following the standard - and there are - it's a real > problem. interoperability conferences are very useful but expensive, and this expense disenfranchises for example, this year m$ paid for serious interoperability testing (along the lines you suggestsed) in the web services space including flying FOSS developers around the world. i've heard very good things about this but most interesting long term movement is that M$ is now engaging with many of these developers to develop automated test suites which demonstrate continued interoperability amongst the group. AIUI the problem is that F2F is very expensive but interoperability is an ongoing problem. automated testing is now mainstream and the techiques developed should be applicable more widely. > Now, Sieve presents special problems for the sort of testing that works best > for protocols. Sieve is a language, not a protocol, and that makes it kinda > tough to throw two implementation on the wire and see if they can communicate. script based protocol testing can - with reasonable ease - be used to automate this style of functional test > So the charter includes the idea of developing an interop test suite. I'm not > entirely clear how that's going to work, but hopefully it will do so by > building up a suite of scripts covering things various implementations have > found to be problematic and making sure other implementations handle them the > same way. (if I have time I'll toss in some of the unit tests we use.) one of the problem with this approach to testing is that it does nothing to help implementors develop the basic function correctly. i understand that this is the skill in developing traditional compliance and interoperability suites, and i can understand how good implementations find this very useful. they do nothing to encourage new implemetations to get the basics right which IMO is equally important. >> all the stuff i help to develop is agile FOSS (TDD and BDD). automated >> tests will need to be developed so it's just a question of whether the >> tests exist already and can be shared and reused between >> implementations. the unit tests themselves are only really effectively >> portable between the moderns (.NET, java, python) but it's the >> verified example mappings which will take the time. > > You now appear to be talking about unit tests. We use agile processes here as > well and unit tests are part of what we do. i would prefer to avoid splitting hairs about what exactly is a unit test > But these are not at all the same > as compliance tests or interop testing. Compliances tests focus on compliance > with the specification - it's implicit in the name. A proper set of unit > tests are written while looking at an actual implementation, and can look at > known edge cases in an implementation, regression tests to make sure reported > bugs don't appear, and code coverage. This is all great stuff and very > valuable. when developing against a specification, a range of automated tests are valuable. these may well include automated created with unit testing tools that aren't really best described as unit test (as well as developer created tests that use other testing tools). for example, when developing an xml serializer for sieve, i would expect to create unit tests for each object in the pipeline, BDD integration tests for partially assembled pipelines and functional sieve script -> xml document tests. tests in the first category will only be use internally, the second are only likely to be useful for libraries sharing the interfaces defined but the functional sieve -> xml document tests should be useful more generally. IMHO pooling the third category would be worthwhile. i agree that this isn't sufficient for a good compliance test suite - it's unlikely to provide good enough coverage of difficult cases which a compliance expert would create - but it should provide a reasonable starting point and reduce the effort involved by allowing experts to focus exclusively on difficult cases. >> > As a result of this and several other experiences I must confess to >> > considerable cynicism that compliance testing represents a path to >> > interoperability. In my experience it does not, and has been for the most part >> > a colossal waste of time. > >> automated testing has come a very long way in the last decade but i >> understand your perspective > > GUI testing may have changed by becoming more automatable (or not - this is > not my area and I don't keep up with the state of the art), but I haven't seen > all that much change in most other areas. Back in the late 70s and early 80s > when I did math modelling software we actually did far more extensive unit > testing and had better code coverage than we do now. automated developer testing (TDD BDD etc) has now mainstreamed. generally testing toolsets such as xUnit and xMock drove the rise of FOSS and the (post-)modern languages (java, .NET, python, ruby) over the last decade. this isn't to say that developers didn't test before but automated testing and the agile methods it enables have transformed the development landscape. >> IMO it is a method (but not the only one) of producing reliable >> relatively bug free software. i think this is a reasonable >> pre-requisite for good interoperability. a good suite should aim to >> reduce the numbers of poor implementations which claim compatibility >> rather than try to ensure that good implementations interoperate >> perfectly. > > I'm always cautious about extrapolating from limited data - and we haven't seen > all that much use of scripts being moved from one implementation to another. > But to the extent we have, the problems that have shown up have been interop > issues. Unfortunately there are several Sieve implementations out there that > have chosen to ignore the extensions we've defined and roll their own to do > things the base specification does not cover. is there any (lightweight) way for implementors to let this group know about the extensions they've rolled? (other than showing on this list) > <snip> > >> > The bottom line, such as it is, is that there's effectively no leeway in the >> > copyright boilerplate you have to use if you want your stuff published as an >> > RFC. So I, and I suspect a lot of others, simply go with the flow and use >> > whatever we're told we have to use. If this is problematic for you, the place >> > to take that up is on the IPR WG list. It is not within our charter here to >> > consider such matters in any case. > >> does the IEFT require copyright assignment? > > Yes, there's a thing called the IETF Trust that rights are assigned to. copyright assignment should be unnecessary unless the IEFT intends to sue (which IMO would not be a positive development) > In fact they're going even further and asking for people to sign an agreement > granting rights to stuff in old RFCs published before copyright assignment > was required. It isn't clear to me how this can handle stuff like the > RFCs I've written with, say. Jon Postel (sadly deceased) as the coauthor, but > again, IANAL and I really try to stay as far away from this as possible. does the IEFT plan to grant back an unlimited exploitation license to the original authors? - robert
- draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Alexey Melnikov
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Alexey Melnikov
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Ned Freed
- [Fwd: Re: draft-freed-sieve-in-xml status?] Alexey Melnikov
- Re: draft-freed-sieve-in-xml status? Jeffrey Hutzelman
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- How to get implementors involved (was Re: draft-f… Alexey Melnikov
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Alexey Melnikov
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Alexey Melnikov
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Alexey Melnikov
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Kjetil Torgrim Homme
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Cyrus Daboo
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Aaron Stone
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Philip Guenther
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Aaron Stone
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: How to get implementors involved (was Re: dra… Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- Re: draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Arnt Gulbrandsen
- draft-freed-sieve-in-xml status? Ned Freed
- Re: draft-freed-sieve-in-xml status? Robert Burrell Donkin
- Re: draft-freed-sieve-in-xml status? Ned Freed