Re: [iola-conversion-tool] Bug in Datatracker Agenda Code?

Henrik Levkowetz <henrik@levkowetz.com> Wed, 21 March 2012 14:51 UTC

Return-Path: <henrik@levkowetz.com>
X-Original-To: iola-conversion-tool@ietfa.amsl.com
Delivered-To: iola-conversion-tool@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 01A8321F8717 for <iola-conversion-tool@ietfa.amsl.com>; Wed, 21 Mar 2012 07:51:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.295
X-Spam-Level:
X-Spam-Status: No, score=-102.295 tagged_above=-999 required=5 tests=[AWL=0.305, BAYES_00=-2.599, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CrKvFYEm8f7O for <iola-conversion-tool@ietfa.amsl.com>; Wed, 21 Mar 2012 07:51:22 -0700 (PDT)
Received: from grenache.tools.ietf.org (grenache.tools.ietf.org [IPv6:2a01:3f0:1:2::30]) by ietfa.amsl.com (Postfix) with ESMTP id D49A721F8715 for <iola-conversion-tool@ietf.org>; Wed, 21 Mar 2012 07:51:21 -0700 (PDT)
Received: from [2a01:3f0:1:0:21e:c2ff:fe13:7e3e] (port=49408 helo=brunello.netnod.se) by grenache.tools.ietf.org with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.77) (envelope-from <henrik@levkowetz.com>) id 1SAMsi-00058u-HC; Wed, 21 Mar 2012 15:51:20 +0100
Message-ID: <4F69EAE8.50804@levkowetz.com>
Date: Wed, 21 Mar 2012 15:51:20 +0100
From: Henrik Levkowetz <henrik@levkowetz.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: Ryan Cross <rcross@amsl.com>
References: <8F28AC7A-8D1D-465C-A92B-DCC07197DADF@amsl.com> <4F54DC2D.1000302@levkowetz.com> <11DE49D4-2C52-492F-A467-EEE22F19382A@amsl.com> <4F54EDF2.7020200@levkowetz.com> <D26A6771-EEE8-4819-9305-FCF3CC757FD8@amsl.com> <4F5500BF.9050604@levkowetz.com> <874725C6-E6A1-4DE5-B3FE-3EA0618ECE6A@amsl.com>
In-Reply-To: <874725C6-E6A1-4DE5-B3FE-3EA0618ECE6A@amsl.com>
X-Enigmail-Version: 1.4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-SA-Exim-Connect-IP: 2a01:3f0:1:0:21e:c2ff:fe13:7e3e
X-SA-Exim-Rcpt-To: rcross@amsl.com, amorris@amsl.com, iola-conversion-tool@ietf.org, henrik-sent@levkowetz.com
X-SA-Exim-Mail-From: henrik@levkowetz.com
X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000)
X-SA-Exim-Scanned: Yes (on grenache.tools.ietf.org)
Cc: Alexa Morris <amorris@amsl.com>, iola-conversion-tool@ietf.org
Subject: Re: [iola-conversion-tool] Bug in Datatracker Agenda Code?
X-BeenThere: iola-conversion-tool@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Discussion of the IOLA / DB Schema Conversion Tool Project <iola-conversion-tool.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/iola-conversion-tool>, <mailto:iola-conversion-tool-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/iola-conversion-tool>
List-Post: <mailto:iola-conversion-tool@ietf.org>
List-Help: <mailto:iola-conversion-tool-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/iola-conversion-tool>, <mailto:iola-conversion-tool-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Mar 2012 14:51:23 -0000

Hi Ryan,

On 2012-03-21 15:17 Ryan Cross said:
> Hi Henrik,
> 
> On Mar 5, 2012, at 10:06 AM, Henrik Levkowetz wrote:
> 
>> 3. Even after I fixed the things above, I went 'round and 'round a
>> couple of times, as I only saw a blank space where the agenda
>> should have been.  This seems to have been caused by the uploaded
>> document having a mac character set encoding and mac line-endings
>> -- Ryan, you maybe should investigate that further, and see if you
>> should do character set and line-ending conversion.  I think what's
>> on disk should either be ascii or unicode -- for drafts we check
>> that it's pure ascii, but the agendas are different, and we should
>> maybe enforce unicode?

> Does this mean the code is parsing the agenda file?  Does it need to
> do this?  I'm happy to add character set validation, but this would
> affect everyone submitting agenda's so I want to be sure that is the
> official requirement.

No, the code isn't parsing the agenda file, but the file is being
displayed through web browsers, and if you tell the browsers it's
latin-1 when it's utf-8, or vice versa, things don't look good.
And for mac line-endings things can get so bad that the browser
doesn't show anything at all in some circumstances -- that seemed to
be what happened above.

Best regards,

	Henrik