Re: [yaco-idsubmit-tool] Title extraction failure

[Henrik Levkowetz <[e-mail]>]

(I changed the subject line to match the new issue)

On 2011-03-13 20:04 Tony Hansen said:
> Here's a difficult one for the tool:
> 
> draft-ietf-yam-pre-evaluation-template-04.txt
> draft-ietf-yam-pre-evaluation-template-04.xml
> 
> (I adjusted the dates to get past the date checks.)
> 
> The title is being picked up as
> 
>      Group
> 
> instead of as
> 
>   Preliminary Evaluation of RFC XXX "[PLACEHOLDER: INSERT TITLE HERE]",
> for advancement from Draft Standard to Full Standard by the YAM Working
>                                   Group
> 
> Obviously, the long lines that are part of the title are messing things 
> up and the submission tool is only picking up the last line.

Yes.

Emilio, I'd suggest changing the regex to require one or more blank
lines before and after the title, and making the first group explicitly
include all lines of the {1, 3} repeat (this is what messes up this
case).  Also, we should strip out newlines and extra space from the title.

Suggested patch:

12:44 ~/src/db/yaco/idsubmit/ietf/utils
henrik@merlot $ svn diff draft.py 
Index: draft.py
===================================================================
--- draft.py	(revision 2886)
+++ draft.py	(working copy)
@@ -575,11 +575,12 @@
     def get_title(self):
         if self._title:
             return self._title
-        title_re = re.compile('(.+\n){1,3}(\s+<?draft-\S+\s*\n)')
+        title_re = re.compile('(?:\n\s*\n\s*)((.+\n){1,3})(\s+<?draft-\S+\s*\n)\s*\n')
         match = title_re.search(self.pages[0])
         if match:
             title = match.group(1)
             title = title.strip()
+            title = re.sub('\s*\n\s*', ' ', title)
             self._title = title
             return self._title
         # unusual title extract

12:45 ~/src/db/yaco/idsubmit/ietf/utils
henrik@merlot $ 

> Given that the XML is also being uploaded, why aren't the title, 
> abstract, and author info being extracted from there? I thought that was 
> going to be one of the enhancements being put into the new I-D 
> submission tool? That would avoid all issues with the issue-prone patterns.

Yes, but it's not part of this work order -- what we're trying to do with
this round is to replace the old tool as soon as possible, both to relieve
the pain of the old one, and to make it possible to migrate to the new
database schema.  There will more work done on the submission tool, to
cover _all_ of the requirements of RFC 4228, but that will come after we
have this round deployed.  Metadata extraction from XML files, if available,
is among the remaining 4228 requirements to be handled in the next round.

At least this has been the plan so far ...


Best,

	Henrik