Re: [mile] Review of rfc5070bis and schema

["Roman D. Danyliw" <[e-mail]>]

Hello Panos!

I found Issue #42 (https://trac.tools.ietf.org/wg/mile/trac/ticket/42) tracking that the "-watchlist-" attributes were underspecified and then dropped in -07.  The class and attributes names have changed since the resolution text was written so it should read "removed ... in favor or @observable-id and IndicatorData class".  These changes in -07 also closed Issue #41, the undocumented @indicator-* attributes, and #14, predicate logic for indicators.  Examples of this approach (which evolved since then) were presented at IETF 90 (Toronto) -- specifically slide 5 of https://www.ietf.org/proceedings/90/slides/slides-90-mile-5.pdf.  I can't find detailed mailing list conversations beyond what's already in the tracker.

For an IODEF-Document with no incident report and only indicators, I believe the examples in Section 4.4 of draft-ietf-mile-iodef-guidance-05 using the -21 data model would look as follows:

Section 4.4, Example 1:
===================
Indicator: "(10.10.10.104 OR 10.10.10.106) AND target address 10.1.1.1"
===================
<IODEF-Document ...>
...
 <IndicatorData>
   <Indicator>
     <IndicatorID name="csirt.example.com" version="1">
     G90823490
     </IndicatorID>
     <Description>C2 domains</Description>    
     <IndicatorExpression operator="and">
       <IndicatorExpression operator="or">
         <Observable>
           <System category="source" spoofed="no">
             <Node>
               <Address category="ipv4-addr">
                 10.10.10.104
               </Address>
             </Node>
           </System>
         </Observable>
         <Observable>
           <System category="source" spoofed="no">
             <Node>
               <Address category="ipv4-addr">
                 10.10.10.106
               </Address>
             </Node>
           </System>
         </Observable>
       </IndicatorExpression>
       <Observable>
         <System category="target" spoofed="no">
           <Node>
             <Address category="ipv4-addr">
               10.1.1.1
             </Address>
           </Node>
         </System>
       </Observable>
     </IndicatorExpression>
   </Indicator>
 </IndicatorData>
...
</IODEF-Document>

Section 4.4, Example 2:
===================
Indicator: 
Dummy.txt (SHA256 = 141accec23e7e5 ...) OR
Dummy2.txt (SHA256 = 141accec23e7e5 ...)
===================
<IODEF-Document ...>
...
 <IndicatorData>
   <Indicator>
     <IndicatorID name="csirt.example.com" version="1">
     A4399IWQ
     </IndicatorID>
     <Description>File hash wathlist</Description>    
     <IndicatorExpression operator="or">
         <Observable>
           <FileData>
             <File>
               <FileName>dummy.txt</FileName>
               <HashData>
                 <Hash>
                  <ds:DigestMethod Algorithm=
                  "http://www.w3.org/2001/04/xmlenc#sha256"/>
                  <ds:DigestValue>
                   141accec23e7e5157de60853cb1e01bc38042d
                   08f9086040815300b7fe75c184
                  </ds:DigestValue>
                 </Hash>
               </HashData>
             </File>
           </FileData>
         </Observable>
         <Observable>
           <FileData>
             <File>
               <FileName>dummy2.txt</FileName>
               <HashData>
                 <Hash>
                  <ds:DigestMethod Algorithm=
                  "http://www.w3.org/2001/04/xmlenc#sha256"/>
                  <ds:DigestValue>
                   141accec23e7e5157de60853cb1e01bc38042d
                   08f9086040815300b7fe75c184
                  </ds:DigestValue>
                 </Hash>
               </HashData>
             </File>
           </FileData>
         </Observable>
     </IndicatorExpression>
   </Indicator>
 </IndicatorData>
...
</IODEF-Document>

Note that Observable/System was added in -21 after I read your note.  Previously, the more cumbersome Observable/EventData/Flow/System would have been used.  With the current data model, embedding indicators into EvenData is not necessary.  They can be directly represented with IndicatorData.  The IndicatorExpression class specifies the operator by which to construct the boolean algebraic expression and applies it to a wide variety of possible observables.

Have we lost something we need?

Roman

> -----Original Message-----
> From: Panos Kampanakis (pkampana) [mailto:[e-mail]]
> Sent: Tuesday, April 26, 2016 12:17 PM
> To: Roman D. Danyliw <[e-mail]>; [e-mail]
> Cc: Mio SUZUKI <[e-mail]>
> Subject: RE: Review of rfc5070bis and schema
> 
> Hi Roman,
> 
> Sorry that I haven't reviewed the bis doc recently.
> 
> As we were revamping the guidance doc https://tools.ietf.org/html/draft-
> ietf-mile-iodef-guidance-05 we realized that the watchlist elements have
> been removed from the bis document after the draft-ietf-mile-rfc5070-bis-
> 06 version. The watchlists were used in the predicate logic for indicators of
> the same incident.
> 
> Do you remember if there was a decision to remove the watchlist attributes
> from the classes? I couldn't find more information in the archives and I don't
> remember if there was a discussion in the list.
> 
> Panos
> 
> 
> 
> 
> -----Original Message-----
> From: mile [mailto:[e-mail]] On Behalf Of Roman D. Danyliw
> Sent: Monday, March 21, 2016 5:08 PM
> To: Waltermire, David A. <[e-mail]>; [e-mail]
> Subject: Re: [mile] Review of rfc5070bis and schema
> 
> Hi David!
> 
> Thanks again for this review.  Almost all have been address in versions -16 to
> -18.  Inline is a response to each item.
> 
> > -----Original Message-----
> > From: mile [mailto:[e-mail]] On Behalf Of Waltermire,
> > David
> > Sent: Monday, November 02, 2015 8:29 PM
> [snip]
> 
> > Section 2.4:
> >
> > The following text is diffecult to parse. Consider rephrasing:
> >
> > This relationship between multiple
> > classes being translated instances of the same text is indicated by a
> > common identifier set in the translation-id attribute.
> 
> Rewritten.
> 
> > Section 2.12:
> >
> > Isn't this implemented as the PostalAddressType in the schema not
> xs:string?
> 
> Yes.  Fixed.
> 
> > Section 2.16:
> >
> > There is a cross reference for the text "Section 3.3.8" that looks
> > like it is incorrect.
> 
> No, the cross reference is right.  Section 3.3.8 of
> https://www.w3.org/TR/xmlschema-2/.
> 
> > General schema Comments:
> > - The schema is missing an XML declaration at the beginning.
> 
> Fixed.
> 
> > - Some type names are inconsistant with the overall type naming scheme
> > (e.g., systemimpact-type-type)
> 
> That's intentional.  The convention to name the types is "<element name> -
> dash- <attribute name> -type-".  There are several attributes named type so
> that's why it looks odd.
> 
> > - The type "yes-no-type" is not referenced in the schema.
> 
> Removed.
> 
> > - The outline numbering has many singular subsections (e.g., 3.13.1).
> > This issue may get raised by the RFC editor and should probably get
> > fixed/clarified before then.
> 
> There are fewer now.  I don't know how easily fix that and keep the
> document flow.
> 
> > - Schema types for the REAL i/d type vary between float and double. Is
> > there a reason for this (e.g., Counter -> xs:double)? If so, it may
> > make sense to have specific types in the i/d to reflect the distinction.
> 
> Fixed.  REAL is now consistently xs:float.
> 
> > - The prefixes used in section 3 and subsections (e.g., ds, xml)
> > should be documented at the beginning of section 3 for clarity.
> 
> Could you please clarify.  Do you mean where a namespace is used in Section
> 3, it should explained (e.g., xml:lang or ds:DigestAlgorithm)?
> 
> > Section 3.2:
> >
> > - The cardinality of IndicatorData in the schema is 0..1 while in the
> > i/d it is 0..*
> 
> Fixed.
> 
> > - The observable-id attribute is marked as the STRING type, but is an
> > xs:ID type in the schema. All instances of xs:ID are required to be
> > unique in an XML instance. For this reason it should be marked as an
> > ID type in the i/d. This means we need a datatype for ID in section 2.
> 
> Fixed.  ID is a type in Section 2.  Currently, Section 2.14.
> 
> > - The definition for status -> ext-value uses the terminology "An
> > escape value used to extend this attribute". I recommend against using
> > the term "escape" here to avoid confusion with escaping. Perhaps it
> > would be better to use "A value used to indicate that this attribute
> > is extended and the actual value is provided using the ext-status attribute."
> 
> Fixed.
> 
> > Section 3.3:
> > - The colors values in the restriction class provide mappings to other
> > provided values, but they do not define the meaning of the colors.
> > They seem to relate to TLP. Perhaps we should reference the related UK
> TLP work?
> 
> It is a reference to the UK.  I wasn't able to find a reference.  Can you help?
> 
> > Section 3.6:
> > - The cardinality of IncidentId, URL, ThreatActor, and Campaign in the
> > schema are 1..*. In the i/d this matches in the UML text, but not in
> > the diagrams where they are 0..*. The cardinality on the parent choice
> > (1..*) should allow the individual elements to be 1..1. Let me know if
> > you have any questions on this.
> 
> Fixed.
> 
> > Section 3.7:
> > - ThreatActor -> ThreatActorId is inconsistant in the schema (1..1),
> > diagram (0..1), and text (1..*).
> > - ThreatActor -> Description is inconsistant in the schema and diagram
> > (0..*), and text (1..*).
> 
> Fixed.
> 
> > Section 3.8
> > - Campaign -> CampaignID and Description are inconsistant in the
> > schema and diagram (0..1), and text (1..*).
> 
> Fixed
> 
> > Section 3.9
> > - The schema is missing the ext-dtype attribute documented in the i/d.
> 
> Fixed
> 
> > Section 3.10
> > - Is there a reason that Fax is limited to 0..1? I would guess some
> > organizations may have more than one Fax #.
> 
> Fixed.
> 
> > Section 3.10.2
> > - The attribute 'translation-id' is missing in the i/d diagram and text.
> 
> Fixed.  ML_STRING classes are now only depicted once in the document.
> 
> > Section 3.12.1
> > - The requirement for at least one of Description or
> > DetectionConfiguration can be modeled a xs:choice with 1..*
> > cardinality in the schema. Other classes can probably be improved in the
> same way.
> 
> True.  I mentioned this in Prague (?).  I thought the WG wasn't concerned
> about this as long as the text provided the clarity.
> 
> WG: disagree?
> 
> > Section 3.13:
> > - The element 'enum:Reference' in the i/d is not in the enum namespace
> > in the schema.
> >
> > Section 3.13.1:
> > - The element 'ReferenceName' should be 'enum:ReferenceName' in the
> > i/d text.
> > - The text starting with "At least one of these classes MUST be present."
> > appears to be improperly nested in the i/d.
> >
> > Section 3.14:
> > - The text:
> >
> > A least one instance of the possible three impact classes (i.e.,
> > Impact, TimeImpact, or MonetaryImpact) MUST be present.
> >
> > Should read as:
> >
> > A least one instance of the possible five impact classes (i.e.,
> > SystemImpact, BusinessImpact, TimeImpact, MonetaryImpact, or
> > IntendedImpact) MUST be present.
> >
> > - The IntendedImpact class doesn't appear to be documented in this
> section.
> > The text is confusing on this:
> >
> > Identically defined as Section 3.14.2 but describes intent rather
> >       than the realized impact.
> >
> > Should be:
> >
> > Identical to the BusinessImpact class defined in Section 3.14.2, but
> > describes the intent rather
> >       than the realized impact.
> 
> Fixed.
> 
> 
> > Section 3.14.1:
> > - The attribute 'type' does not have the 'unknown' default value
> > defined in the schema. 'unknown' is not a valid enum value.
> > - The attribute 'type' allow the enum value admin in the schema. This
> > value is not documented in the i/d.
> >
> > Section 3.14.3:
> > - The schema is missing the 'ext-metrics' and 'ext-duration'
> > attributes defined in the i/d.
> >
> > Section 3.14.5:
> >
> > The 'Confidence' class is defined in the schme as:
> >
> >   <xs:element name="Confidence">
> >     <xs:complexType mixed="true">
> >       <xs:attribute name="rating" use="required"
> > type="confidence-rating- type"/>
> >     </xs:complexType>
> >   </xs:element>
> >
> > To match the i/d it should be defined in the schema as:
> >
> >   <xs:element name="Confidence">
> >     <xs:complexType>
> >       <xs:simpleContent>
> >         <xs:extension base="iodef:PositiveFloatType">
> >           <xs:attribute name="rating" use="required"
> > type="confidence-rating- type"/>
> >         </xs:extension>
> >       </xs:simpleContent>
> >     </xs:complexType>
> >   </xs:element>
> 
> Fixed.
> 
> > Section 3.15:
> > - The cardinality for 'HistoryItem' is "One or many." This is
> > different than the use of "One or more." in the rest of the document.
> > This should be made consistant with the rest of the draft. Also, check for
> other cases of this.
> 
> Fixed.  This was a pervasive problem.
> 
> > Section 3.19:
> > - The schema allows Node to be 1..* while the i/d diagram and text is 1..1.
> 
> Fixed 1:1.
> 
> > Section 3.20:
> > - The schema defines a NodeRole element that is not documented in the
> i/d.
> > - The i/d defines a DateTime element that is not defined in the schema.
> >
> > Section 3.20.1:
> > - The base type of this class is xs:string in the schema. The i/d
> > diagram does not reflect this.
> > - Should the default for category be ipv6-addr given the emphasis on
> > transition?
> 
> Fixed.
> 
> > Section 3.20.2:
> > - The base type of this class is MLString in the schema. The i/d
> > diagram does not reflect this.
> 
> Fixed.  Classes no longer derive the MLString type.
> 
> > - The i/d diagram and text is also missing the attribute translation-id.
> > - NodeRole -> Category is inconsistant where the schema has
> > 'c2-server' and the i/d has 'c2'.
> 
> Fixed.
> 
> > Section 3.20.3:
> > - The schema is missing the ext-unit attribute.
> 
> Fixed.
> 
> > Section 3.21.1:
> > - The values for 'record-type' are not documented in the i/d. There is
> > a "TODO" placeholder in the i/d.
> 
> Fixed.  RelatedDNS was redefined as an EXTENSION type.
> 
> > Section 3.21.3:
> > - The 'SameDomainContact' element is 1..1 in the schema and 0..1 in
> > the i/d diagram and text.
> 
> No, the schema is right.  These elements are in an xs:choice, not
> xs:sequence:
> 
>       <xs:choice>
>         <xs:element ref="iodef:SameDomainContact"/>
>         <xs:element ref="iodef:Contact"
>                     minOccurs="1" maxOccurs="unbounded"/>
>       </xs:choice>
> 
> > Section 3.22:
> > - The choice between 'Port' and 'Portlist' should be changed to be
> > 1..1 in the schema to reflect the requirement for one of the two to be used
> in the i/d.
> 
> Fixed.
> 
> > Section 3.22.1:
> > - The 'IANAService' element is 1..1 in the schema and 0..1 in the i/d
> > diagram and text,
> 
> Fixed.
> 
> > Section 3.22.2:
> > - The attribute 'proto-name' is an xs:integer in the schema and
> > xs:string in the i/d. It looks like the schema needs to be changed based on
> the i/d text.
> >> - The attribute 'ext-dtype' is missing in the schema.
> > - the attribute 'dtype' is missing values "portlist" and "path" in the schema.
> 
> Fixed.  Now of type EXTENSION.
> 
>  > Section 3.22.4:
> > - The attribute has the name 'ext-dtype' in the schema and
> > 'enum-dtype' in the i/d diagram.
> > - SoftwareReference -> spec-name -> swid entry needs to have the
> > actual reference provided. Look for "fix me."
> 
> Fixed.
> 
> > Section 3.24:
> > - The cardnalities of EmailHeaderField, HashData, and SignatureData
> > are 0..* in the i/d diagram and 0..1 in the schema and i/d text.
> 
> Fixed.
> 
> > Section 3.25.1:
> > - The 'RecordItem' element is 1..* in the schema and 0..* in the i/d
> > diagram and text.
> 
> Fixed.
> 
> > Section 3.26.1:
> > - The attribute 'observable-id' is missing in the schema.
> 
> Fixed.
> 
> > Section 3.27.1:
> > - The element 'ds: X509Data' has a space in the i/d diagram that
> > should be removed.
> 
> Fixed.
> 
> > Section 3.28.1:
> > - The element 'FileType' has the type xs:integer in the schema and
> > xs:string in the i/d. The schema should be changed to xs:string based on
> the i/d text.
> > - The elements 'SignatureData' and 'AssociatedSoftware' in the i/d are
> > named differently in the schema.
> 
> Fixed.
> 
> > Section 3.29:
> > - The attribute 'ext-scope', which is defined in the schema, is
> > missing in the i/d diagram.
> 
> Fixed.
> 
> > Section 3.29.1:
> > - The element 'ds:CannonicalizationMethod' is 0..1 in the i/d and 1..1
> > in the schema.
> 
> Fixed.  0..1
> 
> > Section 3.29.2:
> > - The element AdditionalData is 0..* in the i/d and 1..1 in the schema.
> 
> Fixed.
> 
> > Section 3.32:
> > - The element 'AlternativeIndicatorID' is 0..* in the schema and 0..1 in the
> i/d.
> > - The element 'AdditionalData' is missing in the schema.
> 
> Fixed.
> 
> > Section 3.32.3:
> > - The attributes 'restriction' and 'ext-restriction' are defined in
> > the schema and are missing in the i/d.
> > - The element 'Service' is missing in the schema.
> > - The element 'AdditionalData' is 0..1 in the schema and 0..* in the i/d.
> > - A choice should be used to model the "extactly one" requirement.
> 
> Fixed.
> 
> > Section 3.32.3.1:
> > - The element 'BulkObservableList' is 0..1 in the schema and 1..1 in the i/d.
> > - The element 'AdditionalData' is 0..1 in the schema and 0..* in the i/d.
> 
> Fixed.
> 
> > Section 3.32.4:
> > - The cardinality of the elements do not match in the schema and i/d.
> 
> Fixed.
> 
> > Section 3.32.6:
> > - It is not clear what the type "EMPTY" means here.
> > - I don't recall if the "IDREF" data type was defined in section 2.
> 
> Fixed.  Yes, IDREF is a type in Section 2.
> 
> > Section 3.32.7:
> > - It is not clear what the type "EMPTY" means here.
> > - I don't recall if the "IDREF" data type was defined in section 2.
> 
> Fixed.  Yes, IDREF is a type in Section 2.
> 
> Thank you again!
> Roman
> _______________________________________________
> mile mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/mile