Re: [pkix] Clarification on "zero" hash value in SigPolicyHash (CAdES)

[Niklas Matthies <[e-mail]>]

Coming back to this topic...

To summarize, I believe it's fairly clear now that option 3 was the 
intended interpretation, despite being poorly communicated by the 
wording in the standard. I've also now come to strongly favor option 3 
due to its aim of maximizing compatibility with prior implementations 
that know nothing about zero-hash values and expect a real hash value 
to be present in accordance with earlier versions of the standard.

In particular, existing implementations that only check for the hash 
length, possibly as part of their format checks, or that implicitly 
rely on the hash value having the correct length without checking it, 
would remain unaffected, and in the worst case would merely report a 
policy hash mismatch -- as opposed to a format error, or possibly a 
crash if the implementation blindly reads n bytes based on the 
algorithm identifier value.

In the meantime, there's the additional compatibility consideration 
that signatures generated by implementations that have correctly 
adopted the interpretation of option 3 should be honored. On the other 
hand, the wording in the standard is sufficiently unclear that some 
implementers have interpreted it differently than intended, and thus 
signatures have been produced and are "in the wild" with zero hashes 
of different length.

In the light of that situation, I would propose the following approach 
for a future revision of the standard:

1. Signature validation applications SHALL interpret a zero hash value 
of any length (including zero-length) as an unknown/unspecified hash 
value.

2. Signature creation applications outputting a zero hash SHOULD 
generate it with a length consistent with the algorithm identifier, 
but MAY generate it with any length (including zero-length).

The first point ensures that existing signatures created under either 
interpretation remain valid. The second point maintains the originally 
intended interpretation (option 3) as the recommended form of a zero 
hash, while allowing other forms in accordance with the first point. 

Signature creators are then still free to use e.g. an empty OCTET 
STRING if they care less about interoperability with legacy 
implementations and more about the following objection.

Regarding the objection that a real hash value may happen to be 
all-bits-zero and would be incorrectly interpreted as an unknown hash 
value: While such an occurrence cannot be ruled out, it is exceedingly 
unlikely. In probabilistic terms, a bit flip due to cosmic radiation 
or other soft error is vastly more likely. It's hard to conceive how 
an attacker could exploit that possibility, assuming strong hash 
algorithms. I simply don't think it is an issue.

As a side note: I wonder why the authors of V1.7.3 didn't just 
introduce a new algorithm OID defined to indicate an unknown hash 
value. (It could even formally be defined to denote the trivial hash 
algorithm that maps all inputs to the same empty value.) Possibly 
there were concerns that existing implementations might somehow fail 
badly due to the unknown algorithm. Or maybe setting the hash value to 
zero just seemed the simpler solution overall.

Anyway, please feel free to relay my thoughts to the ESI group.

Niklas


On Wed 2019-07-17 at 09:53h, Stefan Santesson wrote on pkix:
>Niklas,
>
>Thanks for your analysis, which quite possibly is very true.
>
>However, I would just add that an empty OCTET STRING value would also be "syntactically correct" on the ASN.1 level.
>
>Stefan Santesson
>
>On 2019-07-16, 16:58, "pkix on behalf of Niklas Matthies" <[e-mail] on behalf of [e-mail]> wrote:
>
>    It now occurred to me that sigPolicyHash field being optional in ETSI
>    TS 101 733 V1.6.3 (2005-09) probably was discovered to be a mistake
>    because (1) it introduces a parsing ambiguity with
>    sigPolicyQualifiers, as both are optional SEQUENCEs, and (2) because
>    it broke compatibility with existing implementation.
>
>    So it would seem that the zero hash value was introduced as a
>    workaround to still be able to have an optional hash, syntactically
>    masquerading as a real hash, and thus maintaining compatibility with
>    the earlier versions at least on the ASN.1 level. Then the note about
>    backwards compatibility would not be referring to the support of
>    "zero" hash values, but to the fact that keeping the sigPolicyHash
>    field mandatory for backwards compatibility reasons leaves no other
>    choice than to use a "compatible" encoding for a missing hash value.
>
>    This interpretation would favor option 3 (using the length implied by
>    the hash algorithm), as this arguably causes the least disruption for
>    pre-V1.6.3 (and RFC-3126-based) implementations.
>
>    Niklas
>
>
>    On Fri 2019-07-12 at 22:05h, Niklas Matthies wrote on pkix:
>    >Dear all,
>    >
>    >I hope that someone can shed some light on the definition and origin
>    >of the special-case "zero" hash value for SigPolicyHash in CAdES. The
>    >main question is what exactly constitutes a "zero" hash value on the
>    >ASN.1 level.
>    >
>    >In the most current CAdES specification, ETSI EN 319 122-1 V1.1.1
>    >(2016-04), the zero hash value is described as follows (clause
>    >5.2.9.1, page 20):        The sigPolicyHash field shall contain the
>    >identifier of the hash    algorithm and the hash of the value of the
>    >signature policy.
>    >   The hashValue within the sigPolicyHash may be set to zero to
>    >indicate that the policy hash value is not known.
>    >
>    >       NOTE: The use of a zero-sigPolicyHash value is to ensure
>    >backwards compatibility with earlier versions of ETSI TS 101 733 [1].
>    >
>    >The ASN.1 definition of SigPolicyHash is as follows:
>    >
>    >   SignaturePolicyId ::= SEQUENCE {
>    >       sigPolicyId SigPolicyId,
>    >       sigPolicyHash SigPolicyHash,
>    >       sigPolicyQualifiers SEQUENCE SIZE (1..MAX) OF
>    >SigPolicyQualifierInfo OPTIONAL
>    >   }
>    >
>    >   SigPolicyHash ::= OtherHashAlgAndValue
>    >
>    >   OtherHashAlgAndValue ::= SEQUENCE {
>    >       hashAlgorithm   AlgorithmIdentifier,
>    >       hashValue       OtherHashValue }
>    >
>    >   OtherHashValue ::= OCTET STRING
>    >
>    >The question is what constitutes a "zero" hash value. A zero-length
>    >octet string? Any octet string where all octets are zero? Is the
>    >length of the octet string relevant, e.g. can or should it match the
>    >length implied by the hash algorithm?
>    >
>    >I spent some time going back the history of the standards to find the
>    >"earlier version" relevant for the note about backwards compatibility,
>    >however that search was inconclusive.
>    >
>    >All versions of ETSI TS 101 733 from the latest V2.2.1 (2013-04) back
>    >to V1.7.4 (2008-07) include the following note, which had a second
>    >sentence added:
>    >
>    >       NOTE: The use of a zero-sigPolicyHash value is to ensure
>    >backwards compatibility with earlier versions of the
>    >       current document. If sigPolicyHash is zero, then the hash
>    >value should not be checked against the calculated hash value
>    >of the signature policy.
>    >
>    >Confusingly, this now talks about sigPolicyHash being zero (whatever
>    >that would mean), not about the hashValue field within sigPolicyHash.
>    >
>    >The same text appears in RFC 5126 (February 2008).
>    >
>    >The previous version ETSI TS 101 733 V1.7.3 (2007-01) is the first
>    >version to talk about a zero hash value, and the note is just one
>    >sentence again:
>    >
>    >   The hashValue within the sigPolicyHash max [sic] be set to zero to
>    >indicate that the policy hash value is not known.
>    >
>    >       NOTE: The use of zero policy hash value is to ensure backward
>    >compatibility with earlier versions of the current document.
>    >
>    >In addition, the annex states the following:
>    >
>    >   Annex K (informative): Changes from the previous version
>    >
>    >       • if the hash of the signature policy is unknown, then, by
>    >convention, the sigPolicyHash shall be set to all zero
>    >
>    >Here again, the annex talks about sigPolicyHash being set to zero, not
>    >the hashValue within sigPolicyHash.
>    >
>    >On the RFC side, the same note first appears in
>    >https://tools.ietf.org/html/draft-ietf-smime-cades-02 (May 2007), and
>    >the sentence above (without the note) first appears in
>    >https://tools.ietf.org/html/draft-ietf-smime-cades-01 (April 2006).
>    >
>    >In the previous ETSI TS 101 733 V1.6.3 (2005-09), the sigPolicyHash
>    >field is optional (which is mandatory in all previous and all
>    >subsequent versions), and the following note is included:
>    >
>    >       NOTE: In the previous version of TS 101 733 (i.e. version
>    >1.5.1)        sigPolicyHash was mandatory. Implementations requiring
>    >to be        backward compatible with version 1.5.1 and previous
>    >versions        of the current document MUST include SigPolicyHash.
>    >
>    >None of the previous versions, from ETSI TS 101 733 V1.5.1 (2003-12)
>    >back to ETSI TS 101 733 V1.2.2 (2000-12) (the earliest version
>    >available at
>    >https://www.etsi.org/deliver/etsi_ts/101700_101799/101733/), as well
>    >as RFC 3126 (September 2001), do mention anything about zero hash
>    >values.
>    >
>    >So it remains unclear to which specification backwards compatiblity is
>    >intended. The point where the wording about zero hash values was
>    >introduced suggests that this may be about V1.6.3, where the
>    >sigPolicyHash field was optional. However, it's not clear how a "zero"
>    >hash value would achieve compatibility with the case of a missing
>    >sigPolicyHash field.
>    >
>    >I would appreciate any clarification on that topic.
>    >
>    >Kind regards,
>    >Niklas
>    >
>    >_______________________________________________
>    >pkix mailing list
>    >[e-mail]
>    >https://www.ietf.org/mailman/listinfo/pkix
>
>    _______________________________________________
>    pkix mailing list
>    [e-mail]
>    https://www.ietf.org/mailman/listinfo/pkix
>
>
>