[pkix] Clarification on "zero" hash value in SigPolicyHash (CAdES)

[Niklas Matthies <[e-mail]>]

Dear all,

I hope that someone can shed some light on the definition and origin 
of the special-case "zero" hash value for SigPolicyHash in CAdES. The 
main question is what exactly constitutes a "zero" hash value on the 
ASN.1 level.

In the most current CAdES specification, ETSI EN 319 122-1 V1.1.1 
(2016-04), the zero hash value is described as follows (clause 
5.2.9.1, page 20): 
    
    The sigPolicyHash field shall contain the identifier of the hash 
    algorithm and the hash of the value of the signature policy.
    The hashValue within the sigPolicyHash may be set to zero to 
    indicate that the policy hash value is not known.

        NOTE: The use of a zero-sigPolicyHash value is to ensure 
        backwards compatibility with earlier versions of ETSI TS 101 733 [1].

The ASN.1 definition of SigPolicyHash is as follows:

    SignaturePolicyId ::= SEQUENCE {
        sigPolicyId SigPolicyId,
        sigPolicyHash SigPolicyHash,
        sigPolicyQualifiers SEQUENCE SIZE (1..MAX) OF 
            SigPolicyQualifierInfo OPTIONAL
    }

    SigPolicyHash ::= OtherHashAlgAndValue

    OtherHashAlgAndValue ::= SEQUENCE {
        hashAlgorithm   AlgorithmIdentifier,
        hashValue       OtherHashValue }

    OtherHashValue ::= OCTET STRING

The question is what constitutes a "zero" hash value. A zero-length 
octet string? Any octet string where all octets are zero? Is the 
length of the octet string relevant, e.g. can or should it match the 
length implied by the hash algorithm?

I spent some time going back the history of the standards to find the 
"earlier version" relevant for the note about backwards compatibility, 
however that search was inconclusive.

All versions of ETSI TS 101 733 from the latest V2.2.1 (2013-04) back 
to V1.7.4 (2008-07) include the following note, which had a second 
sentence added:

        NOTE: The use of a zero-sigPolicyHash value is to ensure 
        backwards compatibility with earlier versions of the
        current document. If sigPolicyHash is zero, then the hash 
        value should not be checked against the calculated hash value 
        of the signature policy.

Confusingly, this now talks about sigPolicyHash being zero (whatever that 
would mean), not about the hashValue field within sigPolicyHash.

The same text appears in RFC 5126 (February 2008).

The previous version ETSI TS 101 733 V1.7.3 (2007-01) is the first 
version to talk about a zero hash value, and the note is just one 
sentence again:

    The hashValue within the sigPolicyHash max [sic] be set to zero to 
    indicate that the policy hash value is not known.

        NOTE: The use of zero policy hash value is to ensure backward 
        compatibility with earlier versions of the current document.

In addition, the annex states the following:

    Annex K (informative): Changes from the previous version

        • if the hash of the signature policy is unknown, then, by 
        convention, the sigPolicyHash shall be set to all zero

Here again, the annex talks about sigPolicyHash being set to zero, 
not the hashValue within sigPolicyHash.

On the RFC side, the same note first appears in 
https://tools.ietf.org/html/draft-ietf-smime-cades-02 (May 2007), 
and the sentence above (without the note) first appears in 
https://tools.ietf.org/html/draft-ietf-smime-cades-01 (April 2006).

In the previous ETSI TS 101 733 V1.6.3 (2005-09), the sigPolicyHash 
field is optional (which is mandatory in all previous and all 
subsequent versions), and the following note is included:

        NOTE: In the previous version of TS 101 733 (i.e. version 1.5.1) 
        sigPolicyHash was mandatory. Implementations requiring to be 
        backward compatible with version 1.5.1 and previous versions 
        of the current document MUST include SigPolicyHash.

None of the previous versions, from ETSI TS 101 733 V1.5.1 (2003-12) 
back to ETSI TS 101 733 V1.2.2 (2000-12) (the earliest version available 
at https://www.etsi.org/deliver/etsi_ts/101700_101799/101733/), as 
well as RFC 3126 (September 2001), do mention anything about zero hash 
values.

So it remains unclear to which specification backwards compatiblity is 
intended. The point where the wording about zero hash values was 
introduced suggests that this may be about V1.6.3, where the
sigPolicyHash field was optional. However, it's not clear how a "zero" 
hash value would achieve compatibility with the case of a missing 
sigPolicyHash field.

I would appreciate any clarification on that topic.

Kind regards,
Niklas