Re: [regext] Extension Prefixes, JSON Values, and URI Path Segments

[Tom Harrison <[e-mail]>]

Hi Mario,

On Mon, May 30, 2022 at 09:51:21AM +0200, Mario Loffredo wrote:
> Il 29/05/2022 06:42, Tom Harrison ha scritto:
>> On Fri, May 27, 2022 at 12:41:27PM +0200, Mario Loffredo wrote:
>>> Think the matter is that even the possible backwards-compatible changes
>>> would result in being hardly backwards-compatible.
>>> 
>>> Let te me give an example to make myself clear and move the discussion on a
>>> practical perspective.
>>> 
>>> Let's call "ext1" the rdapConformance value signaling the support of
>>> "ext1_data" response extension. The response would be:
>>> 
>>> {
>>> 
>>> "rdapConformance" : [ ....., "ext1"],
>>> 
>>> ...
>>> "ext1_data" : { ... },
>>> ....
>>> 
>>> }
>>> 
>>> Now, let's suppose to add the field "newfield" to "ext1_data" and signal
>>> this change by a version update from "ext1" to "ext2". The response should
>>> be (in theory):
>>> 
>>> {
>>> 
>>> "rdapConformance" [ ....., "ext2"],
>>> 
>>> ...
>>> 
>>> "ext2_data" : { ... },   // where ext2_data includes all the member of
>>> "ext1_data" plus "newfield"
>>> ....
>>> 
>>> }
>>> 
>>> To avoid a breakage in the REST contract, the RDAP server should implement a
>>> deprecation process ending with the replacement of "ext1_data" with
>>> "ext2_data". This means that, for a period of time, both the version of the
>>> response extensions should be provided:
>>> 
>>> {
>>> 
>>> "rdapConformance" [ ....., "ext1", "ext2"],
>>> 
>>> ...
>>> "ext1_data" : { ... },
>>> "ext2_data" : { ... },
>>> ....
>>> 
>>> }
>>> 
>>> 
>>> This situation would result in the following paradox as well: being the
>>> introduction of new field in a JSON response widely considered a non
>>> breaking change, it should be signaled by a minor version update (e.g.
>>> "ext1_1). But, since the final effect would be the replacement of a response
>>> field with another,  a major version change should be used instead. In
>>> short, every new version of the response extension would imply a major
>>> version update ("ext1", "ext2", "ext3", ..) and  a consequent deprecation
>>> process the server should support.
>> 
>> I don't understand the relevance of the scenario given above to the
>> approach I set out for handing backwards-compatible changes.  To avoid
>> any doubt about the models I was proposing, I'll step through them in
>> more detail, per your approach above.
>> 
>>   - Scenario A: add single field (backwards-compatible):
>>      - Initial: register extension "ext1", which notes that "ext1_data"
>>        will be included in responses, with a single field named
>>        "field1":
>> 
>>          "rdapConformance": [ ..., "ext1" ],
>>          ...
>>          "ext1_data": { "field1": ... }
>> 
>>      - Later: update the existing registration for extension "ext1",
>>        noting that some implementations of this extension may now
>>        return "field2".  Implementations in accordance with the
>>        original version of "ext1" continue working as they do today.
>>        Implementations in accordance with the new version of "ext1"
>>        return responses like so:
>> 
>>          "rdapConformance": [ ..., "ext1" ],
>>          ...
>>          "ext1_data": { "field1": ...,
>>                         "field2": ... }
>> 
>>      - The presence of "field2" is enough to distinguish the new
>>        version from the original version.  Clients written to work with
>>        the original version do not change, and clients written to work
>>        with the new version are also fine, because they can determine
>>        whether the new version has been implemented by way of the
>>        presence of "field2".
> 
> [ML] Usually, REST-JSON consumers configure JSON libraries to ignore unknown
> fields in deserialization. This just to reduce to prevent from breaking
> changes in responses.
> 
> Hence, the presence of "field2" wouldn't be detected unless it was managed
> by the client (e.g. by putting every unknown field on a given map).

I'm not sure how this is different from what I wrote.  A client
configured in this way is one that works against the original version
only, which is fine.  If/when the client is updated to support the new
version, then it can make use of the new field.

>>   - Scenario B: add multiple fields (backwards-compatible):
>>      - Initial: register extension "ext1", which notes that "ext1_data"
>>        will be included in responses, with a single field named
>>        "field1".  Subversioning is managed by way of an additional
>>        "conformance" field, which is used to note minor
>>        (backwards-compatible) updates:
>> 
>>          "rdapConformance": [ ..., "ext1" ],
>>          ...
>>          "ext1_data": { "conformance": "ext1",
>>                         "field1": ... }
>> 
>>      - Later: update the existing registration for extension "ext1",
>>        noting that a new "ext1_data.conformance" value of "ext1_1" may
>>        be used, implementations of which will return "field2" and
>>        "field3" in the "ext1_data" element.  Implementations in
>>        accordance with the original version of "ext1" continue working
>>        as they do today.  Implementations in accordance with the new
>>        version of "ext1" return responses like so:
>> 
>>          "rdapConformance": [ ..., "ext1" ],
>>          ...
>>          "ext1_data": { "conformance": "ext1_1",
>>                         "field1": ...,
>>                         "field2": ...,
>>                         "field3": ... }
>> 
>>      - The "ext1_data.conformance" field is enough to distinguish the
>>        new version from the original version.  Clients written to work
>>        with the original version do not change, and clients written to
>>        work with the new version are also fine, because they can
>>        inspect at the "conformance" field.
> 
> [ML]  As a  consequence of this scenario,  every RDAP response extension
> should include a fixed property named "conformance".

Not necessarily.  An extension author might decide that the chance of
change is so low that they don't see the need to incorporate this sort
of signalling into their specification.

> If this could be admissible for response extensions defined in the RDAP
> context, it couldn't fit for external specifications.
> 
> See JSContact for example. VCard itself is an external specification used
> for defining an RDAP response.

Wrapping the external specification object inside another object that
provides the version/conformance information for that wrapped object
seems like an adequate way to deal with this concern.

>> Backwards-incompatible changes continue to require a new top-level
>> "rdapConformance" value and a transition process where both
>> "ext1_data" and "ext2_data" (or their equivalents) are returned, per
>> the example you gave above.
> 
> [ML]  Both Scenario A and B seem to me requiring some additional effort from
> implementers. Can't understand why we should make more difficult what could
> be managed straightforwardly.
> 
> Think we should accomplish the solution requiring the lowest (to zero)
> implementation effort for both client and server. If this means that some
> RFCs must be corrected to clarify the extension mechanism and make it more
> straightforward, we should do it.

There are a few considerations here:

 - There appears to be general consensus that clarification of the
   extension mechanism in RDAP is required.  At one end, that
   clarification could restate the 'strict' model more clearly.  At
   the other, it might be something like James' "Approach C", from
   https://mailarchive.ietf.org/arch/msg/regext/e7hZPYUoD3GmjdGiC444TmrCsjg/.
 - The closer that clarification is to the current 'strict' model, the
   better (IMHO), since that will help to minimise change in existing
   RDAP clients.
 - If people are noting apparent limitations of the 'strict' model,
   then demonstrating that the limitations may not be so serious as
   initially thought may help with keeping something like the 'strict'
   model in place.

Of course, the consensus may be that more extensive change is
worthwhile notwithstanding that the 'strict' model can be made to
accommodate the various requirements that people are raising here.
However, I think it's useful all the same to talk about what can and
can't be done with that model today.

(The above is also relevant to the question of how the existing
documents should be interpreted, separately from a potential
clarifying document.  For example, if the 'strict' model meant that
backwards-compatible changes were as onerous as you posited, that
would be an argument in favour of an alternative reading.)

>> Again, my concern here is not to say that the above are ideal
>> mechanisms for versioning or similar, but simply to note that the
>> existing strict reading is not so fundamentally problematic that it
>> necessitates an alternative reading/approach.
> 
> [ML]  Maybe it isn't problematic but it seems a bit cumbersome to me.
> 
>>> A possible inelegant and misleading workaround would be to treat any new
>>> version as a new extension like in the following:
>>> 
>>> {
>>> 
>>> "rdapConformance" [ ....., "ext1", "newfield1"],
>>> 
>>> ...
>>> "ext1_data" : {
>>>                            ...
>>>                           "newfiled1" : { ... }
>>>                          },
>>> ....
>>> 
>>> }
>> 
>> If "newfield1" were named e.g. "ext1_1", I'm not sure that it would be
>> particularly misleading, at least on a current strict reading of the
>> text.
> 
> [ML] The example was based on the assumption that there should be a tight
> coupling between the name prefix of a new response extension and the
> rdapConformance value signaling the support for that extension.

Fair enough.  Scenario A is still an adequate way of dealing with the
concern here, I think.

>> It's probably fair to say that it's inelegant, but I think the
>> key thing about this contention between a strict reading and some
>> other potential reading is whether the strict reading is so
>> problematic that some other reading must have been intended, and I
>> don't think this (arguable) inelegance is so serious that that is the
>> case here.
> 
> [ML] The strict reading results in operational drawbacks that make the
> implementers' work harder.
> 
>>> Stripping the version information from the extension prefix (Approach B or
>>> C)  would simplify the management of this case:
>>> 
>>> {
>>> 
>>> "rdapConformance": [ ....., "ext_1_1"], // (or "ext_level_1_1") to be
>>> determined if the RDAP server should include both "ext_1" and "ext_1_1" at
>>> least for a period of time
>>> 
>>> ...
>>> "ext_data" : {
>>>                            ...
>>>                           "newfiled" : { ... }
>>>                          },
>>> ....
>>> 
>>> }
>>> 
>>> 
>>> Definitively, no breakages in the REST API contract, no deprecation process
>>> to be managed by the server, no extra effort for both client and server
>>> owing to the fact that JSON is schemeless and, by default, JSON libraries
>>> don't block the deserialization of an object including an unknown property.
>> 
>> The above example would require the extension author to document their
>> intent to occupy a namespace of sorts in the extension registry,
>> because otherwise a client written to operate against "ext_1" will not
>> know what to do when it encounters "ext_1_1" (i.e. it won't be
>> possible to make backwards-compatible changes).
> 
> [ML] Every change on a stable version implies the change needs to be
> documented and this is should be done regardless the approach defined to
> extend RDAP.
> 
> The presence of a minor version in the rdapConformance array would signal
> the clients that a non breaking change have been introduced either in the
> request or in the response.
> 
> Client implementers could take all their time to gather information about
> that change and then update their client being confident that, in the
> meantime, it would keep working.
> 
>> The idea of an
>> extension declaring ownership of a namespace is at odds with the text
>> in 9083 about extensions being identified by "a unique string literal
>> value registered in the IANA RDAP Extensions registry", too.  Given
>> that, I'm not sure that it would be open to take this approach with
>> the text as it stands.
> 
> [ML] I intended that you, like some others in this WG, agreed that current
> RFCs should be corrected to clarify the argument.
> 
> The "conformance" property itself should be defined and such a definition
> should be harmonized with the current RFCs.
> 
> If we agree that the RDAP extension mechanism must be reviewed, let's define
> a new one that doesn't introduce ambiguities, drawbacks and is the easiest
> to be implemented.

Per your comments, I was concerned to note that this proposed approach
would not work as against the current text, rather than that it was a
bad idea in the context of a broader rethinking of extensions in RDAP.

>>> A similar example is about adding a new optional query parameter to an URI
>>> path defined by a request extension.
>> 
>> I think scenario B that I set out previously (i.e. embed further
>> versioning information in the extension response) would work for this
>> aspect as well.
> 
> [ML] Sorry but I didn't catch it. Can you clarify how the presence of a new
> optional query parameter would be signaled in the response?

To work through another scenario:

 - Initial: register extension "ext1", which notes that the "help"
   response will include an additional "ext1_version" element
   describing the exact version of the specification being used
   (initially "1.0").  The extension also supports a new path,
   "/ext1_query".
 - Later: update the existing registration for extension "ext1",
   noting that "ext1_version" is now "2.0", and that the "/ext1_query"
   path now supports the new optional parameter.
 - Clients written to work with the original version do not change.
   Clients written to work with the new version can inspect the
   "ext1_version" field in order to determine whether the server
   supports the optional parameter.

> Anyway, don't see much difference between signaling by adding a new
> version in the rdapConformance array and signaling by introducing an
> ad-hoc response field.

The difference is that the ad hoc response field can be used with the
current 'strict' model.

-Tom