Re: [CDNi] [E] Cache management interface | Triggers v2

["Mishra, Sanjay" <[e-mail]>]

Hi Alan - Please see comments embedded below. Also, I suggest a 3-way call
as that will make it easier to sort out any changes that get added into v9
and into the new cache management draft.


>>>>    1.  Items 5 & 6 to generalize playlist to an object list is
>>>>    reasonable and we can add those in the next revision (v10) early Dec.
>>>>
>>>> Great, please let me know what is the best way to add those
>>> In the spirit of iterative changes, I propose to consider three minor
>>> features that may belong in the core CI/T v2 spec, rather than the cache
>>> management extension:
>>> - add trigger execution priority in Trigger.v2, to allow uCDN to
>>> introduce urgent triggers
>>> [NS/SM] While working on the different drafts, we already discussed
>>> capabilities that allows dependencies between triggers.
>>>
>> We concluded that any such capability can be simply defined and added as
>>> an extension, without forcing the dCDN to support it. We suggest to push it
>>> to a separate doc.
>>>
>> [AA] I don't think I agree here.
> I see CI/T triggers as a spec for queue-based task execution and
> extensions as a mechanism to enrich the processing of individual tasks.
> Task prioritization is a core feature of the trigger framework, not an
> individual trigger feature.
> Trigger priority support should likely go beyond individual trigger
> extensions.
> For example, when viewing pending and complete triggers, uCDN may be
> interested to see trigger creation time (mtime), the time when trigger
> execution started, and trigger priority,
> so it can make decisions on new triggers priority.
>
[NS/SM] prioritization requires further definition: how a priority is
specified, how strict is the prioritization, do we interrupt one trigger
from running when a new, higher priority trigger, is introduced? etc.
Beyond definition, adding it as an integral part of the trigger structure,
forces the dCDN to support that.
Note that one of the fundamental concepts in the new structure of the
trigger object is its extendability. We do not want to redefine the trigger
object again and again (creating trigger.v3, trigger.v4...). Therefore the
trigger object should be lean as possible.
We have the specs list to define "on what to run"
We have the extensions list to define "how to run"
Both have registries for new types.
A prioritization algorithm can be simply defined as an extension, and a
future extension can improve it.

- add an optional object list to Trigger Status v2; this is to show to uCDN
>>> how dCDN resolved playlist (aka objectlist) provided to it via
>>> conteMnt.playlist(objectlist) spec
>>> [NS/SM] makes sense, also for url patterns. However, wouldnt it
>>> dramatically enlarge the status object and make the transaction heavier?
>>>
>> Addiiotnally note that the extra complexity the dCDN would have to manage
>>> such a list, especially in a multiple CDN hierarcy topology (e.g. when the
>>> dCDN has 2 further dCDNs that may provide different replies)
>>>
>> Bottom line, we can add it to the respones structure, specifying that it
>>> should be included only if the "status request" specify so via a flag. However,
>>> due to the complexity and requirement for concise definitions, maybe to
>>> add such feature we should add it in a separate doc, with a designated API
>>> endpoint.
>>>
>>
>>
> [AA] Yes, it might. For that reason the proposal is to allow dCDN to
> specify such object list via URL to dCDN-side resource, using MI.Link
> mechanism.
> In this way, there may be no need to have a flag in the status request,
> because uCDN may choose not to retrieve the object list if it's not
> interested in it.
> I am not sure what the purpose of the designated API endpoint would be.
> To make sure, all these proposals have already been written up in the SVTA
> spec draft.
>
 [NS/SM] +1 for having it in another link.
The "designated endpoint" is just a predefined link that we do not need to
list as part of the status object.
e.g. /Triggers/0/targets-list

On Fri, Nov 3, 2023 at 3:42 PM Alan Arolovitch <[e-mail]> wrote:

> Sanjay, Nir,
> Thanks for the feedback, please find my responses embedded below
>
> On Fri, Nov 3, 2023 at 5:11 AM Mishra, Sanjay <[e-mail]>
> wrote:
>
>> Hi Alan - Nir and discussed your comments and please see our further
>> clarifications to your comments
>>
>>
>>>>    1. For items 1 to 3 are new extensions you are proposing and those
>>>>    may be just easy to define in a new draft.
>>>>
>>>> Sounds good, I'll work on the draft once we finalize the cache
>>> management spec at SVTA.
>>>
>>>
>>>>    1. For #4, this appears a heavy lift in the current draft as some
>>>>    of the capabilities such as duration and recurrence and the RFC7265 brings
>>>>    added complexity for dCDN and some dCDN may not support. It may be best to
>>>>    leave Timepolicy as is defined in v2 but JCAL policy can be added in the
>>>>    new draft.
>>>>
>>>> I can see that JCAL/iCalendar support can be somewhat of a lift, even
>>> though there's a number of JCAL libraries out there.
>>> At the same time, the TimePolicy extension in its current form is a
>>> partial ad-hoc scheduling implementation, and should be obsoleted by
>>> RFC7265,
>>> which is a well-established IETF standard with implementation base.
>>> Do we want to release CI/T v2 that is obsolete at the time of release?
>>> [NS/SM] We do not think TimePolicy would be obsoleted. Let the dcdn
>>> choose what it supports and publish via the FCI (as already defined in the
>>> draft)
>>>
>>
> [AA]  If so, and we think that TimePolicy in its current form has value, I
> propose we break out the JCAL support into a new extension SchedulePolicy,
> and have it specified together in the same draft with other extensions
> coming out of the SVTA cache management spec
>
>>
>>
>>> If dCDN doesn't support some advanced JCAL scheduling features like
>>> recurrence, there's ample room to manage that using eextension error code
>>> in Error.V2
>>>
>>> [NS/SM] This should be managed by a capability object - such a solution
>>> would be cleaner and mor sustaninable than emitting an error, which is too
>>> late.
>>>
>> We support a capability object to specify which spec can be used with
>>> which extension. However, the usecase of complex extension is not covered
>>> by such a general capability.
>>>
>> We should probably need to define for JCAL (and every complex extension)
>>> an additional capability object specifying what part of JCAL it is
>>> supported.
>>> I think that both the new JCAL extension and its capability object
>>> should be pushed to a separate doc.
>>>
>> [AA] I am okay with a capability object specifying support per extension.
> I don't think it makes sense to specify subsets of extension functionality,
> as we would be guessing
> what part of the extension future implementations may choose to opt in or
> out of.
> In my mind, this is exactly the purpose of having the eextension error
> code in Error.V2 object, to allow extension support negotiation without
> prescribing support specifics.
>
>
>>>>    1.  Items 5 & 6 to generalize playlist to an object list is
>>>>    reasonable and we can add those in the next revision (v10) early Dec.
>>>>
>>>> Great, please let me know what is the best way to add those
>>> In the spirit of iterative changes, I propose to consider three minor
>>> features that may belong in the core CI/T v2 spec, rather than the cache
>>> management extension:
>>> - add trigger execution priority in Trigger.v2, to allow uCDN to
>>> introduce urgent triggers
>>> [NS/SM] While working on the different drafts, we already discussed
>>> capabilities that allows dependencies between triggers.
>>>
>> We concluded that any such capability can be simply defined and added as
>>> an extension, without forcing the dCDN to support it. We suggest to push it
>>> to a separate doc.
>>>
>> [AA] I don't think I agree here.
> I see CI/T triggers as a spec for queue-based task execution and
> extensions as a mechanism to enrich the processing of individual tasks.
> Task prioritization is a core feature of the trigger framework, not an
> individual trigger feature.
> Trigger priority support should likely go beyond individual trigger
> extensions.
> For example, when viewing pending and complete triggers, uCDN may be
> interested to see trigger creation time (mtime), the time when trigger
> execution started, and trigger priority,
> so it can make decisions on new triggers priority.
>
>>
>>
>>> - add an optional object list to Trigger Status v2; this is to show to
>>> uCDN how dCDN resolved playlist (aka objectlist) provided to it via
>>> conteMnt.playlist(objectlist) spec
>>> [NS/SM] makes sense, also for url patterns. However, wouldnt it
>>> dramatically enlarge the status object and make the transaction heavier?
>>>
>> Addiiotnally note that the extra complexity the dCDN would have to manage
>>> such a list, especially in a multiple CDN hierarcy topology (e.g. when the
>>> dCDN has 2 further dCDNs that may provide different replies)
>>>
>> Bottom line, we can add it to the respones structure, specifying that it
>>> should be included only if the "status request" specify so via a flag. However,
>>> due to the complexity and requirement for concise definitions, maybe to
>>> add such feature we should add it in a separate doc, with a designated API
>>> endpoint.
>>>
>>
>>
> [AA] Yes, it might. For that reason the proposal is to allow dCDN to
> specify such object list via URL to dCDN-side resource, using MI.Link
> mechanism.
> In this way, there may be no need to have a flag in the status request,
> because uCDN may choose not to retrieve the object list if it's not
> interested in it.
> I am not sure what the purpose of the designated API endpoint would be.
> To make sure, all these proposals have already been written up in the SVTA
> spec draft.
>
>>
>>>
>> - add an optional object list to Error.v2
>>> [NS/SM] Again. It might be too heavy and complex.
>>>
>> A full description may create an [object * error * cdn-heirarchy] matrix,
>>> which may be too much.
>>>
>> Lets decide how to proceed after resolving which option we agree to take
>>> wrt the above (status object) question.
>>>
>> [AA] As above, the way out of burdening the response is by supplying the
> list via an external URL.
>
>>
>> - update trigger spec objects that use URLs (which is all trigger specs
>>> except for CCID) to add a field that indicates the type of URL to use -
>>> published URL vs. cache key URL
>>>
>>
>>> If we do that, we can have a clean break between the core CI/T v2 draft
>>> on hand, and 2 new drafts to be submitted later - cache management trigger
>>> extensions and
>>> cache management interface.
>>> Also, this would be a higher priority than adding JCAL support to v10.
>>>
>>
>> [NS/SM] We see 2 options to support that, and need to think about it.
>>>
>> Let's take the below example:
>>
>>     {
>>          "trigger-subject": "content",
>>          "generic-trigger-spec-type": "content-playlist",
>>          "generic-trigger-spec-value": {
>>              "playlist": "https://www.example.com/hls/title/index.m3u8 <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.example.com_hls_title_index.m3u8&d=DwMFaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=XniVbishGiO2Ao9hKqSc-hTVIWCi3T-x6GdHR4ZTgoM&m=971EsgFLP9ZLZjNqyHBn_eP79-nLrsreGAuMd4CdJ1bGkZZhLGpP43_nqnMc4Fh-&s=WNpDRf7l_aoJVrQzvMwJZXKtjzmEmn83XzqmqILOIaA&e=>",
>>              "media-protocol": "hls"
>>           }
>>        }
>>
>>
>>
>>> Option A. We already have a "trigger-subject" proprety ("metadata" /
>>> "content"). Lets enrich the list of subjects to support the separation of
>>> URLs type, or add an additional field, e.g. 'trigger-subject-variant"
>>>
>>
>>     {
>>          "trigger-subject": "content",
>>
>>           "trigger-subject-variant": "published-url",
>>
>> "generic-trigger-spec-type": "content-playlist",
>> "generic-trigger-spec-value": { "playlist": "
>> https://www.example.com/hls/title/index.m3u8
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.example.com_hls_title_index.m3u8&d=DwMFaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=XniVbishGiO2Ao9hKqSc-hTVIWCi3T-x6GdHR4ZTgoM&m=971EsgFLP9ZLZjNqyHBn_eP79-nLrsreGAuMd4CdJ1bGkZZhLGpP43_nqnMc4Fh-&s=WNpDRf7l_aoJVrQzvMwJZXKtjzmEmn83XzqmqILOIaA&e=>",
>> "media-protocol": "hls"
>>
>>          }
>>     }
>>
>>
>> This approach would mean changes in the capability objects.
>> We currently list the supported (subject, spec, extension) tuples.
>> Instead we would now have (subject, variants, spec, extension) tuples.
>>
>> Option B. Define it within the different spec value objects, e.g. with a
>> "url-type" field
>>     {
>>
>>          "trigger-subject": "content",
>>          "generic-trigger-spec-type": "content-playlist",
>>          "generic-trigger-spec-value": {
>>              "playlist": "https://www.example.com/hls/title/index.m3u8 <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.example.com_hls_title_index.m3u8&d=DwMFaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=XniVbishGiO2Ao9hKqSc-hTVIWCi3T-x6GdHR4ZTgoM&m=971EsgFLP9ZLZjNqyHBn_eP79-nLrsreGAuMd4CdJ1bGkZZhLGpP43_nqnMc4Fh-&s=WNpDRf7l_aoJVrQzvMwJZXKtjzmEmn83XzqmqILOIaA&e=>",
>>
>>              "url-type": "published",             "media-protocol": "hls"
>>           }
>>        }
>>
>> In this case no capability tuple would be changed. However, we would not
>> be able to specify whether the different url types are supported or not.
>>
>> Any preference from your side?
>>
> [AA] I prefer Option B. The URL type is an instruction on how to interpret
> the spec, along the lines of "media-protocol".
> If there's a need to publish an individual capability around this, it may
> be added just like the CI/T protocol capability list (section 9.3).
>
>
>>
>>>
>>>
>>>    1. For #7, can you provide more information as to what changes to
>>>    error codes you are considering?
>>>
>>> The only error code that we have for now is etimeout, to support
>>> TimePolicy
>>> [SM/NS] Adding desginated error codes whenever an extestion is defined,
>>> would not scale. There is an "eextension" error code that should be
>>> used for every extension related error. Furthermore, the error object
>>> includes a "description" field that can be used - so extension specific
>>> error information can be placed in the "description" field.
>>>
>> When defining a new extension (like a TimePolicy), that may have specific
>>> erros (e.g. "timeout"), we should also define the required format of the
>>> error description to allow parsing by the uCDN. So the error would be
>>> of type "eextension" and the description would be
>>>
>> "{subtype: timeout}". Alternatively, we can consider adding a general
>>> "description-dict" field to the error and utilze it in the same manner
>>> instead (having a per extension, predefined structre)
>>>
>> [AA] etimeout is different from eextension. The latter is defined now as
> an error in parsing the extension (e.g. malformed extension syntax), or
> lack of support for the extension.
> etimeout communicates back to uCDN that dCDN supports the extension and
> parsed it correctly, however, couldn't execute the trigger within policy
> constraints specified in the extension.
> This has broader use in the extension subsystem than just time-out. It is
> different from ereject, which indicates that the trigger was not picked up
> for execution.
>
>>
>> On Mon, Oct 30, 2023 at 10:22 AM Alan Arolovitch <[e-mail]> wrote:
>>
>>> Sanjay, Nir,
>>> Thanks, see my comments below
>>>
>>> Best regards
>>> Alan
>>>
>>> On Sun, Oct 29, 2023 at 5:09 PM Mishra, Sanjay <
>>> [e-mail]> wrote:
>>>
>>>> Hi Alan - Nir and I discussed your email below, and recommend the
>>>> following:
>>>>
>>>>
>>>>    1. For items 1 to 3 are new extensions you are proposing and those
>>>>    may be just easy to define in a new draft.
>>>>
>>>> Sounds good, I'll work on the draft once we finalize the cache
>>> management spec at SVTA.
>>>
>>>
>>>>    1. For #4, this appears a heavy lift in the current draft as some
>>>>    of the capabilities such as duration and recurrence and the RFC7265 brings
>>>>    added complexity for dCDN and some dCDN may not support. It may be best to
>>>>    leave Timepolicy as is defined in v2 but JCAL policy can be added in the
>>>>    new draft.
>>>>
>>>> I can see that JCAL/iCalendar support can be somewhat of a lift, even
>>> though there's a number of JCAL libraries out there.
>>> At the same time, the TimePolicy extension in its current form is a
>>> partial ad-hoc scheduling implementation, and should be obsoleted by
>>> RFC7265,
>>> which is a well-established IETF standard with implementation base.
>>> Do we want to release CI/T v2 that is obsolete at the time of release?
>>> If dCDN doesn't support some advanced JCAL scheduling features like
>>> recurrence, there's ample room to manage that using eextension error code
>>> in Error.V2
>>>
>>>>
>>>>    1.  Items 5 & 6 to generalize playlist to an object list is
>>>>    reasonable and we can add those in the next revision (v10) early Dec.
>>>>
>>>> Great, please let me know what is the best way to add those
>>> In the spirit of iterative changes, I propose to consider three minor
>>> features that may belong in the core CI/T v2 spec, rather than the cache
>>> management extension:
>>> - add trigger execution priority in Trigger.v2, to allow uCDN to
>>> introduce urgent triggers
>>> - add an optional object list to Trigger Status v2; this is to show to
>>> uCDN how dCDN resolved playlist (aka objectlist) provided to it via
>>> content.playlist(objectlist) spec
>>> - add an optional object list to Error.v2
>>> - update trigger spec objects that use URLs (which is all trigger specs
>>> except for CCID) to add a field that indicates the type of URL to use -
>>> published URL vs. cache key URL
>>>
>>> If we do that, we can have a clean break between the core CI/T v2 draft
>>> on hand, and 2 new drafts to be submitted later - cache management trigger
>>> extensions and
>>> cache management interface.
>>> Also, this would be a higher priority than adding JCAL support to v10.
>>>
>>>    1. For #7, can you provide more information as to what changes to
>>>    error codes you are considering?
>>>
>>> The only error code that we have for now is etimeout, to support
>>> TimePolicy
>>>
>>>> Thanks
>>>> Sanjay/NIr
>>>>
>>>> On Thu, Oct 26, 2023 at 1:26 PM Alan Arolovitch <[e-mail]> wrote:
>>>>
>>>>>
>>>>> Hello
>>>>>
>>>>> I'm leading the cache management interface project at SVTA. The scope
>>>>> is to define a full-fledged CDNi-compliant cache management interface as
>>>>> part of the open caching architecture
>>>>> We are heading towards completion of the initial spec draft, and I
>>>>> would like to share my thoughts on how to best bring this work into CDNi.
>>>>> The current high-level scope includes:
>>>>> - Trigger-based cache data operations that rely on and extend CDNi
>>>>> triggers v2 as currently defined in CDNI CI/T 2nd edition
>>>>> - Additional CDNi metadata objects allowing uCDN to assign cache
>>>>> objects in configuration hierarchy to cache buckets; tag cache objects for
>>>>> use in triggers via content.ccid trigger type; indicate cache object
>>>>> priority
>>>>> - Cache bucket management API
>>>>>
>>>>> The current thinking is to split the work into two documents - one
>>>>> that specifies extensions to triggers v2, and the other that pertains to
>>>>> cache management specific metadata and related API.
>>>>>
>>>>> If so, would it make sense to incorporate the triggers v2 changes into
>>>>> the current draft - all or some of them, or introduce them as a separate
>>>>> document?
>>>>>
>>>>> The proposed triggers v2 change scope is as follows:
>>>>> 1. New trigger extension PrepositionPolicy, per paragraph 8 of CDNI
>>>>> CI/T v2, governing prepositioning policies like retry, patrial retrieval,
>>>>> preposition methods, concurrency and bandwidth, and use of CDNi metadata
>>>>> 2. New trigger extension PurgePolicy, governing purge semantics
>>>>> 3. New trigger extension CommonPolicy, governing trigger execution
>>>>> priority and type of URLs referenced in triggers (published vs. cache-key
>>>>> URLs)
>>>>> 4. Changes to trigger extension TimePolicy (8.2 in CDNi CI/T v2),
>>>>> adding trigger execution scheduling, based on JSON-based iCAL [RFC7265],
>>>>> allowing specifying start, stop, recurrence, duration, timezone etc.
>>>>> 5. Changes of new content.playlist trigger spec(7.5.1 in CDNi CI/T v2)
>>>>> to content.objectlist spec, in conjunction of introducing new types of
>>>>> object list, in addition to hls, dash and mss types per 7.5.2 in CDNi CI/T
>>>>> v2, allowing specifying lists of objects for trigger in JSON-encoded and
>>>>> plain text formats
>>>>> 6. Changes to Trigger Status Resource v2 (6.1.3 in CDNi CI/T v2) to
>>>>> specify objectlists that were derived from content.objectlist triigger spec
>>>>> 7. Changes to error codes in Error v2 to support the new extension
>>>>> functionality
>>>>>
>>>>> One possible option is to incorporate changes to existing objects
>>>>> (items 4-7) in the current draft, and defer the new trigger extensions
>>>>> (1-3) to a separate document.
>>>>>
>>>>> Sanjay, Kevin,
>>>>> I would appreciate some time in the open mic section in Prague to talk
>>>>> about this
>>>>>
>>>>> Kind regards
>>>>> Alan
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> CDNi mailing list
>>>>> [e-mail]
>>>>>
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_cdni&d=DwIGaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=XniVbishGiO2Ao9hKqSc-hTVIWCi3T-x6GdHR4ZTgoM&m=klLmU7WL8P8gyd8m8z7YB4NCt4EbVkWgNY8xAAHx_jHe9zZVF9x85ykXTw7n3KgG&s=9VNsXLtJaYQ0fxZWKYHdXOx9VLDAIzhrpd1OxV4QrR8&e=
>>>>>
>>>>
>>>
>
> --
> Alan Arolovitch
> CEO, 2You IO
> *E* [e-mail] | *M* +1.617.8939400 <(617)%20893-9400> | *W* www.2you.io
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.2you.io_&d=DwMFaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=XniVbishGiO2Ao9hKqSc-hTVIWCi3T-x6GdHR4ZTgoM&m=c-5SyPLTVB1nIyBEtOXxvVcxPojbcaHEQc17TO-ElGhM-EWAiIsBwCwxkn3mp0bb&s=AdXGBx_lSJ5IRkEx_vi95QKoWPQJlzL8vYt1vpXEGFE&e=>
>