Re: [scim] Closing adoption call for draft-peterson-scim-cursor-pagination/

[Danny Zollner <[e-mail]>]

Thanks Nancy,

I've recently uploaded draft-ietf-scim-cursor-pagination-00, identical to the version that was up at the time of the call for adoption.

Additionally, @Dean H. Saxe<mailto:[e-mail]> and I have written responses to the concerns raised by Phil Hunt during the call for adoption. Please see below:

DZ = Danny Zollner
DHS = Dean H. Saxe
Asterisked lines  = Phil's comments

* Amount of data transferred which each cycle could incur high data transfer/exfiltration costs and data exposure risks while transferring entire data sets on a repeating basis
[DZ] Concern about cost of implementation is a concern for those implementing. Not all applications have underlying architectures that make this a concern. As this draft is expanding on pagination and offering an additional optional method to use, there is no requirement for applications to implement this if it is not feasible from a cost standpoint. Support on the working group during the call for adoption shows that a number of individuals find the proposed functionality feasible to implement. Data exposure risk is no more than proposed alternatives such as doing GET /<Resource> with all results on one page.
* Currency - how often can an entire data set be downloaded using paging and keep systems close enough in sync.  In the past I had customers paging LDAP lasting over 48 hours to accomplish a sync cycle.
[DZ] There are existing large-scale RESTful APIs with similar pagination capabilities. This draft helps address problems for client/server relationships where high scale (hundreds of thousands/millions of results) need to be retrieved while paginating results due to the lack of feasibility of receiving all results in a single web response. Concerns about how frequently data needs to be retrieved and the associated performance requirements is an implementation concern.

[DHS] Scaling could be an issue, but it is unlikely to be an issue for the majority of use cases. Where high throughput is required, services will have to build to meet that ability or choose different mechanisms. SCIM should not attempt to solve all use cases.

* DoS - Despite some imporovement avoiding indexing, a number of server times will be unable to keep state of an entire data set for extended periods of time (e.g. minutes to hours) unless paging allows unreturned entties to keep changing.  This raises other challenges when using for synchronization.  In my experience this requires storying "copies" of information in some form. This means a relatively few number of clients could exhaust service provider resources very quickly leading to denial of service.  Does paging impose "locking" preventing a service provider from accepting changes while paging in progress?
[DZ] Today some SCIM implementations must do the behavior being highlighted here (keeping data in memory) in order to provide index-based pagination. Underlying data sources for SCIM service providers are varied and for some cursor-based pagination is the preferred solution already. Concerns about how to prevent resource exhaustion, denial of service attacks related to multiple states being held at once, etc. are all implementation concerns, although some commentary/guidance on the topic can be included in a future draft of the cursor-based pagination spec.
[DHS] Strongly consistent read queries are supported with a cursor based pagination out of the box on AWS and Azure based on a quick scan of the docs:
https://learn.microsoft.com/en-us/sql/relational-databases/cursors?view=sql-server-ver16<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fsql%2Frelational-databases%2Fcursors%3Fview%3Dsql-server-ver16&data=05%7C01%7CDanny.Zollner%40microsoft.com%7Cb43fc0b0f0cd477dd71408db0e47213f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638119470909884837%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sI5Dlmnxbbu%2F%2FylqS%2FrAk8r%2FWqW2Coq2oQYMVahrGPs%3D&reserved=0>  https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.Pagination.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Famazondynamodb%2Flatest%2Fdeveloperguide%2FQuery.Pagination.html&data=05%7C01%7CDanny.Zollner%40microsoft.com%7Cb43fc0b0f0cd477dd71408db0e47213f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638119470909884837%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oEqNuL5494DgyvOnhrqsh7U9PLXLimDL%2BOUz5Dknku4%3D&reserved=0> and https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.ReadConsistency<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Famazondynamodb%2Flatest%2Fdeveloperguide%2FQuery.html%23Query.ReadConsistency&data=05%7C01%7CDanny.Zollner%40microsoft.com%7Cb43fc0b0f0cd477dd71408db0e47213f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638119470909884837%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j13n%2Febd89sa%2BUZtxGxQpHF9ERl5I2w3BkPZSLrd7TQ%3D&reserved=0>
* Last IETF meeting presenters indicated this spec requires additional "delta" processing capabilioties
[DZ] This is not the case. If this was previously stated, it was in error. Delta query/change detection functionality complements cursor-based pagination, but both are useful standing alone.
* History:  This was tried before with LDAP VLV where replcation remained out of scope.  Many meta-directory implementations focused on bullk transfer sync tech.

[DHS] I'm curious about this and what happened with VLV?  Why did it fail? What can we learn from it?

* Utility:   There is no way to express information about how information changes or how many changes took place in an interval.
[DZ] This is out of scope for this draft.

[DHS] We welcome further ideas on how to represent this now, and in the future. Maybe this is part of a delta-query capability model?

* Mis-use:  I heard a number of cases where paging was to be used to bootstrap new or failed server nodes.  Is this really a good use?
[DZ] This can be summarized as "retrieval of large result sets efficiently" - as mentioned above. It has been suggested previously that an alternative to cursor-based pagination is to retrieve the entire result set in a single web response. This is not always feasible at high scale across distributed/SaaS systems, while cursor-based pagination can be. Going back to the original summarization of "retrieval of large result sets efficiently" - if retrieving 100Ks/millions of objects in a single web response is not feasible, then yes, using cursor-based pagination to accomplish this is a good use.

[DHS] Alt-view - Is there a reason that this is misuse?  Identifying this as "misuse" is not constructive criticism of the draft.  We shouldn't constrain the specification to only the use cases we envision as "valid". Equally we should not seek to enable all use cases.

* Databse mis-use:  Almost all examples from database vendors refer to paging a "small" set of rows for situations like user interface design

[DHS] The argument seems to be both that this breaks at scale and is not necessary for small scale use cases.  Adopting a cursor-based mechanism does not preclude the use of the existing index-based mechanisms which work for the smaller result sets.  Cursor based pagination can also work in this scenario.  As noted above, support for cursors and strongly consistent reads exist in various database implementations today, so this is unlikely to be a blocker for implementation.


* Simplify:  For the planned use, I would expect it would be much simpler for SCIM Service Providers to allow clients to GET large data sets in exceess of the normal server result set limit.  Clients can download 100s of GB of data to disk very quickly these data.  That data then can hold state while the client processes it.

[DHS]I don't believe this is prevented by the spec today, yet it is not in use.

[DZ] The resource requirements to receive and process a single large response are significant and do not align with the scaling, distributed multi-server/node/worker model that many modern systems utilize which relies on a fleet of smaller servers/nodes/workers that have smaller resource allotments.


Thank you,

Danny Zollner
From: scim <[e-mail]> On Behalf Of Nancy Cam-Winget (ncamwing)
Sent: Wednesday, January 18, 2023 6:14 PM
To: SCIM WG <[e-mail]>
Subject: [EXTERNAL] [scim] Closing adoption call for draft-peterson-scim-cursor-pagination/

Hello SCIMers,

Hope your 2023 is off to a great start!

We have received consensus for the adoption of  draft-peterson-scim-cursor-pagination,  however, we've also had discussions on potential issues
to the applicability of specific use cases as noted by Phil's email.  To that extent, the chairs are calling the approval to move this draft as a working
group document but request that the use cases targeted for this draft are articulated either in a section of this draft or the use cases draft.
Further, there are some points Phil has raised on his (no) adoption comments that we the chairs would like to see responses from the authors (or
others who support the draft adoption).

Authors (e.g. Matt and Danny) please resubmit the draft as draft-ietf-scim-cursor-pagination. ou can also use the
SCIM GitHub https://github.com/ietf-scim-wg<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fietf-scim-wg&data=05%7C01%7Cdanny.zollner%40microsoft.com%7Caf2ff3e0f83c458b11e108daf9b22546%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638096841152926001%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=kownsGzUjyk28Z0DmsL1Dvmm8ULgDBaTQyoCrkaZKOg%3D&reserved=0> so that we can track issues/comments on the draft as well.

Best, Nancy (on behalf of the SCIM chairs)