Re: [scim] Charter discussion item: What are the use cases for having a SCIM cursors

["Matt Peterson (mpeterso)" <[e-mail]>]

Phil,

In order to implement index/offset pagination in SCIM on top of an API or database that uses cursor pagination, the SCIM Service Provider must fetch the entire result set (i.e. iterate on the cursor to completion), then store the full result set in order to serve (via SCIM) a specific indexed page.   This resource intensive translating between cursor pagination (the backend protocol/database) and SCIM index pagination (SCIM) was a showstopper for our implementation which uses low cost “serverless” (AWS Lambda, Azure Functions) architecture.

We have implemented “front end” SCIM Service Provider for all of the “back-end” services in this list: https://www.cloud.oneidentity.com/products/connect/connectors and we found that the majority of the web protocols that we were calling on the “back end” use SCIM service provider use cursor pagination.  We would only have been able to support a fraction of these application without implementing the scim-cursor-pagination draft.

I recognize that my use case of a SCIM Service provider with dozens of “backends” is not common.  However, the frequency with which we encounter cursor pagination in existing APIs and protocols seems like an important consideration for the SCIM workgroup.  For example, one does not even need to venture farther than adding a SCIM interface on top of LDAP to encounter the pagination translation problem described above.

As for why SCIM clients need to query SCIM Service Provider to obtain large result sets, here is what we have encountered:


  1.  Constructing and enforcing Application authorization models -  An application (acting as a SCIM client) downloads users/groups from an IdP in order to present views to application administrators that are used to create RBAC, Policy and ACL authorization rules.  Enforcement-time authorization decisions need to be made quickly such that authorization-time calls to SCIM service provider is not viable which is why many applications maintain an “sync’d” application-side cache of users and groups.
  2.  Identity Management and Governance systems – Use a “canonical” identity model where all accounts and groups are represented.  This model is used to create provisioning rules and calculate separation of duty violations, attestations, and approvals etc.   Management-time evaluation of the model needs to be done efficiently without blocking calls to connected systems.

In both 1 and 2 (above) the there is an *initial load* of objects into the SCIM client’s data model and then subsequent use of SCIM to keep client’s data model with changes on SCIM Service Provider.

Pagination is desirable for the initial load of all objects.  However, the subsequent up-to-date maintenance of a client’s copy (cache) of the results is not necessarily a good use case for pagination.  For example, the need to re-request all objects in order to detect a deleted object is something that could be made much more efficient and should be addressed separately from the pagination topic.

I have posted to this mailing list about “Maintaining SCIM client-side cache consistency with SCIMv2”.  See post with this same subject: (https://mailarchive.ietf.org/arch/msg/scim/P5DVTqWLmVKqyvD0dgUszADQZrE/).  There were no public responses to this thread, but I still believe that this is a VERY common pattern.   I did receive a good private response from Phil Hunt who suggested use of “ETags and RFC7232 Http Conditional requests” – an approach definitely worth discussing.

As feedback to our charter draft, I believe that there is enough interest in two distinct areas to merit mention in the charter:


  *   “Pagination of large data sets” - including interest in pagination of multi-valued attributes.  Yes, I think that object pagination and attribute pagination could be considered in the same topic.  Group membership is the primary use case for large multi-valued attributes. I believe the group membership use cases can be addressed with object pagination and best practice filters (see https://mailarchive.ietf.org/arch/msg/scim/oZ3DcI15AOvA519rH5sRtJOJV2A)”.  Another alternative Dale Old’s suggestion this morning, of a “GroupMembers collection” (which would allow use of object pagination for group members and, probably easier detection of group membership changes).
  *   “Maintaining SCIM client-side cache consistency with SCIMv2” (https://mailarchive.ietf.org/arch/msg/scim/P5DVTqWLmVKqyvD0dgUszADQZrE/).  )

--
Matt Peterson
[e-mail]

From: scim <[e-mail]> On Behalf Of Phil Hunt
Sent: Wednesday, July 7, 2021 10:17 AM
To: SCIM WG <[e-mail]>
Subject: [scim] Charter discussion item: What are the use cases for having a SCIM cursors

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

As promised on the call today, this is to follow up and ask for use cases for people who want to use cursors. I think this would be helpful to understand in order to drive to a common purpose and solution.

For example, today I think I heard:
*  Some clients want to confirm resources have been deleted.   Why does this come about? SCIM already returns a definitive success/fail upong HTTP DELETE. Is it a case of co-ordination between multiple clients?

* Is it used to re-concile (e.g. meta-directory style)  between disparately managed systems periodically?

* Others reasons?

Can the use of cursors be confined to “specialized” clients where cursor might be consider a special “priviledge”.  IOW….would you allow javascript UI components to use cursors against your SCIM server?

Phil Hunt
@independentid
[e-mail]<mailto:[e-mail]>