Re: [scim] Queries on SCIM Cursor-based Pagination Draft

["Matt Peterson (mpeterso)" <[e-mail]>]

Anurada,

Thank you for the feedback on SCIM Cursor-based Pagination Draft.

You asked:  "No explanation on backward traversal (traverse to the previous page result set)"

I agree.  The draft could describe more clearly the use of previousCursor (when supported).   Would the following rewording of the cursor query parameter be clearer?

cursor - To request the next page, pass the nextCursor value from
the current result page. For the previous page, pass the
previousCursor from the current result page. The cursor parameter
SHOULD be omitted for the first request of a paginated query.


You asked: "As per the following paragraph in the introduction, is the cursor-based pagination supported only when all resources are fetched from the server?"

Not only is filtering an OPTIONAL capability,for service providers,
it is also a very common for SCIM clients to not to use a query
filter in order to intentionally to retrieve all resources.
Therefore, pagination of results (more so than filtering) is a
primary scalability mechanism for SCIM service providers.

No, cursor-based pagination should be supported for filtered and unfiltered requests.  This paragraph in the introduction was intended to counter the opinion that high performance pagination would not be needed if clients would constrain queries with filter.   In a future version of this draft, I would omit this paragraph as there is less controversy on this point than I originally thought.

--
Matt

From: scim <[e-mail]> On Behalf Of Anuradha Karunarathna
Sent: Monday, February 7, 2022 9:24 AM
To: [e-mail]
Subject: [scim] Queries on SCIM Cursor-based Pagination Draft

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.


Hi all,


WSO2 Identity Server(IS) is a SCIM service provider which supports SCIM 2.0, and SCIM is the main protocol used in identity management in WSO2-IS.

After seeing performance and functionality improvements that can be achieved with the cursor-based pagination we are planning to implement the cursor-based pagination draft [1] in WSO2 IS. While reading the draft spec we encountered the following concerns. Appreciate your clarification and input on these points.


  1.
  2.
  3.  No explanation on backward traversal (traverse to the previous page result set)
  4.

*

*

*        As per the draft, count and cursor are the URL pagination parameters for cursor-based pagination.

*        (eg: GET /Users?cursor=VZUTiyhEQJ94I&count=10).
*
*

*

*        Also, the definition of the "cursor" attribute is

*        "The

*        string

*         value from the nextCursor attribute

*         from the previous result page. A  cursor parameter SHOULD be  omitted for the first request of the paginated query"
*
*

*

*        If the SCIM service provider supports previousCursor, what should be the API request format to

*         traverse back to the previous page?
*
o

o

o   Suggestions:
o
1.

2.

3.      Instead of "cursor" query param, support two query params as "before" and "after" as similar to

4.       Facebook API[2]
5.
6.

7.

8.      Introduce new query param as "cursorDirection"  to indicate whether the client needs the before/next

9.       set of results from the cursor, and change cursor attribute definition
10.
11.

12.

13.   Only if "count" and "cursor" params are supported, add a prefix to the cursor such as prev_ and

14.    next_ that can be parsed before sending to the DB layer[3].
15.

Get next page results: GET /Users?cursor=base64encode<next_cursor1>&count=10

Get previous page results: GET /Users?cursor=base64encode<pre_cursor2>&count=10

2.
3.
4.      Supporting filtering params and cursor-based pagination params together in the API request
5.


As per the following paragraph in the introduction, is the cursor-based pagination supported only when all resources are fetched from the server?


"  Not only is filtering an OPTIONAL capability,for service providers,

   it is also a very common for SCIM clients to not to use a query

   filter in order to intentionally to retrieve all resources.

   Therefore, pagination of results (more so than filtering) is a

   primary scalability mechanism for SCIM service providers.

"


[1] https://datatracker.ietf.org/doc/html/draft-peterson-scim-cursor-pagination-00<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-peterson-scim-cursor-pagination-00&data=04%7C01%7Cmatt.peterson%40quest.com%7Cb71a260276744b4a0b2908d9ea564b70%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637798478629490490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=UkfdM6WJPc5kO%2FYXW8XEAOMK%2BJo7Yu6Tz84JLWYbTKM%3D&reserved=0>

[2] https://developers.facebook.com/docs/graph-api/results<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdevelopers.facebook.com%2Fdocs%2Fgraph-api%2Fresults&data=04%7C01%7Cmatt.peterson%40quest.com%7Cb71a260276744b4a0b2908d9ea564b70%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637798478629490490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=w8VD1YVMi3ZY9ugyo2OwqTILNfcYnR7GPc39oQj5ogo%3D&reserved=0>
[3]  https://medium.com/swlh/how-to-implement-cursor-pagination-like-a-pro-513140b65f32<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Fswlh%2Fhow-to-implement-cursor-pagination-like-a-pro-513140b65f32&data=04%7C01%7Cmatt.peterson%40quest.com%7Cb71a260276744b4a0b2908d9ea564b70%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637798478629490490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=P%2B8mICK6yoZpnWCcgIB%2FwtSN1ntI%2B8%2Ba64YRi4baVYQ%3D&reserved=0>


Regards,
Anuradha