[tsvwg] Re: Question on TSVWG Scope and Feedback on draft-shenzhihong-dacp-00

["[e-mail]" <[e-mail]>]

Hi Sophie,
 
Thank you for your thorough and insightful feedback on the DACP draft. We're very encouraged by your positive comments and appreciate your support for discussing it within the TSVWG.
 
Here are our thoughts on your excellent points, all of which will help improve the next version of the draft.
 
A. Unstructured Data Framing
blob Column: Your understanding is correct. The blob column is a pass-through for data that isn't easily structured (like images), and it relies on lazy loading—the content is only streamed upon explicit request. To your point, making the name mandatory is too restrictive. In our revision, we will make the inclusion of a binary content column OPTIONAL.
FileListSchema: You are absolutely right. We will define a standard, baseline FileListSchema in the next draft to ensure maximum compatibility.
 
B. RPC Mapping & DoS Concerns
This is an important security consideration. To mitigate the DoS risk from long transformation chains, we will update the "Security Considerations" section to detail required server-side mitigations, such as enforcing transformation chain maxium length and per-user resource quotas.
 
C. Provenance and Integrity
This is an excellent suggestion. To strengthen the audit trail, we will specify an OPTIONAL mechanism for integrity, based on your proposal of using cryptographic chaining or signatures.
 
Thank you again for this high-quality feedback. Your suggestions are invaluable as we work towards the -01 version.
 
Best regards,
Xiaojie Zhu
(on behalf of the authors)


[e-mail]
 
From: Sophie Harper
Date: 2025-10-11 23:40
To: [e-mail]
CC: tsvwg; bluejoe; zjcheng
Subject: [tsvwg] Re: Question on TSVWG Scope and Feedback on draft-shenzhihong-dacp-00
Dear Authors and TSVWG Chairs/Members,
I am writing as an individual interested in high-performance data protocols to offer some initial feedback on draft-shenzhihong-dacp-00.
We highly appreciate the effort in proposing the Data Access and Collaboration Protocol (DACP). The protocol is an elegant solution to the challenges of modern distributed computing by providing "secure, high-performance, and auditable streaming of data across distributed systems."
The choice to build DACP "directly on top of Apache Arrow Flight" is excellent, immediately leveraging columnar data transfer and minimizing "high serialization overhead." The introduction of the Streaming DataFrame (SDF) as a "standardized data unit" with Immutability and Deferred Execution principles is a powerful, high-level abstraction that will greatly benefit the scientific community.
Regarding your primary question on TSVWG scope, while DACP's core functionality is largely Application Layer (defining the SDF model, operations, and provenance), its entire existence is driven by the need for "low-latency data access" and "high-performance stream-framed data transport." Therefore, I believe the TSVWG would be the appropriate group to review the transport behavior and ensure robust performance and interaction with underlying transport protocols.
Here are a few specific points for consideration:
A. Clarity on Unstructured Data Framing
Section 7.1.2 defines the "Framing Files and File Collections," which maps a list of files into an SDF where the file's raw content is in a "special column, typically named blob and of type Binary."
To ensure interoperability, it would be valuable to clarify:
1.Is the column name blob RECOMMENDED or REQUIRED?
2.Should the schema for this specific File List SDF (referred to as a FileListSchema) be standardized within the protocol for maximum client/server compatibility?
B. Implications of RPC Mapping
Section 5.3 states that Transformation operations "MUST NOT trigger network requests," while Actions trigger server-side execution via DoGet, DoAction, or GetFlightInfo.
This clear separation is crucial for optimization. Could the authors expand on the security or resource allocation implications if a malicious client constructs an extremely long Transformation Chain before executing an Action? Does the server impose any limit on the complexity of the accumulated actions array to prevent Denial of Service (DoS) attacks on compute resources?
C. Provenance and Integrity
Section 9 mentions that the integrity of the provenance trail relies on the security of each hop. Given that "Each intermediate node MUST append its own entry to the provenance trail," have the authors considered a lightweight mechanism (e.g., cryptographic chaining or signature) to allow the final server or client to verify the non-tampering of the intermediate HopInfo entries by untrusted proxies? This could enhance the auditable nature of the trail.
Thank you again for this excellent draft. I look forward to its progression.
Best regards,
Sophie Harper

2025年10月10日 星期五 下午 4:24，[e-mail] <[e-mail]> 来信：
Dear TSVWG Chairs and Members,

We are writing to propose our new Internet-Draft, DACP (Data Access and Collaboration Protocol), for consideration and potential adoption by the Transport Area Working Group (TSVWG).
 
The Internet-Draft is available here:
https://datatracker.ietf.org/doc/draft-shenzhihong-dacp/

What is DACP?
DACP is a communication protocol designed to support cross-node, cross-process data access, primarily for scientific and distributed computing environments. Modern data processing, particularly in scientific and distributed computing, requires unified, low-latency data access across diverse nodes and processes. However, the fragmented and heterogeneous nature of scientific data currently hinders effective data sharing and collaboration. To address this, DACP introduces a high-performance solution built upon Apache Arrow Flight, creating a zero-copy, columnar streaming framework that unifies access to both structured and unstructured data. DACP’s goal is to provide a secure, auditable, and low-latency data channel that finally enables the efficient sharing and collaborative processing of scientific data.

Our primary question for the group is whether the problem that DACP addresses falls within the scope and interest of the TSVWG. We would be very grateful for your expert opinion on this. If you feel this work is not a good fit, any suggestion for a more appropriate working group would be highly appreciated.
 
Thank you for your time and expertise.
 
Best regards,  
Xiaojie Zhu
(on behalf of the authors)



[e-mail]