Re: [Json] JData: A general-purpose data storage and interchange format - Draft 1

[Qianqian Fang <[e-mail]>]

FYI, after a list of deliberated updates, I felt comfortable to tag the 
"Draft-1" of the JData specification.

https://github.com/fangq/jdata/blob/Draft_1/JData_specification.md


The major changes include

1. switch from column-major to row-major order to serialize N-D arrays 
(in both annotated and direct form)
2. define _ArrayData_ as 2-D arrays for better grouping of similar data 
types for space-efficiency
3. add support for generic byte-streams, non-string keyed maps, and 
weighted graphs
4. fix the incorrect UBJSON extended syntax for N-D array storage


I also noticed recently that the MATLAB Production Server (mps) included 
in MATLAB R2018a or later
had adopted a very similar data annotation schemes like in 
JSONLab/JData: it uses "mwsize",
"mwtype", "mwdata" instead of "_ArraySize_", "_ArrayType_" and 
"_ArrayData_".  The serialization
for complex arrays is also very similarly defined.

Eventually, all software that intends to handle complex arrays/data 
structures will have to consider
the additional serialization schemes for complex data, which I believe 
JData will be able to make this
process easier.


Qianqian


On 5/13/19 7:52 PM, Qianqian Fang wrote:
>
> Dear list,
>
> (I am new to this mailing list, apologize if this is not the right 
> place to post proposals of new JSON-based specifications - in that 
> case, I am appreciated if you can point me to the right direction).
>
> I am a researcher/professor working in a university. A big part of my 
> work, aside from teaching, involves writing computing software and 
> processing medical image data. Over the past 10 years, I gradually 
> migrated the software I wrote, most of them are open-source, some 
> funded by the NIH, to use JSON as the input/output - I really love 
> this format because it is human readable, easy to manipulate, compact, 
> with parsers widely available.
>
> In 2011, I wrote a JSON encoder/decoder MATLAB toolbox 
> <https://www.mathworks.com/matlabcentral/fileexchange/33381-jsonlab-a-toolbox-to-encode-decode-json-files>, 
> called JSONLab <https://github.com/fangq/jsonlab>, and the toolbox has 
> grown a small user community since. In 2013, I added support for 
> UBJSON <http://ubjson.org> (http://ubjson.org), a simple binary JSON 
> format, into my toolbox. Around 2015, I felt strongly that a 
> combination of text and binary JSON is well capable in handling a wide 
> variety of scientific data that I, and many of my colleagues, handle 
> on a daily basis. Compared to the more "advanced" and "feature-rich" 
> data formats such as HDF5, CDF and NetCDF, JSON/UBJSON has clear 
> advantage of being so simple, excellently readable and requiring much 
> low programming overhead to implement. Many other less complicated but 
> still somewhat "opaque" imaging data formats such as DICOM, Analyze7.5 
> and NifTi, can also benefit from a more human-readable version if one 
> can find a data mapping to JSON/UBJSON.
>
> So I started a project <https://github.com/fangq/jdata/commits/master> 
> called "JData" to use JSON constructs to map common data structures, 
> such as N-D arrays, hashes, tables, trees, graphs etc, as the 
> foundation to store/interchange scientific data in a more readable and 
> easy-to-operate fashion (many of these are already supported in 
> JSONLab). After much procrastination, I finally finished the first 
> draft of this specification, and would like your thoughts.
>
> The current draft of the specification can be found here
>
> https://github.com/fangq/jdata/blob/master/JData_specification.md
>
> the repository dedicated to the development and maintenance this 
> specification is
>
> https://github.com/fangq/jdata
>
> The overall idea is to define complex data structures using a set of 
> dedicated "name" tags in JSON/UBJSON without changing the syntax of 
> the format. This makes the generated file JSON/UBJSON compatible and 
> can be readily parsed by most existing parsers.
>
> Currently, this specification supports the following major features:
>
>  1. N-D arrays with and without data compression
>  2. Trees, tables, hashes, graphs, linked lists
>  3. Inline metadata and metadata node append-able to all elements
>  4. Data grouping tags similar to HDF5
>  5. Indexing and query interface
>  6. Referencing and link support
>  7. dual interface text <-> binary
>
> The keyword names were choose to minimize conflict with other JSON 
> features that are under development (such as JSON-LD, JSON schema).
>
> I am sure there are typos and minor issues that I overlooked as an 
> early draft. What I would like to hear from this community are
>
>  1. well, what do you think? is this a project that you would consider
>     useful (in general and for the research community)?
>  2. any major loopholes in the design of the specification? I am new
>     to writing a specification from scratch, I don't want to miss
>     anything important from the start
>  3. if there is a value to continue developing this specification/file
>     format, what is the typical path way for such development? what
>     are the appropriate community/group to discuss ideas and get
>     suggestions?
>
> again, I have no experience writing an RFC or specification from 
> scratch, so, please be gentle, and I appreciate your guidance and 
> pointers.
>
> Qianqian
>