[Json] JData: A general-purpose data storage and interchange format

[Qianqian Fang <[e-mail]>]

Dear list,

(I am new to this mailing list, apologize if this is not the right place 
to post proposals of new JSON-based specifications - in that case, I am 
appreciated if you can point me to the right direction).

I am a researcher/professor working in a university. A big part of my 
work, aside from teaching, involves writing computing software and 
processing medical image data. Over the past 10 years, I gradually 
migrated the software I wrote, most of them are open-source, some funded 
by the NIH, to use JSON as the input/output - I really love this format 
because it is human readable, easy to manipulate, compact, with parsers 
widely available.

In 2011, I wrote a JSON encoder/decoder MATLAB toolbox 
<https://www.mathworks.com/matlabcentral/fileexchange/33381-jsonlab-a-toolbox-to-encode-decode-json-files>, 
called JSONLab <https://github.com/fangq/jsonlab>, and the toolbox has 
grown a small user community since. In 2013, I added support for UBJSON 
<http://ubjson.org> (http://ubjson.org), a simple binary JSON format, 
into my toolbox. Around 2015, I felt strongly that a combination of text 
and binary JSON is well capable in handling a wide variety of scientific 
data that I, and many of my colleagues, handle on a daily basis. 
Compared to the more "advanced" and "feature-rich" data formats such as 
HDF5, CDF and NetCDF, JSON/UBJSON has clear advantage of being so 
simple, excellently readable and requiring much low programming overhead 
to implement. Many other less complicated but still somewhat "opaque" 
imaging data formats such as DICOM, Analyze7.5 and NifTi, can also 
benefit from a more human-readable version if one can find a data 
mapping to JSON/UBJSON.

So I started a project <https://github.com/fangq/jdata/commits/master> 
called "JData" to use JSON constructs to map common data structures, 
such as N-D arrays, hashes, tables, trees, graphs etc, as the foundation 
to store/interchange scientific data in a more readable and 
easy-to-operate fashion (many of these are already supported in 
JSONLab). After much procrastination, I finally finished the first draft 
of this specification, and would like your thoughts.

The current draft of the specification can be found here

https://github.com/fangq/jdata/blob/master/JData_specification.md

the repository dedicated to the development and maintenance this 
specification is

https://github.com/fangq/jdata

The overall idea is to define complex data structures using a set of 
dedicated "name" tags in JSON/UBJSON without changing the syntax of the 
format. This makes the generated file JSON/UBJSON compatible and can be 
readily parsed by most existing parsers.

Currently, this specification supports the following major features:

 1. N-D arrays with and without data compression
 2. Trees, tables, hashes, graphs, linked lists
 3. Inline metadata and metadata node append-able to all elements
 4. Data grouping tags similar to HDF5
 5. Indexing and query interface
 6. Referencing and link support
 7. dual interface text <-> binary

The keyword names were choose to minimize conflict with other JSON 
features that are under development (such as JSON-LD, JSON schema).

I am sure there are typos and minor issues that I overlooked as an early 
draft. What I would like to hear from this community are

 1. well, what do you think? is this a project that you would consider
    useful (in general and for the research community)?
 2. any major loopholes in the design of the specification? I am new to
    writing a specification from scratch, I don't want to miss anything
    important from the start
 3. if there is a value to continue developing this specification/file
    format, what is the typical path way for such development? what are
    the appropriate community/group to discuss ideas and get suggestions?

again, I have no experience writing an RFC or specification from 
scratch, so, please be gentle, and I appreciate your guidance and pointers.

Qianqian