Re: [Json] Proposed minimal change for duplicate names in objects

[Tatu Saloranta <[e-mail]>]

On Sat, Jul 6, 2013 at 9:58 PM, Tim Bray <[e-mail]> wrote:

> I’ll assume you’re right when you say dupe detection has been measured as
> expensive at run time, but I’m thinking that if I were writing a reader I'd
> implement a hash table with a test-and-set method, so I admit I'm surprised
> by the finding.  I think I’d need to see a little more research before I’d
> accept that as a given.  -T
>
>
Point is that no hash table need be created at all. All one has is simple
tokenizer (lexer). No tree is built, just parent stack with information to
match close markers, report error locations. And bit of state information
to validate well-formedness wrt separators.
I agree that if a per-document symbol table/map is maintained, extra cost
would probably be negligible and not worth worrying too much about.

Hash table will typically be used at higher level, if building an object
representation, such as tree model, or native objects: either as structure
to build (tree), or mapping to object fields (name to method/field). At
this level catching duplicates is easier and more efficient to handle, and
I would not be worried about mandating such handling.

So why separation to layers like this? While writing a naive tokenizer is
an easy task, it takes much more effort to create an efficient one. But
there are not many variations there, so once an efficient one exists, it is
easy to use that as a building block, save time and have some confidence
that simple encoding/decoding works correctly and efficiently.
Conversely, higher level components vary a lot regarding functionality.
This is similar to division between low-level SAX or Stax APIs, and
higher-level DOM (tree) and JAXB (data binding) for XML on Java. I think
C#/dotnet has similar separation between pull-parsing and higher-level
abstractions, but it seems absent from most scripting languages.

As to research, I don't know if this --
https://github.com/eishay/jvm-serializers/wiki -- helps, but contrasting
these two entries:

                           create     ser   deser   total   size  +dfl

  json/jackson/manual           62    1137    1519  2656    468   253
  json/jackson/tree             63    2045    2650  4695    485   259

gives difference between manually written data-binding (streaming parser,
hand-written code to match names to fields), and building a HashMap-based
simple tree model (built on streaming parser) as 2-to-1 performance
difference (2656 nsec vs 4695 nsec).
Part of this is with overhead of Maps vs. plain java objects, but not all.

I could prototype with actual detection too (in fact, there is an RFE for
streaming parser to do just that -- and I consider it a legitimate thing to
do as optional settings) and see how much actual overhead a highly-tuned
minimal-state parser will have on Java platform.

-+ Tatu +-


>
> On Sat, Jul 6, 2013 at 9:37 PM, Tatu Saloranta <[e-mail]>wrote;wrote:
>
>> On Sat, Jul 6, 2013 at 7:57 PM, Nico Williams <[e-mail]>wrote;wrote:
>>
>>> On Sat, Jul 6, 2013 at 8:44 PM, Tim Bray <[e-mail]> wrote:
>>> > This feels like a no-brainer to me, but that’s probably because (as
>>> I’ve
>>> > said before) I’m an API guy, and the only use for JSON objects in my
>>> world
>>> > is to transfer a hash table or database record or whatever from here to
>>> > there, and there, and in such a situation dupes can never be useful or
>>> > intended and can only be a symptom of breakage (or, in the JOSE case, a
>>> > symptom of a malicious attack on my crypto).
>>>
>>> I agree.  As a security guy I would prefer if one way or another we
>>> end up with no dup names, but as an "API guy" myself I think of the
>>> streaming parsers (they offer an API after all).  Just say the magic
>>> words: "to hell with minimal state streaming parsers" or perhaps
>>> something to the effect that *some* component of a layered application
>>> MUST reject objects with dup names.  It's either or.  Let's choose.
>>>
>>> I'm happy with "some component of a layered application MUST reject
>>> objects with duplicate names" -- I prefer this to the "no minimal
>>> state streaming parsers" alternative.
>>>
>>> I will assume that in general objects rarely have lots of names, so
>>> that parsers need not keep that much state in order to check for dups.
>>>  Requiring parsers to reject objects with dup names is my second
>>> choice.
>>>
>>
>> Just to make sure: I also do not have any use for duplicates, and
>> consider them flaws in processing. I have never used duplicates for
>> anything, nor find that interesting approach.
>> The only concern really is that of mandating (or not) detection and/or
>> prevention at lowest level components of commonly used processing stacks
>> (low-level push/pull parser, higher-level library or app code that builds
>> full representations), since this is significant cost, based on extensive
>> profiling I have done at this level.
>>
>> Case of application code directly using streaming parser/generators is
>> not nearly as common as that of frameworks using them to produce higher
>> level abstractions.
>> These higher level abstraction (JSON tree representation, binding to
>> native objects) do either report errors such as duplicates, and at very
>> least can detect it and use consistent handling. They can do it much more
>> efficiently than low-level components since they have to build
>> representations that then serve as data structures for detecting/preventing
>> duplicates.
>>
>> Specific concern for me is this: if specification mandates detection
>> and/or prevention for parser, generators, without any mention that 'parser'
>> and 'generator' are logical concepts (thing that reads JSON, thing that
>> writes JSON), I will have lots of users who promptly demand low-level
>> components to force checks.
>> And then I get to spend lots of time discussing why such checks can (and
>> IMO should) still be pushed to higher level of processing. It is amazing
>> how much FUD can be generated based on cursory reading of specifications.
>>
>> This concern extends to suggested "Internet JSON messages" specification.
>>
>> I would like simple suggestion that some component of the processing
>> should/must detect and report duplicates; and prevent producing of same;
>> or, lengthier explanation of what "parser" and "generator" mean (parser is
>> such a horrible misnomer -- there is very very little parsing involved,
>> it's just a lexer, and optional object builder -- but I digress).
>>
>> -+ Tatu +-
>>
>>
>