[Atp] Re: ATP records from arbitrary JSON

[Ryan Barrett <[e-mail]>]

On Wed, Jul 29, 2026 at 2:49 AM bumblefudge von CASA <
[e-mail]> wrote:

>
> Yes, as our slides mentioned, the `$type=unknown` hatch is how many of
> the illicit floats have dodged validation and snuck into the protocol's
> current data plane, but our argument is that this is the wrong tradeoff for
> a global protocol, and that floats deserve first-class treatment rather
> than "void your warranty" status.
>

Sure, of course. I understand the relationship between arbitrary JSON and
float handling, but I interpreted OP here as mainly about the former, not
the latter. I don't think $type: unknown or "void your warranty" arbitrary
JSON are the right ways to do floats long term either. Apologies if I
implied otherwise.




> `type=unknown` embedding of otherwise bog-standard JSON suspends
> validation at the lexicon level and basically signs over the JSON as
> serialized, which is suboptimal for many reasons! Our basic argument is
> that "floats are hard" is an inadequate justification for pushing float
> verification into userspace at the expense of protocol-wide efficiencies of
> scale and stability, when float validation and translation are essentially
> solved problems.
>
> The "suspend validation, here are some bytes" option makes sense for
> things that genuinely break the limitations of a data language expressed in
> JSON Schema or protocol-specific constraints (like Bryan's examples of
> absolute size limits, or top-level versus embedded requirements), or things
> that are inherently unstable in JSON itself.  Our argument is that floats
> should be taken out of that category and integrated into the lexicon type
> system so that floats can get the same "first-class treatment" other JSON
> Numbers have grown accustomed to.
>
> Bryan wrote:
> > One motivating question is whether developers should expect to be able
> to embed existing JSON data cleanly in to atproto records. Can you just
> take arbitrary DID documents and stuff them in there? GeoJSON? JSON-LD?
>
> This feels more like a reductio ad absurdum than devilish advocacy, as we
> aren't arguing that 100% of the JSON on today's web should be embeddable,
> just that the vast majority of scientific data would instantly become
> embeddable if floats were handled natively. I agree that basic constraints
> on what JSON can be embedded are fair for any data/schema language-- size
> constraints and top-level constraints are both perfectly reasonable
> tradeoffs for this data system.  But unlike the case of JSON-LD (which in
> some cases gets canonicalizes differently than "flat JSON"),
> reasonably-sized floats get canonicalized as IEEE754/CBOR-42-compatible
> floats *already* by JCS[^1] and from there get translated stably to and
> from CBOR, and to and from every major language... in the vast majority of
> modern JSON APIs on the web, that's how floats are handled, so the actual
> question here is whether banning floats is a justifiable trade-off for this
> set of JSON APIs.
>
> If [the IETF version] of ATP is intended to be a global-scale,
> general-purpose protocol rather than a sync protocol for microblogging
> apps, I'm not convinced it should drop floats just because the flagship
> implementation didn't need them. But maybe "how much free reign should
> future, independent implementations have" is a bigger upstream question
> than just the type system of Atproto's schema language, about which I
> should probably start a separate thread.
>
> Thanks,
> __bf
>
> [^1]: https://www.rfc-editor.org/rfc/rfc8785.html#section-appendix.b
>
> ---
> bumblefudge
> janitor @Chain Agnostic Standards Alliance
> <https://github.com/chainagnostic/CASA>
> contractable via learningProof UG <https://learningproof.xyz>
> mostly berlin-based
>
> On Tuesday, July 28th, 2026 at 6:36 PM, Bryan Newbold <bryan=
> [e-mail]> wrote:
>
> I think this is a great framing question. And i'm not sure if my answer
> would be a "Yes".
>
> One motivating question is whether developers should expect to be able to
> embed existing JSON data cleanly in to atproto records. Can you just take
> arbitrary DID documents and stuff them in there? GeoJSON? JSON-LD? In many
> cases I would at least carefully consider *not* doing this, even using the
> 'unknown' Lexicon schema type. Even if the data types and structure were
> compatible today, they might not be in the future. We even debated this
> when designing the 'com.atproto.lexicon.schema' Lexicon publication
> meta-schema, and nearly went with JSON blobs or JSON wrapped as 'bytes' in
> the record.
>
> One place friction can come up is reasonable limits on record data size
> and complexity. I wrote up a record data validation guide at one point[0],
> and I think the operational considerations document under our current
> charter should cover a similar scope. It touches on things like maximum
> record data size, nesting depth, string and byte-array lengths, etc. This
> results in another large caveat around permitting arbitrary JSON data.
>
> Another caveat is that the top-level record data "MUST" be of object type
> (not arbitrary JSON data, which might be a string or number), and "MUST"
> contain a '$type' field, and that field "MUST" be of string type, and that
> string "MUST" not be empty, etc, etc. This is a slightly different caveat
> from data simply not containing a dollar-prefixed field.
>
> Instead of maximizing permissiveness of arbitrary JSON data, we might
> instead say that record data is intended to support record data which
> conforms to some Lexicon schema. This might give us more flexibility in
> evolving the data model over time, though there could be tension with the
> 'unknown' data type.
>
> I do think that there is interest/demand in being able to insert arbitrary
> JSON data in to atproto. But it might be better to provide a mechanism (or
> at least a recommended design pattern) for achieving this which has fewer
> caveats.
>
> This reply is a bit of a provocation / devil's advocacy. Maybe the caveats
> aren't actually too hard to satisfy in most situations. But I do think
> there are multiple considerations to take in to account.
>
> --bryan
>
> [0]: https://atproto.com/guides/data-validation
>
> On Tue, Jul 28, 2026 at 1:34 AM Ryan Barrett <[e-mail]> wrote:
>
>> This is a good one! And the good news here is, ATProto largely already
>> supports arbitrary JSON-shaped data in records. It's not encouraged, for a
>> number of reasons, but it is explicitly provided for. The lexicon spec includes
>> an unknown field type <https://atproto.com/specs/lexicon#unknown> that
>> designers can use to indicate that they expect arbitrary data, but you can
>> also put arbitrary data (of any type, including objects) into fields that
>> *aren't* defined in a lexicon. PDSes and other consuming services are
>> explicitly required to allow all this
>> <https://atproto.com/specs/lexicon#authority-and-control>, for a few
>> reasons, eg changing lexicons (backward compatibly) over time.
>>
>> (...subject to some caveats, like you mention, eg floats and some $...
>> names!)
>>
>> On Fri, Jul 24, 2026 at 5:04 AM Volker Mische <[e-mail]>
>> wrote:
>>
>>> Hi all,
>>>
>>> over the past few weeks I realized that I started the whole floats
>>> discussion from within too deep into the rabbit hole. I want to start
>>> again from a higher level, hence a separate thread: Do we want ATP
>>> records to be able to support arbitrary JSON?
>>>
>>> I think the answer is: yes, as much as we can.
>>>
>>> In order to reduce the edge cases, let's assume for this thread that the
>>> "arbitrary JSON" is valid I-JSON [1].
>>>
>>> Of course fully arbitrary isn't possible, which is true for all JSON
>>> based systems that need to signal custom things. In Apache CouchDB it is
>>> top-level fields starting with an underscore which are reserved, hence
>>> cannot be used in your arbitrary JSON input. For ATProto it is all
>>> fields starting with a dollar sign.
>>>
>>> So if your data doesn't contain any key starting with a dollar sign,
>>> then I think it should be possible to store it as an ATP record without
>>> any further limitations.
>>>
>>> I hope we can find agreement that this is a goal to pursue and we can
>>> then take the next step and discuss on how to get there.
>>>
>>> [1]: https://www.rfc-editor.org/info/rfc7493/
>>>
>>> Cheers,
>>> Volker
>>>
>>> _______________________________________________
>>> Atp mailing list -- [e-mail]
>>> To unsubscribe send an email to [e-mail]
>>>
>>
>>
>> --
>> https://snarfed.org/
>> _______________________________________________
>> Atp mailing list -- [e-mail]
>> To unsubscribe send an email to [e-mail]
>>
>
>

-- 
https://snarfed.org/