Re: [wasm] sqlite as rfc-spec'd web-interchange-format?

[kai zhu <[e-mail]>]

> it is lousy at handling concurrent queries
lousy compared to what? localstorage / indexeddb / redux ?
as frontend-developer, our data-store options all suck at concurrency and
are either 1) too toy-case (localstorage) or 2) too difficult to use vs.
wasm-sqlite queries.


> In my experience a lot gets lost in translation between sqlite tables and
other databases
its not perfect for sure, but say i want to serialize/revive following
datasets in browser:
    - catalog of shop-items with fulltextsearch
    - customer's online shopping-cart
    - [subset of] enron-email with fulltextsearch
    - queryable 10-year historical-data of all s&p500 stocks

if not as [wasm] sqlite-database, what better serialization-options
are there for interchanging above datasets over the web?  also if ietf is
not a suitable forum, let me know if there are better places to raise this
issue.

On Thu, Nov 26, 2020 at 7:58 PM Keith Moore <[e-mail]>
wrote:

> On 11/26/20 7:08 PM, kai zhu wrote:
>
> i'm noob to mailing-list, and if you think there's more appropriate place
> for subject, please let me know.
>
> motivation:
> as product-developer, i waste huge amount of time reinventing error-prone
> serializers/revivers to exchange complex-datasets across web.
> standardizing sqlite's file-format would reduce integration-headaches
> between me / teammates / 3rd-parties, since we could more easily agree to
> use wasm-sqlite's standard serializers/revivers (instead of everyone
> reinventing their own for the web).
>
> why sqlite as web-interchange-format?
> 1.
> wasm-sqlite's serializers/revivers are opensource and accessible to
> web-clients
> 2.
> The [U.S.] Library of Congress Recommended Formats Statement (RFS)
> includes SQLite as a preferred format for datasets
> <https://www.loc.gov/preservation/resources/rfs/data.html>. [1]
> 3.
> sqlite's last file-format (v4) remains stable and unchanged since 2006
> 4.
> all sqlite file-formats since 2004 (v1, v2, v3, v4) are intended to be
> forward-compatible with future sqlite releases
> 5.
> sqlite is the "Most Widely Deployed and Used Database Engine" (and
> perhaps second most widely deployed software library, after libz). [2]
>
> the spec for sqlite's file-format is here [3].  i'm no expert at reading
> it, but am wondering how feasible to translate it to an rfc-spec (minus the
> journaling part)?
>
> -kai
>
>
> I've been badly burned by the sqlite library more times than I can count.
> Unless it's been fixed fairly recently, it is lousy at handling concurrent
> queries, so for that reason alone promoting the file format might be a
> disservice.
>
> But maybe it makes some sense as a database interchange format.   Maybe.
> In my experience a lot gets lost in translation between sqlite tables and
> other databases, but maybe not so much that it's useless as a way to convey
> datasets. It's probably a useful format for use in information transfer and
> maybe even simple single-user queries, but beware of using the sqlite
> library to make any kind of services providing concurrent access to more
> than one process.
>
> I'm also not sure whether IETF can attract enough of the right kind of
> expertise to do a good job at this.
>
> But I do suspect there's a need for something like this to be
> standardized, by someone.
> Keith
>
>
>