Re: [I18ndir] I18ndir early review of draft-ietf-dispatch-javascript-mjs-07

["Matthew A. Miller" <[e-mail]>]

Thank you very much for the review, John.  This is very helpful, and
will work on changes to resolve the concerns.

Regarding the encoding:

UTF-8 is strongly encouraged, and required for a source to be a module.
 Preferring BOM sniffing over a charset declaration is the result of
general purpose browsers having to deal with a myriad of misconfigured
servers; this being the defense mechanism that permitted the best
interoperability.

The part about "longest matching octet sequence" is a missed remnant
from the original document which included UTF-32BE/LE entries.  As
UTF-32BE/LE is no longer found in the wild*, those entries were removed
but I missed cleaning up that sentence.

Implementations today do ignore the BOM once the character encoding is
worked out, so this was portion from Section 4.2 was kept to reflect
that.  I'm not sure what changes to make here regarding that, although
the ending "to source text" is another missed remnant from my editing
that will be removed.

Section 4.2 still includes step #3 to deal with the (in practice quite
common) case of a missing BOM and the media type missing a charset
parameter.  There are also too many servers that set this to
"ISO-8859-1" without otherwise examining the sources being served.
We'll make it clearer this is a default/fallback case.


- m&m

Matthew A. Miller

* Web browsers haven't accepted UTF-32 encodings of JavaScript for quite
a long while
On 20/05/08 17:24, Myles Borins wrote:
> Regarding the mime type "text/javascript"
> 
> The HTML Specification is rather explicit
> <https://html.spec.whatwg.org/multipage/scripting.html#scriptingLanguages>
> 
>     Servers should use text/javascript for JavaScript resources. Servers
>     should not use other JavaScript MIME types for JavaScript resources,
>     and must not use non-JavaScript MIME types.
> 
> 
> Browsers validate that source text being loaded is delivered with the
> "text/javascript" mime type and fails if it is any other mime is used.
> There is no in-band way to determine the goal of a Module, and Node.js
> needed a signal to be able to determine how to interpret source text.
> This was the motivation behind creating .mjs.
> 
> The problem this created was that webservers, not knowing the extension,
> were not serving the test/javascript mimetype and browsers were then in
> turn failing to load them as modules. This was a pretty awful developer
> experience, especially for folks experimenting with a newer technology
> they didn't entirely understand to begin with.
> 
> With this context in mind I don't think it is really worth going and
> making a distinct mimetype for modules, at least not one that would be
> strictly enforced, at the very least this probably shouldn't be
> pursued without broader socialization and buy-in from browser vendors.
> 
> My gut is that we are best to use this proposal to document the existing
> status quo... what people are using and is working, rather than
> introduce new constraints that do not have any existing implementation
> or adoption.
> 
> I'll let folks more familiar with your other notes field those questions.
> 
> On Fri, May 8, 2020 at 7:19 PM Barry Leiba <[e-mail]
> <mailto:[e-mail]>> wrote:
> 
>     Thanks, John, for taking the time on this.  It’s a really helpful
>     review, and appreciated.
> 
>     Barry
> 
>     On Fri, May 8, 2020 at 4:17 PM John Levine via Datatracker
>     <[e-mail] <mailto:[e-mail]>> wrote:
> 
>         Reviewer: John Levine
>         Review result: Ready with Issues
> 
>         This is my take on issues with this document mostly from my personal
>         review but also after some discussion we've had on the i18ndir list.
> 
>         Some parts of this draft are quite hard to follow, so I'm giving my
>         understanding of the parts I'm commenting on in case I got them
>         wrong.
>         I realize that a lot of this is unchanged from 4329, which we should
>         have reviewed more carefully 15 years ago.
> 
>         Section 4 on Encoding: I believe it says that the preferred encoding
>         for all javascript is UTF-8, but some sources use other
>         encodings and
>         sometimes mislabel them.  So for anything that you don't know is a
>         module, you have to sniff the contents to see if starts with a BOM,
>         and if so, use the BOM's encoding and delete the BOM.  If the
>         BOM uses
>         an encoding the consumer doesn't support, fail.  If there's no BOM,
>         use the declared character set, or if it's one the consumer doesn't
>         understand, treat it as UTF-8 anyway.
> 
>         Step 1 says "The longest matching octet sequence determines the
>         encoding."
>         which I don't understand, since none of the encodings overlap. 
>         Does that
>         mean it should interpret a partial BOM, e.g., EF BB 20 for
>         UTF-8? Also,
>         why is the BOM deleted?  ECMAscript says a BOM is a space so it
>         should be
>         harmless.
> 
>         While I understand that there is a lot of history here, I'm
>         wondering if
>         the range mislabeling is really as extreme as this implies.  Is
>         there,
>         say, text labelled Shift-JIS which is really UTF-8 or UTF-16? If
>         the
>         mislabelled stuff is consistently mislabelled as one of
>         UTF-8/16/16BE/16LE
>         perhaps it could say to try the BOM trick on those encodings and
>         fail otherwise.
> 
>         I don't understand step 3, "The character encoding scheme is
>         determined to be UTF-8."  How can it be determined to be UTF-8 other
>         than by steps 1 and 2?  Or is it saying that if the declared charset
>         is one the consumer doesn't understand such as KOI8-U, assume it's
>         UTF-8 anyway.
> 
>         I'd suggest rewriting the section to make it clearer that if
>         it's not
>         a module, you look for a BOM, use its encoding if you find one,
>         and (I
>         think) otherwise use the declared encoding.
> 
>         Section 4.3 on error handling: I think it says that if there's a
>         byte
>         sequence that isn't a valid code point in the current encoding,
>         it can
>         fail or it can turn the bytes into Unicode replacement
>         characters, but
>         MUST NOT try anything else.  I agree with this advice but again it
>         could be clearer.
> 
>         Section 3 on Modules: I believe it says that JS scripts and
>         modules have
>         different syntax but you can't easily tell them apart by
>         inspection. 
>         (The term "goal" is familiar since I used to write books about
>         compiler
>         tools, and I realize it's what the ECMAscript spec uses, but it's
>         confusing if you're not a programming language expert.  How
>         about just
>         saying that scripts and modules have different syntax?)
> 
>         Hence some software uses a .mjs filename as a hint that
>         something is a
>         module.  Again I realize that there is a bunch of existing code but
>         this is not great MIME practice.  If the difference matters, it's
>         worth providing a new MIME type such as text/jsmodule, which could
>         have consistently accurate content encodings.  It would coexist with
>         all of the other old MIME types and the .mjs hints. Since this draft
>         deprecates a bunch of existing types and de-deprecates another, this
>         seems as good a time as any to do it.
> 
>         I also wonder whether it's worth making a distinction in MIME
>         processing between modules and scripts.  Would there be any harm in
>         saying to sniff everything for a BOM?  If a .mjs file turns out to
>         have a UTF-16 BOM, it's wrong, but is it likely to be anything other
>         than a javascript module in UTF-16?
> 
> 
>