[media-types] Default charset parameter for text media types

[Peter Occil <[e-mail]>]

What is the default charset parameter when that parameter is absent from a text media type?  Before RFC 6657, the answer was simple: US-ASCII.  However, things have become more complicated since that RFC was published -- RFC 6657 says:

   [A]ll new "text/*" registrations [that is, registrations after July 2012] MUST clearly specify how the charset is determined[.] However, existing "text/*" registrations [before July 2012] that fail to specify how the charset is determined still default to US-ASCII.

This would be easy if all existing and new text media types "specif[ied] how the charset is determined".  Unfortunately this is not the case.

Although I have managed to categorize all text media types into three categories: "no default", "US-ASCII default", and "UTF-8 default", deciding which category a media type falls in is not always trivial, since it depends among other things on when the media type was registered, whether it treats the charset parameter as required, optional, or not used, and whether the text is always ASCII or always UTF-8.

What follows are notes that explain how each text media type is assigned one of those three categories (source is my MailLib library's MediaType.cs).  They may be helpful in the process of revising some of those media types so that they "clearly specify how the charset is determined", as RFC 6657 requires for media type registrations after July 2012.

      // NOTE: RFC6657 changed the rules for the default charset in text
      // media types, so that there is no default charset for as yet
      // undefined media types. However,
      // media types defined before this RFC (July 2012) are grandfathered
      // from the rule: those
      // media types "that fail to specify how the charset is determined" still
      // have US-ASCII as default. The text media types defined as of
      // Jul. 11, 2018, are listed below:
      //
      // -- No default charset assumed: --
      //
      // RTP payload types; these are usually unsuitable for MIME,
      // and don't permit a charset parameter, so a default charset is
      // irrelevant:
      // -- 1d-interleaved-parityfec, fwdred, red, parityfec, encaprtp,
      // raptorfec, rtp-enc-aescm128, t140, ulpfec, rtx, rtploopback
      //
      // Charset determined out-of-band:
      // -- vnd.motorola.reflex*(5)*(10)
      //
      // Special procedure defined for charset detection:
      // -- ecmascript*(8), javascript*(8), html,
      // rtf*(5)
      //
      // XML formats (no default assumed if charset is absent, according
      // to RFC7303, the revision of the XML media type specification):
      // -- xml, xml-external-parsed-entity,
      // vnd.in3d.3dml*, vnd.iptc.newsml, vnd.iptc.nitf,
      // vnd.ms-mediapackage*(5),
      // vnd.net2phone.commcenter.command, vnd.radisys.msml-basic-layout,
      // vnd.wap.si, vnd.wap.sl, vnd.wap.wml
      //
      // Behavior deliberately undefined (so whether US-ASCII or another
      // charset is treated as default is irrelevant):
      // -- example
      //
      // These media types don't define a charset parameter (after
      // RFC6657):
      // -- grammar-ref-list*(9), vnd.hgl*(6)*(9), vnd.gml*(9)
      //
      // Uses charset parameter, but no default charset specified (after
      // RFC6657):
      // -- markdown*
      //
      // -- US-ASCII assumed: --
      //
      // These media types don't define a charset parameter (before
      // RFC6657):
      // -- dns, mizar, vnd.latex-z,
      // prs.lines.tag, vnd.dmclientscript,
      // vnd.dvb.subtitle, rfc822-headers,
      // vnd.si.uricatalogue*(7), vnd.si.fly*(7)
      //
      // No charset parameter defined, but does specify ASCII only (after
      // RFC6657):
      // -- vnd.ascii-art****, prs.prop.logic*
      //
      // These media types don't define a default charset:
      // -- css, richtext, enriched, tab-separated-values*,
      // vnd.in3d.spot*, vnd.abc, vnd.wap.wmlscript, vnd.curl,
      // vnd.fmi.flexstor, uri-list, directory*
      //
      // US-ASCII default:
      // -- plain, sgml, troff
      //
      // -- UTF-8 assumed: --
      //
      // UTF-8 only:
      // -- vcard, jcr-cnd, cache-manifest
      //
      // Charset parameter defined but is "always ... UTF-8":
      // -- n3, turtle, vnd.debian.copyright, provenance-notation
      //
      // UTF-8 default:
      // -- csv, calendar**, vnd.a***, parameters, prs.fallenstein.rst,
      // vnd.esmertec.theme.descriptor, vnd.trolltech.linguist,
      // vnd.graphviz, vnd.sun.j2me.app-descriptor, strings*(5),
      // csv-schema*(5)
      //
      // * Required parameter.
      // ** No explicit default, but says that "[t]he charset supported
      // by this revision of iCalendar is UTF-8."
      // *(5) No charset parameter defined.
      // *(6) 8-bit encoding.
      // *(7) Says "US-ASCII" is always used, or otherwise says
      // the media type contains ASCII text
      // *(8) RFC4329: If charset unrecognized, check for UTF-8/16/32 BOM if it
      // exists; otherwise use UTF-8. If UTF-8, ignore UTF-8 BOM
      // *(9) After RFC 6657.
      // *(10) Charset determined "in an a priori manner", rather than
      // being stated in the payload.
      // *** Default is UTF-8 "if 8-bit bytes are encountered" (even if
      // none are found, though, a 7-bit ASCII text is still also UTF-8).
      // **** Content containing non-ASCII bytes "should be rejected".

--Peter