[Xml-sg-cmt] Latin characters in xml2rfc
Kesara Rathnayake <kesara@staff.ietf.org> Mon, 25 April 2022 23:47 UTC
Return-Path: <kesara@staff.ietf.org>
X-Original-To: xml-sg-cmt@ietfa.amsl.com
Delivered-To: xml-sg-cmt@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D6EA4C18D830 for <xml-sg-cmt@ietfa.amsl.com>; Mon, 25 Apr 2022 16:47:53 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.279
X-Spam-Level: *
X-Spam-Status: No, score=1.279 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, RDNS_NONE=1.274, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BsOdS8C2ermA for <xml-sg-cmt@ietfa.amsl.com>; Mon, 25 Apr 2022 16:47:50 -0700 (PDT)
Received: from ietfx.amsl.com (unknown [50.223.129.196]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 46074C1D2DB3 for <xml-sg-cmt@ietf.org>; Mon, 25 Apr 2022 16:46:47 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by ietfx.amsl.com (Postfix) with ESMTP id 6C5234050F99 for <xml-sg-cmt@ietf.org>; Mon, 25 Apr 2022 16:46:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
Received: from ietfx.amsl.com ([50.223.129.196]) by localhost (ietfx.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EG8zTftDt2oS for <xml-sg-cmt@ietf.org>; Mon, 25 Apr 2022 16:46:47 -0700 (PDT)
Received: from [192.168.1.87] (122-58-104-37-adsl.sparkbb.co.nz [122.58.104.37]) by ietfx.amsl.com (Postfix) with ESMTPSA id DEDE54050F98 for <xml-sg-cmt@ietf.org>; Mon, 25 Apr 2022 16:46:46 -0700 (PDT)
Message-ID: <714a67f8-905e-f64f-267c-bee2ed7e48ac@staff.ietf.org>
Date: Tue, 26 Apr 2022 11:46:43 +1200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Thunderbird/100.0
To: "xml-sg-cmt@ietf.org" <xml-sg-cmt@ietf.org>
Content-Language: en-NZ
From: Kesara Rathnayake <kesara@staff.ietf.org>
Organization: IETF Administration LLC
Content-Type: text/plain; charset="UTF-8"; format="flowed"
Content-Transfer-Encoding: 7bit
Archived-At: <https://mailarchive.ietf.org/arch/msg/xml-sg-cmt/g4edN5ILSYJRShj989J-CMApCb8>
Subject: [Xml-sg-cmt] Latin characters in xml2rfc
X-BeenThere: xml-sg-cmt@ietf.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Working list for the xml and style guide change management team <xml-sg-cmt.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/xml-sg-cmt>, <mailto:xml-sg-cmt-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/xml-sg-cmt/>
List-Post: <mailto:xml-sg-cmt@ietf.org>
List-Help: <mailto:xml-sg-cmt-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/xml-sg-cmt>, <mailto:xml-sg-cmt-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2022 23:47:53 -0000
Hi all, I have included the Unicode character ranges that xml2rfc regards as Latin. This list was generated at *a certain point*, using the following resources: * http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt * http://www.unicode.org/Public/UCD/latest/ucd/ScriptExtensions.txt * http://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt * http://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedBidiClass.txt I can dig deep into how this set was compiled if required. While figuring out if a character is Latin, xml2rfc ignores two sets of characters named "common" and "inherited". I have included all sets below. Note that the ranges are represented as decimal value of the Unicode characters. For example, 65 is 0x0041, i.e. A We will probably have to represent these in a better way for documenting purposes. :) These are the Unicode character ranges that xml2rfc regards as Latin: ``` 'Latin': [range(65, 91), range(97, 123), range(170, 172), range(186, 188), range(192, 215), range(216, 247), range(248, 443), range(443, 445), range(444, 448), range(448, 452), range(452, 660), range(660, 662), range(661, 688), range(688, 697), range(736, 741), range(7424, 7462), range(7468, 7517), range(7522, 7526), range(7531, 7544), range(7545, 7579), range(7579, 7615), range(7680, 7936), range(8305, 8307), range(8319, 8321), range(8336, 8349), range(8490, 8492), range(8498, 8500), range(8526, 8528), range(8544, 8579), range(8579, 8581), range(8581, 8585), range(11360, 11388), range(11388, 11390), range(11390, 11392), range(42786, 42864), range(42864, 42866), range(42865, 42888), range(42891, 42895), range(42895, 42897), range(42896, 42938), range(42999, 43001), range(43000, 43002), range(43002, 43004), range(43003, 43008), range(43824, 43867), range(43868, 43872), range(43872, 43877), range(64256, 64263), range(65313, 65339), range(65345, 65371), range(867, 880), range(1157, 1159), range(4347, 4349), range(8432, 8434), range(43310, 43312), range(2386, 2388), range(2385, 2387)] ``` xml2rfc ignores the following set of characters as "inherited" characters: ``` 'Inherited': [range(768, 880), range(1157, 1159), range(1611, 1622), range(1648, 1650), range(2385, 2387), range(6832, 6846), range(6846, 6848), range(7376, 7379), range(7380, 7393), range(7394, 7401), range(7405, 7407), range(7412, 7414), range(7416, 7418), range(7616, 7674), range(7675, 7680), range(8204, 820 6), range(8400, 8413), range(8413, 8417), range(8417, 8419), range(8418, 8421), range(8421, 8433), range(12330, 12334), range(12441, 12443), range(65024, 65040), range(65056, 65070), range(66045, 66047), range(66272, 66274), range(70459, 70461), range(119143, 119146), range(119163, 119171), range(119173, 119180), r ange(119210, 119214), range(917760, 918000)] ``` xml2rfc also ignores the following set of characters as "common" characters: ``` 'Common': [range(0, 32), range(32, 34), range(33, 36), range(36, 38), range(37, 40), range(40, 42), range(41, 43), range(42, 44), range(43, 45), range(44, 46), range(45, 47), range(46, 48), range(48, 58), range(58, 60), range(60, 63), range(63, 65), range(91, 93), range(92, 94), range(93, 95), range(94, 96), ra nge(95, 97), range(96, 98), range(123, 125), range(124, 126), range(125, 127), range(126, 128), range(127, 160), range(160, 162), range(161, 163), range(162, 166), range(166, 168), range(167, 169), range(168, 170), range(169, 171), range(171, 173), range(172, 174), range(173, 175), range(174, 176), range(175, 177), range(176, 178), range(177, 179), range(178, 180), range(180, 182), range(181, 183), range(182, 184), range(184, 186), range(185, 187), range(187, 189), range(188, 191), range(191, 193), range(215, 217), range(247, 249), range(697, 706), range(706, 710), range(710, 722), range(722, 736), range(741, 746), range(748 , 750), range(749, 751), range(750, 752), range(751, 768), range(884, 886), range(894, 896), range(901, 903), range(903, 905), range(1417, 1419), range(1541, 1543), range(1548, 1550), range(1563, 1565), range(1567, 1569), range(1600, 1602), range(1757, 1759), range(2274, 2276), range(2404, 2406), range(3647, 3649), range(4053, 4057), range(4347, 4349), range(5867, 5870), range(5941, 5943), range(6146, 6148), range(6149, 6151), range(7379, 7381), range(7393, 7395), range(7401, 7405), range(7406, 7410), range(7410, 7412), range(7413, 7415), range(7415, 7417), range(8192, 8203), range(8203, 8205), range(8206, 8208), range(8208, 8214), range(8214, 8216), range(8216, 8218), range(8217, 8219), range(8218, 8220), range(8219, 8221), range(8221, 8223), range(8222, 8224), range(8223, 8225), range(8224, 8232), range(8232, 8234), range(8233, 8235), range(8234, 8239), range(8239, 8241), range(8240, 8249), range(8249, 8251), range(8250, 8252), rang e(8251, 8255), range(8255, 8257), range(8257, 8260), range(8260, 8262), range(8261, 8263), range(8262, 8264), range(8263, 8274), range(8274, 8276), range(8275, 8277), range(8276, 8278), range(8277, 8287), range(8287, 8289), range(8288, 8293), range(8294, 8304), range(8304, 8306), range(8308, 8314), range(8314, 8317 ), range(8317, 8319), range(8318, 8320), range(8320, 8330), range(8330, 8333), range(8333, 8335), range(8334, 8336), range(8352, 8384), range(8448, 8450), range(8450, 8452), range(8451, 8455), range(8455, 8457), range(8456, 8458), range(8458, 8468), range(8468, 8470), range(8469, 8471), range(8470, 8472), range(847 2, 8474), range(8473, 8478), range(8478, 8484), range(8484, 8486), range(8485, 8487), range(8487, 8489), range(8488, 8490), range(8489, 8491), range(8492, 8494), range(8494, 8496), range(8495, 8498), range(8499, 8501), range(8501, 8505), range(8505, 8507), range(8506, 8508), range(8508, 8512), range(8512, 8517), ra nge(8517, 8522), range(8522, 8524), range(8523, 8525), range(8524, 8526), range(8527, 8529), range(8528, 8544), range(8585, 8587), range(8586, 8588), range(8592, 8597), range(8597, 8602), range(8602, 8604), range(8604, 8608), range(8608, 8610), range(8609, 8611), range(8611, 8613), range(8612, 8614), range(8614, 86 16), range(8615, 8622), range(8622, 8624), range(8623, 8654), range(8654, 8656), range(8656, 8658), range(8658, 8660), range(8659, 8661), range(8660, 8662), range(8661, 8692), range(8692, 8960), range(8960, 8968), range(8968, 8970), range(8969, 8971), range(8970, 8972), range(8971, 8973), range(8972, 8992), range(8 992, 8994), range(8994, 9001), range(9001, 9003), range(9002, 9004), range(9003, 9084), range(9084, 9086), range(9085, 9115), range(9115, 9140), range(9140, 9180), range(9180, 9186), range(9186, 9255), range(9280, 9291), range(9312, 9372), range(9372, 9450), range(9450, 9472), range(9472, 9655), range(9655, 9657), range(9656, 9665), range(9665, 9667), range(9666, 9720), range(9720, 9728), range(9728, 9839), range(9839, 9841), range(9840, 10088), range(10088, 10090), range(10089, 10091), range(10090, 10092), range(10091, 10093), range(10092, 10094), range(10093, 10095), range(10094, 10096), range(10095, 10097), range(10096, 1 0098), range(10097, 10099), range(10098, 10100), range(10099, 10101), range(10100, 10102), range(10101, 10103), range(10102, 10132), range(10132, 10176), range(10176, 10181), range(10181, 10183), range(10182, 10184), range(10183, 10214), range(10214, 10216), range(10215, 10217), range(10216, 10218), range(10217, 10 219), range(10218, 10220), range(10219, 10221), range(10220, 10222), range(10221, 10223), range(10222, 10224), range(10223, 10225), range(10224, 10240), range(10496, 10627), range(10627, 10629), range(10628, 10630), range(10629, 10631), range(10630, 10632), range(10631, 10633), range(10632, 10634), range(10633, 106 35), range(10634, 10636), range(10635, 10637), range(10636, 10638), range(10637, 10639), range(10638, 10640), range(10639, 10641), range(10640, 10642), range(10641, 10643), range(10642, 10644), range(10643, 10645), range(10644, 10646), range(10645, 10647), range(10646, 10648), range(10647, 10649), range(10648, 1065 0), range(10649, 10712), range(10712, 10714), range(10713, 10715), range(10714, 10716), range(10715, 10717), range(10716, 10748), range(10748, 10750), range(10749, 10751), range(10750, 11008), range(11008, 11056), range(11056, 11077), range(11077, 11079), range(11079, 11085), range(11085, 11124), range(11126, 11158 ), range(11160, 11209), range(11210, 11263), range(11776, 11778), range(11778, 11780), range(11779, 11781), range(11780, 11782), range(11781, 11783), range(11782, 11785), range(11785, 11787), range(11786, 11788), range(11787, 11789), range(11788, 11790), range(11789, 11791), range(11790, 11799), range(11799, 11801) , range(11800, 11802), range(11802, 11804), range(11803, 11805), range(11804, 11806), range(11805, 11807), range(11806, 11808), range(11808, 11810), range(11809, 11811), range(11810, 11812), range(11811, 11813), range(11812, 11814), range(11813, 11815), range(11814, 11816), range(11815, 11817), range(11816, 11818), range(11817, 11819), range(11818, 11823), range(11823, 11825), range(11824, 11834), range(11834, 11836), range(11836, 11840), range(11840, 11842), range(11841, 11843), range(11842, 11844), range(11843, 11855), range(12272, 12284), range(12288, 12290), range(12289, 12292), range(12292, 12294), range(12294, 12296), range(12296, 12298), range(12297, 12299), range(12298, 12300), range(12299, 12301), range(12300, 12302), range(12301, 12303), range(12302, 12304), range(12303, 12305), range(12304, 12306), range(12305, 12307), range(12306, 12308), range(12308, 12310), range(12309, 12311), range(12310, 12312), range(12311, 12313), r ange(12312, 12314), range(12313, 12315), range(12314, 12316), range(12315, 12317), range(12316, 12318), range(12317, 12319), range(12318, 12320), range(12320, 12322), range(12336, 12338), range(12337, 12342), range(12342, 12344), range(12348, 12350), range(12349, 12351), range(12350, 12352), range(12443, 12445), ra nge(12448, 12450), range(12539, 12541), range(12540, 12542), range(12688, 12690), range(12690, 12694), range(12694, 12704), range(12736, 12772), range(12832, 12842), range(12842, 12872), range(12872, 12880), range(12880, 12882), range(12881, 12896), range(12927, 12929), range(12928, 12938), range(12938, 12977), ran ge(12977, 12992), range(12992, 13008), range(13144, 13312), range(19904, 19968), range(42752, 42775), range(42775, 42784), range(42784, 42786), range(42888, 42890), range(42889, 42891), range(43056, 43062), range(43062, 43064), range(43064, 43066), range(43065, 43067), range(43310, 43312), range(43471, 43473), rang e(43867, 43869), range(64830, 64832), range(64831, 64833), range(65040, 65047), range(65047, 65049), range(65048, 65050), range(65049, 65051), range(65072, 65074), range(65073, 65075), range(65075, 65077), range(65077, 65079), range(65078, 65080), range(65079, 65081), range(65080, 65082), range(65081, 65083), range (65082, 65084), range(65083, 65085), range(65084, 65086), range(65085, 65087), range(65086, 65088), range(65087, 65089), range(65088, 65090), range(65089, 65091), range(65090, 65092), range(65091, 65093), range(65092, 65094), range(65093, 65095), range(65095, 65097), range(65096, 65098), range(65097, 65101), range( 65101, 65104), range(65104, 65107), range(65108, 65112), range(65112, 65114), range(65113, 65115), range(65114, 65116), range(65115, 65117), range(65116, 65118), range(65117, 65119), range(65118, 65120), range(65119, 65122), range(65122, 65124), range(65123, 65125), range(65124, 65127), range(65128, 65130), range(6 5129, 65131), range(65130, 65132), range(65279, 65281), range(65281, 65284), range(65284, 65286), range(65285, 65288), range(65288, 65290), range(65289, 65291), range(65290, 65292), range(65291, 65293), range(65292, 65294), range(65293, 65295), range(65294, 65296), range(65296, 65306), range(65306, 65308), range(65 308, 65311), range(65311, 65313), range(65339, 65341), range(65340, 65342), range(65341, 65343), range(65342, 65344), range(65343, 65345), range(65344, 65346), range(65371, 65373), range(65372, 65374), range(65373, 65375), range(65374, 65376), range(65375, 65377), range(65376, 65378), range(65377, 65379), range(653 78, 65380), range(65379, 65381), range(65380, 65382), range(65392, 65394), range(65438, 65440), range(65504, 65506), range(65506, 65508), range(65507, 65509), range(65508, 65510), range(65509, 65511), range(65512, 65514), range(65513, 65517), range(65517, 65519), range(65529, 65532), range(65532, 65534), range(6579 2, 65795), range(65799, 65844), range(65847, 65856), range(65936, 65948), range(66000, 66045), range(66273, 66300), range(113824, 113828), range(118784, 119030), range(119040, 119079), range(119081, 119141), range(119141, 119143), range(119146, 119149), range(119149, 119155), range(119155, 119163), range(119171, 11 9173), range(119180, 119210), range(119214, 119273), range(119520, 119540), range(119552, 119639), range(119648, 119673), range(119808, 119893), range(119894, 119965), range(119966, 119968), range(119970, 119972), range(119973, 119975), range(119977, 119981), range(119982, 119994), range(119995, 119997), range(1199 97, 120004), range(120005, 120070), range(120071, 120075), range(120077, 120085), range(120086, 120093), range(120094, 120122), range(120123, 120127), range(120128, 120133), range(120134, 120136), range(120138, 120145), range(120146, 120486), range(120488, 120513), range(120513, 120515), range(120514, 120539), rang e(120539, 120541), range(120540, 120571), range(120571, 120573), range(120572, 120597), range(120597, 120599), range(120598, 120629), range(120629, 120631), range(120630, 120655), range(120655, 120657), range(120656, 120687), range(120687, 120689), range(120688, 120713), range(120713, 120715), range(120714, 120745) , range(120745, 120747), range(120746, 120771), range(120771, 120773), range(120772, 120780), range(120782, 120832), range(126065, 126124), range(126124, 126126), range(126125, 126128), range(126128, 126130), range(126129, 126133), range(126976, 127020), range(127024, 127124), range(127136, 127151), range(127153, 1 27168), range(127169, 127184), range(127185, 127222), range(127232, 127245), range(127248, 127340), range(127344, 127405), range(127462, 127488), range(127489, 127491), range(127504, 127548), range(127552, 127561), range(127568, 127570), range(127584, 127590), range(127744, 127995), range(127995, 128000), range(128 000, 128725), range(128736, 128749), range(128752, 128762), range(128768, 128884), range(128896, 128985), range(129024, 129036), range(129040, 129096), range(129104, 129114), range(129120, 129160), range(129168, 129198), range(129280, 129292), range(129296, 129343), range(129344, 129393), range(129395, 129399), ran ge(129402, 129404), range(129404, 129443), range(129456, 129466), range(129472, 129475), range(129488, 129536), range(129632, 129646), range(917505, 917507), range(917536, 917632)] ``` Cheers, Kesara PS: Sorry about dumping source code on you. -- Kesara Rathnayake Senior Software Development Engineer - IETF LLC kesara@staff.ietf.org
- [Xml-sg-cmt] Latin characters in xml2rfc Kesara Rathnayake