[Forces-protocol] Re: Data encoding -- first part

[Alan DeKok <[e-mail]>]

Zsolt Haraszti wrote:

>>   I think the reference is "ANSI/IEEE Standard 754-1985"
>
> I will check (or let me know when you know for sure :).

http://grouper.ieee.org/groups/754/

http://stevehollasch.com/cgindex/coding/ieeefloat.html

   Yup.

> There may some misunderstanding here, so let me rewind a bit.
> 
> N in STRING[N] defines that max size of the string _in the model_.
> This is analogous to the
> 
> 	char 	if_name[16]
> 
> C construct (as opposed to the char *if_name).

   Ah, OK.  Having the protocol avoid the zeros is nice, but it means 
that many structs are now packed to variable-sized data.

> Let me try to clarify it a bit:  Suppose you define a STRUCT A data type
> per ForCES data model.  Suppose furthermore that you implement that
> struct in the FE/CE software by means of a C struct: struct A.  The C
> struct will have a certain memory layout in the host memory.  That 
> layout is what I referred to as "ANSI C struct representation" (I may
> have used the wrong reference here; I shall look it up to be sure).

   The problem is that ANSI C struct representation is 
platform-specific.  Some platforms can pack multiple 'uint8_t' into a 
32-bit word.  Others can't.

   As an extreme example (not applicable here), big/little-endian 
systems pack bit arrays differently.  So I wouldn't depend on anything 
other than order, and maybe 32-bit alignment, for platform-independent C 
structs.  This makes mapping ForCES protocol packed data into C structs 
somewhat problematic.

> The statement above is that for structs which do not have variable
> size fields, the ForCES encoded STRUCT will have the same layout as
> the C struct in memory (minus endianness).  The obvious advantage of
> this is that you can memcpy() between the message buffer and you
> C struct.

   I would suggest adding a note that implementors are STRONGLY 
CAUTIONED to validate that the sizes of the structs are the same, and 
that the padding is the same between ForCES and the C compiler.  This 
kind of "memcpy" of complex structures has historically been open to 
implementation flaws and security attacks.

> Almost, but not quite.  Since we want to make this layout compatible
> with C implementations, we need to include padding between some of the
> elements.  The stated alignment rules serve this purpose.

   OK.  It's just that if you have variable rules for padding, the PACK 
function can't really be recursive, as it varies depending on whether or 
not it's in a struct, or not.

> I believe the provided example illustrates the rules well.

   Yes.

>>   If that's true, then it will be possible to write a function which 
>>takes a ForCES STRUCT definition and a pointer to a C structure 
>>containing the relevant data, and return a PACKed DATA field.  While 
>>this process often won't be necessary, it may be useful to give a sample 
>>function in pseudo-code, or C, to do this packing.  That way people 
>>implementing the protocol have a "known good" place to start from.
> 
> Are you suggesting putting a pseudo-code into the description
> here or into the final standard (or both)?

   Yes.  Code will clarify a LOT of issues.

>>   It would be great to have some sample code which read in simple 
>>STRUCT definitions, and produced the packed DATA.  Having a function to 
>>produce ASCII art would be even better...
> 
> 
> You mean "STRUCT definition and actual value, and produced the packed
> DATA"?                      ^^^^^^^^^^^^^^^^

   Sure.

> We can have a contest on who could post that program first.
> Input: a) an the XML data specification per current ForCES model
>           document
>        b) value assignment for the fields
> Output: an ASCII art showing the packet data.

   Hmm... I don't know if I'll have time.  But it would be extremely 
useful, even for other RFC's containing ASCII art.

>>   Variable-sized entries are severely problematic.
> 
> I agree that they are more trouble-some.  But I find it too restrictive
> to prohibit them.  Arrays of arrays will have this.  Or, if your array
> is based on a struct that has string field(s), will have this problem.

   How about having a packed length for structures?  That is, reserve a 
32-bit word at the start of the struct for it's length, PACK the struct, 
and then update the length field with the resulting value.

   For structs of variable size, this should make it easy to quickly 
find the N'th element in an array of those structs.  The clearest 
benefit is that you don't have to parse or even know the entries of the 
struct, in order to find the N'th element.

> This is one of those places where I would think good designs will avoid
> such ARRAYs as much as possible, but in some cases it cannot be avoided
> so we must support it in the protocol.

   I agree.  I would, however, like to see the difference between 
fixed-size structs && variable-size structs highlighted.

   The problem is that if a data structure has a sub-structure 
*anywhere* in it which contains a STRING[N], then the entire data 
structure is "tainted", and it's size is variable instead of fixed.

   If we expect that most STRUCTs will contain STRING[N], then it may be 
reasonable to assume that ALL STRUCT packing includes a 32-bit prefix of 
the size of the struct, as this will decrease the complexity of the parser.


   If STRING[N] is the only variable-sized data type in the protocol, 
then it may be reasonable instead to separate the string packing from 
the struct packing.  i.e. Rather than packing the string in place, pack 
all of the strings together, and pack a "pointer" in the struct to the 
string.

   I'm not sure I like that idea, though, as packing will require 
multiple passes to get all of the cross-references correct.

   Alan DeKok.



_______________________________________________
Forces-protocol mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/forces-protocol