[IPFIX] string vs octetArray (for non UTF-8 character sets)

[Andrew Feren <[e-mail]>]

Hi all,

I ran into something the other day that doesn't appear to be an issue 
with any current standard information elements, but given the number of 
vendors exporting URL information seems like the issue will come up 
eventually.

I originally gave URLs a data type of string as this seemed like the 
most appropriate type for a human readable string, but then my parser 
squawked about the following URL in an export

http://www.plixer.com/blog/scrutinizer/the-null-scan-you're-being-watched/  <http://www.plixer.com/blog/scrutinizer/the-null-scan-you%E2%80%99re-being-watched/>

In the above URL the "you're" was seen on the wire as "you<92>re" (<92> 
representing one hex byte).  I don't know what character set that is, 
but Windows thinks it is an apostrophe (or RIGHT SINGLE QUOTATION MARK 
if we were speaking UTF-8)

This got me wondering.  What is the right thing to do when monitoring 
text that is not necessarily UTF-8.

a) treat it as an octetArray?  This works, but doesn't feel quite 
right.  It seems useful to have a distinction between raw bytes and 
readable strings.  For presentation purposes for example.

b) expect the exporter to convert to something that is UTF-8 and still 
accurately reports what was observed.  (For example in this case 
converting byte in question to an ascii ' (<27>) looks right and is 
valid UTF-8, but results in a 404, but converting to a real UTF-8 RIGHT 
SINGLE QUOTATION MARK works)

c) define a new data type with character set information.

d) whistle and walk away

e) something else

Thoughts?
-Andrew