Re: [icnrg] Comments on draft-irtf-icnrg-ccninfo-02

[Hitoshi Asaeda <[e-mail]>]

Hi Dave,

Thanks for your review again.
I've been unfortunately fully occupied recently but I'll later reply and then update the CCNinfo draft for sure.
And thanks for pointing your path steering draft. I'll read the draft as well.

Please give me some time.

Regards,

Hitoshi



> On Oct 26, 2019, at 5:01, David R. Oran <[e-mail]> wrote:
> 
> I want to re-activate our discussion on CCNInfo, as Singapore is coming up and we want to push forward on progressing this important work.
> 
> In addition to the exchange below, I’d like to get the opinion of the CCNInfo authors on whether the draft on path steering that Ilya Moiseenko and I recently submitted (https://datatracker.ietf.org/doc/draft-oran-icnrg-pathsteering/) might be helpful in making CCNInfo as useful as possible. What do you think? It would be great if you could comment generally on this work as well, since one of the main use cases for path steering is network diagnosis.
> 
> DaveO.
> 
> On 20 Aug 2019, at 4:32, Hitoshi Asaeda wrote:
> 
>> Hi Dave,
>> 
>> Thank you very much for your review (and sorry for delay).
>> 
>>> On Aug 2, 2019, at 23:16, David R. Oran <[e-mail]> wrote:
>>> 
>>> I did a full re-read of the latest CCNInfo draft. These comments are with <Chair Hat off>. I’ll be sending a separate message with my <chair hat on> comments shortly.
>>> 
>>> General comments:
>>> 
>>> 	• The specification is a lot more complete and easy to understand. It’s already implemented (by the authors), and I suspect another implementer could get pretty close to an interoperable implementation from the material in the current version.
>>> 
>>> 	• The design seems fundamentally solid. It’s not the design I would have chosen, but given this is for research purposes and heading toward experimental status rather than a standard, I think progressing this approach is fine. We can experiment with it (and other approaches) to learn more about how to instrument and manage networks based on CCNx. There is of course other complementary work in ICNRG (e.g. ping and traceroute) so researchers would be encouraged to do both qualitative and quantitative evaluation of the tradeoffs in the various approaches. There is also work under submission to the ICN’19 conference that takes yet another approach so soon we may have a rich set of things to compare and contrast.
>> 
>> I agree.
>> 
> Now that the ICN paper is out (https://conferences.sigcomm.org/acm-icn/2019/proceedings/icn19-20.pdf) a really useful exercise would be to have a discussion about the various tools and where they overlap, where they complement each other, and where they might conflict. Is there some chance we could get this together for Singapore?
> 
>>> 	• The forwarding model is pretty much separate from the Interest/Data forwarding procedures. I understand the authors’ motivation for this, but it does mean a lot of extra code in all the forwarders in order to have appropriate coverage. It also raises a number of possibly tricky resource allocation issues, since CCNInfo will be competing for both memory (i.e. PIT) and link bandwidth with Interest/Data while having somewhat different dynamic characteristics. For example, the Request messages will be large compared to regular Interests (especially so if they are signed), and the set of state stored in the PIT somewhat different, meaning you either have a separate data structure or a more complex joint PIT data structure.
>> 
>> I don't deny that CCNinfo requires additional implementation costs for forwarders. One (big) positive situation is that we've already implemented CCNinfo specified in the current draft into Cefore, which is an open source and can be referred by developers. Hope it contributes to the other forwarders' implementations.
>> In addition, the current CCNinfo specification allows to return null values for several fields such as First/Last Seqnum or Elapsed Cache Time fields in the Reply sub-block. (Section 3.2.1.1 says these values MAY be null.) It means that the forwarder can not only hide these values because of privacy/security policy, but also can skip the implementations of the complex functions to report these values.
>> 
> This makes sense. CCNinfo makes different tradeoffs from other approaches - we can analyze these with ongoing evaluation as we start to see deployed ICN networks.
> 
>>> 	• It was somewhat difficult for me to get a holistic grasp of the security properties of the CCNInfo Request/Reply protocol. For example:
>>> 
>>> 		• Requests can be signed by the initiator, but the request blocks are not individually signed. This seems to result in a fully transitive trust model upstream toward the FHR (CCNInfo terminology identifying the forwarder directly attached to a producer). Is this a problem? I’m not sure.
>> 
>> CCNinfo inherits the manner of the regular CCN for signing the messages only with publisher/consumer keys. CCNinfo additionally considers the lightweight access control using the node IDs. Except this access control, the current CCNinfo does not provide additional secure mechanisms by itself.
>> 
> I’m still hung up on the introduction of this new “namespace” of node ids. Why can’t we just name the nodes with CCNx names?
> 
>> Imposing on each router to verify and authenticate all of the request blocks is a heavy duty. However, if people prefer CCNinfo to verify whether the request/reply messages are sent from adjacent routers, I can add that requirement with SHOULD (or MAY? MUST is too strong) in the revision.
> No, I don’t think that’s necessary or even a good tradeoff since it would involve an initialization protocol to do the key exchanges hop-by-hop - unlike Hopauth, which provides the necessary transitivity without this extra work. At any rate, I said above “I’m not sure” and I’m still not sure what’s the right approach - it would be great if others could weigh in here.
> 
>> Note that HopAuth we have proposed in a separate draft is the general mechanism to verify and authenticate CCN messages along the forwarding path (as well as consumers/publishers). CCNinfo can cooperate with such external secure mechanism. I think mentioning such cooperation is another choice.
>> 
>> What do you think?
>> 
> Referencing Hopauth as a possible way to deal with the transitivity question would be good, in my opinion.
> 
>>> 		• Conversely, recording the identity of every upstream forwarder may represent a privacy problem. Again, the tradeoffs are difficult to assess, since there is the ability in the protocol for information hiding, and administrative controls at domain boundaries. However, the anonymity characteristics of Interest handling (the further you are away from the consumer, the harder it is to localize the consumer in the topology, discover the identity, or establish linkability) are compromised. Does this matter? I’m not sure.
>> 
>> Thank you for your comments.
>> 
>> Regarding "router/forwarder" privacy, in section 10.1, we say;
>> "according to the policy configuration, the Node Identifier field in the Report block MAY be null (i.e., all-zeros), but the Request Arrival Time field SHOULD NOT be null."
>> This means that CCNinfo allows forwarders to hide their Node IDs (e.g., node names, IP addresses) if they wish. However, if a forwarder sets its Node ID to null, its upstream routers cannot recognize and verify the forwarder with the Node ID. This is a dilemma of authentication vs privacy (or anonymity).
> Yes, modulo my desire to not introduce node ids in the first place.
> 
>> Note that as seen in section 10.2, currently it is not allowed consumers (i.e., CCNinfo users) to hide their Node IDs. It means that the Node Identifier field in the Request block MUST NOT be null.
>> Section 10.3 describes "topology" privacy and section 10.4 describes "content (or publisher)" privacy. However, I agree that all of these sections should be improved.
>> 
> Ok, good idea.
> 
>> As said above I can add the requirement of message verification from adjacent routers, but I believed we don't need to define other "CCNinfo oriented or original" secure mechanisms in this document as the expected threats are not CCNinfo oriented but resided in CCN itself.
>> What do you think?
>> 
> My sense is that this might be overkill, but I’d like to hear what others think.
> 
>>> 		• On a similar note, it appears the individual reply blocks in Reply messages are not signed so again CCNInfo appears to rely on transitive trust to protect the returned information. Even if they were to be signed, deciding whether the right key was used is not at all clear (this is related to another comment later on the question of how forwarders are named/identified).
>> 
>> Right, it is not addressed by the regular CCN nor CCNinfo at this moment as of above discussions. Should I add the requirement or method of message verification? Or can we rely on some other mechanisms, e.g., HopAuth?
>> 
> Designing something specific to CCNinfo strikes me as not the best way to address these problems. They aren’t uinque to CCNinfo; any distributed multi-object situation where there are intermediaries presents the same issues. In general this is a question of managing provenance and detecting things like dropping parts of the data in order to mislead the consumers. It’s probably not appropriate to saddle CCNinfo with attempting to solve this problem in a bespoke manner.
> 
>>> 	• Some of the statistics reported in CCNInfo are likely to be expensive to compute, and possibly quite algorithmically difficult to implement efficiently in high-performance implementations that shard the PIT & CS. It isn’t clear whether these ought to be done by CCNInfo or obtained instead by a direct application-layer management protocol talking to the management application in a forwarder node. It may be worth scrubbing these to only return basic information about an individual Data Object in a cache, repo or producer, and only return a return handle to use (via a separate protocol) to obtain the details for groupings like prefixes.
>> 
>> As I replied above, CCNinfo is not obliged to implement several complex functions, e.g., to report the values such as First/Last Seqnum or Elapsed Cache Time in the Reply sub-block. They can be reported with null.
> Ok. However having both basic information and extended information while good, doesn’t by itself deal with the question of computational attacks against forwarders using CCNinfo protocols. I’m still leaning toward leaving the complex stuff out of CCNinfo and using CCNinfo as a way to “bootstrap” more heavy weight instrumentation, either using RICE mechanisms to ask for the computations and data return, or a pub/sub system like that on the ICN paper referenced above. In the latter case you’d use CCNinfo to discover what you ought to be subscribing to.
> 
>> The current draft may not clearly mention that implementing the complex functions can be omitted or simplified in order to prioritize higher throughput.
>> We'll add some clarification in the revision. Thanks.
>> 
> Ok, and mentioning how omission might mitigate computational attacks would be useful in the security considerations section.
> 
>>> 	• CCNInfo creates a non-CCNx namespace for naming/identifying forwarders. Why? This seems to me both fraught with management complexity (allocation, duplication, security, etc.), at odds with the naming architecture of CCNx (or NDN) and frankly unnecessary. If instead CCNinfo used the existing namespace structure to name forwarders, things would (at least in my opinion) be much better:
>> 
>> I'm not clear your question.
>> The draft mention the possibility to use an IP address to identify a node, but it is not obliged at all. The current CCNinfo can use any identifier, say node name, for forwarder/publisher/consumer.
>> Or, are you saying we should not allow to use IP addresses for node identification because of some reasons? I prefer to less limitation, though.
>> 
> I’m objecting to a separate naming method for nodes/forwarders. I still don’t understand why the network operator doesn’t just give them the CCNx names like everything else.
> 
>>> 		• name allocation for forwarders is no different from that for any other data producer
>>> 		• by providing these names in CCNInfo Replies to the initiator, you now have a direct and easy coupling to a more comprehensive management protocol driven by regular interest/data exchanges (or for more sophisticated uses, by RICE).
>>> 		• security becomes a lot better integrated, as you can use all the existing trust schema work to decide what to sign, what to encrypt, and using what keys.
>> 
>> I'm sorry that I cannot understand your points. Are they addressed if CCNinfo uses "node name" for the request/reply messages? Or are you saying other things?
>> 
> Sorry if I’m being obtuse here. If we simply say that nodes have names in the operator’s namespace, all the regular machinery for signing, encrypting, and executing trust models becomes identical to what would be used for all other CCNx applications. Maybe we need to sit down and chat about this as we do seem to have a disconnect that I’m not explaining well.
> 
>>> 	• It appears flow balance is seriously violated by the discovery form of CCNInfo request/reply exchanges, as a single request can generate multiple replies (in pathological cases, an exponential number of replies since Request aggregation is turned off explicitly). We really need to deal with this. Some possible alternatives:
>>> 
>>> 		• quarantine relies at intermediate forwarders and combine them; returning only one “aggregate” reply on each downstream link.
>> 
>> We thought the reply aggregation previously, but to aggregate multiple replies, we impose on forwarders need to keep replies for some period and merge them into a single message. The concern here is that intermediate forwarders ought to deal with the message aggregation, which is an additional complex task and delays the reply. In addition, some replies are potentially lost or delayed due to various conditions, and it is impossible to recover the lost/delayed replies in any case.
> Yes, I agree that reply aggregation is likely to be an unattractive point in the design space. On the other hand, there are important implications for congestion control that (I think) have to be dealt with in some way. Perhaps using the machinery in my flow balance draft (https://datatracker.ietf.org/doc/draft-oran-icnrg-flowbalance/) might help here?
> 
>> There are papers (mainly for IoT to gather sensor data within a network) that adopt CCN/NDN message aggregation inside networks, but I couldn't find a perfect answer for in-network aggregation. Do you think we should mention some (maybe optional) mechanism for the message aggregation in the revision?
>> 
> No, let’s keep CCNinfo as is since this would be a pretty major addition without a clear benefit. I’d prefer to look at mechanisms that deal with the general problem of one logical request returning a multiplicative amount of data. One approach that I think may have merit (again not specific to CCNinfo) is to do things in two phases. The first phase explores through CCNinfo and takes a snapshot so the set of distributed data is close in time (i.e. within a single RTT), and returns a small handle (e.g. a name or name suffix) for each instance via CCNinfo. Then you go and issue individual unicast requests with conventional Interest/Dta or RICE to return the actual data, which would maintain flow balance and loosen the need for long timeout timers.
> 
> Let’s talk about this some time. Could be made general and integrated with CCNinfo and RICE if we want.
> 
>>> 		• Explicitly bound the number of replies as well as their size and communicate this in the protocol so the resource allocators can account for this in congestion management.
>>> Here are some more detailed comments on things that I found in reading the specification:
>>> 
>>> 	• Report blocks can get pretty big. The recommendation is to just give up and not report anything more if your reply would exceed an IPv6 MTU. Why? CCNx supports Data objects as large as 64K and current NDN restricts to 4K. Admittedly this has all sort of “interesting” fragmentation and congestion control implications (which are the subject of a new I.D. I plan to submit soon…but I digress). Given that there is no exclusion capability, nor steering capability it’s not possible to get beyond the “horizon” created by this limitation. I’m not advocating necessarily adding either of these, however the small MTU for replies does seem to be problematic for a network management tool.
>> 
>> Are you saying a big "request" or "reply"? In the current spec, CCNinfo returns an error for a fragmented "request" message, while it does not prohibit a fragmented "reply".
>> Honestly speaking, I forgot the precise reason why I had restricted fragmented "request".
>> I guess I considered the "request" message must not be fragmented as the "reply" message includes all request blocks of the original request message and then causes additional fragmentation on its way back. But I'm not sure..
>> 
>> Anyway, since CCNinfo does not rely on underlay, say IP networks, and I cannot remember the precise reason, we can remove the restriction of the fragmentation for any CCNinfo message (request/reply) and the NO_SPACE error code in the revision. What do you think?
>> 
> Well, you need a ”NO_SPACE” type error no matter what. Fragmenting requests is a lot trickier than fragmenting replies - if fragments are lost or rejected, the error cases are pretty hard to deal with in a simple way. I’d say CCNinfo is fine in not supporting request fragmentation.
> 
>>> 	• In describing the information returned from caches, it wasn’t entirely clear if the these sum all the things in the cache matching a given prefix or something else. If they are in fact prefix-matched, this really needs to be done in the background and not in the forwarding path given the expense of traversing the cache data structures. (Folks will recall we got rid of prefix-match on data in CCNx for exactly this reason, and NDN recently changed to prefer exact match as well). Once you do this it may be appropriate to ask why not just layer these capabilities of CCNInfo as an application (as does the NIST work for NDN) rather than bake it into the basic network layer packet forwarding. I don’t necessarily advocate this; in fact I would prefer to simplify CCNInfo to make it feasible to implement at high performance (see my general comment earlier about only returning simple stuff in CCNInfo and doing the complex stuff on top).
>> 
>> The exact name includes the chunk or segment number of the content, such as ccn:/abc/xyz/Chunk=10. CCNinfo supports exact match for chunk level trace, e.g., ccn:/abc/xyz/Chunk=10. In addition, it supports prefix match; it can "only" omit the chunk or segment numbers for the request. e.g., ccn:/abc/xyz/. CCNinfo does not trace with a prefix search like FIB that finds longer prefix match.
>> 
> Well, unless I misunderstand, even that would require a prefix match in the CS, since the CS might include some chunks and not others, and therefore an exact match would miss if the CS had only some of the chunks. So, if you get a CCNinfo request with a name the omits the chunk number, the CS lookup can’t be based on exact match. That means you either have LPNM in the CS (expensive) or you have to implement a special lookup just for CCNinfo that keeps track of whole objects and does an exact match that succeeds if and only if there is at least one chunk in the cache. It also means that any cache eviction algorithm has to go and check to see if the last chunk of an object has been evicted and remove the data structure entry that is searched for that specialized match.
> 
>> For content retrieval, you may want to only allow the exact match (i.e., chunk/sequence number must be specified). However, for discovering content/cache location/status, I thought we can allow to specify chunk numbers because of its usability.
>> I have a question. What do you think if we change the prefix matching is optional? I still believe the prefix matching is highly beneficial especially for research. How about changing the last paragraph of section 3.2.1.1 to the following sentence:
>>  "CCNinfo allows to specify an exact name of content (such as
>>   "ccn:/news/today/Chunk=10"). It is OPTIONAL to support a request
>>   with a content prefix name, which omits a chunk or segment number
>>   (such as "ccn:/news/today"). When a CCNinfo user specifies an exact
>>   name, s/he will obtain only about the specified content object in
>>   the content forwarder. When a CCNinfo user specifies a prefix name,
>>   s/he will obtain the summary information of the matched content
>>   objects in the content forwarder."
>> 
> My preference would be that instead of doing expensive stuff in the forwarding path, even if just for CCNinfo, we have a separate protocol based on RICE or the pub/sub scheme in the ICN’19 paper. This is in keeping with my general sense that CCNinfo might be better returning pointers/names of the things to be returned, and let the actual fetching be done separately.
> 
>> For the next comment. Application-layer vs. network-layer is the fundamental discussion. Both have pros and cons. Previously, we had developed an application-layer CCN measurement tool and published an article ("Contrace: A Tool for Measuring and Tracing Content-Centric Networks", IEEE ComMag, Mar. 2015), which is ref [6] in the draft. We inherited the major concept of this one to CCNinfo, yet we chose the network-layer measurement tool. The reasons are, in short, the standardized approach embedded into the standardized protocol is advantageous from the viewpoint of its deployment and interoperability, and type value reservation is better than name reservation or port number reservation for this kind of tool as sequentially increased type values are more manageable, and conceptually CCN works with any name for data forwarding or doesn't rely on TCP/UDP port number.
>> 
> So, I definitely agree that to do good instrumentation you need support in the network layer, and in that sense CCNinfo, like ping and traceroute, is a really important contribution. Where I have a different view is in whether a tree exploration protocol like CCNinfo should also be the mechanism by which rich data is computed and returned.
> 
>> For the last comment, I agree to high performance measurement (or minimized negative performance at least) and hence this draft allows to skip implementations of several complex (or heavy duty) functions.
>> 
> I get it, but of course if implementation is optional then many people won’t do it and the value is diminished. Perhaps there’s a slightly different “sweet spot” that is sufficiently cheap that it can be made mandatory yet still provide a “bootstrapping” capability to give you handles that can be used in the application layer to return complex data.
> 
>>> 	• I didn’t see use of the existing T_MTU_TOO_LARGE error when you run out of space (page 11). Did I miss it?
>> 
>> Instead of MTU_TOO_LARGE, we defined NO_SPACE error code. But as I said above, we can completely remove such fragmentation error and its code from the revision, if you agree.
>> 
> Ok, makes sense.
> 
>>> 	• Is 16 bits enough entropy for RequestID (page 12)? it may be ok if you don’t do any aggregation across consumers (which you currently don’t) and lifetimes are bounded very small as you currently do (4 seconds). It still makes me nervous though.
>> 
>> CCNinfo is a tool to discover the path information toward the caching forwarder/publisher along FIB and the cache status in the caching forwarder. It could be used by researchers and operators to measure/recognize the CCN conditions. CCNinfo request/reply are not very frequently happened like the ordinary interest/data exchange in my expectation. Hence IMO 16 bits Request ID is currently enough. But if people agree on enlarging the size, I can do it in the revision. The Reply timeout for full discovery is now 3 seconds as its default.
>> 
> Given that the RequestID isn’t a significant contributor to the size of the CCNinfo exchanges, making it big enough to avoid any questions about its entropy might be a good tradeoff.
> 
>>> 	• On page 14 you RECOMMEND routers have synchronized clocks. This too strong in my opinion, for three reasons:
>>> 		• you are throwing away the low order 16 bits of the NTP timestamp anyway
>> 
>> We truncate the 16 bits of the lower part of 32 bit fraction part of a second. This method (see the formula in page 14) is used by the standard protocol, e.g., Mtrace2 (RFC 8487, ref [8]). Precisely, it truncates about 15 micro sec as the maximum. We don't need this micro sec level preciseness for RTT measurement.
>> 
> Sure, I agree - I’m arguing that you don’t need closely synchronized clocks if you don’t need the resolution.
> 
>>> 		• loose synchronization is sufficient for the kinds of uses I think you want from CCNInfo.
>> 
>> To measure one-way latency or end-to-end RTT, time synchronization among routers can be omitted. For per-hop RTT measurement, however, this time synchronization is required.
>> 
> But then you’re also saying that you don’t need sub 15µs resolution on a given hop? 15µs is an eternity on a 100 gig datacenter link.
> 
>>> 		• If you expect CCNInfo to give you accurate per-hop and total RTT estimation using these clocks this isn’t terribly helpful given that the whole protocol runs on a different forwarding model, so you can’t use the CCNInfo RTT measurements to say much of anything about what real Interest/Data exchanges will experience. The alternative Ping and Traceroute proposals should do a much better job of this.
>> 
>> As said above, for the total (end-to-end) RTT measurement, time synchronization is not a mandate.
>> I don't deny the alternative approaches, but it is very useful for CCNinfo to have the functionality. If the word, "RECOMMEND", the current draft uses is strong for you/people, I may suggest to change the statement to the following one (it uses MUST but totally sounds optional as I use "if").
>>  "CCNinfo measures one-way latency and end-to-end RTT; however, if one
>>   wants to measure per-hop RTT as well, all the routers on the path MUST
>>   have synchronized clocks."
>> Is this statement acceptable?
>> 
> Sure, it’s acceptable to say that. Where I’m coming from is that uncertainties due to CCNinfo using different forwarding machinery from regular Interest/Data is likely to be a greater source of divergence from the actual performance of forwarding for normal Interest and Data that the fact that the numbers are really accurate for CCNinfo exchanges especially might not actually be all that helpful for instrumenting the forwarding plane. The likely different queuing behavior alone could cause order-of-magnitude or greater differences.
> 
>>> 	• Is the count of “received interests” (page 18) all those received or only those satisfied?
>>> 
>>> 	• I didn’t see much value in the Reply block information on First/Last Segnum or Elapsed Cache time. For these and some others it might help to given some examples of how one might use this.
>> 
>> First/Last Seqnum are the values to roughly expect the consecutiveness of in-network cache. They give a hint of better cache allocation in the network. You may be interested in the paper, "Consecutive Caching and Adaptive Retrieval for In-Network Big Data Sharing," Proc. IEEE ICC, May 2018. In the revision, we will add some text for and usage of these values and explain the situation with that reference. BTW, returning these values are "MAY", so that some CCN implementations can omit to report these values (by filling with null).
>> Elapsed Cache Time is used to design cache algorithms. We will explain a bit more for this value, too. Note this value is allowed to (MAY) be null as well.
>> 
> I’ll take a look at the paper. Thanks!
> 
>>> 	• in Section 4.2 you say you have to compare the number of report blocks with the hop limit. I think this means you have to remember the received hop limit in the PIT entry for the request, but I didn’t see that in the list of state you have to keep.
>> 
>> We mentioned it in section 4.1,
>>  "CCNinfo user's program MUST keep the following information; Request
>>   ID and Flags specified in the Request block, Node Identifier and
>>   Request Arrival Time specified in the Report block, and HopLimit
>>   specified in the fixed header."
>> 
> Ah, you’re right. Got it.
> 
>>> Minor stuff:
>>> 
>>> 	• move the actual packet type and TLV allocations to the IANA considerations section, and mark the values as “TBS” rather than picking them (yes…this might force you to change the implementation, but them’s the rules for RFCs…).
>> 
>> According to my experiments of IANA section in I-Ds, we can specify the pointers (i.e., sections) with the related statements such as "Initial values for the TLV Types are given in the table at the beginning of Section 3.2" in the IANA section, without completely moving the statements or tables. In fact, I prefer to this style since it's much readable.
>> For the values, Ok, I mark them as TBS. (What's TBS? Not TBA?)
>> 
> I meant TBA. And pointing to the tables is fine for now; if IANA objects, it can get changed during final RFC editor work.
> 
>>> 	• page 23 s/the fill discovery request/the full discovery request/
>> 
>> Will be done in the revision, thanks.
>> 
>> Thank you very much for your careful review.
>> It'd be great to hear your subsequent opinions.
>> 
> Hope you find this discussion helpful in moving CCNinfo forward!
> I hope you can come to Singapore and we can discuss this and other instrimentation/management ideas in further detail.
> 
> Best, DaveO.
> 
>> Best regards,
>> 
>> Hitoshi
>> 
>> 
>> 
>>> [End of comments]