Re: [dtn-users] A problem with dtntunnel

["[e-mail]" <[e-mail]>]

Hi David,

I want dtntunnel to be as transparent to a TCP application as possible.

Also I am not sure whether it will work or not, but it would be nice if in
case of connection disruption, or even in case of no connection at the time
when the sending application initiates the transfer, dtn could cache the
data that it received from the application and send it to the receiving
application when connection is restored. This will work only for limited
class of applications that only send data and don't expect replies from the
peer.

But in case if an application on either sending or receiving side *
explicitly* terminates the connection, closes socket, sends FIN (not the
case of lost connectivity), dtntunnel must do the same to the application
on the other side, so that it terminates correctly. This is exactly what
happens in case of a direct connection between application client and
server without dtntunnel.


2012/12/13 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <
[e-mail]>

> Hi Sergey,****
>
> I see now the end to end effect you are wanting and I’ll experiment some
> more and get back with you but it may be a few days.****
>
> ** **
>
> I guess one question is: ****
>
> If both ends are not up in “near real time” would you want the sender to
> stay connected until the receiving end comes on line and can return the FIN
> in all circumstances?****
>
> DZ****
>
> ** **
>
> David Zoller****
>
> COLSA Corporation****
>
> HOSC / C107  ****
>
> (Office: [phone]
> *EMail: [e-mail]****
>
> ** **
>
> *From:* [e-mail] [mailto:[e-mail]]
> *Sent:* Thursday, December 13, 2012 3:23 AM
>
> *To:* Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
> *Cc:* [e-mail]
> *Subject:* Re: [dtn-users] A problem with dtntunnel****
>
> ** **
>
> Hi David,
>
> I'll start from the end. I have repeated the test with 100 parallel flows
> twice, and it passed flawlessly. I think that was my mistake, I could have
> counted
> the number of erased connections too early, before all of 100 connections
> have terminated.
>
> Now to the first part. I have replaced while(1) with while(!sock_eof) as
> you suggested. This leads to very strange results when sending a small file
> with nc. Note the last two SYN and RST packets on the receiver side.
>
> Sender: tcpdump -nn -npi lo port 9999 or port 19999
> 13:10:35.136283 IP 192.168.1.2.32880 > 192.168.1.2.19999: Flags [S], seq
> 2925076366, win 5840, options [mss 1460,sackOK,TS val 90705589 ecr
> 0,nop,wscale 7], length 0
> 13:10:35.136346 IP 192.168.4.1.9999 > 192.168.1.2.32880: Flags [S.], seq
> 2931196344, ack 2925076367, win 32768, options [mss 16396,sackOK,TS val
> 90705589 ecr 90705589,nop,wscale 7], length 0
> 13:10:35.136371 IP 192.168.1.2.32880 > 192.168.1.2.19999: Flags [.], ack
> 2931196345, win 46, options [nop,nop,TS val 90705589 ecr 90705589], length 0
> 13:10:35.136799 IP 192.168.1.2.32880 > 192.168.1.2.19999: Flags [P.], seq
> 0:1, ack 1, win 46, options [nop,nop,TS val 90705590 ecr 90705589], length 1
> 13:10:35.136819 IP 192.168.4.1.9999 > 192.168.1.2.32880: Flags [.], ack 2,
> win 256, options [nop,nop,TS val 90705590 ecr 90705590], length 0
> 13:10:35.136892 IP 192.168.1.2.32880 > 192.168.1.2.19999: Flags [F.], seq
> 1, ack 1, win 46, options [nop,nop,TS val 90705590 ecr 90705590], length 0
> 13:10:35.176755 IP 192.168.4.1.9999 > 192.168.1.2.32880: Flags [.], ack 3,
> win 256, options [nop,nop,TS val 90705630 ecr 90705590], length 0
> 13:10:35.178602 IP 192.168.4.1.9999 > 192.168.1.2.32880: Flags [F.], seq
> 1, ack 3, win 256, options [nop,nop,TS val 90705631 ecr 90705590], length 0
> 13:10:35.178623 IP 192.168.1.2.32880 > 192.168.1.2.19999: Flags [.], ack
> 2, win 46, options [nop,nop,TS val 90705631 ecr 90705631], length 0
> 13:10:35.253829 IP 192.168.1.2.32881 > 192.168.1.2.19999: Flags [S], seq
> 2934914533, win 5840, options [mss 1460,sackOK,TS val 90705707 ecr
> 0,nop,wscale 7], length 0
> 13:10:35.253860 IP 192.168.4.1.9999 > 192.168.1.2.32881: Flags [S.], seq
> 2932249125, ack 2934914534, win 32768, options [mss 16396,sackOK,TS val
> 90705707 ecr 90705707,nop,wscale 7], length 0
> 13:10:35.253878 IP 192.168.1.2.32881 > 192.168.1.2.19999: Flags [.], ack
> 2932249126, win 46, options [nop,nop,TS val 90705707 ecr 90705707], length 0
> 13:10:36.380942 IP 192.168.4.1.9999 > 192.168.1.2.32881: Flags [F.], seq
> 1, ack 1, win 256, options [nop,nop,TS val 90706834 ecr 90705707], length 0
> 13:10:36.381495 IP 192.168.1.2.32881 > 192.168.1.2.19999: Flags [.], ack
> 2, win 46, options [nop,nop,TS val 90706835 ecr 90706834], length 0
> 13:10:36.398070 IP 192.168.1.2.32881 > 192.168.1.2.19999: Flags [F.], seq
> 0, ack 2, win 46, options [nop,nop,TS val 90706851 ecr 90706834], length 0
> 13:10:36.398131 IP 192.168.4.1.9999 > 192.168.1.2.32881: Flags [.], ack 2,
> win 256, options [nop,nop,TS val 90706851 ecr 90706851], length 0
>
> Receiver: tcpdump -nn -npi lo port 9999 or port 19999
> 13:10:35.185211 IP 192.168.4.1.44550 > 192.168.4.1.9999: Flags [S], seq
> 2927123945, win 32792, options [mss 16396,sackOK,TS val 90696412 ecr
> 0,nop,wscale 7], length 0
> 13:10:35.185266 IP 192.168.4.1.9999 > 192.168.4.1.44550: Flags [S.], seq
> 2929621602, ack 2927123946, win 32768, options [mss 16396,sackOK,TS val
> 90696412 ecr 90696412,nop,wscale 7], length 0
> 13:10:35.185286 IP 192.168.4.1.44550 > 192.168.4.1.9999: Flags [.], ack 1,
> win 257, options [nop,nop,TS val 90696412 ecr 90696412], length 0
> 13:10:35.206818 IP 192.168.4.1.44550 > 192.168.4.1.9999: Flags [P.], seq
> 1:2, ack 1, win 257, options [nop,nop,TS val 90696434 ecr 90696412], length
> 1
> 13:10:35.206861 IP 192.168.4.1.9999 > 192.168.4.1.44550: Flags [.], ack 2,
> win 256, options [nop,nop,TS val 90696434 ecr 90696434], length 0
> 13:10:35.222032 IP 192.168.4.1.44550 > 192.168.4.1.9999: Flags [F.], seq
> 2, ack 1, win 257, options [nop,nop,TS val 90696449 ecr 90696434], length 0
> 13:10:35.222133 IP 192.168.4.1.9999 > 192.168.4.1.44550: Flags [F.], seq
> 1, ack 3, win 256, options [nop,nop,TS val 90696449 ecr 90696449], length 0
> 13:10:35.222146 IP 192.168.4.1.44550 > 192.168.4.1.9999: Flags [.], ack 2,
> win 257, options [nop,nop,TS val 90696449 ecr 90696449], length 0
> 13:10:36.325396 IP 192.168.4.1.44551 > 192.168.4.1.9999: Flags [S], seq
> 2945716889, win 32792, options [mss 16396,sackOK,TS val 90697552 ecr
> 0,nop,wscale 7], length 0
> 13:10:36.325447 IP 192.168.4.1.9999 > 192.168.4.1.44551: Flags [R.], seq
> 0, ack 2945716890, win 0, length 0
>
> Another strange thing happend when I have run sending nc and forgot to run
> the receiving nc. Just try it yourself and see tcpdump.
>
> I have also conducted another test with the original dtntunnel code. I
> have run nc in the interactive mode, i.e. nc <ip> <port> on the client side
> and nc -l <port> on the server side. Then I terminated the connection on
> the client side by pressing CTRL+D. The server received a FIN packed from
> the client, replied with another FIN, and terminated. The client did not
> receive the FIN packet from the server and did not terminate. After that I
> have established a new connection and terminated it by pressing CTRL+D on
> the server side. This time server did send a FIN packet, the client got it,
> replied with its own FIN and terminated. The server did not receive the FIN
> from the client and continued running.
>
> This makes me think that dtntunnel, that have received the final FIN from
> a TCP application, doesn't transmit this FIN to its remote dtntunnel peer.
> Or the latter one doesn't retransmit this final FIN to the TCP application
> on its side. I'm not a TCP and sockets guru, so I cannot figure out how to
> translate this to the language of sockets, which I think is important to do
> in order to understand how to fix dtntunnel.
>
> I don't think that implementing an option that configures dtntunnel
> behavior in relation to closing connections is a good idea. I believe that
> there is a flaw in dtntunnel logic, and it could be fixed in a way that
> dtntunnel determines itself when to close a connection to a TCP application.
> ****
>
> 2012/12/13 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <
> [e-mail]>****
>
> Hi Sergey,****
>
> Research indicates that the TIME_WAIT state is a feature of TCP to prevent
> delayed packets for connection from being delivered to a later connection
> between the same hosts. [
> http://tools.ietf.org/html/draft-faber-time-wait-avoidance-00]****
>
>  ****
>
> Based on that, the added “if (sock_eof)” block can be removed and the
> while condition can be changed to “while (!sock_eof)” and you get the same
> result.****
>
>  ****
>
> But, looking at the intent of the code and at the oasys::IPSocket, it
> appears that it is possible to establish a socket connection and then only
> close the read side while keeping the write side open. I’ve never run
> across a socket used in that manner but that is what it looks like to me.
> So, the TCPTunnel::Connection::run  while loop allows for the read side to
> close and continue transmitting bundle payloads on the write side until it
> gets a bundle with the EOF bit set. If there are no bundles coming back the
> other way as is the case here then the while loop never exits which keeps
> the netcat connection open.****
>
>  ****
>
> So, based on that, I think a better solution would be to implement an
> option that configures dtntunnel to close the connection when the read side
> terminates. (Someone in the know can weigh in) Your solution is probably
> sufficient for your test.****
>
>  ****
>
> --****
>
>  ****
>
> As to connections not being removed from the table and sockets not being
> closed…****
>
>  ****
>
> I set up a test to kick off 100 netcats on an 11MB file so that the
> connection would be open for several seconds:****
>
> #!/bin/bash****
>
> ctr=0****
>
> while [ $ctr -lt 100 ]; do****
>
>     nc -w 10 x.x.x.x 12345 < data11mg  &****
>
>     let ctr=ctr+1****
>
> done****
>
>  ****
>
> I also added some debug code to print out a connection accepted counter
> and closed counter as they are accepted and closed.****
>
> After 30 connections, the first close kicks in and then there is a mix of
> connects and closes until finished.****
>
> All of the connections were closed and I did not have any sockets left
> open.****
>
>  ****
>
> On my first attempt I exceeded my payload quota and started seeing “error
> sending bundle: 141” messages so I don’t think that is the issue you are
> running into.****
>
>  ****
>
> You might try putting in similar debug to see if you can determine what is
> happening.****
>
>  ****
>
> Hope this helps,****
>
> DZ****
>
>  ****
>
>  ****
>
> David Zoller****
>
> COLSA Corporation****
>
> HOSC / C107  ****
>
> (Office: [phone]
> *EMail: [e-mail]****
>
>  ****
>
> *From:* [e-mail] [mailto:[e-mail]]
> *Sent:* Wednesday, December 12, 2012 2:07 PM****
>
>
> *To:* Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
> *Cc:* [e-mail]
> *Subject:* Re: [dtn-users] A problem with dtntunnel****
>
>  ****
>
> Hi David,
>
> I meant exactly the same position before the end of the while loop, just
> described it incorrect.
>
> Please let me know if you find the solution for this problem.
>
> Best regards,
> Sergey****
>
> 2012/12/12 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <
> [e-mail]>****
>
> Hi Sergey,****
>
> I was narrowing in on the same track... The FIN results in a returned
> length of zero on the read and errno is also zero so you have to key on the
> return length which is already in the code.****
>
>  ****
>
> You may have added a couple of lines of debug code and it should be
> inserted just before the end of the while(1) loop at line 491 in the latest
> version on Sourceforge.****
>
>  ****
>
> I am seeing the sending socket stay in the TIME_WAIT mode even if I add a
> call to sock_.shutdown(SHUT_RDWR) before closing the socket;****
>
> Getting close I think.****
>
>  ****
>
> The 0.0.0.0 was because I did not have anything at the other end of the
> tunnel and just stuck some easy numbers in there.****
>
> Thanks,
> DZ****
>
>  ****
>
> David Zoller****
>
> COLSA Corporation****
>
> Marshall Space Flight Center****
>
>  ****
>
> *From:* [e-mail] [mailto:[e-mail]]
> *Sent:* Wednesday, December 12, 2012 11:06 AM****
>
>
> *To:* Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
> *Cc:* [e-mail]
> *Subject:* Re: [dtn-users] A problem with dtntunnel****
>
>  ****
>
> Hi David,
>
> I added the following code at the line 492 of TCPTunnel.cc and it seems
> that it helped.
>
>     if (sock_eof) {
>         sock_.close();
>         goto done;
>     }
>
> The senders, both nc and iperf, exit after having finished sending the
> data. There are
> no more sockets in CLOSE_WAIT state. However I am completely unsure
> whether
> my code is correct and is in correct place.
>
> Now I have encountered another problem. When iperf test is run with 100
> parallel flows,
> some dtntunnel connections are not removed from dtntunel's connection
> table, and the
> related sockets are not closed. I suppose this happens because my code
> addition is far
> from perfect.
>
> And one more question, what does 0.0.0.0 mean in -T tunnel specification?
> Is it just an
> example host, or 0.0.0.0 has some special meaning for dtntunnel?****
>
> 2012/12/12 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <
> [e-mail]>****
>
> Hi Sergey,****
>
> I have duplicated what you are reporting and have to retract my earlier
> statement saying that this is the netcat design and not a problem J****
>
> The sender netcat attempts to close the socket when it gets the [EOF] and
> not in response to the receiver closing its end.****
>
>  ****
>
>  ****
>
> *netcat sender:*****
>
> cat /etc/passwd | nc x.x.x.x 12345****
>
>  ****
>
> *scenario 1 - netcat receiver with the keep listener option:*****
>
>                 nc -k -l 12345****
>
> ·         Sender terminates and receiver stays alive and listening for a
> re-connect****
>
> ·         FIN is sent in both directions initiated by the sender****
>
>  ****
>
> *scenario 2 - dtntunnel receiver:*****
>
>                 dtntunnel -t -T 12345:0.0.0.0:54321 dtn://desteid/xxxx****
>
> ·         Sender does not terminate****
>
> ·         FIN is sent from sender to receiver but not the other way****
>
> ·         Sender socket is in state FIN_WAIT2****
>
> ·         Receiver socket is in state CLOSE_WAIT****
>
> Kill the sender and repeat...****
>
> ·         Sender does not terminate****
>
> ·         FIN is sent from sender to receiver but not the other way****
>
> ·         New Sender socket is in state FIN_WAIT2 and the first one has
> timed out and died****
>
> ·         Both Receiver sockets are in state CLOSE_WAIT****
>
> Kill the dtntunnel receiver…****
>
> ·         2 FINs are sent from the receiver – 1 to each of the sockets
> (even though the other ends have expired)****
>
>  ****
>
>  ****
>
> I believe the issue would be in the oasys IPSocket or possibly the
> IPClient. I’ll have a look at it unless there is a low-level socket expert
> out there that wants to give it a go…****
>
>  ****
>
> Best regards,****
>
> DZ****
>
>  ****
>
> David Zoller****
>
> COLSA Corporation****
>
> Marshall Space Flight Center****
>
>  ****
>
> *From:* [e-mail] [mailto:[e-mail]]
> *Sent:* Wednesday, December 12, 2012 4:38 AM****
>
>
> *To:* Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]****
>
> *Subject:* Re: [dtn-users] A problem with dtntunnel****
>
>  ****
>
> Hi David,
>
> I have investigated that issue a little. I have found, that no matter
> whether netcat is run with or without the -w1 option, something
> strange happens. According to netstat -ntp on the sending node, dtntunnel
> remains in CLOSE_WAIT state after nc finishes sending
> data. I have run tcpdump on both sender and receiver, and it showed that
> final FIN from the receiver doesn't reach the sender.
> This looked like a bug in dtntunnel to me. I decided to re-check with
> iperf, and again, tcpdump showed the same problem.
>
> Here are the last lines of the tcpdump. Remember that in my setup all
> outgoing TCP packets with destination port 9999 are redirected
> with the help of IPTables to the local dtntunnel process, listening om
> port 19999. That is why I run tcpdump on loopback interface.
>
> Receiver (192.168.4.1): tcpdump -nn -npi lo port 9999 or port 19999
> 15:05:02.496366 IP 192.168.4.1.34284 > 192.168.4.1.9999: Flags [P.], seq
> 123995:131097, ack 1, win 257, options [nop,nop,TS val 11163723 ecr
> 11163718], length 7102
> 15:05:02.496428 IP 192.168.4.1.9999 > 192.168.4.1.34284: Flags [.], ack
> 131097, win 1154, options [nop,nop,TS val 11163724 ecr 11163723], length 0
> 15:05:03.457421 IP 192.168.4.1.34284 > 192.168.4.1.9999: Flags [F.], seq
> 131097, ack 1, win 257, options [nop,nop,TS val 11164684 ecr 11163724],
> length 0
> 15:05:03.497482 IP 192.168.4.1.9999 > 192.168.4.1.34284: Flags [.], ack
> 131098, win 1154, options [nop,nop,TS val 11164725 ecr 11164684], length 0
> 15:05:04.457762 IP 192.168.4.1.9999 > 192.168.4.1.34284: Flags [F.], seq
> 1, ack 131098, win 1154, options [nop,nop,TS val 11165685 ecr 11164684],
> length 0
> 15:05:04.457856 IP 192.168.4.1.34284 > 192.168.4.1.9999: Flags [.], ack 2,
> win 257, options [nop,nop,TS val 11165685 ecr 11165685], length 0
>
> Sender (192.168.2.1): tcpdump -nn -npi lo port 9999 or port 19999
> 15:05:02.433755 IP 192.168.4.1.9999 > 192.168.1.2.43360: Flags [.], ack
> 127449, win 386, options [nop,nop,TS val 11172887 ecr 11172875], length 0
> 15:05:02.433776 IP 192.168.1.2.43360 > 192.168.1.2.19999: Flags [.], seq
> 127448:130344, ack 1, win 46, options [nop,nop,TS val 11172887 ecr
> 11172887], length 2896
> 15:05:02.433783 IP 192.168.1.2.43360 > 192.168.1.2.19999: Flags [P.], seq
> 130344:131096, ack 1, win 46, options [nop,nop,TS val 11172887 ecr
> 11172887], length 752
> 15:05:02.444839 IP 192.168.4.1.9999 > 192.168.1.2.43360: Flags [.], ack
> 131097, win 386, options [nop,nop,TS val 11172898 ecr 11172887], length 0
> 15:05:03.421915 IP 192.168.1.2.43360 > 192.168.1.2.19999: Flags [F.], seq
> 131096, ack 1, win 46, options [nop,nop,TS val 11173875 ecr 11172898],
> length 0
> 15:05:03.461742 IP 192.168.4.1.9999 > 192.168.1.2.43360: Flags [.], ack
> 131098, win 386, options [nop,nop,TS val 11173915 ecr 11173875], length 0
>
> I believe that the cause of why the sending netcat without -w1 option
> doesn't exit, is that it doesn't receive the final FIN from the dtntunnel.
> Can this issue be fixed with a small amount of work?****
>
> 2012/12/5 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <
> [e-mail]>****
>
> Hi Sergey,****
>
> I would not even call this a problem. netcat is designed to be extremely
> flexible and by default the sender keeps its connection open as long as the
> other end does. On the other hand the receiver by default terminates after
> receipt of an EOF but there is a switch to keep listening if that is the
> desired behavior. ****
>
>  ****
>
> With DTN, your sending nc may do its thing while the destination node is
> not even on line and then an hour later it becomes available and completes
> the transmission to the receiving nc. In this scenario, the sender would
> still be “hung up” possibly indefinitely waiting for a terminating signal.
> ****
>
>  ****
>
> I am not the DTN2 authHi ority, but, I don’t see a change to dtntunnel for
> this as it would assume a specific usage that would probably break someone
> else’s usage (like mine J).****
>
>  ****
>
> Best regards,****
>
> DZ****
>
>  ****
>
> David Zoller****
>
> COLSA Corporation****
>
> Marshall Space Flight Center****
>
>  ****
>
> *From:* [e-mail] [mailto:[e-mail]]
> *Sent:* Wednesday, December 05, 2012 2:56 AM
> *To:* Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
> *Cc:* [e-mail]
> *Subject:* Re: [dtn-users] A problem with dtntunnel****
>
>  ****
>
> Hi David,
>
> Thanks for your help, nc -w1 did the trick. But shouldn't dtntunnel
> behavior be changed? The receiving dtntunnel could signal the sending
> dtntunnel, that the listening nc has disconnected, so that the sending
> dtntunnel close its connection to the sending nc. Or is this solely the
> problem of nc?****
>
> 2012/12/5 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <
> [e-mail]>****
>
> Hi Sergey,****
>
> On RHEL 5.7, I am running dtn-2.9.0 plus modifications that should not
> impact the behavior of dtntunnel.****
>
> I just ran your test without the iptables redirect and I see the same
> behavior.****
>
> I believe that the sender exits if you go directly from nc to nc because
> the receiver exits when it gets an end of file.****
>
> You can add a “-w 1” option to the sender nc so that it will timeout and
> exit after stdin is idle for 1 second.****
>
> Best regards,****
>
> DZ****
>
>  ****
>
> David Zoller****
>
> COLSA Corporation****
>
> HOSC / C107  ****
>
> (Office: [phone]
> *EMail: [e-mail]****
>
>  ****
>
> *From:* [e-mail] [mailto:[e-mail]] *On
> Behalf Of *[e-mail]
> *Sent:* Tuesday, December 04, 2012 11:13 AM
> *To:* [e-mail]
> *Subject:* [dtn-users] A problem with dtntunnel****
>
>  ****
>
> Hello all!
>
> I am using dtn-2.9.0 on a RHEL 6 based Linux distro and I am having a
> problem when using netcat (nc) with dtntunnel.
> On the sender node I run "cat /etc/passwd | nc receiver_ip". On the
> receiver node I run "nc -l 9999". With the help
> of Iptables port 9999 gets redirected to the port 19999, which is listened
> by dtntunnel.
>
> The file /etc/passwd is successfully delivered to the receiver and is
> shown on the screen. After this the receiving nc exists.
> No problem here. However, nc on the sender node doesn't exit after it
> sends the file. It continues to run forever. When I
> run nc in the opposite direction, I get the same problem - the sending nc
> doesn't exit.
>
> My configuration on both nodes is symmetric:
> iptables -t nat -A OUTPUT -d $REMOTE_HOST -p tcp --dport 9999 -j DNAT --to
> $LOCAL_HOST:19999
> dtntunnel -T $LOCAL_HOST:19999:$REMOTE_HOST:9999 $REMOTE_NODE/dtntunnel/nc
> -d
> dtntunnel -L --local-eid $LOCAL_NODE/dtntunnel/nc -d
>
> dtnping works ok in both directions.
>
> Please give me any advice.
>
> With best regards,
> Sergey Sireskin ****
>
>
>
>
> --
> Kindest Regards
>
> Sergey Sireskin
> FGUP CNII EISU****
>
>
>
>
> --
> Kindest Regards
>
> Sergey Sireskin
> FGUP CNII EISU****
>
>
>
>
> --
> Best regards,
> Sergey Sireskin****
>
>
>
>
> --
> Best regards,
> Sergey Sireskin****
>
>
>
>
> --
> Best regards,
> Sergey Sireskin
>
> ****
>



-- 
Best regards,
Sergey Sireskin