Re: [dtn-users] A problem with dtntunnel

"Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]" <david.a.zoller@nasa.gov> Wed, 12 December 2012 15:31 UTC

Return-Path: <david.a.zoller@nasa.gov>
X-Original-To: dtn-users@ietfa.amsl.com
Delivered-To: dtn-users@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F19C921E804A for <dtn-users@ietfa.amsl.com>; Wed, 12 Dec 2012 07:31:08 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.598
X-Spam-Level:
X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id i8z+3UN9nLJh for <dtn-users@ietfa.amsl.com>; Wed, 12 Dec 2012 07:31:03 -0800 (PST)
Received: from ndmsnpf01.ndc.nasa.gov (ndmsnpf01.ndc.nasa.gov [198.117.0.121]) by ietfa.amsl.com (Postfix) with ESMTP id 42DF621E80A3 for <dtn-users@irtf.org>; Wed, 12 Dec 2012 07:30:42 -0800 (PST)
Received: from ndmsppt02.ndc.nasa.gov (ndmsppt02.ndc.nasa.gov [198.117.0.101]) by ndmsnpf01.ndc.nasa.gov (Postfix) with ESMTP id 1A14C2606A8; Wed, 12 Dec 2012 09:30:31 -0600 (CST)
Received: from ndmshub06.ndc.nasa.gov (ndmshub06.ndc.nasa.gov [198.117.2.165]) by ndmsppt02.ndc.nasa.gov (8.14.5/8.14.5) with ESMTP id qBCFUUtm020717; Wed, 12 Dec 2012 09:30:30 -0600
Received: from NDMSSCC05.ndc.nasa.gov ([198.117.2.174]) by ndmshub06.ndc.nasa.gov ([198.117.2.165]) with mapi; Wed, 12 Dec 2012 09:30:30 -0600
From: "Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]" <david.a.zoller@nasa.gov>
To: "ssireskin@gmail.com" <ssireskin@gmail.com>, "dtn-users@irtf.org" <dtn-users@irtf.org>
Date: Wed, 12 Dec 2012 09:30:29 -0600
Thread-Topic: [dtn-users] A problem with dtntunnel
Thread-Index: Ac3YVLcoQqtPNWquR9O+gQw5+4BGUQAIaVug
Message-ID: <04E3D99A62496240BCD6A576813E6E31E0BDECD834@NDMSSCC05.ndc.nasa.gov>
References: <CAJR8z9--cVk67ac-aJ2haKpc=7LSVWHFXhTykaGcdpLQeevtiQ@mail.gmail.com> <04E3D99A62496240BCD6A576813E6E31E0BDBBA4FA@NDMSSCC05.ndc.nasa.gov> <CAJR8z98cbUhEPMyzR4Syp+Cd1xcg3Ei3u-UjykCQCGo2rBe9QA@mail.gmail.com> <04E3D99A62496240BCD6A576813E6E31E0BDBBA685@NDMSSCC05.ndc.nasa.gov> <CAJR8z98JoR2BGaSLer+u9k=Ok6iFroO0puqkDG2RpCzf=AxXng@mail.gmail.com>
In-Reply-To: <CAJR8z98JoR2BGaSLer+u9k=Ok6iFroO0puqkDG2RpCzf=AxXng@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative; boundary="_000_04E3D99A62496240BCD6A576813E6E31E0BDECD834NDMSSCC05ndcn_"
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.9.8327, 1.0.431, 0.0.0000 definitions=2012-12-12_05:2012-12-12, 2012-12-12, 1970-01-01 signatures=0
Subject: Re: [dtn-users] A problem with dtntunnel
X-BeenThere: dtn-users@irtf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "The Delay-Tolerant Networking Research Group \(DTNRG\) - Users." <dtn-users.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/dtn-users>, <mailto:dtn-users-request@irtf.org?subject=unsubscribe>
List-Archive: <http://www.irtf.org/mail-archive/web/dtn-users>
List-Post: <mailto:dtn-users@irtf.org>
List-Help: <mailto:dtn-users-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/dtn-users>, <mailto:dtn-users-request@irtf.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Dec 2012 15:31:09 -0000

Hi Sergey,
I have duplicated what you are reporting and have to retract my earlier statement saying that this is the netcat design and not a problem :)
The sender netcat attempts to close the socket when it gets the [EOF] and not in response to the receiver closing its end.


netcat sender:
cat /etc/passwd | nc x.x.x.x 12345

scenario 1 - netcat receiver with the keep listener option:
                nc -k -l 12345

*         Sender terminates and receiver stays alive and listening for a re-connect

*         FIN is sent in both directions initiated by the sender

scenario 2 - dtntunnel receiver:
                dtntunnel -t -T 12345:0.0.0.0:54321 dtn://desteid/xxxx

*         Sender does not terminate

*         FIN is sent from sender to receiver but not the other way

*         Sender socket is in state FIN_WAIT2

*         Receiver socket is in state CLOSE_WAIT
Kill the sender and repeat...

*         Sender does not terminate

*         FIN is sent from sender to receiver but not the other way

*         New Sender socket is in state FIN_WAIT2 and the first one has timed out and died

*         Both Receiver sockets are in state CLOSE_WAIT
Kill the dtntunnel receiver...

*         2 FINs are sent from the receiver - 1 to each of the sockets (even though the other ends have expired)


I believe the issue would be in the oasys IPSocket or possibly the IPClient. I'll have a look at it unless there is a low-level socket expert out there that wants to give it a go...

Best regards,
DZ

David Zoller
COLSA Corporation
Marshall Space Flight Center

From: ssireskin@gmail.com [mailto:ssireskin@gmail.com]
Sent: Wednesday, December 12, 2012 4:38 AM
To: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
Subject: Re: [dtn-users] A problem with dtntunnel

Hi David,

I have investigated that issue a little. I have found, that no matter whether netcat is run with or without the -w1 option, something
strange happens. According to netstat -ntp on the sending node, dtntunnel remains in CLOSE_WAIT state after nc finishes sending
data. I have run tcpdump on both sender and receiver, and it showed that final FIN from the receiver doesn't reach the sender.
This looked like a bug in dtntunnel to me. I decided to re-check with iperf, and again, tcpdump showed the same problem.

Here are the last lines of the tcpdump. Remember that in my setup all outgoing TCP packets with destination port 9999 are redirected
with the help of IPTables to the local dtntunnel process, listening om port 19999. That is why I run tcpdump on loopback interface.

Receiver (192.168.4.1): tcpdump -nn -npi lo port 9999 or port 19999
15:05:02.496366 IP 192.168.4.1.34284 > 192.168.4.1.9999: Flags [P.], seq 123995:131097, ack 1, win 257, options [nop,nop,TS val 11163723 ecr 11163718], length 7102
15:05:02.496428 IP 192.168.4.1.9999 > 192.168.4.1.34284: Flags [.], ack 131097, win 1154, options [nop,nop,TS val 11163724 ecr 11163723], length 0
15:05:03.457421 IP 192.168.4.1.34284 > 192.168.4.1.9999: Flags [F.], seq 131097, ack 1, win 257, options [nop,nop,TS val 11164684 ecr 11163724], length 0
15:05:03.497482 IP 192.168.4.1.9999 > 192.168.4.1.34284: Flags [.], ack 131098, win 1154, options [nop,nop,TS val 11164725 ecr 11164684], length 0
15:05:04.457762 IP 192.168.4.1.9999 > 192.168.4.1.34284: Flags [F.], seq 1, ack 131098, win 1154, options [nop,nop,TS val 11165685 ecr 11164684], length 0
15:05:04.457856 IP 192.168.4.1.34284 > 192.168.4.1.9999: Flags [.], ack 2, win 257, options [nop,nop,TS val 11165685 ecr 11165685], length 0

Sender (192.168.2.1): tcpdump -nn -npi lo port 9999 or port 19999
15:05:02.433755 IP 192.168.4.1.9999 > 192.168.1.2.43360: Flags [.], ack 127449, win 386, options [nop,nop,TS val 11172887 ecr 11172875], length 0
15:05:02.433776 IP 192.168.1.2.43360 > 192.168.1.2.19999: Flags [.], seq 127448:130344, ack 1, win 46, options [nop,nop,TS val 11172887 ecr 11172887], length 2896
15:05:02.433783 IP 192.168.1.2.43360 > 192.168.1.2.19999: Flags [P.], seq 130344:131096, ack 1, win 46, options [nop,nop,TS val 11172887 ecr 11172887], length 752
15:05:02.444839 IP 192.168.4.1.9999 > 192.168.1.2.43360: Flags [.], ack 131097, win 386, options [nop,nop,TS val 11172898 ecr 11172887], length 0
15:05:03.421915 IP 192.168.1.2.43360 > 192.168.1.2.19999: Flags [F.], seq 131096, ack 1, win 46, options [nop,nop,TS val 11173875 ecr 11172898], length 0
15:05:03.461742 IP 192.168.4.1.9999 > 192.168.1.2.43360: Flags [.], ack 131098, win 386, options [nop,nop,TS val 11173915 ecr 11173875], length 0

I believe that the cause of why the sending netcat without -w1 option doesn't exit, is that it doesn't receive the final FIN from the dtntunnel.
Can this issue be fixed with a small amount of work?
2012/12/5 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <david.a.zoller@nasa.gov<mailto:david.a.zoller@nasa.gov>>
Hi Sergey,
I would not even call this a problem. netcat is designed to be extremely flexible and by default the sender keeps its connection open as long as the other end does. On the other hand the receiver by default terminates after receipt of an EOF but there is a switch to keep listening if that is the desired behavior.

With DTN, your sending nc may do its thing while the destination node is not even on line and then an hour later it becomes available and completes the transmission to the receiving nc. In this scenario, the sender would still be "hung up" possibly indefinitely waiting for a terminating signal.

I am not the DTN2 authHi ority, but, I don't see a change to dtntunnel for this as it would assume a specific usage that would probably break someone else's usage (like mine :)).

Best regards,
DZ

David Zoller
COLSA Corporation
Marshall Space Flight Center

From: ssireskin@gmail.com<mailto:ssireskin@gmail.com> [mailto:ssireskin@gmail.com<mailto:ssireskin@gmail.com>]
Sent: Wednesday, December 05, 2012 2:56 AM
To: Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT]
Cc: dtn-users@irtf.org<mailto:dtn-users@irtf.org>
Subject: Re: [dtn-users] A problem with dtntunnel

Hi David,

Thanks for your help, nc -w1 did the trick. But shouldn't dtntunnel behavior be changed? The receiving dtntunnel could signal the sending dtntunnel, that the listening nc has disconnected, so that the sending dtntunnel close its connection to the sending nc. Or is this solely the problem of nc?
2012/12/5 Zoller, David A. (MSFC-EO60)[HOSC SERVICES CONTRACT] <david.a.zoller@nasa.gov<mailto:david.a.zoller@nasa.gov>>
Hi Sergey,
On RHEL 5.7, I am running dtn-2.9.0 plus modifications that should not impact the behavior of dtntunnel.
I just ran your test without the iptables redirect and I see the same behavior.
I believe that the sender exits if you go directly from nc to nc because the receiver exits when it gets an end of file.
You can add a "-w 1" option to the sender nc so that it will timeout and exit after stdin is idle for 1 second.
Best regards,
DZ

David Zoller
COLSA Corporation
HOSC / C107
*Office: (256) 544-1820
*EMail: david.a.zoller@nasa.gov<mailto:david.a.zoller@nasa.gov>

From: dtn-users-bounces@irtf.org<mailto:dtn-users-bounces@irtf.org> [mailto:dtn-users-bounces@irtf.org<mailto:dtn-users-bounces@irtf.org>] On Behalf Of ssireskin@gmail.com<mailto:ssireskin@gmail.com>
Sent: Tuesday, December 04, 2012 11:13 AM
To: dtn-users@irtf.org<mailto:dtn-users@irtf.org>
Subject: [dtn-users] A problem with dtntunnel

Hello all!

I am using dtn-2.9.0 on a RHEL 6 based Linux distro and I am having a problem when using netcat (nc) with dtntunnel.
On the sender node I run "cat /etc/passwd | nc receiver_ip". On the receiver node I run "nc -l 9999". With the help
of Iptables port 9999 gets redirected to the port 19999, which is listened by dtntunnel.

The file /etc/passwd is successfully delivered to the receiver and is shown on the screen. After this the receiving nc exists.
No problem here. However, nc on the sender node doesn't exit after it sends the file. It continues to run forever. When I
run nc in the opposite direction, I get the same problem - the sending nc doesn't exit.

My configuration on both nodes is symmetric:
iptables -t nat -A OUTPUT -d $REMOTE_HOST -p tcp --dport 9999 -j DNAT --to $LOCAL_HOST:19999
dtntunnel -T $LOCAL_HOST:19999:$REMOTE_HOST:9999 $REMOTE_NODE/dtntunnel/nc -d
dtntunnel -L --local-eid $LOCAL_NODE/dtntunnel/nc -d

dtnping works ok in both directions.

Please give me any advice.

With best regards,
Sergey Sireskin



--
Kindest Regards

Sergey Sireskin
FGUP CNII EISU



--
Kindest Regards

Sergey Sireskin
FGUP CNII EISU