[dtn-users] DTN2 - Frequent Spinlock errors

[Nik Ansell <[e-mail]>]

Hello All,

I am running some experiments to identify bundle delivery characteristics
of DTN2 (+LTPlib), ION and IBR-DTN on the Raspberry Pi B2. Currently I am
trying to identify reliable bundle delivery scenarios for all 3 DTN
implementations, so I can then apply certain network simulation scenarios
to observe how each DTN implementation and convergence layer behaves.

The scope of my experimentation is below for info:

ION: UDPCL, TCPCL, LTPCL
DTN2: UDPCL, TCLCP, LTPCL - using LTPlib
IBR-DTN: UDPCL, TCPCL

I have determined reliable, repeatable scenarios for both ION and IBR-DTN,
however DTN2 is producing produce a number of errors when using UDP, TCP or
LTP. The errors also seem to appear randomly after any number of for loop
iterations, e.g. (5,13,50,48,95,413, 567, 988, etc etc).

Errors that cause dtnd to use 200% CPU and stop delivering bundles:
warning: deliver_front is waiting for spin lock held by (null), which has
reached spin limit
warning: Bundle::del_ref is waiting for spin lock held by (null), which has
reached spin limit
warning: Bundle::is_queued_on is waiting for spin lock held by (null),
which has reached spin limit
warning: LinkBlockSet::find_blocks is waiting for spin lock held by (null),
which has reached spin limit
warning: BundleList::erase is waiting for spin lock held by (null), which
has reached spin limit
warning: Bundle::add_ref is waiting for spin lock held by (null), which has
reached spin limit
warning: ForwardingLog::get_count is waiting for spin lock held by (null),
which has reached spin limit

Errors: that cause dtnd to quit:
LTP:ASSERTION FAILED (is_locked_by_me()) at thread/SpinLock.cc:91

Errors that seem to loop forever but not cause dtnd to quit or use 200% CPU
(This error seems to happen consistently after sending 99 LTP bundles):
/dtn/cl/ltp/sender error] Unable to create Sender LTP Socket in 5 seconds -
retrying

To send files I am calling dtnsource in a for loop as below, which allows
me to adjust the time between each bundle transmission, or each block of 10
bundle transmissions. For UDP and LTP I am using bundles of 63K, for TCP I
am using 1M bundles. I have tried several combinations for $WAIT (0-5
seconds) and $BULKWAIT (0-20 seconds), but cannot seem to find a working
combination.

for ((i=1;i<=END;i++)); do
    echo "($COUNT) dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE"
    dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE
    sleep $WAIT
    let "COUNT++"

    # Sleep for x secs after each 10
    if (( $COUNT % 10 == 0 ))
    then
    echo "Sleeping for $BULKWAIT seconds"
    sleep $BULKWAIT
    fi

done

The daemon is stopped and restarted after each test, I have also tried
deleting and recreating the berkley DB each time, but this does not seem to
have much effect. I have also managed to recreate some of the errors on a
virtual Ubuntu test-bed, specifically Bundle::del_ref.

After reading through the oasys SpinLock::lock code, it looks like this is
possibly an infinite loop that tries up to 1,000,000 times to get a lock,
then resets the counter to zero and puts out a warning message.

Has anyone had a similar experience or can suggest any troubleshooting tips?
Any suggestions to workaround or fix the problem will be very gratefully
received!

Kind Regards,
Nik