[p2pi] More workshop notes

This is a bit late in coming, but I also took notes at the workshop on  
May 28 and wanted to share them with the list (my colleague John  
Morris was on the program committee). I tried to capture anything that  
wasn't on a slide (and some of what was on the slides if it was  
necessary to understand the rest), so my notes ended up quite a bit  
longer than Spencer's.

I grouped paragraphs and back-and-forths by topic, so a blank line  
between paragraphs indicates a shift in the topic of conversation.

Being somewhat unfamiliar with the IETF community, there were several  
speakers whose names I didn't catch. They are listed simply as  
"Speaker" (or "Speaker 1" and "Speaker 2" where they might be confused  
for the same person within a back-and-forth).

Cheers,
Alissa

IETF p2pi Workshop
May 28, 2008
Cambridge, MA

3.1 Welcome/Note Well/Intro Slides – Cullen Jennings

Goals for the meeting:
1) Understand the technical problem behind Comcast/BitTorrent
2) Understand what IETF could do about the problem
	• figure out where in IETF this could happen
	• tasks that are feasible to finish
	• engineering, not research or politics
	• applicable to the entire Internet

We’re not dealing with illegal traffic.
Jessop (RIAA): We should deal in an engineering sense with illegal  
traffic.
Cullen: We have a congestion problem either way, and since we’re  
international, illegal/legal distinctions are tricky.

3.2 Comcast - Jason Livingood/Rich Woundy [16]

Livingood:

Apps providers learning they need to be more friendly, networks  
learning they need to be more apps friendly.

Comcast’s commitments:
1) protocol-agnostic network management
2) new focuses on transparency, disclosure, openness, fairness
3) DOCSIS 3.0 roll out

Woundy – What the Comcast network looks like:

In 2005-06 Comcast was getting lots of complaints about Vonage, other  
VoIP, some gaming. There was a congested link between Comcast  
transport provider and Vonage transport provider in Chicago area. At  
the same time, p2p traffic was growing, as was the correlation between  
heavy p2p and packet loss.

Comcast started managing traffic in 2006 (not an early adopter among  
ISPs). Customer complaints dropped as a result. Last official  
communication from Vonage was March 2006.

DOCSIS architecture: CPE --> CM --> HFC --> CMTS --> onward. CMTS  
allocates slots to transmit upstream. A single CMTS serves thousands  
of homes, has multiple DOCSIS domains. Usually a domain is single  
downstream with a number of upstream links.

Limiting resource is typically bandwidth. That’s what Comcast measures  
for capacity planning. CMTS has a notion of mini-slots. Large packets  
take up a lot of mini-slots. It’s not like ATM cells, or like WiFi  
where dominant cost is acquiring the link. Some discussion about size  
of mini-slots – answers offered are 6.25 microseconds worth of time,  
64 bytes (minimum Ethernet MTU), or 16 bytes (according to Google).

Any cable modem can make a request to CMTS, with requests randomized  
to minimize collisions. CM may piggyback next request in data PDU  
(next request is attached to previous data getting sent). CMTS sees  
these requests in reliable manner

With many CMs trying to do upstream requests, they collide, resulting  
in delays. Packet could be queued up to be next packet to CMTS for a  
full second.

Stanislav: Some motorola cable modems have a 4-second buffer
Speaker: If you measure capacity and speed, buffer size is always 32  
or 64 KB.

Back to Woundy:
It’s vendor-dependent. DOCSIS specification does not specify.

How TCP flow fairness applies in upstream direction is unclear. CMTS  
sees a bunch of service flows – could be many TCP flows or one (or  
UDP). CMTS doesn’t know source IP, dest IP, etc. until it arrives at  
the port, at which point flow has already been transmitted upstream.

Key point: With a lot of upstream traffic, piggybacking happens more,  
which sends heavy upstream users to front of line, over top of  
interactive but less-heavy apps. For example: With MAPs coming down  
between 1 and 3 ms, all requests are going through. If we assume  
packetization delay for a Vonage customer is 20-30 ms, the queue gets  
cleared out. Then when you go back to the request process, you get  
back into contention mode. This could explain how heavy users dominate  
upstream traffic more than interactive users.

A particular cable modem can have multiple service flows. Inbound  
traffic can be classified to particular service flow. Take the example  
of CM which is also a VoIP endpoint (e.g., for Comcast voice). You can  
have a service flow provisioned for voice so that it wouldn’t go  
through the upstream process just described – it wouldn’t need to keep  
requesting upstream space. There would be a separate service flow for  
regular best efforts.

Henning: How can Vonage request to be put in that service flow for VoIP?
Woundy: A cable company in Canada lets people pay extra for a service  
and it will provision a separate service flow for it (could be used  
for Vonage). The other way is to create a service flow on the fly, but  
Comcast doesn’t do this.

Cullen: Would looking at the DiffServ bits coming up from the CPE  
equipment be enough to classify into two different flows?
Woundy: Yes.

Speaker: Does Comcast voice service get affected by upstream  
congestion? What about for emergency calls?
Woundy: Each time transmissions are allowed, we will provision a  
reserved lane in next frame for voice.

Clark: Can you distinguish between interfering with yourself and  
interfering with your neighbor? Which problem do you think you have?  
Priority bits solve the first problem, not the second problem.
Woundy: We have the neighbor problem.

Livingood:

Engineering Activities
1) Congestion Management Improvements
2) P2P optimizations
	• near-term: tracker optimizations, caching, p2p client optimizations
	• long-term: signaling from network to clients (increase  
localization), signaling from clients to network (can declare yourself  
a scavenger service), improved resource allocation
3) DOCSIS 3.0

Speaker 1: We already have ECN and no one uses it. We also could  
resurrect scavenger class work.
Speaker 2: Woundy said problem is on access link, but tracker  
optimizations are for backhaul. And caching only helps on upstream.
Woundy: Yes, tracker optimizations are only for backhaul, but we still  
have to pay for transit. Caching could improve either upstream or  
downstream.

Robb: I’m worried about caching and physical network capacity. Peak-to- 
trough ratios will be higher with caches, and there’s more likelihood  
for contention with caches. If contention is from HFC and you’re  
trying to get timeslots on RF network, contention will be greater  
because you have caching.

Back to Livingood:
Will begin rolling out protocol-agnostic network management very soon.  
With this new net management, all traffic will be marked “priority.”  
When reaching “near congestion state,” we will start marking traffic  
down to best efforts. Threshold will be based on usage over certain  
length of time (“Long Duration Bulk Consumption State”). Length will  
be varied in trial. “Near congestion state” can occur on upstream or  
downstream.

Speaker: What feedback will users get? Anything a user can do to find  
out his/her traffic has been marked down to best efforts?
Livingood: There’s no mechanism for feedback right now.

Cullen: How do you get out of the penalty box?
Livingood: People come out of it quickly, but this is part of what we  
want to discover in trial.

Henning: I can see how this can be done for downstream. But how do you  
lower priority on upstream?
Woundy: We’re creating a new default DOCSIS service flow that has same  
provisioned bandwidth but lower priority.

Bob: This was an interesting decision to change QoS rather than vary  
bandwidth. BT is varying bandwidth allocation on its DSL network. Why  
did you chose QoS over reducing bandwidth?
Livingood: Reducing bandwidth is perceived negatively by users to  
reduce bandwidth.
Bob: But with your system, if an app wants to set its own priority, it  
has lost that ability, whereas reducing bandwidth doesn’t affect that.
Woundy: If congestion works itself out in 5 or 7 minutes it won’t be a  
problem.

Speaker: Will you factor in the subscriber’s tier?
Livingood: If you have 8 Mbit/s downstream service, you’ll have a  
different thresholds than lower service tiers.

Henning: So many tweakable parameters! There’s a lot of work in  
autonomic networking, using control theory approaches to adjust  
parameters based on feedback. Lots of people want to do this kind of  
traffic research, but problem is often unavailability of traffic load  
data.
Livingood: We want to work with research community on these things.

Oran: Does my VPN traffic not get marked down to best efforts when my  
son is using p2p upstairs?
Woundy: That is a desirable angle, but not something that we will do  
this year. DiffServ would be helpful with this.

3.3 BitTorrent -  Stanislav Shalunov [22]

Most important problem is increased end-user latency. Uplink is  
250-500 KB/s. Buffer is 32-64KB. Cheap devices don’t do active queue  
management (AQM). So TCP fills the buffer, and you end up with a 1-4  
second delay, which is a catastrophe from end-user standpoint. Mostly  
you’re interfering with yourself, but on cable you might affect the  
neighborhood.

BitTorrent peer selection is very random by design. “Rarest first” is  
useful because if original seed goes away, swarm will be able to  
continue; max amount of capacity can be registered away from original  
seed; and the rarer the pieces you get, the more likely you’ll have  
pieces somebody wants.

Tracker doesn’t track global piece distribution. “Rarest” is out of  
the peers that you’re talking to. The best approximation of global  
rarest first is an unbiased random sample.
Having a large selection of peers averages out problems. Best peers  
can be far away (connect to Japan or campus network).

Solving the problem of “RTT in Seconds:” We won’t get rid of the 4- 
second buffer, practically speaking. We won’t upgrade every end node  
in the Internet. But there are practical things that can be done
1) congestion control that avoids filling the buffer
2) scavenger service
3) more explicit congestion notification (not necessarily the two-bit  
ECN in IP header)

There’s a good reason to make peer selection random, so everyone  
should realize that not doing rarest first is a trade-off. But if we  
want to make overlays more efficient, the simplest way would be to  
prefer peers in same AS. Since they’re close, they might be faster. We  
have already added the IP to AS mapping table to our client. Not sure  
how useful this will be.

Could also prefer peers in a set of ASes (ISP could have multiple  
ASes), or
prefer peers in list of IP prefixes. For both of these, client needs  
to know the lists.

Could also run a cost minimization algorithm, but we need to know the  
costs to do this.

We did a case study with the four most popular torrents. With >= 20  
peers in an AS, they can just trade with each other. 50% of peers were  
in ASes like this. Two thirds of interdomain traffic went away. Note  
that user experience could get worse if ASes are asymmetric  
(inequality of upload/download) and there’s a flash crowd.

Since the most popular swarm size we had was 9984, the probability of  
two peers being in same neighborhood is so low that there’s no point  
in trying out localization on the neighborhood level first. Simple  
theories of localization should be tried first, and there should be  
substantial gains for doing this kind of localization.

Henning: Do you know the swarm size distributions?
Stanislav: Looks like polynomial distribution. Straight line on log- 
log scale. Not sure about what the tail looks like.

Stanislav: If last mile uplink is most expensive part, caching reduces  
uplink utilization.
Henning: Caches only help if they’re not part of the HFC, outside the  
access network. They can be run by anybody except someone who is on a  
cable modem.

Back to Stanislav:
For 30% hit rate inside 1 AS, cache need to be 1 TB; for 80% hit rate  
inside 1 AS, 100 TB.

Oran: What’s the bandwidth on and off the cache? We may not be storage- 
limited, we may be hotspot limited. You can’t just plunk down a 100 TB  
cache.

BitTorrent congestion control is already in 7 million clients in the  
wild. Using delay instead of loss as the metric. Continuously estimate  
one-way delay, separating propagation from queuing delay. Ramps up  
faster than TCP, but will yield to anything else that is not delay- 
based. Approximates scavenger service class with end-to-end congestion  
control.

Laird: Do you see congestion control as a best practice or standard  
technique?
Stanislav: It may be useful to standardize an experimental framework  
where things are not proprietary. Doesn’t make sense to standardize  
particular settings.
Lars: We already have an area in IETF for new congestion control  
protocols, this could be presented there.

Back to Stanislav:
Lower-hanging fruit for this group is DNS-based cache discovery  
protocol. Also getting information about the network for smarter peer  
selection – perhaps an export of something from BGP communities, or  
anything that can give clients cost information about different paths.

Clark: Back to question of whether you interfere with your neighbors.  
CMTS is involved in deciding whether to hand out slots to you or your  
neighbor. BitTorrent only works on certain slot allocation algorithms.

Oran: I’d like to know how this looks to a 10x under provisioned DSL  
network.
Bob: We run a DSL network that doesn’t have the same problems as cable.

Back to Stanislav:
IETF could also standardize a way for applications to find cache that  
provider might have installed. BitTorrent Engineering Proposal (BEP)  
22, which came out last week, is an attempt to sketch a way of how  
this could be done. We want better ideas of how to do it without DNS,  
without rewriting packets on the fly.

3.4 Lightning Talks

3.4.1 Robb Topolski [10]

P2P doesn’t really open hundreds of active connections or trick  
congestion control. It’s not designed to take up all bandwidth.

We should not rush to solutions before we understand the problems --  
why are networks so congested? Where is the problem? How bad is it,  
how often does it occur, how long does it last, what are the trends?  
What are the effects and deeper root causes? Can we leverage prior  
experiences and insight from the history of the Internet?

Quick fixes will only solve a problem for a particular application,  
but we should fix problems as close to root as possible.

3.4.2 Nick Weaver [1]

P2P does cost-shifting, not cost savings, and it does so  
inefficiently. P2P has as much data crossing ISP boundaries as  
uncached HTTP traffic, and for unpopular files you always have 2x bits  
crossing a boundary as HTTP because of uploading.

Caches are not the solution because of legal liability. ISPs also have  
no incentive to build caches to help someone shifting costs onto them.

3.4.3 Leslie Daigle [18]

We want continued creativity and innovation on the Internet. This  
conversation is not strictly about p2p.

There is an assumption that a network operator has no incentive to  
adjust network to meet demand.

We need to leave today understanding what problem we need to solve:
	• Make p2p work better?
		• Is it enough? Or what happens with next kind of app that comes
		• along?
	• Make networks and p2p apps aware of each others’ requirements?
	• General case of bandwidth-intensive apps, beyond p2p networks?
	• Deal with in-network issues, or impact of cross-network traffic?

3.5 Localization

3.5.1 P4P - Laird Popkin, Haiyong Xie [13/14]

Haiyong:

Caveats: P4P is not just for p2p. Localization is just a by-product of  
p4p, not a goal. It’s also not a tracker optimization approach.

Traditional feedback to apps is limited – routing is limited, rate  
control through TCP is coarse. At application level, TCP hides  
everything except success, failure and responsiveness. We want to  
expose more of what’s going on to apps.

Bob: But TCP hides it because it takes care of it.
Laird: If I’m trying to pick peers this might be useful information.

P4P Control Plane consists of an iTracker with interfaces for static  
topology, policy, provider capability, virtual costs, others (it’s  
extensible). iTracker can
be identified through DNS SRV, or run by third parties.

Virtual costs are provided by ISPs. Can reflect network status and  
policy (OSPF weights, higher costs on links with higher utilization,  
etc.).

Bob: How dynamic are costs expected to be?
Haiyong: We can compute costs by querying the router SNMP database, so  
can be very dynamic, or can be more static, it’s up to the ISP.
Laird: One ISP is instrumenting in real time (every 5 minutes). Others  
have precomputed daytime vs. nighttime distributions. Communication  
channel can update very frequently.

Laird:

Ran a test with Verizon and another ISP (missed the name?). With  
regular p2p, 98% of traffic is from sources external to ISP. With P4P:  
29% local, 21% more internal, 50% external, where “internal” means IP  
for both source and dest were within AS of Verizon. “Local” vs.  
“internal” was based on map of IPs provided by Verizon. This was a  
large swarm – 35,000 nodes global peak, 1000 nodes after that. 300-500  
peers internally on Verizon.

Nick: Would transit across border be same volume as uncached HTTP?
Laird: Yes, but nobody pays for transit that way.

Back to Laird:
P4P is a trade group, not a standards group, so it’s looking at ways  
to extend IETF protocols. A few ideas:
	• trackerless p2p: some mechanism (DNS?) to find iTrackers
	• tracker-based p2p: mechanism for peers to find their ASNs more  
easily than IP-lookup mechanism
	• enable p2p to “play nice”
	• enable apps to determine ISP usage policies and consumption against  
quota (like cell phone minutes)

Jon: Does P4P have its own DNS resource records?
Haiyong: It’s only a proposal.
Jon: What about your own TCP header? Anything else you’re doing with  
IETF standards?
Laird: The TCP header is for flagging information as being not-time- 
sensitive.

Henning: The peer discovery aspect relates to many other things (STUN,  
geo-location). If we discover things in one way, it makes sense to  
leverage that. We shouldn’t rush to deploy solutions.
Laird: We’re validating the premise, but turning it into a standard  
requires a lot more consensus. All we’ve proved is that localizing  
traffic improves performance and reduces cost.

3.5.2 Microsoft - Yu-Shun Wang [12]

Localization is useful for several reasons:
	• shifting costs to where we can get comfortable with current  
capacity of network
	• mitigate asymmetric capacity provisioning
	• limit traffic within an administrative scope – QoS, throttling,  
admission control

However, localization provides a non-uniform user experience (depends  
on your scope or location).

Designed a Multi-Layer Tracker-Based Architecture. Global tracker can  
delegate or redirect a peer to a local tracker. Provision local caches  
among seeds and peers. Nothing to prevent operator from setting up  
further sub-trackers. Content server and tracker could sit at same  
place (could be a hybrid CDN).

Good work for IETF to do may be tracker redirection/delegation. Could  
standardize a generic tracker format – can think of it in the abstract  
as going from global entity to local entity.

Haiyong: Don’t ISPs want to dump traffic off to other ISPs? Why would  
they localize?
Yu-Shun: What I’m hearing today is that localization could help.
Jon: Localization would not be the sole factor, just something to  
balance against other considerations.

Speaker: Everyone has a disincentive to do this kind of protocol.
Jon: Yu-Shun chopped it up into workable pieces. IETF is not going to  
take everything on. The tracker formats look like a good piece.
Henning: This is as much an economics problem. If there are incentive  
mechanisms available, then we don’t have to pick one – customer care  
is a good example. These won’t be the same everywhere, but there are  
plausible incentives out there.

Bob: Difficult decision for IETF – ISP won’t want to use this if you  
pick all the swarm hosts and work out the costs between them, because  
then the ISP is complicit in copyright infringement.
Speaker: Is merely answering a localization request about an IP  
address a legal issue? Without knowing who requests what?
Bob: We need a lawyer.

3.5.3 Vinay Aggrawal [28]

P2P is doing what link layer should have been doing (except at app  
layer). To us this means we need better ISP-P2P cooperation. ISP knows  
bandwidth, geography, service class, OSPF/BGP metrics, distance to  
peers, routing policy, etc. Instead of trying to reverse-engineer  
these, P2P should be able to find out some of this information.

Oracle service is hosted by ISP at known location. It gets list of IPs  
from each p2p node, sorts the list, and gives list back to node.  
Criteria is up to the ISP.

Henning: It would be in the interest of an ISP with upstream problems  
is to offload all of them onto Columbia University, and Oracle could  
facilitate that. Also, with oracle list mechanism, tracker gives list  
of randomly selected nodes – swarm can be large but tracker only picks  
a small handful to give to oracle. 10 out of 1000s, and you can’t send  
1000s of IPs to be sorted.

Oran: By rooting source at each individual swarm, result is not close  
to optimal Steiner tree for entire swarm. Input could instead be  
everyone else’s maps.

Laird: What are the scaling properties of this approach? In large  
swarm every peer will need a unique recommended list.
Vinay: Perhaps tracker could come to Oracle instead of peers dealing  
with Oracle. Scaling may need some DNS help for look-ups. Oracle is  
designed to be app-independent – helps ISPs remain outside copyright  
liability. Consulting Oracle also does not imply participation in file- 
sharing.
Speaker: Need to focus on scalability in light of growth of legal  
content. 9000 is not the top end of swarm size.
Stanislav: When you first join a swarm you would need to rank about  
200 peers, and once you’re established (within 1 minute), you will  
know of 2500 peers.

Jon: Modularity and application-independence are the appeal here. If  
IETF can leverage that, there may be a lot we can do. The computation  
that an Oracle does would not necessarily be an IETF focus.
Speaker: With localization built into routing layer already, this kind  
of system can scale even with very large lists.
Vinay: One possibility is to standardize the interface for the kinds  
of messages that could be sent between ISPs, peers, trackers.

Speaker: US and EU both have legal regimes that say caching is  
acceptable, as long as it looks just like Web caching.
Bob: It’s not about whether you supply the caching service, but if  
you’re pointing to content, that may be illegal even if you’re not the  
one caching it.

Haiyong: Peer ranking is a special case of virtual cost. Could use  
virtual cost to rank on the peers. How to handle interdomain ranking?
Vinay: Two ways. ISP could rank its own peers first, or could base  
ranking on AS number. Or, you could get multiple ISPs working  
together, have Oracles send lists to each other.

Speaker: When ISP is supposed to expect that clients will magically  
follow their guidance, that seems too much to expect. Why would  
clients do it?
Jon: There is an argument that there is an incentive to do this.

3.5.4 Cullen – What could IETF do on localization? [Summary slide from  
the morning]

Leslie: What problem are we trying to solve?

Stanislav: Finding AS numbers is not as valuable as other things that  
are on the slide because we already know how to do that.
Others: We don’t know how to do it.

Speaker: We’ve heard about three kinds of things: making clients less  
aggressive in bandwidth usage, moving data into caches so it’s not  
consuming uplink and downlink, and variety of ways to open up consumer  
backhaul capacity. It’s important to ask which of these three general  
approaches is worthwhile to do. Reducing backhaul doesn’t seem to be  
the highest priority. I’m terrified of legal issues with caching.

Robb: AS number discovery seems like a good idea.
Speaker: AS doesn’t help mobile IP and mobile – AS has nothing to do  
with where you actually are in the world.

Henning: Suggest that we follow the do-no-harm principle. Not  
everybody is going to be upstanding. If we provide info that cannot be  
verified by end user, can lead to undesirable behavior (hot potato  
routing magnified). Don’t want to provide info to users that gets them  
into trouble either.

Cullen: What interest is there in standardized tracker formats?
Stanislav: Don’t understand the notion of global tracker. Whatever  
local trackers there could be would be in addition to global  
(original) tracker?
Yu: Take current tracker format and make it recursive, to make  
selection easier. Stay within the local tracker once you’re there (can  
fall back to global tracker if necessary).
Lisa: HTTP servers often build their own indexes of mirrors in  
completely non-standard human-readable formats. This seems like  
another use case for this kind of solution.
Lars: Don’t need to do anything new, just deploy what we have.
Robb: This is very application-specific.
Stanislav: Would be cool to consult another tracker to find other  
peers, but for most tracker operators redirecting to other peers/ 
trackers is not going to happen.

Greg: Why don’t we consider the mechanism as a whole as being  
standardized? IM was successful, but there were boundaries about  
interoperability, so we brought it into the IETF.
Jon: The manner in which we brought IM into IETF was by modularizing.  
We brought in core pieces first to foster interoperability.
Lars: Transport area is doing some related stuff, like AQM schemes to  
help create shorter queues. Hasn’t been deployed yet because long  
queues weren’t a problem. Did DiffServ but no incentive to mark stuff  
if network doesn’t respond, and no incentive to respond if end points  
don’t mark. Experimental congestion control for high-speed long-delay  
networks is in the works, but nothing stops us from doing friendlier- 
than-TCP congestion control. Incentives are better for those sorts of  
protocols because they don’t require network to change with it.

Jon: A lot of people in industry talk about localization as a help to  
ISPs. People are going to be doing this regardless. Are people  
refuting the claim that localization is useful?
Speaker: It’s possible that localization has value, but we had a  
specific concern about a particular traffic management practice.
Jon: We’re looking for incrementally beneficial solutions.

Livingood: Next month Comcast is starting a localization trial and  
will be willing to share a great deal of data.
Jon: We need data.
Speaker: We have a localization implementation. We need to know who is  
requesting localization and who is delivering.
Henning: Alternative is to guess at data when we don’t have data.  
Lying about your own AS is detectable. Lying about which IP addr is  
closer is hard to detect. Not purely a problem related to p2p, we will  
be facing this is p2psip – how to find relay nodes so I don’t go to  
Australia to make a call to NY.

Bob: Network cannot say what costs are in other networks. Overlay can  
talk about costs in whole paths. Shouldn’t necessarily assume that  
network is source of information – overlay may be able to furnish that  
data across networks.

Lisa: Some things are feasible until you try to standardize them.  
Localization may be one of them. Some things work when you constrain  
it to one ISP and one kind of modem, but not more broadly. Problems  
with trust, motivation, and accuracy – how could oracles share lists  
even when they trust each other?

3.6 Congestion

3.6.1 Bob Briscoe [32]

Compare DiffServ and weighted congestion control. IETF shouldn’t judge  
balance of control between network and hosts – there is a role for  
this to be done on hosts if we get it right

DiffServ
	• APIs
		• DiffServ is done without APIs, not done on hosts.
		• App would need to know what the class choices are.
	• Control
		• Need API for apps to be able to do this, otherwise it’s network- 
controlled.
		• Even with APIs, how to know network is complying?
	• Policing
		• Wasn’t designed to work down to granularity of individual users
		• Could be done by volume? time of day? on-net vs. off-net?
			• But the whole problem is that peak period will shift once it’s  
declared.
		• Totally sender-based, but in p2p world people talk about limiting  
download, which is on receiver side.
	• Interconnect
		• if two networks work out deal, and I’m a light user sending to a  
heavy user, I use up my budget to be brought down in their network  
because the recipient is getting reduced priority.
	• Moving the problem: with heavy hitters in each class, you end up  
with class for every sort of traffic.

Weighted Congestion Control
	• APIs
		• Just a matter of setting the weight – extension of socket API.
		• Need consensus for it to be portable across OSes.
		• IETF doesn’t standardize socket APIs typically.
	• Control
		• Unambiguously with the host.
	• Policing
		• Why not always set weight to max? how to deal with this?
		• Don’t want light traffic bursts to puncture real-time apps.
		• Sender-based issue just life DiffServ.

There are a few features of good solutions. ISP should not have to  
fiddle around inside the user’s envelope (but this doesn’t mean ISPs  
can’t offer optional prioritization). There should also be no “wriggle- 
room” for suspicion about what ISP is doing.

We can view congestion as a harm (cost) metric, where the cost equals  
the amount of data that wasn’t served to its destination. This follows  
“no wriggle-room” – cost is greatest when peaks are greatest. Using  
this metric can allow for a policy that says not to send too much data  
during congestion, without specifying exactly how much.

Danny: Can you relate this metric to Comcast’s metric? Part of  
thinking about what is important to do from standardization  
perspective has to take into account what is going to be done by ISPs.
Bob: Two ways of looking at it. One is to set volume limits  
arbitrarily, but other is to set limits based on lengths of queues.

ISPs can’t limit what the can’t see. Need to give information about  
gaps to the network. This is the idea behind re-ECN, which makes it in  
sender’s interest to reveal the congestion it knows about.

Quick fixes need to be compatible with long-term goals. IETF/IRTF  
should setup longer-term design team to asses the general resource  
sharing problem and the long-term effects of any proposed quick fixes.

Lars: You did not mention p2p, which suggests that the problems that  
we have are with massive uses of bandwidth, not just p2p.

3.6.2 Marcin Matuszewski

Interviewed 20 ISPs. Their profit margins are low, with acquisition  
cost higher than monthly revenue. Cost structure varies (fixed vs.  
mobile). Heavy users are a perceived problem.

Tools available to ISPs (though none widely used): volume-based  
accounting, fixed price up to certain volume, shaping/blocking/ 
charging for excessive traffic, higher priority assigned to new flows,  
DPI, adding capacity, limiting subscriber flows, banning servers from  
residential access, superpeers/content caches, blocking heavy users.

Bob: BT’s retail department uses DPI and they’re open about it, and  
they think it’s 10x more cost effective than adding bandwidth.

Conclusion: heavy hitters are a business/marketing problem, not a  
technical problem.

3.7 QoS

3.7.1 Mary Barnes [24]

DPI and traffic analysis are not viable in long term. DiffServ Code  
Points (DSCP) could be helpful.

DS devices at edges mark packets so nodes don’t need to remember. This  
could support charging based on usage, which would be helpful.

Peterson: How does DSCP particularly lend itself to the p2p problem  
space?
Bob: P2P traffic is less than it used to be, relative to stream  
traffic. Problem has been around for 7 years, why the rush to solve it  
now?
Jon: Because of what happened last year.

Clark: Academics have done a lot of hand-wringing about lack of DS  
deployment, but DS is used in 50% of Cisco enterprise customers. We  
have an economic problem to solve on the broader internet.
Oran: Actually it’s larger than that now because VoIP is so popular.  
But some DSL networks are just not ready for DS. There are not a lot  
of layer 2 people in the room, but there are severe problems in layer  
2 networks. How do you trust the markings for DS? That issue will  
cause everything to come to a screeching to a halt.
Mary: That’s a common problem to a lot of these solutions.
Robb: Incentive for a host to do the right thing needs to be because  
it will benefit him to do the right thing.

3.7.2 Henning Schulzrinne [25]

Just minimizing cost to network provider while maximizing cost to  
content provider doesn’t help because content provider will just move  
networks.

It’s most useful to talk about total cost of delivery, including cost  
of media storage, media server bandwidth, and delivery bandwidth. Do  
you re-use existing components or use new components (e.g., personal  
DVR vs. cache)?

Rational cost shifting exists where people who can save the money  
funnel savings to those who can’t save the money. Rentable caches are  
an example – ISPs get paid to lend storage area so they are incented  
to keep caches. In some versions this is the Akamai model.

Transit bandwidth has flattened out, is no longer getting cheaper.  
Cost of providing content must take into account different costs on  
different links (costs look like a step function).

Bob: Cost of empty links is different than cost with traffic on the  
links. Whether you use long link or short link if it’s empty you pay  
the same.
Henning: Assume an end-user centric model where you buy bandwidth by  
the bit. The price that is charged to the customer and the price you  
pay for transit are not the same. In a fixed-price market there is not  
incremental cost. Each additional bit costs nothing until you reach  
capacity. Congestion doesn’t play a factor in cost today – you pay the  
same amount of money regardless of congestion on the link.
Bob: That is only true in the US.

Back to Henning:
One question is whether localization matters with long tails. One  
percent of the U.S. population saw Superbad on one weekend. If this  
was VoD, each neighborhood would have a copy.

With diurnal variations on some networks, average verses peak times  
doesn’t leave much left to shift demand around. On Columbia network  
only trough is in early morning.

There’s a general discovery problem for things like STUN, HELD, LoST,  
SIP local network config. These could all be informed by ISP. There  
are a number of discovery mechanisms: DHCP, DNS (SRV, NAPTR), anycast.

Scavenger service helps avoid self-interference and requires no  
admission control – can only do worse than you were doing before. The  
missing piece is to define the code point -- don’t want to hand code  
for each network.

For volume-based pricing, we need application-visible charging  
indicator. App could offer to defer a data transfer until later when  
bandwidth gets cheaper. Perhaps IETF could help here? Although  
business models are infinite. Also, charging or rate-limiting exposes  
users to risk of malware running up your traffic bill.

Conclusion: simple mechanisms are needed that help prevent self- 
interference and work for both symmetric and asymmetric. We also need  
more data about content distribution.

3.8 Conclusion and Wrap-Up

Jon: This idea of a generalized way to talk to your local network  
keeps coming up.
Henning: It would be useful to be able to ask a node how close I am to  
the limit.

Speaker: VoD costs or the quality of a VoIP call are all application- 
specific. Too hard to create generic transport-layer framework.
Henning: The ability to query local network provider to be able to  
detect what I’m getting myself into can be generic. Right now service  
providers cannot easily add metered services that are observable by a  
user (like electricity). Without a meter, you have no scalable way to  
built apps.
Jon: Even in P4P, ability for service providers to express policy is  
at the root of this. We want to extract out where the policies come  
from, interconnection charges, etc. Abstract it into a black box or  
express it as a consequence.
Speaker: Network can say it’s expensive or not, but not sure how much  
more granular we can get.

Cullen: We heard people saying there would be bandwidth caps.  
Scavenger class might be worth implementing if it doesn’t count  
against your bandwidth limit.

Robb: Two issues: congestion and cost. Congestion in last mile is a  
technical issue. Cost issue is a choice or decision not to part with  
your money.
Jon: But they’re related.
Robb: But the root cause matters. No reasonable amount of money will  
solve the congestion problem. But the existence of available bandwidth  
that is unused is a cost issue. Congestion control doesn’t fix the  
cost issue. When we don’t continue to grow the network, we’re just  
deciding how to order the packets at a gateway. IETF should decide  
what is strictly within business purview. IETF should focus on  
congestion issues.
Jon: This does not all fall into one bucket. People have raised the  
problem of how to make inelastic traffic compete reasonably with  
elastic traffic.

Danny: Big divide here in design areas to work in – pricing/signaling  
about what things cost and signaling about congestion and fairness  
behavior. There’s a long history of failure to match up network  
protocols with economic terms, so I’m wary of the first bucket. Skype  
figured it out, but it’s very hard to get it right (think  
micropayments). P3P tried to get between social and economic  
interactions and we didn’t get it right. More modest task of signaling  
fairness is more realistic.
Jon: Where does scavenger class fall?
Danny: If marking is related to action ISP will take, it’s in the  
first category. If it’s proposed as a currency, it’s in second category.
Clark: I’m uncomfortable with division between cost and congestion.  
Congestion which arises when there’s a scarcity needs to be resolved  
through some cost mechanism. People seem to be making value judgments  
that a p2p person will suffer less than other people. If in times of  
congestion I get to use the network and you don’t, you paid because  
you didn’t get to use it (exclusion). Can push back by asking users to  
pay, or slow down. Distinction between cost and congestion is false.  
ISPs are adding more capacity – they’re paying to relieve congestion.  
Many cost propositions haven’t worked, but that doesn’t mean they  
won’t in the future.
Danny: There’s a relationship between congestion and cost. We don’t  
understand it or how to communicate it in a transparent manner, and  
that’s the challenge. Cost questions may get worked out out-of-band  
from here.
Bob: If congestion isn’t limited, then cost and congestion aren’t the  
same thing. When congestion becomes part of the contract of what  
you’re not meant to do, then they become the same thing. Because an  
operator can’t see congestion, it can’t make it part of the contract.  
Can’t see congestion that user is causing elsewhere.
Speaker: An IRTF workgroup should go back to economics. In economics  
pricing is a protocol. It indicates when we should spend money. The  
problem we have is that people have discovered the Internet –  
assumption was that grandma would get a big web page back, but now she  
has a server.

Greg: There are a decent number of people interested in looking at  
caching, at least as a superpeer. Optimizing peer selection is  
something IETF could look at. I also encourage us to look at bringing  
the entire thing in, like with SSL and Jabber. As a vendor, DSCP is  
costly because whatever is in provider edge box has to do application  
identification in order to trust DS markings.
Cullen: Latest Windows Vista API is one where voice traffic is marked  
CS7, above priority reserve for router traffic. Every switch is going  
to have to remark all traffic from all Windows boxes, which means  
every box because you can’t distinguish a Windows box.

Nick: We might end up in a situation where every party is not  
necessarily happy. Instead the question is how to minimize damage in  
war between ISPs and p2p. Notions towards user fairness enforced in  
the network are where IETF should focus. That seems to be the least  
unpleasant for users who aren’t heavy users. Bandwidth caps, etc.  
cause those people harm.
Other Danny (not Weitzner): I agree with Nick. IETF might want to  
consider application employment of transport layer functions – apps  
opening all those TCP connections isn’t necessarily a good thing. IETF  
could say something about that. Mentality on mobile side – nights,  
weekends, distances for billing. All of this is transparent on  
network, but it costs more to deliver bits further away, and some ISPs  
are thinking about pricing based on some of these factors.

Lars: RFC 2914 explains what congestion control is good for. 1) Back  
off and, 2) being sort of fair to other users. Much of what we brought  
up today is not specific to p2p. If you’re a heavy user and you build  
up a queue, you will have delay – same for IPTV. I wonder if we can  
isolate these problems away from p2p. Also fairness – need to deal  
with heavy vs. light whether you’re using p2p or not.
Jon: Coming back to modularity and how to unify the modules. All the  
components we’ve talked about add up to p2p system.
Lars: No, the current cause is p2p. In the future the cause could be  
other things. Solving them for p2p might not solve them for next thing.

Laird: Issue of p2p opening lots of connections should end up in  
summary notes.
Stanislav: Multiple connections is not the cause of the problems. If  
you open one connection, you will get same delay that’s equal to  
buffer size in bottleneck link divided by bitrate of bottleneck link.  
Caused by Facebook photo uploader in same way as BitTorrent.
Nick: That is only true if you are the only person the link. With  
multiple users sharing one link, TCP flow fairness gives people with  
multiple open connections a larger portion of the bandwidth. Users  
with multiple torrents open can saturate the network.
Cullen: Gmail, Gmaps, Mosaic browser use 4-5 TCP connections. It’s not  
specific to BitTorrent.
Nick: How BitTorrent is commonly used by users is 4 flows per torrent  
with multiple torrents.
Oran: There are non-evil reasons to open multiple connections and you  
can’t tell the difference.
Stanislav: Piece dissipation is the reason for opening multiple  
connections. If you only trade with one person you end up with  
disjoint pairs. Your graph looks better with 3-4 peers.
Livingood: Ted Hardie talked about unattended apps – it’s not just  
p2p. P2P is as important as all the other apps out there.
Bob: I’m not saying multiple flows is a problem – the internet is a  
problem because it can’t judge how many flows people are using.
Jon: If we do not solve the problem for p2p, then we have not solved  
the problem.

_______________________________________________
p2pi mailing list
p2pi@ietf.org
https://www.ietf.org/mailman/listinfo/p2pi