Re: [alto] IETF 114 ALTO Hackathon projects
Harvey Newman <newman@hep.caltech.edu> Mon, 25 July 2022 02:41 UTC
Return-Path: <hbnewman@gmail.com>
X-Original-To: alto@ietfa.amsl.com
Delivered-To: alto@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 07BA7C157B43; Sun, 24 Jul 2022 19:41:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.43
X-Spam-Level:
X-Spam-Status: No, score=-1.43 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.248, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fdFl5JoKuD-S; Sun, 24 Jul 2022 19:41:48 -0700 (PDT)
Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 48AEFC14CF01; Sun, 24 Jul 2022 19:41:48 -0700 (PDT)
Received: by mail-pg1-f179.google.com with SMTP id bh13so9146815pgb.4; Sun, 24 Jul 2022 19:41:48 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:reply-to :subject:content-language:to:cc:references:from:organization :in-reply-to; bh=lid9APAnhBnwLaO4JXoXbM0DhuntfhVUKL9egUQGkN8=; b=v3WDQa65jMDHUHQcvkSO1dXYdCYh/Diyz4Cvfn7jlqfiprf4PVVs+saXV1trxWUHXZ XMvg0ZQKrDTn2MOHtBn5MmSaV9UG+L1nKYd9xagjzxOJvp9g0OEGiQVlRTcdzuOJmt4v WqrNB0ol93BLfP6wfoZFVZYUga2PJtIcKTq5fnE3Y7PNe1Uc+bk0dPZf1UtwDcbvKrV1 O2IieD9a9CkcJvRsQnKAA4cu3gkovkdDocmERGNgNzodgdJOUJFzoAopsK9y5N6wrvSz VAWEq71yw56lda7q5OHUgfhnc6AdIr97hC0oQP9RSVKpu8Yp5wjHybrTHQSewt40n5k+ VQnA==
X-Gm-Message-State: AJIora+9DIce79bu245S6xULoXQy7OR1Sl1SQbX9Pxq4kQkVkT2oTv1z +Jg2QMKWgrj25hCfAKZJ3aQ=
X-Google-Smtp-Source: AGRyM1uXr8wrbt/FEEIbTeS3EESK+yu1sEK70vjLgZB1DwuCbtNeql/Q5TiwJV/sgZ7mnrbT3uOTIg==
X-Received: by 2002:a65:49c5:0:b0:412:6e3e:bd91 with SMTP id t5-20020a6549c5000000b004126e3ebd91mr9212116pgs.221.1658716907275; Sun, 24 Jul 2022 19:41:47 -0700 (PDT)
Received: from [172.16.5.251] ([23.152.80.66]) by smtp.gmail.com with ESMTPSA id 189-20020a6206c6000000b005254344bf48sm8177479pfg.5.2022.07.24.19.41.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 24 Jul 2022 19:41:47 -0700 (PDT)
Content-Type: multipart/alternative; boundary="------------YDRFdD7ZSpL0A0l0HD4Yebjw"
Message-ID: <356d3d7e-e1fe-11b8-fda9-bf3e08d79f66@hep.caltech.edu>
Date: Sun, 24 Jul 2022 19:41:44 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0
Reply-To: newman@hep.caltech.edu
Content-Language: en-US
To: "Y. Richard Yang" <yry@cs.yale.edu>, Mihai Patrascoiu <mihai.patrascoiu@cern.ch>
Cc: Jordi Ros Giralt <jros@qti.qualcomm.com>, Mario Lassnig <mario.lassnig@cern.ch>, Steven Murray <Steven.Murray@cern.ch>, "alto@ietf.org" <alto@ietf.org>, "alto-chairs@ietf.org" <alto-chairs@ietf.org>, "fts-devel@cern.ch" <fts-devel@cern.ch>, Ryan Yang <y.ryan.yang@gmail.com>, Jacob Dunefsky <jacob.dunefsky@yale.edu>
References: <SN6PR02MB5375149B197B2278E061009EF6869@SN6PR02MB5375.namprd02.prod.outlook.com> <CANUuoLrh5OiJ5OyHhqaJz-X08Zbh2s1x+p8Z-2jTz-ddi=4sQg@mail.gmail.com> <806717396.52304.1658390894774@cernmail.cern.ch> <CANUuoLoWdcFCcP3mPT+A4cgEVZOrXTyykgG-4g+-h22J9YMiZQ@mail.gmail.com>
From: Harvey Newman <newman@hep.caltech.edu>
Organization: Caltech
In-Reply-To: <CANUuoLoWdcFCcP3mPT+A4cgEVZOrXTyykgG-4g+-h22J9YMiZQ@mail.gmail.com>
Archived-At: <https://mailarchive.ietf.org/arch/msg/alto/HX93ulSCGn5JvUT-Bv7yTOa8ADg>
X-Mailman-Approved-At: Mon, 25 Jul 2022 01:34:52 -0700
Subject: Re: [alto] IETF 114 ALTO Hackathon projects
X-BeenThere: alto@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: "Application-Layer Traffic Optimization \(alto\) WG mailing list" <alto.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/alto>, <mailto:alto-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/alto/>
List-Post: <mailto:alto@ietf.org>
List-Help: <mailto:alto-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/alto>, <mailto:alto-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Jul 2022 02:41:52 -0000
That's great to hear Richard. And great thanks to the team. http: we should see that a modern version of http gets into components of XrootD.... Best regards Harvey On 7/24/2022 6:54 PM, Y. Richard Yang wrote: > Hi Mihai, Steve, Mario, all, > > This is an update on the IETF hackathon which just finished today > (July 24) in Philadelphia, PA. Mahdi presented the results to the > whole hackathon audience at slightly over 2 pm on behalf of the team. > The initial integration of ALTO and FTS and the zero-order algorithm > are fully functional (at least in the concurrent setting). > > This excellent progress is the result of hard work that takes a > village; Jensen stayed up 24 hours non-stop until 2 pm to finish up > the FTS production code integration framework, Mahdi was sleeping only > 4 hours per day in consecutive days to implement the full zero-order > gradient algorithm, Kai and his team made great progress on > implementing openalto with both standard features (path vector) and > next step features (FCS), and the team worked under the great > organization of Jordi. We plan to do some more demos and socials > tomorrow (Monday) at the IETF Happy Hour to try to get more IETF > feedback. The working slides are at: > https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true > <https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true> > > As soon as the IETF this week is over, we will schedule an overall > group meeting to go over the technical details, and plan for the next > steps. I consider HTTP a great example of a hugely successful > collaboration between the networking community (IETF) and the data > sciences (CERN). Let's use this project to promote collaboration > between these two communities again. > > Cheers, > > Richard for the team > > > > On Thu, Jul 21, 2022 at 4:08 AM Mihai Patrascoiu > <mihai.patrascoiu@cern.ch> wrote: > > Hello Richard, > > I'll summarize my answers from the other thread. > > Unfortunately, due to other arrangements, we won't be able to > attend the hackathon this weekend. > > However, we are very excited about the proposed objectives. We > already established a way for the FTS Optimizer zero-order > algorithm to make it's way into production. Once the feature is > ready, with the help of Mario, we can organize a large-scale test > and compare the results we get in production with the simulated > results. > > On further ALTO and FTS integration, one aspect I see particularly > promising is: > 1. Use ALTO information to assess the network capability of a > given RSE > 2. Set a maximum limit for how much network percentage FTS can > occupy. In a feedback loop via ALTO, FTS would find whether it may > increase or should decrease traffic involving that RSE > > This maps directly to what we see in production: FTS can hit some > sites too hard. > Any extra knowledge FTS gets on this aspect would allow it to > improve. > > Best Regards, > Mihai for the FTS team >> On 20/07/2022 03:58 Y. Richard Yang <yry@cs.yale.edu> wrote: >> >> >> Hi all, >> >> During the weekly ALTO meeting today, we discussed the coming >> hackathon and Qin suggested that we send an update to the mailing >> list on the FTS+ALTO project, and here is the update: >> >> - In short, if you want to work on ALTO integration >> with production, open projects, this can be a wonderful project >> to work on: it may help both ALTO (in terms of its crucial >> deployment mandate) and the largest data-intensive science >> projects at CERN. >> >> Specifically: >> - For those who have not tracked the hackathon, as Jordi said, it >> is a continuation of the 113 Hackathon. In particular, the 113 >> Hackathon focused on the integration with Rucio, which is a >> wonderfully designed tool used widely in CERN and some >> other projects for data movement of a large amount of data (PB -> >> EB). In 113, the hackathon modified the manual workflow, which is >> the workflow to select the source to download a dataset when a >> client issues a download command. The other workflow of Rucio is >> the automatic workflow, which selects the sources and >> destinations to realize user-specified replication rules. In our >> original plan, 114 would focus on the automatic workflow. >> However, the team realized that Rucio is built on top of FTS, >> which is the main tool used at CERN to schedule which transfer is >> scheduled at what time, where Rucio is at a higher layer >> providing the transfers for FTS to schedule. >> >> - The 114 hackathon includes 2 objectives: (1) introducing >> resource control to FTS, and (2) evaluating an alternative design >> of FTS core: the FTS Optimizer. Hence, the work includes 3 >> components, integrating the FTS production code and framework: >> (1/basic) allow FTS to specify resource control goal and use ALTO >> to map FTS control state (called links in FTS, where a link is a >> pair consisting of a source node to a destination node, where a >> node is called RSE) to network state; >> (2/basic) implement a full zero-order algorithm, to achieve fully >> efficient, zero-order gradient control as FTS Optimizer; >> (3/stretch) realize a composition framework to compose end-to-end >> resource performance function, including both zero-order and >> first-order gradient, covering the bottleneck structure. >> >> The project is exciting both in terms of its technical content >> and also in terms of its potential impacts. The hackathon project >> will work under the guidance of the Rucio project lead (Dr. Mario >> Lassnig) and the FTS project lead (Mihai Patrascoiu and Steven >> Murray); all cc'd in this email. This can be a wonderful >> opportunity for IETF, the networked systems community and the >> data-intensive sciences communities to work together. >> >> Please feel free to reach out to us (Jordi, Mahdi and me) if you >> want to join the hackathon or later. >> >> Thanks, >> Richard on behalf of the IETF 114 ALTO+FTS Hackathon Team >> >> On Tue, Jul 12, 2022 at 1:14 AM Jordi Ros Giralt >> <jros@qti.qualcomm.com> wrote: >> >> Hi all, >> >> I have uploaded to the IETF Hackathon Wiki the project >> description for ALTO. The proposed project is a natural >> evolution from the work done during the 113 Hackathon. I want >> to also thank and credit Jensen for providing the initial >> plan for the Hackathon. >> >> https://trac.ietf.org/trac/ietf/meeting/wiki/114hackathon >> >> >> You will also see that in addition to the ALTO >> hackathon, Ziyang will be leading a second project under the >> name "The SDN-based MPTCP-aware and MPQUIC-aware Transmission >> Control Model using ALTO". Many thanks Ziyang for leading >> this very interesting project too. >> >> Jordi >> On behalf of ALTO Hackathon group >> _______________________________________________ >> alto mailing list >> alto@ietf.org >> https://www.ietf.org/mailman/listinfo/alto >>
- [alto] IETF 114 ALTO Hackathon projects Jordi Ros Giralt
- Re: [alto] IETF 114 ALTO Hackathon projects Y. Richard Yang
- Re: [alto] IETF 114 ALTO Hackathon projects Mihai Patrascoiu
- Re: [alto] IETF 114 ALTO Hackathon projects Y. Richard Yang
- Re: [alto] IETF 114 ALTO Hackathon projects Harvey Newman
- Re: [alto] IETF 114 ALTO Hackathon projects Mario Lassnig