Re: [alto] IETF 114 ALTO Hackathon projects

Harvey Newman <newman@hep.caltech.edu> Mon, 25 July 2022 02:41 UTC

Return-Path: <hbnewman@gmail.com>
X-Original-To: alto@ietfa.amsl.com
Delivered-To: alto@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 07BA7C157B43; Sun, 24 Jul 2022 19:41:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.43
X-Spam-Level:
X-Spam-Status: No, score=-1.43 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.248, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fdFl5JoKuD-S; Sun, 24 Jul 2022 19:41:48 -0700 (PDT)
Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 48AEFC14CF01; Sun, 24 Jul 2022 19:41:48 -0700 (PDT)
Received: by mail-pg1-f179.google.com with SMTP id bh13so9146815pgb.4; Sun, 24 Jul 2022 19:41:48 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:reply-to :subject:content-language:to:cc:references:from:organization :in-reply-to; bh=lid9APAnhBnwLaO4JXoXbM0DhuntfhVUKL9egUQGkN8=; b=v3WDQa65jMDHUHQcvkSO1dXYdCYh/Diyz4Cvfn7jlqfiprf4PVVs+saXV1trxWUHXZ XMvg0ZQKrDTn2MOHtBn5MmSaV9UG+L1nKYd9xagjzxOJvp9g0OEGiQVlRTcdzuOJmt4v WqrNB0ol93BLfP6wfoZFVZYUga2PJtIcKTq5fnE3Y7PNe1Uc+bk0dPZf1UtwDcbvKrV1 O2IieD9a9CkcJvRsQnKAA4cu3gkovkdDocmERGNgNzodgdJOUJFzoAopsK9y5N6wrvSz VAWEq71yw56lda7q5OHUgfhnc6AdIr97hC0oQP9RSVKpu8Yp5wjHybrTHQSewt40n5k+ VQnA==
X-Gm-Message-State: AJIora+9DIce79bu245S6xULoXQy7OR1Sl1SQbX9Pxq4kQkVkT2oTv1z +Jg2QMKWgrj25hCfAKZJ3aQ=
X-Google-Smtp-Source: AGRyM1uXr8wrbt/FEEIbTeS3EESK+yu1sEK70vjLgZB1DwuCbtNeql/Q5TiwJV/sgZ7mnrbT3uOTIg==
X-Received: by 2002:a65:49c5:0:b0:412:6e3e:bd91 with SMTP id t5-20020a6549c5000000b004126e3ebd91mr9212116pgs.221.1658716907275; Sun, 24 Jul 2022 19:41:47 -0700 (PDT)
Received: from [172.16.5.251] ([23.152.80.66]) by smtp.gmail.com with ESMTPSA id 189-20020a6206c6000000b005254344bf48sm8177479pfg.5.2022.07.24.19.41.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 24 Jul 2022 19:41:47 -0700 (PDT)
Content-Type: multipart/alternative; boundary="------------YDRFdD7ZSpL0A0l0HD4Yebjw"
Message-ID: <356d3d7e-e1fe-11b8-fda9-bf3e08d79f66@hep.caltech.edu>
Date: Sun, 24 Jul 2022 19:41:44 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0
Reply-To: newman@hep.caltech.edu
Content-Language: en-US
To: "Y. Richard Yang" <yry@cs.yale.edu>, Mihai Patrascoiu <mihai.patrascoiu@cern.ch>
Cc: Jordi Ros Giralt <jros@qti.qualcomm.com>, Mario Lassnig <mario.lassnig@cern.ch>, Steven Murray <Steven.Murray@cern.ch>, "alto@ietf.org" <alto@ietf.org>, "alto-chairs@ietf.org" <alto-chairs@ietf.org>, "fts-devel@cern.ch" <fts-devel@cern.ch>, Ryan Yang <y.ryan.yang@gmail.com>, Jacob Dunefsky <jacob.dunefsky@yale.edu>
References: <SN6PR02MB5375149B197B2278E061009EF6869@SN6PR02MB5375.namprd02.prod.outlook.com> <CANUuoLrh5OiJ5OyHhqaJz-X08Zbh2s1x+p8Z-2jTz-ddi=4sQg@mail.gmail.com> <806717396.52304.1658390894774@cernmail.cern.ch> <CANUuoLoWdcFCcP3mPT+A4cgEVZOrXTyykgG-4g+-h22J9YMiZQ@mail.gmail.com>
From: Harvey Newman <newman@hep.caltech.edu>
Organization: Caltech
In-Reply-To: <CANUuoLoWdcFCcP3mPT+A4cgEVZOrXTyykgG-4g+-h22J9YMiZQ@mail.gmail.com>
Archived-At: <https://mailarchive.ietf.org/arch/msg/alto/HX93ulSCGn5JvUT-Bv7yTOa8ADg>
X-Mailman-Approved-At: Mon, 25 Jul 2022 01:34:52 -0700
Subject: Re: [alto] IETF 114 ALTO Hackathon projects
X-BeenThere: alto@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: "Application-Layer Traffic Optimization \(alto\) WG mailing list" <alto.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/alto>, <mailto:alto-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/alto/>
List-Post: <mailto:alto@ietf.org>
List-Help: <mailto:alto-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/alto>, <mailto:alto-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Jul 2022 02:41:52 -0000

That's great to hear Richard.

And great thanks to the team.

http: we should see that a modern version of http gets into components of
XrootD....

Best regards
Harvey


On 7/24/2022 6:54 PM, Y. Richard Yang wrote:
> Hi Mihai, Steve, Mario, all,
>
> This is an update on the IETF hackathon which just finished today 
> (July 24) in Philadelphia, PA. Mahdi presented the results to the 
> whole hackathon audience at slightly over 2 pm on behalf of the team. 
> The initial integration of ALTO and FTS and the zero-order algorithm 
> are fully functional (at least in the concurrent setting).
>
> This excellent progress is the result of hard work that takes a 
> village; Jensen stayed up 24 hours non-stop until 2 pm to finish up 
> the FTS production code integration framework, Mahdi was sleeping only 
> 4 hours per day in consecutive days to implement the full zero-order 
> gradient algorithm, Kai and his team made great progress on 
> implementing openalto with both standard features (path vector) and 
> next step features (FCS), and the team worked under the great 
> organization of Jordi. We plan to do some more demos and socials 
> tomorrow (Monday) at the IETF Happy Hour to try to get more IETF 
> feedback. The working slides are at: 
> https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true 
> <https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true>
>
> As soon as the IETF this week is over, we will schedule an overall 
> group meeting to go over the technical details, and plan for the next 
> steps. I consider HTTP a great example of a hugely successful 
> collaboration between the networking community (IETF) and the data 
> sciences (CERN). Let's use this project to promote collaboration 
> between these two communities again.
>
> Cheers,
>
> Richard for the team
>
>
>
> On Thu, Jul 21, 2022 at 4:08 AM Mihai Patrascoiu 
> <mihai.patrascoiu@cern.ch> wrote:
>
>     Hello Richard,
>
>     I'll summarize my answers from the other thread.
>
>     Unfortunately, due to other arrangements, we won't be able to
>     attend the hackathon this weekend.
>
>     However, we are very excited about the proposed objectives. We
>     already established a way for the FTS Optimizer zero-order
>     algorithm to make it's way into production. Once the feature is
>     ready, with the help of Mario, we can organize a large-scale test
>     and compare the results we get in production with the simulated
>     results.
>
>     On further ALTO and FTS integration, one aspect I see particularly
>     promising is:
>     1. Use ALTO information to assess the network capability of a
>     given RSE
>     2. Set a maximum limit for how much network percentage FTS can
>     occupy. In a feedback loop via ALTO, FTS would find whether it may
>     increase or should decrease traffic involving that RSE
>
>     This maps directly to what we see in production: FTS can hit some
>     sites too hard.
>     Any extra knowledge FTS gets on this aspect would allow it to
>     improve.
>
>     Best Regards,
>     Mihai for the FTS team
>>     On 20/07/2022 03:58 Y. Richard Yang <yry@cs.yale.edu> wrote:
>>
>>
>>     Hi all,
>>
>>     During the weekly ALTO meeting today, we discussed the coming
>>     hackathon and Qin suggested that we send an update to the mailing
>>     list on the FTS+ALTO project, and here is the update:
>>
>>     - In short, if you want to work on ALTO integration
>>     with production, open projects, this can be a wonderful project
>>     to work on: it may help both ALTO (in terms of its crucial
>>     deployment mandate) and the largest data-intensive science
>>     projects at CERN.
>>
>>     Specifically:
>>     - For those who have not tracked the hackathon, as Jordi said, it
>>     is a continuation of the 113 Hackathon.  In particular, the 113
>>     Hackathon focused on the integration with Rucio, which is a
>>     wonderfully designed tool used widely in CERN and some
>>     other projects for data movement of a large amount of data (PB ->
>>     EB). In 113, the hackathon modified the manual workflow, which is
>>     the workflow to select the source to download a dataset when a
>>     client issues a download command. The other workflow of Rucio is
>>     the automatic workflow, which selects the sources and
>>     destinations to realize user-specified replication rules. In our
>>     original plan, 114 would focus on the automatic workflow.
>>     However, the team realized that Rucio is built on top of FTS,
>>     which is the main tool used at CERN to schedule which transfer is
>>     scheduled at what time, where Rucio is at a higher layer
>>     providing the transfers for FTS to schedule.
>>
>>     - The 114 hackathon includes 2 objectives: (1) introducing
>>     resource control to FTS, and (2) evaluating an alternative design
>>     of FTS core: the FTS Optimizer. Hence, the work includes 3
>>     components, integrating the FTS production code and framework:
>>     (1/basic) allow FTS to specify resource control goal and use ALTO
>>     to map FTS control state (called links in FTS, where a link is a
>>     pair consisting of a source node to a destination node, where a
>>     node is called RSE) to network state;
>>     (2/basic) implement a full zero-order algorithm, to achieve fully
>>     efficient, zero-order gradient control as FTS Optimizer;
>>     (3/stretch) realize a composition framework to compose end-to-end
>>     resource performance function, including both zero-order and
>>     first-order gradient, covering the bottleneck structure.
>>
>>     The project is exciting both in terms of its technical content
>>     and also in terms of its potential impacts. The hackathon project
>>     will work under the guidance of the Rucio project lead (Dr. Mario
>>     Lassnig) and the FTS project lead (Mihai Patrascoiu and Steven
>>     Murray); all cc'd in this email. This can be a wonderful
>>     opportunity for IETF, the networked systems community and the
>>     data-intensive sciences communities to work together.
>>
>>     Please feel free to reach out to us (Jordi, Mahdi and me) if you
>>     want to join the hackathon or later.
>>
>>     Thanks,
>>     Richard on behalf of the IETF 114 ALTO+FTS Hackathon Team
>>
>>     On Tue, Jul 12, 2022 at 1:14 AM Jordi Ros Giralt
>>     <jros@qti.qualcomm.com> wrote:
>>
>>         Hi all,
>>
>>         I have uploaded to the IETF Hackathon Wiki the project
>>         description for ALTO. The proposed project is a natural
>>         evolution from the work done during the 113 Hackathon. I want
>>         to also thank and credit Jensen for providing the initial
>>         plan for the Hackathon.
>>
>>         https://trac.ietf.org/trac/ietf/meeting/wiki/114hackathon
>>
>>
>>         You will also see that in addition to the ALTO
>>         hackathon, Ziyang will be leading a second project under the
>>         name "The SDN-based MPTCP-aware and MPQUIC-aware Transmission
>>         Control Model using ALTO". Many thanks Ziyang for leading
>>         this very interesting project too.
>>
>>         Jordi
>>         On behalf of ALTO Hackathon group
>>         _______________________________________________
>>         alto mailing list
>>         alto@ietf.org
>>         https://www.ietf.org/mailman/listinfo/alto
>>