Re: [alto] IETF 114 ALTO Hackathon projects

[Mario Lassnig <[e-mail]>]

Hi everyone,

just came back from holidays. This is fantastic news, and a big thank 
you to the people who invested so much of their time.

I'm really excited to try out the new algorithm for ATLAS.

Cheers,
Mario



On 25/07/2022 04:41, Harvey Newman wrote:
> That's great to hear Richard.
> 
> And great thanks to the team.
> 
> http: we should see that a modern version of http gets into components of
> XrootD....
> 
> Best regards
> Harvey
> 
> 
> On 7/24/2022 6:54 PM, Y. Richard Yang wrote:
>> Hi Mihai, Steve, Mario, all,
>>
>> This is an update on the IETF hackathon which just finished today 
>> (July 24) in Philadelphia, PA. Mahdi presented the results to the 
>> whole hackathon audience at slightly over 2 pm on behalf of the team. 
>> The initial integration of ALTO and FTS and the zero-order algorithm 
>> are fully functional (at least in the concurrent setting).
>>
>> This excellent progress is the result of hard work that takes a 
>> village; Jensen stayed up 24 hours non-stop until 2 pm to finish up 
>> the FTS production code integration framework, Mahdi was sleeping only 
>> 4 hours per day in consecutive days to implement the full zero-order 
>> gradient algorithm, Kai and his team made great progress on 
>> implementing openalto with both standard features (path vector) and 
>> next step features (FCS), and the team worked under the great 
>> organization of Jordi. We plan to do some more demos and socials 
>> tomorrow (Monday) at the IETF Happy Hour to try to get more IETF 
>> feedback. The working slides are at: 
>> https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true 
>> <https://docs.google.com/presentation/d/1CcCMvZ_SeWLRAN32Qz8RFcHrF8spAu_6/edit?usp=sharing&ouid=110882839032059047983&rtpof=true&sd=true>
>>
>> As soon as the IETF this week is over, we will schedule an overall 
>> group meeting to go over the technical details, and plan for the next 
>> steps. I consider HTTP a great example of a hugely successful 
>> collaboration between the networking community (IETF) and the data 
>> sciences (CERN). Let's use this project to promote collaboration 
>> between these two communities again.
>>
>> Cheers,
>>
>> Richard for the team
>>
>>
>>
>> On Thu, Jul 21, 2022 at 4:08 AM Mihai Patrascoiu 
>> <[e-mail]> wrote:
>>
>>     Hello Richard,
>>
>>     I'll summarize my answers from the other thread.
>>
>>     Unfortunately, due to other arrangements, we won't be able to
>>     attend the hackathon this weekend.
>>
>>     However, we are very excited about the proposed objectives. We
>>     already established a way for the FTS Optimizer zero-order
>>     algorithm to make it's way into production. Once the feature is
>>     ready, with the help of Mario, we can organize a large-scale test
>>     and compare the results we get in production with the simulated
>>     results.
>>
>>     On further ALTO and FTS integration, one aspect I see particularly
>>     promising is:
>>     1. Use ALTO information to assess the network capability of a
>>     given RSE
>>     2. Set a maximum limit for how much network percentage FTS can
>>     occupy. In a feedback loop via ALTO, FTS would find whether it may
>>     increase or should decrease traffic involving that RSE
>>
>>     This maps directly to what we see in production: FTS can hit some
>>     sites too hard.
>>     Any extra knowledge FTS gets on this aspect would allow it to
>>     improve.
>>
>>     Best Regards,
>>     Mihai for the FTS team
>>>     On 20/07/2022 03:58 Y. Richard Yang <[e-mail]> wrote:
>>>
>>>
>>>     Hi all,
>>>
>>>     During the weekly ALTO meeting today, we discussed the coming
>>>     hackathon and Qin suggested that we send an update to the mailing
>>>     list on the FTS+ALTO project, and here is the update:
>>>
>>>     - In short, if you want to work on ALTO integration
>>>     with production, open projects, this can be a wonderful project
>>>     to work on: it may help both ALTO (in terms of its crucial
>>>     deployment mandate) and the largest data-intensive science
>>>     projects at CERN.
>>>
>>>     Specifically:
>>>     - For those who have not tracked the hackathon, as Jordi said, it
>>>     is a continuation of the 113 Hackathon.  In particular, the 113
>>>     Hackathon focused on the integration with Rucio, which is a
>>>     wonderfully designed tool used widely in CERN and some
>>>     other projects for data movement of a large amount of data (PB ->
>>>     EB). In 113, the hackathon modified the manual workflow, which is
>>>     the workflow to select the source to download a dataset when a
>>>     client issues a download command. The other workflow of Rucio is
>>>     the automatic workflow, which selects the sources and
>>>     destinations to realize user-specified replication rules. In our
>>>     original plan, 114 would focus on the automatic workflow.
>>>     However, the team realized that Rucio is built on top of FTS,
>>>     which is the main tool used at CERN to schedule which transfer is
>>>     scheduled at what time, where Rucio is at a higher layer
>>>     providing the transfers for FTS to schedule.
>>>
>>>     - The 114 hackathon includes 2 objectives: (1) introducing
>>>     resource control to FTS, and (2) evaluating an alternative design
>>>     of FTS core: the FTS Optimizer. Hence, the work includes 3
>>>     components, integrating the FTS production code and framework:
>>>     (1/basic) allow FTS to specify resource control goal and use ALTO
>>>     to map FTS control state (called links in FTS, where a link is a
>>>     pair consisting of a source node to a destination node, where a
>>>     node is called RSE) to network state;
>>>     (2/basic) implement a full zero-order algorithm, to achieve fully
>>>     efficient, zero-order gradient control as FTS Optimizer;
>>>     (3/stretch) realize a composition framework to compose end-to-end
>>>     resource performance function, including both zero-order and
>>>     first-order gradient, covering the bottleneck structure.
>>>
>>>     The project is exciting both in terms of its technical content
>>>     and also in terms of its potential impacts. The hackathon project
>>>     will work under the guidance of the Rucio project lead (Dr. Mario
>>>     Lassnig) and the FTS project lead (Mihai Patrascoiu and Steven
>>>     Murray); all cc'd in this email. This can be a wonderful
>>>     opportunity for IETF, the networked systems community and the
>>>     data-intensive sciences communities to work together.
>>>
>>>     Please feel free to reach out to us (Jordi, Mahdi and me) if you
>>>     want to join the hackathon or later.
>>>
>>>     Thanks,
>>>     Richard on behalf of the IETF 114 ALTO+FTS Hackathon Team
>>>
>>>     On Tue, Jul 12, 2022 at 1:14 AM Jordi Ros Giralt
>>>     <[e-mail]> wrote:
>>>
>>>         Hi all,
>>>
>>>         I have uploaded to the IETF Hackathon Wiki the project
>>>         description for ALTO. The proposed project is a natural
>>>         evolution from the work done during the 113 Hackathon. I want
>>>         to also thank and credit Jensen for providing the initial
>>>         plan for the Hackathon.
>>>
>>>         https://trac.ietf.org/trac/ietf/meeting/wiki/114hackathon
>>>
>>>
>>>         You will also see that in addition to the ALTO
>>>         hackathon, Ziyang will be leading a second project under the
>>>         name "The SDN-based MPTCP-aware and MPQUIC-aware Transmission
>>>         Control Model using ALTO". Many thanks Ziyang for leading
>>>         this very interesting project too.
>>>
>>>         Jordi
>>>         On behalf of ALTO Hackathon group
>>>         _______________________________________________
>>>         alto mailing list
>>>         [e-mail]
>>>         https://www.ietf.org/mailman/listinfo/alto
>>>