[Tools-discuss] Re: Agenda Data Loading Slowly

[Robert Sparks <[e-mail]>]

Turns out it's already much closer.

I was distracted by the wrong view. The agenda page uses a different, 
but very similar view, at 
https://github.com/ietf-tools/datatracker/blob/ed6b061cfe4279328dbb7b914f5f7f76644521f9/ietf/meeting/views.py#L1745-L1746, 
which calls 
https://github.com/ietf-tools/datatracker/blob/ed6b061cfe4279328dbb7b914f5f7f76644521f9/ietf/meeting/views.py#L1678-L1742, 
which is also called by a celery task that forces that cache update, but 
only for the "current" meeting, where current is defined here: 
https://github.com/ietf-tools/datatracker/blob/ed6b061cfe4279328dbb7b914f5f7f76644521f9/ietf/meeting/helpers.py#L54-L56.

RJS

On 10/13/25 12:56 PM, Robert Sparks wrote:
>
> Sure, we already have such celery tasks for several of those slow views.
>
> Again, on a quick read, because the local cache for _this_ view is 
> controlled by the decorator I first pointed to, a celery task wouldn't 
> be able to invalidate that cache the way it should do. We'd need to 
> tweak the way the cache is managed to allow that to happen.
>
> The lift to serve from the edge (realized through putting the bits in 
> a bucket) is separable, but still important - most of those slow views 
> return large responses.
>
> RjS
>
> On 10/13/25 12:31 PM, Eric Rescorla wrote:
>>
>>
>> On Mon, Oct 13, 2025 at 10:15 AM Robert Sparks <[e-mail]> 
>> wrote:
>>
>>     First line at
>>     https://github.com/ietf-tools/datatracker/blob/ed6b061cfe4279328dbb7b914f5f7f76644521f9/ietf/meeting/views.py#L2312-L2443.
>>
>>     It's a simple origin-side cache, time is the only invalidator.
>>     More complicated invalidation would bring a new set of slow
>>     calculations.
>>
>>     Again, reworking this to use a celery task to push the result to
>>     a bucket on a similar period +/- so the browser never sees the
>>     time the work takes would be one incremental step we could make.
>>
>> It seems like an even more incremental step would just be to use 
>> celery to re-generate the top X slow view, thus keeping the cache 
>> warm. This would have similar timeliness and performance consequences 
>> properties but with no need to mess around with buckets.
>>
>> -Ekr
>>
>>     There's not a concrete schedule for doing that. It's highly
>>     likely that we can do this between 124 and 125, but there are a
>>     lot of other things in flight.
>>
>>     A larger refactor that would make a lot of the calculation in
>>     that view, such as a data structure that makes finding the
>>     times/rooms of all scheduled sessions on the official agenda in
>>     the set of all sessions scheduled in different time/rooms in
>>     draft agendas or those not scheduled at all,  be more efficient
>>     manner is needed - what's grown incrementally from the original
>>     scheduling management tooling is over-complicated at this point.
>>     _That_ level of refactor is much further out.
>>
>>     In the meantime, if you let me know what you're trying to take
>>     from the page, I might be able to point you to places that aren't
>>     doing a lot of the work that endpoint does that you might not be
>>     using.
>>
>>     RjS
>>
>>     On 10/13/25 11:50 AM, Eric Rescorla wrote:
>>>     On Mon, Oct 13, 2025 at 9:41 AM Robert Sparks
>>>     <[e-mail]> wrote:
>>>
>>>         Yes, the agenda is cached, and when the cache expires it is
>>>         recalculated - that recalculation is very complicated (as it
>>>         returns, for instance, the current versions of each
>>>         (agenda/slides/drafts) for each group).
>>>
>>>     Can you say more about how the caching works here? Is it just a
>>>     simple
>>>     time-based cache or something else?
>>>
>>>         We could (and will) refactor the page to distribute the the
>>>         work that's currently done in that one massive json load. We
>>>         can also move keeping a computed version published so the
>>>         browser never sees the time the computation takes, but
>>>         that's not going to happen before IETF 124.
>>>
>>>     Is there a timeline for this work?
>>>
>>>     -Ekr
>>>
>>>         RjS
>>>
>>>         On 10/13/25 10:59 AM, Eric Rescorla wrote:
>>>>
>>>>
>>>>         On Mon, Oct 13, 2025 at 8:54 AM Bob Hinden
>>>>         <[e-mail]> wrote:
>>>>
>>>>             Ekr,
>>>>
>>>>             For what it’s worth, I just went to both links, only
>>>>             took a few seconds for me.
>>>>
>>>>
>>>>         Yes, presumably it's being cached somewhere. However, it's
>>>>         not at all uncommon
>>>>         for this to be slow, and I've seen the then-current agenda
>>>>         take a long time on
>>>>         a number of occasions, so I do think it's worth addressing.
>>>>
>>>>         -Ekr
>>>>
>>>>
>>>>             Bob
>>>>
>>>>
>>>>>             On Oct 13, 2025, at 8:49 AM, Eric Rescorla
>>>>>             <[e-mail]> wrote:
>>>>>
>>>>>             Hi folks,
>>>>>
>>>>>             I just went to load the agenda for IETF 123
>>>>>             (https://datatracker.ietf.org/meeting/123/agenda) for
>>>>>             <reasons>.
>>>>>
>>>>>             It took 17s for the datatracker to send me the
>>>>>             underlying JSON file for the agenda
>>>>>             (https://datatracker.ietf.org/api/meeting/123/agenda-data)
>>>>>             It seems like it's faster on repeated load, so
>>>>>             presumably there is some caching going on, but 17s
>>>>>             still seems pretty excessive.
>>>>>
>>>>>             -Ekr
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>             <image.png>
>>>>>
>>>>>             -----------------------------------------------
>>>>>             Tools-discuss mailing list -- [e-mail]
>>>>>             To unsubscribe send an email to
>>>>>             [e-mail]
>>>>>             https://mailarchive.ietf.org/arch/browse/tools-discuss/
>>>>
>>>>
>>>>         -----------------------------------------------
>>>>         Tools-discuss mailing list [e-mail]
>>>>         To unsubscribe send an email [e-mail]
>>>>         https://mailarchive.ietf.org/arch/browse/tools-discuss/
>>>         -----------------------------------------------
>>>         Tools-discuss mailing list -- [e-mail]
>>>         To unsubscribe send an email to [e-mail]
>>>         https://mailarchive.ietf.org/arch/browse/tools-discuss/
>>>
>>
>> -----------------------------------------------
>> Tools-discuss mailing list [e-mail]
>> To unsubscribe send an email [e-mail]
>> https://mailarchive.ietf.org/arch/browse/tools-discuss/
>
> -----------------------------------------------
> Tools-discuss mailing list [e-mail]
> To unsubscribe send an email [e-mail]
> https://mailarchive.ietf.org/arch/browse/tools-discuss/