Re: [TOOLS-DEVELOPMENT] Backing up Github WG repositories

"Martin Thomson" <mt@lowentropy.net> Wed, 26 February 2020 19:43 UTC

Return-Path: <mt@lowentropy.net>
X-Original-To: tools-development@ietfa.amsl.com
Delivered-To: tools-development@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6CFF93A1294 for <tools-development@ietfa.amsl.com>; Wed, 26 Feb 2020 11:43:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level:
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=lowentropy.net header.b=dEKnfYwt; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=ZxpMAKWT
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n2dTMDKPAULh for <tools-development@ietfa.amsl.com>; Wed, 26 Feb 2020 11:43:10 -0800 (PST)
Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 78D9A3A1291 for <tools-development@ietf.org>; Wed, 26 Feb 2020 11:43:10 -0800 (PST)
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id D19A76DE; Wed, 26 Feb 2020 14:43:08 -0500 (EST)
Received: from imap2 ([10.202.2.52]) by compute1.internal (MEProxy); Wed, 26 Feb 2020 14:43:09 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lowentropy.net; h=mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type; s=fm1; bh=6IwCJcSb93sYauMKjahd7mzq7WU5tSt dovQisED/Eek=; b=dEKnfYwtNBJAcqzhXZQDdngtbkUpAj/RXpgf/biZcMXfXa9 Z+jcHEDh4rT8ebw6lCvRMJ54tJJFheryL7gGAe4rbaYJnNIjQ9W6jUbeypuvazVj 2S97B9LNaxOykDUIV4cUV9fahvf6TXsP1FG6AYRHWsN3U9aRuCoXD9PsldAZnVmw TFn4pFR1FdftD0nCbEKyTLxYJS6mE6CT5AB4iNr2GxCgrA4r/tC14HgnCdA9sIb2 utQ/sBN3FAebSUb9PGPqmLSQUFycD2+8m2xC2EAFpQMsmHgxAYYj0mHb0xx2rbYz IauZYoPwWIOo1in9+i8+NrveGChYx4SbsFQWH4g==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=6IwCJc Sb93sYauMKjahd7mzq7WU5tStdovQisED/Eek=; b=ZxpMAKWTHqY6Z2KsirRUlP oEFbJXp3iyZRP/QQeas08nz5CTJlCycq+CUdf8WRzdXOLWpUsBdtpisAY9bTeFpV NKJXloJ3GBd1UZMClM5LTMenysmP/MEeWytag0Y3wT6Ei3GwVFjjlD/+DR5wqB0g kpjp/CnSCk5BNv/5T16ZUNlvYZ1jNJmoKCmGxy/yUK6Xm0lDNpFDu9RSiHRLtezh II0UIaCnJGnTIbr4gGpf6O1i32wL51Od4geHUy5cadTBoeFxSAFqTXQVXAmgKr/+ i+bHyhKuWW0L+ERRBHFc/7s0wBamLJctxLTwyLr4ctIZcaXnQa6h8MQXeVclhGTQ ==
X-ME-Sender: <xms:TMpWXv7gg79xmFArO7wxMtYRGtcQFoTHeNhv-x8GJJx-btsCYCGTkg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrleeggdduvdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesthdtredtreertdenucfhrhhomhepfdforghr thhinhcuvfhhohhmshhonhdfuceomhhtsehlohifvghnthhrohhphidrnhgvtheqnecuff homhgrihhnpehgihhthhhusgdrtghomhenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehmtheslhhofigvnhhtrhhophihrdhnvght
X-ME-Proxy: <xmx:TMpWXvKyMsIjt5ad3tY_Scj_RWDmjEuLepyPSYDkwmetyVQA5wXp5A> <xmx:TMpWXr44nKwyimxKaDJlEwjRMUIExtYyv_4sYFUZmJqik6UtH01L3Q> <xmx:TMpWXmb8RTfwAGCMo98JqAX9913wKUch183l11D8qEF4ZxUZ-DqORA> <xmx:TMpWXhIIT4GBrqvM3xHHMOO2n6WmgSKaSniK1EmVhnbZW0M2ERHVSw>
Received: by mailuser.nyi.internal (Postfix, from userid 501) id EDAFCE00AC; Wed, 26 Feb 2020 14:43:07 -0500 (EST)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.1.7-947-gbed3ff6-fmstable-20200220v2
Mime-Version: 1.0
Message-Id: <a3b33093-03d4-4c78-b5d4-8709f60b9fd6@www.fastmail.com>
In-Reply-To: <2acd5e5e-0a1e-7658-964f-ca8d881b76f2@nostrum.com>
References: <9de49701-4477-291a-9a98-a39feb819a44@nostrum.com> <66380131-5b4e-4fa4-860c-08b240929e7f@www.fastmail.com> <2acd5e5e-0a1e-7658-964f-ca8d881b76f2@nostrum.com>
Date: Wed, 26 Feb 2020 12:42:46 -0700
From: Martin Thomson <mt@lowentropy.net>
To: Robert Sparks <rjsparks@nostrum.com>, IETF Tools Development <tools-development@ietf.org>
Content-Type: text/plain
Archived-At: <https://mailarchive.ietf.org/arch/msg/tools-development/A-DFKHUC_WF72QU9Qp8HAqWXql4>
Subject: Re: [TOOLS-DEVELOPMENT] Backing up Github WG repositories
X-BeenThere: tools-development@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Tools Development list server <tools-development.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tools-development>, <mailto:tools-development-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tools-development/>
List-Post: <mailto:tools-development@ietf.org>
List-Help: <mailto:tools-development-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tools-development>, <mailto:tools-development-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 26 Feb 2020 19:43:13 -0000

On Wed, Feb 26, 2020, at 12:00, Robert Sparks wrote:
> Hi Martin -
> 
> Have you tried using the organization migration api 
> (<https://developer.github.com/v3/migrations/orgs/> or something like 
> <https://github.com/josegonzalez/python-github-backup> against one or 
> more of the repositories you have the owner role for? It would be good 
> to know if either of those capture labels or comments.

So I have used the v3 "REST" API, which does offer the ability to capture comments.  (I think that is what you mean by the first reference there; the organization migration option isn't really relevant here, by my understanding.)  I just didn't use that capability, mostly because the API already requires a massive amount of data.  There is a graphql API (https://developer.github.com/v4/) that is better in that regard, but I haven't had time to play with it.

The API has information about labels. I have used that successfully.

The python tool appears to capture labels and comments, but I haven't tried it.  I don't know what format it stores data in. That might be critical to determining its suitability for this project.