Software Heritage: preserving the world’s source code

SoftWare Heritage’s (SWH) mission is to build the universal archive of all available software source code, preserve the precious knowledge it contains, and make it widely accessible [Di Cosmo and Zacchiroli, 2017]. With this mission in mind, it now collects, preserves and shares source code, which is accessible on the browsable archive. Figure 1 shows the most recent capture of the archive’s size. 

Started over 5 years ago by Inria with support from UNESCO, Software Heritage is a non-profit initiative supported by a growing number of research institutions, industry players, and governmental bodies. SWH has a dedicated team who make sure that the infrastructure is running and capable of responding to multiple stakeholders in a variety of situations. All the code developed for SWH is released under a Free and Open Source Software (FOSS) license.

How does it work?

Archiving all the source code is a daunting task and there are different mechanisms put in place to ensure the preservation of source code from different types of origins. Archiving a repository from a forge isn’t the same action as archiving source code from a package manager. It becomes even harder when you realize that version control systems have evolved a lot over the last decades. In Figure 2, you can find the evolution of version control, starting with simple tools that could only operate on a local machine, like RCS, then moving to systems that relied on a central server to allow concurrent modifications to a large software project, like CVS or Subversion, and finally to Distributed Version Control Systems (DVCS) that enabled fully distributed software development, without relying on any central server (e.g Git). The SWH archive harvests source code from different sources and converts all the source code into a single and universal data structure which is an enormous Merkle DAG [Merkle, 1987].

To understand a little better how the process works, you can find in Figure 3, the listing and loading tasks:

   listing 

gathering all the source code urls it can find.

   loading 

importing or updating a source code entry, which technically means inserting blob objects (files) in the object storage, and inserting nodes and edges in the SWH archive’s graph.

When loading the source code artifacts, SWH computes and exposes the SWHID intrinsic identifier for each artifact 

As shown in the Figure 4, the core of a SWHID identifier is made up by a prefix, swh (registered with IANA), a schema version, a tag to identify the type of the artifact it denotes (snapshot, release, revision, directory or content), and the cryptographic hash of the corresponding object, computed like for Merkle trees. It can also contain a set of qualifiers that provide the context of the artifact. Given a software artifact, everybody can compute the corresponding SWHID using a standard cryptographic algorithm, and can verify that it has not been modified.

You can save your code now!

It might take some time to get to every repository in the world, especially if these repositories keep on changing several times a day. This is why the “Save Code Now” service is provided, to give the possibility to notify SWH with a save request.

If you have a code repository, first check the archive by searching on the web platform.

If it is already archived, you can take a new snapshot by clicking on the camera icon on the right saying “save again”. Here an example with the ASPECT project, where you can find the latest visit, or all the visits.

If you can’t locate your source code, you can go ahead and click on “Save code now”, which is available on the menu on the left side of the archive’s platform. For interested developers, there is also a dedicated API endpoint.

By doing so, a priority task will be created for your repository and the content will be archived, after a moderator’s validation, usually a few hours later.