CodeCommons

Open, responsible, and transparent AI: Our shared goal

CodeCommons is an ambitious project to create the world’s most comprehensive digital commons for code

Building on the existing foundation of Software Heritage, the largest publicly available source code archive, CodeCommons aims to bring into one place all the critical and qualified information needed to create smaller, better datasets for the next generation of AI tools.

At its core, the project prioritizes transparency and traceability, enabling model builders and users to respect creators' rights while promoting sovereign and sustainable AI.

Why CodeCommons?

spa

Sustainability

Minimizing the environmental and economic costs associated with repeated data collection

balance

Ethics

Ensuring AI respects authors’ rights and operates transparently.

Transparency

Guaranteeing traceable and reliable results for AI models

gavel

Sovereignty

Empowering nations to develop their own AI technologies

spa

Sustainability

Minimizing the environmental and economic costs associated with repeated data collection

Our vision

The power of AI is accessible to everyone and serves as a force for good
Developers and researchers have an unparalleled resource for building transparent, traceable, and ethical AI systems
Governments and organizations can ensure AI innovation aligns with principles of sustainability and sovereignty

What we're building

integration_instructions <1>
The world’s largest source code commons Enriched with billions of files, historical development data, metadata, and contextual links to scientific literature.
A scalable, unified data platform Allowing quick selection and extraction of specific code subsets designed for advanced AI training, with clear tracking using Software Heritage Identifiers (SWHIDs).
<2> computer
handshake <3>
Tools for ethical AI development Advanced tools to ensure compliance with copyright laws, assess code quality, and enhance AI reproducibility.
Sustainable infrastructure Partner with GENCI supercomputers to enable large-scale training of next-generation models while reducing environmental impact.
<4> lan
description <5>
Clear principles To ensure the ethical use of the Software Heritage archive for AI training, any machine learning models (ML) trained on our archive must be made publicly available under an open license, along with the necessary documentation and tools. The specific training data must be clearly identified using SWHIDs, enabling the assessment of biases, verification of data inclusion, and attribution of generated code. Mechanisms need to be in place to allow legitimate authors to exclude their code from training sets.

<1>
The world’s largest source code commons Enriched with billions of files, historical development data, metadata, and contextual links to scientific literature.
<2>
A scalable, unified data platform Allowing quick selection and extraction of specific code subsets designed for advanced AI training, with clear tracking using Software Heritage Identifiers (SWHIDs).
<3>
Tools for ethical AI development Advanced tools to ensure compliance with copyright laws, assess code quality, and enhance AI reproducibility.
<4>
Sustainable infrastructure Partner with GENCI supercomputers to enable large-scale training of next-generation models while reducing environmental impact.
<5>
Clear principles To ensure the ethical use of the Software Heritage archive for AI training, any machine learning models (ML) trained on our archive must be made publicly available under an open license, along with the necessary documentation and tools. The specific training data must be clearly identified using SWHIDs, enabling the assessment of biases, verification of data inclusion, and attribution of generated code. Mechanisms need to be in place to allow legitimate authors to exclude their code from training sets.

In February 2024, the BigCode project released StarCoder2, a state-of-the-art open AI model trained using the GitHub repositories archived in Software Heritage. This noteworthy release is proof that it's possible to develop high-quality models while adhering to rigorous principles of transparency and openness.

Resources

CodeCommons presentation
LLM issues addressed by CodeCommons:
- Gousios et al. (MSR 2012) GHTorrent: Github’s data from a firehose
- Gunasekar et al. (2023) Textbooks Are All You Need
- Lefeuvre et al. (REP’23) Fingerprinting and Building Large Reproducible Datasets
- Ledivarec et al. (ASE 2023) HyperDiff: Computing Source Code Diffs at Scale
- Sallam et al. (2023) ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns
- Fan et al. (FoSE 2023) Large Language Models for Software Engineering: Survey and Open Problems
ScanCode SCA tool by AboutCode
Software Heritage compressed graph
Software Heritage’s Statement on LLMs

Contact

Partners and teams

Key partners

Software Heritage - INRIA
Universal archive of source code
DiverSE - INRIA
Software engineering, code, programming, languages, managing software variability. Large-scale software evolution and generative AI for software development
ALManaCH - INRIA
Modeling and automatic linguistic analysis and computational humanities
CEDAR - INRIA
Analysis and processing of complex data on a large scale
DILS - CEA
Engineering, software, and systems
DIASI - CEA
Natural language processing, generative AI
Tweag - Modus Create
Machine learning, modeling, natural language processing, distributed computing

Additional partners

AboutCode
The global reference for license detection
Emérite Inria - Patrick Valduriez
Advanced expertise in massive data management
Sant'Anna
Data compression and text algorithms (ACM Paris Kanellakis award 2022)
University of Pisa
Expertise in massively parallel HPC programming
University of Bologna
Expertise in machine learning and text similarity
University of Turin
EuroHPC and expertise in efficient low-level distributed structures