Manual

Getting Started

GHOST.jl can be installed from the repository through:

using Pkg
Pkg.add(url = "https://github.com/uva-bi-sdad/GHOST.jl")

to load the package, use

using GHOST

Licenses

GitHub can recognize certain licenses for repositories per their documentation. We filter out the machine-detectable licenses that are approved by the Open Source Initiative based on the SPDX Working Group SPDX License List data.

The following licenses are machine-detectable OSI-approved licenses.

Collection Strategy

Universe

We are interested in finding every repository on GitHub that fits the following criteria:

• Is public
• Has a machine detectable OSI-approved license
• Is not a fork
• Is not a mirror
• Is not archived

In the GitHub search syntax the following criteria is denoted by

{
  search(query: "is:public fork:false mirror:false archived:false license:$spdx created:2007-10-29T14:37:16+00..2020-01-01T00:00:00+00", type: REPOSITORY) {
    repositoryCount
  }
}

where $spdx a license keyword (e.g., mit).

created:2010-01-01T00:00:00+00..2010-01-01T12:00:00+00 1,850

created:2010-01-01T00:00:00+00..2010-01-01T12:00:00+00 998
created:2010-01-01T12:00:00+00..2010-01-02T00:00:00+00 952

We then prune intervals to obtain the least amount of valid intervals that cover the full time period.

For example,

The queries table is used to store the queries and track their status. Once all the records have been obtained for the repos table their done status becomes TRUE.

Repository base branch

The commit data for a Git repository is dependent on the base branch.

The repos table contains the GitHub repository global node ID and the global node ID for the base branch of the repository.

The various status values include:

• Ready: We will commence collecting commit data from it.
• Unavailable: Repository is not accessible (e.g., deleted of made private NOT_FOUND, DMCA takedown)
• Error: Something weird happened such as someone Git force pushing and changing the history during the scrape process.

Commits

For each repository, we query the commit data based on the time coverage of the data collection.

The commits table contains this data and is used to update the status of the repository commit data at the repos table.

Relational Database

How To Use

In order to use this package, refer to API section in the documentation, the examples in the test suite, the CI and pipeline scripts.

Requirements

• GitHub Personal Access Tokens with public access
• Julia v1 (current release v1.5.3)
• A PostgreSQL database connection (tested with v11-v13)