API

GHOST

This is a module for collecting GitHub data about open source repositories and contributors.

GitHubPersonalAccessToken(login::AbstractString,
                          token::AbstractString,
                          )::GitHubPersonalAccessToken

A GitHub Personal Access Token

Fields

• login::String
• token::String
• client::Client
• limits::Limits

find_queries(spdx::AbstractString)

find_repos(batch::AbstractDataFrame)::Nothing

Takes a batch of 10 spdx/createdat and puts the data in the database.

graphql(obj::GitHubPersonalAccessToken,
        operationName::AbstractString,
        vars::Dict{String};
        max_retries::Integer = 3)

Return JSON of the GraphQL query.

licenses(conn::Connection,
         pat::GitHubPersonalAccessToken,
         schema::AbstractString = "gh_2007_$(Dates.year(floor(now(), Year) - Day(1)))",
         )::Nothing

Uploads the licenses table to the database. It includes every OSI-approved license that is machine readable with Licensee.

queries(conn::Connection,
        spdx::AbstractString,
        schema::AbstractString = "gh_2007_$(Dates.year(floor(now(), Year) - Day(1)))")

This will upload the queries to the database with:

• spdx::text NOT NULL
• created::tsrange NOT NULL
• count::smallint NOT NULL
• asof::time
• done::bool NOT NULL

query_commits(branches::AbstractVector{<:AbstractString}, batch_size::Integer)::Nothing

query_commits(branch::AbstractString)::Nothing

setup(;host::AbstractString = get(ENV, "PGHOST", "localhost"),
       port::AbstractString = get(ENV, "PGPORT", "5432"),
       dbname::AbstractString = get(ENV, "PGDATABASE", "postgres"),
       user::AbstractString = get(ENV, "PGUSER", "postgres"),
       password::AbstractString = get(ENV, "PGPASSWORD", "postgres"),
       schema::AbstractString = "gh_2007_$(year(floor(now(utc_tz), Year) - Day(1)))",
       pats::Union{Nothing, Vector{GitHubPersonalAccessToken}} = nothing)

Sets up your PostgreSQL database for the project.

Example

julia> setup(pats = [GitHubPersonalAccessToken("MyGH_Login", ENV["GH_PAT"])])

julia> setup()

setup_parallel(limit::Integer = 0; password::AbstractString = get(ENV, "PGPASSWORD", "postgres"))::Nothing

Setup workers.

GH_FIRST_REPO_TS::DateTime = 2007-10-29T14:37:16

Timestamp when the earliest public GitHub repository was created (id: "MDEwOlJlcG9zaXRvcnkx", nameWithOwner: "mojombo/grit")

GITHUB_GRAPHQL_ENDPOINT::String = "https://api.github.com/graphql"

GitHub API v4 GraphQL API endpoint.

GITHUB_REST_ENDPOINT::String = "https://api.github.com"

GitHub API v3 RESTful root endpoint.

Limits

GitHub API limits.

It includes how many remaining queries are available for the current time period and when it resets.

Fields

• limit::UInt16
• remaining::UInt16
• reset::DateTime

cleanintervals(row)

Returns the input if the count is 1,000 records or fewer. If there are more than a 1,000 it splits them based on the ratio of the count.

fill_missing_intervals(spdx::AbstractString, data::AbstractDataFrame)

find_repo_count_for_intervals(spdx::AbstractString, created::AbstractVector{<:Interval{DateTime}})

format_tsrange(obj::Interval{ZonedDateTime})

Return the Postgres compatible form.

parse_author(node)::NamedTuple

This parses the email, name, and ID of the author node.

parse_commit(branch, node)::NamedTuple

This parses a commit node and adds the branch it queried.

parse_repo(node, spdx::AbstractString)::NamedTuple

Parses a node and returns a suitable NamedTuple for the table.

prune(data)

Prune the intervals based on the created and count values.

query_intervals(created::AbstractVector{<:AbstractVector{Interval{ZonedDateTime}}})::DataFrame

Returns a

query_intervals(spdx::AbstractString, created::AbstractVector{<:Interval{ZonedDateTime}})

Return count of search results based on the license for each created interval.