Webhooks

Pull Request Risk Assessment

Overview

This webhook provides detailed information about a specific pull request change event in a repository, along with an associated risk score and mitigations.

Ordering

ℹ️ Webhooks are not guaranteed to arrive in order

Lifecycle

Your code will need to handle out of order events. This can be accomplished by using the gh_updated_at  timestamp in the payload. It is a high water mark from the updated timestamp in GitHub.

​

Events are reactive to subscribed GitHub webhook events and propagated immediately after Shepherdly processes each pull request.

Successful events (HTTP Status code 200-299) will not be redelivered or retried.

Retry Behavior

Retries will occur when encountering any status code not within a 200-299 range or networking connection error and timeout.

​

This will occur up to 100 times with an exponential backoff starting at 30 seconds.

Once the above condition is met (100 attempts), the webhook will be discarded.

Webhook Payload Structure

The payload contains several key sections: pull_request , risk_score , mitigations , and resilience_coverage .

pull_request

This section includes some high level information about the PR, however, if you require more datapoints, it's best to consume the GitHub rest api yourself.

The most important field to be aware of is the gh_updated_at  which is serves as the high watermark. Since webhook events can arrive out of order, this field can ensure your system is deterministically in sync and that there are no lost updates.

risk_score

This section provides the risk assessment of the PR.

• pull_id : Unique identifier of the PR.
• true_positive_prob : Probability of the PR being a true positive in terms of risk. This is percentage value so if you need to format this for humans, multiply by 100.

• model_version : Version of the risk assessment model.

  score : Risk score of the PR out for 100.

  buggy_files : Array of files identified as hotspots. By default, these are bugs that are in the top 90th percentile and have had at least 1-2 recent bug fixes.

  prediction_explanations : Array of explanations for the risk prediction, each containing:

  • feature_name : Name of the feature contributing to the risk assessment.
  • value : Value of the feature.
  • threshold : Threshold value for the feature.

  • direction : Threshold direction (i.e. gt, lt, lte, gte). For example, if threshold  1 and direction  is gt , the value  would have to be 2 or higher.

    
    

mitigations

A full list of configured mitigations strategies for the repository and their state. Even if a mitigation is not required, if it's defined as an available option for the repo, it will be present here.

• rule_id : Unique identifier of the mitigation rule.
• name : Identifier of the mitigation, this is effectively a slug.
• detected_count : Number of times the issue was detected.
• added_manually : Boolean indicating if the mitigation was added manually. This is a manual attestation.
• required : Boolean indicating if the mitigation is required give the configured risk threshold for the repo.
• completed : Boolean indicating if the mitigation is completed either manually or detected_count  > 0.
• label : Human label for the mitigation.

• custom : Boolean. If true then it's not using an out of the box rule which has the ability to be autodetected. Custom mitigations can be tracked like any other mitigation.

  
  

resilience_coverage

Integer. Convenience field that calculates the total required completed mitigations as a percentage.

Webhook Headers:

Webhook Payload Example:

{
 "pull_request": {
    "id": 1915898,
    "number": 318,
    "repo_id": 1,
    "repo_name": "api",
    "author_username": "acmeuser",
    "created_at": 1705703140751,
    "updated_at": 1706802105404,
    "state": "closed",
    "merged_at": 1706802093000,
    "closed_at": 1706802094000,
    "additions": 930,
    "commits": 7,
    "deletions": 627,
    "changed_files": 36,
    "requested_reviewers": 1,
    "assignees": 0,
    "title": "Improve SSO flow",
    "html_url": "https://github.com/acmecorp/api/pull/318",
    "total_comments": 65,
    "comments": 2,
    "reviews": 24,
    "review_comments": 63,
    "code_comments": 60,
    "draft": false,
    "merge_commit_sha": "ea172107494224b1e3cc4b9af7f3b0d57425f8bb",
    "scm_created_at": 1705703137000,
    "scm_updated_at": 1706802094000
  },
  "resilience_coverage": 60,
  "risk_score": {
    "created_at": 1705703150953,
    "updated_at": 1706802123444,
    "true_positive_prob": 0.5893021840837632,
    "model_version": 1,
    "score": 80,
    "file_hotspots": [
      {
        "filename": "src/api/auth/user.py",
        "score": 1.106092513447328,
        "percentile": 0.9506172839506167
      },
      {
        "filename": "src/client/team.py",
        "score": 1.0000061399936853,
        "percentile": 0.924741975006946
      }
    ],
    "prediction_explanations": [
      {
        "feature_name": "SumLinkedBugReports",
        "value": 95,
        "threshold": 18.5,
        "direction": "gt"
      },
      {
        "feature_name": "ExecLinesDelta",
        "value": 114,
        "threshold": 38.5,
        "direction": "gt"
      }
    ]
  },
  "mitigations": [
    {
      "rule_id": 2563,
      "name": "code_review",
      "detected_count": 2,
      "added_manually": false,
      "pull_id": 1918122,
      "required": true,
      "custom": false,
      "just_in_time": false,
      "completed": true,
      "label": "Code Review"
    },
    {
      "rule_id": 2567,
      "name": "feature_flag",
      "detected_count": 0,
      "added_manually": false,
      "pull_id": 1918122,
      "required": true,
      "custom": false,
      "just_in_time": false,
      "completed": false,
      "label": "Feature Flag"
    },
    {
      "rule_id": 2565,
      "name": "integration_test",
      "detected_count": 0,
      "added_manually": false,
      "pull_id": 1918122,
      "required": true,
      "custom": false,
      "just_in_time": false,
      "completed": false,
      "label": "Integration Test"
    },
    {
      "rule_id": 2561,
      "name": "observability",
      "detected_count": 7,
      "added_manually": false,
      "pull_id": 1918122,
      "required": true,
      "custom": false,
      "just_in_time": false,
      "completed": true,
      "label": "Observability"
    },
    {
      "rule_id": 2566,
      "name": "unit_test",
      "detected_count": 6,
      "added_manually": false,
      "pull_id": 1918122,
      "required": false,
      "custom": false,
      "just_in_time": false,
      "completed": true,
      "label": "Unit Test"
    }
  ]
}