Shepherdly Repo Config

Setup

Shepherdly customized per repository with a yml config file. It can be placed anywhere in the repo but the filename must end with shepherdly.yml .

The config file must be introduced and subsequently modified through pull requests. Changes directly to the default branch will be ignored.

Basic Behavior

Changes do not take effect until the pull request containing the file are merged into the default (ex: main or master ) branch.

Each change to the yml file will undergo a schema validation. A comment will be posted to the file directly indicating if the schema has passed or if errors exist.

Inheritance

Config files can inherit and extend from parent configurations in a polymorphic manner by referencing a parent_config_name . Shepherdly must have permissions to the repository containing the parent config.

To setup a config so that it can be inherited, specify a config_name .

Example:

Parent:

config_name: ui-shepherdly.yml
configurations:
  - name: scoring_comment_enabled
    value: true

Child:

parent_config_name: ui-shepherdly.yml
configurations:
  - name: scoring_comment_draft_mode
    value: true

In this example above, the child config would inherit the scoring_comment_enabled value but also override the system default scoring_comment_draft_mode .

Configurations

Name	System Default Value	Description
`scoring_comment_enabled`	false	Toggles the PR bot comment.
`scoring_comment_risk_score_threshold`	0	Determines the minimum risk score required to trigger the PR bot comment.
`scoring_comment_draft_mode`	false	Toggles the PR bot for PRs in 'draft' state.
`buggy_file_hotspot_percentile_threshold`	0.90	The minimum percentile needed for a file to be classified as a buggy hotspot. The default is a good balance of signal to noise.
`qa_git_team_name`	none	If exists, name of team that is responsible for QA.
`integration_test_path`	none	Comma separated value of directories that contain integrations tests.
`file_path_exclusions`	none	Comma separated value of directories that don't contain any executable production code.
`email_compatible_pr_comment`	false	Toggles the shields.io badges at the top of the PR comment on or off for email compatibility. These badges are not rendered in the stock email notifications from GitHub. Enabling this flag will use a text-based approach to display this information in the comment and make it compatible with email clients.
`auto_approve_risk_score_threshold`	none	Disabled by default. Enables auto approval if the risk score of a PR is at or below the specified value (0-100). If a PR is not in draft mode and is within the threshold, the bot will approve the PR. If at any point before it is merged, the risk score is over the threshold, the approval will be dismissed, indicating human approval is required.

Mitigations

Out of the box

These are mitigations that are supported by default and can be auto-detected using AST parsing in supported languages.

Name	Risk Score Threshold
`code_review`	0
`integration_test`	50
`unit_test`	50
`feature_flag`	75
`observability`	75

Specific thresholds can be overridden:

mitigations:
  - name: observability
    score_threshold: 80
  - name: feature_flag
    score_threshold: 80

System default mitigations can be disabled entirely and delegate to whatever is specified in the repo mitigations by setting `default_system_mitigations` to `false`. :

configurations:
mitigations:
  - name: code_review
    score_threshold: 0
  - name: feature_flag
    score_threshold: 80
  - name: unit_test
    score_threshold: 50
  - name: observability
    score_threshold: 70
default_system_mitigations: false

Custom Mitigations

Custom mitigations don't have any default auto-detection capability but still allow for them to be marked as required given a precise risk score. Engineers can manually attest to their completion and is trackable in the web app.

Similar to default mitigations, completion of custom mitigations increases resilience coverage metric of the PR.

mitigations:
  - name: run_security_scan
    score_threshold: 80
    custom: true
    label: Run Security Scan

URL & String templating

Custom mitigation labels can be URL's with some string templating to help with deep linking by specifying the label_url property.

The available contextual variables is currently just the Pull Request Number the comment resides in: {{pr_number}} .

mitigations:
  - name: run_security_scan
    score_threshold: 80
    custom: true
    label: Run Security Scan
    label_url: https://api.acme.com/scan/{{pr_number}}

Webhooks

Webhooks can be enabled for users who want to tail the risk score stream. It provides the current risk score and resilience coverage, along with basic information about the pull request.

webhooks:
  - url: https://api.acmecorp.com/some/path