Addepar/RedFlag

Table of Contents

• Batch Mode
• CI Mode
• Evaluation Mode
• Advanced Configuration
• Contributing
• License
• Contact

# Batch Mode RedFlag is able to analyze a large number of commits in a single run. These commits can be specified using commit hashes, branch names, or tags. This is useful for scoping manual security testing of logical groups of code, such as release candidates. ## Workflow  ## Getting Started ### Installation ##### Use a Virtual Environment 1. Create a virtual environment: python -m venv redflag-venv source redflag-venv/bin/activate 2. Install RedFlag: pip install addepar-redflag Alternatively, if you'd like to use Poetry, clone the repo and use `poetry install` and then `poetry run redflag`. ### Setup Credentials Credentials can be set using: 1. Environment variables 2. A `.env` file 3. CLI parameters 4. Configuration file (This is not recommended for security reasons!) ##### AWS Credentials RedFlag uses [Boto3-Compatible Credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) using **Profiles** or **Env Vars**. Ensure that your AWS IAM policy has `InvokeModel` and `InvokeModelWithResponseStream` permissions to [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_service-with-iam.html). Lastly, *make sure you've [requested the necessary Claude models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html)!* ##### GitHub PAT Use a [Personal Access Token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) with `repo` permissions. Set the token as an environment variable: export RF_GITHUB_TOKEN=your-token-here ##### Jira API Token *(Optional)* First, set a Jira URL (`https://your-org.atlassian.net`) in the configuration file (`jira_url`), as a CLI parameter (`--jira-url`), or as an environment variable (`RF_JIRA_URL`). export RF_JIRA_USER=your-username-here export RF_JIRA_TOKEN=your-token-here ##### Slack Token *(Optional)* 1. [Create a Slack App](https://api.slack.com/quickstart) 2. [Request required scopes](https://api.slack.com/quickstart#scopes) 3. [Install and authorize the App](https://api.slack.com/quickstart#installing) 4. Add environment variables or configuration entries when running RedFlag export RF_SLACK_TOKEN=xoxb-slack-token-here export RF_SLACK_CHANNEL=C0123456789 *Don't forget to invite the bot to the channel to avoid a `channel_not_found` error*. ### Usage Here are some examples on how to run RedFlag in batch mode: # Using branch names: redflag --repo YourOrg/SomeRepo --from main --to dev # Using commit hashes: redflag --repo YouOrg/SomeRepo --from a1b2c3 --to d4e5f6 # With a custom configuration file: redflag --config custom-config.yml ## Report Output By default, RedFlag produces an HTML report that can be opened in a browser.

(back to top)

# CI Mode RedFlag can be run in CI pipelines to flag PRs and add the appropriate reviewers. This mode uses GitHub Actions to run RedFlag on every PR and post a comment if the PR requires a review. Additionally, CI Mode is best suited for Slack alerting. [][docs-ci-mode-url]

(back to top)

# Evaluation Mode RedFlag can be run in evaluation mode to evaluate the performance of the AI model using your own custom dataset. This mode is useful for understanding how the model and prompts perform on your codebase and aids in security risk evaluation. [][docs-eval-mode-url]

(back to top)

# Advanced Configuration ## Order of Precedence 1. CLI Parameters 2. Environment Variables 3. Configuration File 4. Default Values On each execution, RedFlag will load the configuration in the order of precedence above and then output a table that shows the final configuration and where each parameter was set. ## Configuration Options and Defaults The following table shows configuration options for each parameter: #### General Settings | Parameter | CLI Param | Env Var | Config File | Default | |---------------------------------------------------------------------------------------|-----------|---------|-------------|---------| | [Configuration File](https://github.com/Addepar/RedFlag/blob/main/config.sample.yaml) | --config | - | - | - | | Repository | --repo | RF_REPO | repo | - | | Branch/Commit From | --from | RF_FROM | from | - | | Branch/Commit To | --to | RF_TO | to | - | #### Integration Settings | Parameter | CLI Param | Env Var | Config File | Default | |-------------------------------------------------------------------------------------------------------------------------------------|------------------|-------------------|---------------|---------| | [GitHub Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) | --github-token | RF_GITHUB_TOKEN | github_token | - | | Jira URL | --jira-url | RF_JIRA_URL | jira.url | - | | Jira Username | --jira-user | RF_JIRA_USER | jira.user | - | | [Jira Token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) | --jira-token | RF_JIRA_TOKEN | jira.token | - | | [Slack Token](https://api.slack.com/concepts/token-types) | --slack-token | RF_SLACK_TOKEN | slack.token | - | | [Slack Channel (ID)](https://slack.com/help/articles/221769328-Locate-your-Slack-URL-or-ID) | --slack-channel | RF_SLACK_CHANNEL | slack.channel | - | | Slack Message Headline | --slack-headline | RF_SLACK_HEADLINE | slack.headline | - | #### LLM Settings | Parameter | CLI Param | Env Var | Config File | Default | |--------------------------------------------------------------------------------------------------------------------|--------------------|---------------------|----------------------------|-------------------------------------------| | Debug LLM | --debug-llm | - | - | `False` | | [Bedrock Model ID](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html) | --bedrock-model-id | RF_BEDROCK_MODEL_ID | bedrock.model_id | `anthropic.claude-3-sonnet-20240229-v1:0` | | [Bedrock Profile](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html) | --bedrock-profile | RF_BEDROCK_PROFILE | bedrock.profile | - | | [Bedrock Region](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) | --bedrock-region | RF_BEDROCK_REGION | bedrock.region | - | | Review Prompt (Role) | - | - | prompts.review.role | Security review (see `sample.config.yaml`) | | Review Prompt (Question) | - | - | prompts.review.question | Security review (see `sample.config.yaml`) | | Test Plan Prompt (Role) | - | - | prompts.test_plan.role | Security review (see `sample.config.yaml`) | | Test Plan Prompt (Question) | - | - | prompts.test_plan.question | Security review (see `sample.config.yaml`) | #### Input/Output Settings | Parameter | CLI Param | Env Var | Config File | Default | |---------------------------|--------------------------|----------------|-------------------------|-----------| | Output Directory | --output-dir | RF_OUTPUT_DIR | output_dir | `results` | | Maximum Commits | --max-commits | RF_MAX_COMMITS | max_commits | `0` (∞) | | Don't Output HTML | --no-output-html | - | - | - | | Don't Output JSON | --no-output-json | - | - | - | | Don't Show Progress Bar | --no-progress-bar | - | - | - | | Don't Strip HTML Comments | --no-strip-html-comments | - | - | - | | Filter Commit Titles | - | - | filter_commits.title | - | | Filter Commit Users | - | - | filter_commits.user | - | | Strip Description Lines | - | - | strip_description_lines | - | #### Evaluation Parameters (`eval` Command) | Parameter | CLI Param | Env Var | Config File | Default | |---------------------------------------------------------------------------------------------------------|------------|------------------|--------------|---------| | [Evaluation Dataset](https://github.com/Addepar/RedFlag/wiki/Evaluation-Mode#building-a-custom-dataset) | --dataset | RF_DATASET | dataset | - |

(back to top)

## License Distributed under the MIT License. See [`LICENSE.md`](https://github.com/Addepar/RedFlag?tab=MIT-1-ov-file#readme) for more information.

(back to top)

## Contact Say hi to Addepar Security Engineering at [security-engineering@addepar.com](mailto:security-engineering@addepar.com).

(back to top)