EXPORTING TO CSV

Getting Started

This page will help you with exporting your Self-Service RESTful API data to CSV format.

This is useful if you need to analyze the data, create reports, or import the data into other tools that require CSV format.

It is possible to programmatically parse the output from the AttackForge Self-Service RESTful API into CSV format using various tools such as JQ.

JQ is a lightweight and flexible command-line JSON processor.

JQ is supported on multiple operating systems (Linux, OSX, Windows) & is freely available. It is written in C and has no runtime dependencies.

This tutorial will show you how to install JQ and use it to convert AttackForge Self-Service RESTful API data into CSV format.

Installing JQ on Linux

1. From command line, type following command: sudo apt-get install jq

2. Other install options are available from https://stedolan.github.io/jq/download/

Installing JQ on Windows

1. Download JQ from here: https://stedolan.github.io/jq/download/. We recommend using JQ v1.6.

2. Go into C:\Users\<username> and create a new folder called Bin.

3. Place the downloaded file into the newly created Bin folder.

4. Rename the file to “jq” (so the full name should be jq.exe).

5. Click on the Windows Start menu, and type Edit the System Environment Variables, and hit Enter.

6. The System Properties window will be displayed. Click on Environment Variables.

7. An Environment Variables window appears and two text boxes are displayed. In the top box, click on Path (third item in the list) and hit Edit.

8. An Edit environment variable windows appears; click on the %USERPROFILE% entry and select Edit Text.

9. An Edit User Variable windows appears. At the end of the %USERPROFILE% string, enter the path to the Bin folder that you created in Step 2. The path should be separated from the %USERPROFILE% string by a semi-colon, and the Bin folder path should contain a semi-colon at the end of it – like following:

10. Click OK to exit the Edit User Variable window. Click OK to exit the Environment Variables window. Click OK to exit System Properties window.

11. Open a command window (Windows Start menu > type CMD > select Command Prompt).

12. Enter this command: echo %PATH%. The new folder that you created in Step 2 should be present at the end of the output text. This indicates that the environment variables have been changed and should now pick up the JQ utility.

13. To check the utility is working correctly, type this command: jq

It should display results similar to below – confirming the tool is installed correctly.

Creating CSV Using API Data

For this tutorial, we are going to focus on the getVulnerabilities API method. You can however use this approach for any of the Self-Service RESTful API methods.

Open a command window / terminal.

Run the following command (replacing your tenant name & API Key). Make sure you have access to the getVulnerabilities API (see https://support.attackforge.com/attackforge-enterprise/modules/users#managing-access-to-self-service-api)

curl -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities" -H "Host: <YOUR-TENANT>" -H "X-SSAPI-KEY: <YOUR-API-KEY>" -H "Content-Type: application/json" -H "Connection: close" | jq -r "(.vulnerabilities[] | [.vulnerability_id, .vulnerability_title, .vulnerability_created, .vulnerability_priority, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, if(try(.vulnerability_affected_assets)) then [.vulnerability_affected_assets[].asset.name] | join(\", \") else \"\" end, .vulnerability_retest, .vulnerability_project_name, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce]) | @csv" > filtered_vulns.csv

The command above will produce a CSV titled filtered_vulns.csv in the working directory where you had run the command from.

The CSV will include the following details for each vulnerability:

• Id

• Title (e.g. Blind SQL Injection)

• Created (time/date)

• Priority (Critical/High/Medium/Low/Info)

• Status (Open/Closed)

• Status Updated (time/date)

• Affected Asset Name (e.g. attackforge.com)

• Affected Assets - if vulnerability is using Grouped Assets, show each asset name

• Retest (Yes/No)

• Project Name (e.g. Web App Pentest)

• Description

• Attack Scenario

• Remediation Recommendation

• Steps to Reproduce

Changing Fields & Re-Ordering Fields

If you would like to change the existing fields in the CSV, or re-order the fields in the CSV – adjust everything between [.vulnerability_created, ... , .vulnerability_id]:

curl -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities" -H "Host: <YOUR-TENANT>" -H "X-SSAPI-KEY: <YOUR-API-KEY>" -H "Content-Type: application/json" -H "Connection: close" | jq -r "(.vulnerabilities[] | [.vulnerability_id, .vulnerability_title, .vulnerability_created, .vulnerability_priority, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, if(try(.vulnerability_affected_assets)) then [.vulnerability_affected_assets[].asset.name] | join(\", \") else \"\" end, .vulnerability_retest, .vulnerability_project_name, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce]) | @csv" > filtered_vulns.csv

Adding Additional Fields

If you would like to add additional fields to the CSV, you can use any of the fields available in the API (see https://support.attackforge.com/attackforge-enterprise/modules/self-service-api/getvulnerabilities)

For example, the command below will also include the Vulnerability SLA (.vulnerability_sla) in the CSV:

curl -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities" -H "Host: <YOUR-TENANT>" -H "X-SSAPI-KEY: <YOUR-API-KEY>" -H "Content-Type: application/json" -H "Connection: close" | jq -r "(.vulnerabilities[] | [.vulnerability_id, .vulnerability_title, .vulnerability_created, .vulnerability_priority, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, if(try(.vulnerability_affected_assets)) then [.vulnerability_affected_assets[].asset.name] | join(\", \") else \"\" end, .vulnerability_retest, .vulnerability_project_name, .vulnerability_sla, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce]) | @csv" > filtered_vulns.csv

Including Column Names

If you would like to include column names in the CSV, you can set the columns using the command below – see [\"Id\", ... , \"Steps To Reproduce\"]:

curl -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities" -H "Host: <YOUR-TENANT>" -H "X-SSAPI-KEY: <YOUR-API-KEY>" -H "Content-Type: application/json" -H "Connection: close" | jq -r "[\"Id\", \"Title\", \"Discovered\", \"Priority\", \"Status\", \"Status Last Updated\", \"Affected Asset\", \"Affected Assets\", \"Retest\", \"Project\", \"Description\", \"Attack Scenario\", \"Recommendation\", \"Steps To Reproduce\"], (.vulnerabilities[] | [.vulnerability_id, .vulnerability_title, .vulnerability_created, .vulnerability_priority, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, if(try(.vulnerability_affected_assets)) then [.vulnerability_affected_assets[].asset.name] | join(\", \") else \"\" end, .vulnerability_retest, .vulnerability_project_name, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce]) | @csv" > filtered_vulns.csv

Note the backslashes \" are necessary to escape the quotation marks for each CSV column title. If you do not include the backslashes, you may get a compilation error.

Filtering Vulnerabilities

If you would like to filter the vulnerabilities, you can use any of the existing filters supported in the API (see https://support.attackforge.com/attackforge-enterprise/modules/users#managing-access-to-self-service-api)

For example, the command below will limit the vulnerabilities retrieved from the API to Open vulnerabilities only using the Advanced Query Filter (https://support.attackforge.com/attackforge-enterprise/modules/self-service-restful-api/advanced-query-filter) --data-urlencode "q={status:{$eq:\"Open\"}}":

curl -G -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities" --data-urlencode "q_vulnerability={status:{$eq:\"Open\"}}" -H "Host: <YOUR-TENANT>" -H "X-SSAPI-KEY: <YOUR-API-KEY>" -H "Content-Type: application/json" -H "Connection: close" | jq -r "[\"Id\", \"Title\", \"Discovered\", \"Priority\", \"Status\", \"Status Last Updated\", \"Affected Asset\", \"Affected Assets\", \"Retest\", \"Project\", \"Description\", \"Attack Scenario\", \"Recommendation\", \"Steps To Reproduce\"], (.vulnerabilities[] | [.vulnerability_id, .vulnerability_title, .vulnerability_created, .vulnerability_priority, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, if(try(.vulnerability_affected_assets)) then [.vulnerability_affected_assets[].asset.name] | join(\", \") else \"\" end, .vulnerability_retest, .vulnerability_project_name, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce]) | @csv" > filtered_vulns.csv

Advanced Examples

Example 1 - Get All Closed Vulnerabilities and Reason Why Closed

Vulnerability Id	Created	Priority	Title	Status	Status Last Updated	Affected Asset	Affected Assets	Project Name	Project Id	Last Closed Reason	Last Closed On	Last Closed By
65bc9d9a4534bd000e50db3f	2024-02-02T07:45:30.179Z	Critical	Out-of-date Version	Closed	2024-02-02T09:36:41.513Z	https://attackforge.com		Test Project 1	65af64ec07bd34000fbac60d	Issue Closed: Risk accepted	2024-02-02T09:36:41.513Z	Admin Test
65b56ceff8b879000f887c7e	2024-01-27T20:51:59.346Z	Critical	Out-of-date Version	Closed	2024-02-02T05:33:38.928Z		https://portal.attackforge.com, https://app.attackforge.com	Test Project 1	65af64ec07bd34000fbac60d	Issue Closed: Issue has been fixed	2024-02-02T05:33:38.928Z	Admin Test

curl -G -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities"
 --data-urlencode "q_vulnerability={status:{$eq:\"Closed\"}}" -H "Content-Type: application/json" -H "Connection: close" -H "X-SSAPI-KEY: <YOUR-API-KEY>" | jq -r "[\"Vulnerability Id\", \"Created\", \"Priority\", \"Title\", \"Status\", \"Status Last Updated\", \"Affected Asset\", \"Affected Assets\", \"Project Name\", \"Project Id\", \"Last Closed Reason\", \"Last Closed On\", \"Last Closed By\"], (.vulnerabilities[] | [.vulnerability_id, .vulnerability_created, .vulnerability_priority, .vulnerability_title, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, if(try(.vulnerability_affected_assets)) then [.vulnerability_affected_assets[].asset.name] | join(\", \") else \"\" end, .vulnerability_project_name, .vulnerability_project_id, (last(.vulnerability_remediation_notes[] | select(.note | contains(\"Issue Closed\"))) | .note, .created, .created_by)]) | @csv" > filtered_vulns.csv

Wrapping up

You can now use the JQ utility for any of the AttackForge Self-Service APIs, to extract any desired information for your reporting purposes.

You can check all the of the available APIs from our support site: https://support.attackforge.com/attackforge-enterprise/modules/self-service-api

Troubleshooting

1. Test jq utility is available

jq

This should result in info/help output in the terminal. This will test to confirm jq utility has been installed & configured successfully on your host.

2. Test cURL works + piping/passing JSON data to jq

curl "https://api.github.com/repos/stedolan/jq/commits?per_page=5" | jq "."

This should output JSON data to the terminal.

3. Test @csv is working

curl "https://api.github.com/repos/stedolan/jq/commits?per_page=5" | jq -r ".[0] | [.url] | @csv"

It should print "https://api.github.com/repos/stedolan/jq/commits/d18b2d078c2383d9472d0a0a226e07009025574f" in the terminal

4. Try output data into csv file

curl "https://api.github.com/repos/stedolan/jq/commits?per_page=5" | jq -r ".[0] | [.url] | @csv" > output.csv

This should create a file output.csv in the directory you are currently running commands from in command terminal, the contents of the file will be https://api.github.com/repos/stedolan/jq/commits/d18b2d078c2383d9472d0a0a226e07009025574f

5. Store AttackForge getVulnerabilities data in a JSON file

curl -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities" -H "Host: <YOUR-TENANT>" -H "X-SSAPI-KEY: <YOUR-API-KEY>" -H "Content-Type: application/json" -H "Connection: close" > vulns.json

This should create a file vulns.json in the directory you are currently running commands from in terminal, the contents of this file will be vulnerabilities which match the query parameter.

6. Try running JQ on vulns.json

jq -r "(.vulnerabilities[] | [.vulnerability_created, .vulnerability_priority, .vulnerability_title, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, .vulnerability_retest, .vulnerability_project_name, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce, .vulnerability_id]) | @csv" vulns.json

This should output csv data in the terminal.

7. Try exporting vulns.json to csv

jq -r "(.vulnerabilities[] | [.vulnerability_created, .vulnerability_priority, .vulnerability_title, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, .vulnerability_retest, .vulnerability_project_name, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce, .vulnerability_id]) | @csv" vulns.json > filtered_vulns.csv

This should create a file filtered_vulns.csv in the directory you are currently running commands from in terminal, the contents of this file will be vulnerabilities which match the specified fields in the JQ command.

If you get to Step 7 above and it is all working correctly, there should be no restrictions from running the original command (see below):

curl -X GET "https://<YOUR-TENANT>/api/ss/vulnerabilities" -H "Host: <YOUR-TENANT>" -H "X-SSAPI-KEY: <YOUR-API-KEY>" -H "Content-Type: application/json" -H "Connection: close" | jq -r "(.vulnerabilities[] | [.vulnerability_id, .vulnerability_title, .vulnerability_created, .vulnerability_priority, .vulnerability_status, .vulnerability_status_updated, .vulnerability_affected_asset_name, if(try(.vulnerability_affected_assets)) then [.vulnerability_affected_assets[].asset.name] | join(\", \") else \"\" end, .vulnerability_retest, .vulnerability_project_name, .vulnerability_description, .vulnerability_attack_scenario, .vulnerability_remediation_recommendation, .vulnerability_steps_to_reproduce]) | @csv" > filtered_vulns.csv