Unit, integration and acceptance tests are essential quality controls to running a reliable continuous integration or continuous delivery process. Automated security tests are a natural shoe-in to the process, but too often these are excluded because security tests aren’t automated, or there is the belief that they can’t be automated.
Security tests are not special.
They can, and should be automated wherever possible so that they can form part of a CD pipeline and provide the same benefits as other automated tests: early feedback of security issues and halting the process until either the reported vulnerability- or the test- is fixed.
The BDD-Security framework provides a set of ready-made tests to form the baseline for a program of security-tests-as-software-tests. These tests can be divided into a number of different groups, all of which lend themselves to automation to a great or lesser degree:
- Functional Security Tests to verify that key security features such as authentication and password reset work as expected.
- The bulk of security tests would fall into the non-functional category and can be further divided into:
- Specific tests again non-functional aspects, e.g. is the HttpOnly flag set on session cookies? Does the session terminate when the user logs out?
- Automated security scanning against the application, and automated scans against the underlying infrastructure
These types of tests are typically already included in manually driven vulnerability assessment exercises. Manual tests should of course also include an element of exploratory testing, driven by a security testing specialist. This is the primary reason why automated tests cannot entirely replace human security testing- but they can act as wonderful compliments to the repetitive elements of it. As Morpheus said: Never send a human to do a machine’s job.
…and if you have sent a human, how about recording the issues identified as automated tests so that they’re part of the continuous testing suite?
BDD-Security effectively acts as an orchestration framework to perform the tests 1 through 4 listed above. Since many of these touch on independent domains, e.g. testing the SSL configuration, automated scanning for vulnerabilities of the infrastructure and of the application; they can be performed in parallel.
The picture painted by the Build Pipeline Plugin isn’t quite right because there’s only a single Ropey-Deploy-PreProd job at the end of the process which is executed only if all the security jobs complete successfully.
The Jenkins configuration uses the following plugins: – Build Pipeline – Join – HTML Publisher – JBehave Jenkins – xUnit (required by JBehave)
The join plugin is used to launch the security tests after the deploy into the test environment and at the same time call the job immediately after the security tests- if they all pass. (This is the relationship that the Build Pipeline fails to understand in its rendering of the process- although this is just a cosmetic issue and has no impact on actually running the pipeline) The JBehave and xUnit plugins are used to pull in the BDD-Security xml reports and can also be used to define thresholds for test failures. Finally the BDD-Security reports are published as HTML so that the full detail of the specification and the test results can be viewed through Jenkins
Without the HTML reports we can still use the xUnit results although they’re less complete and don’t provide the whole test context: