Engineering shops often won’t have the expertise or the time to do deep security testing on all areas of the product. Using external security firms who have specific security knowledge can extend security testing well beyond their own capabilities.
External security testing — often called penetration testing or pen testing — can be effective when the scope is well-defined and narrow. It is better to do four small tests on specific areas rather than one large test across much of the product. Two to three week engagements are relatively inexpensive and should be run once per quarter. Even four engagements a year are generally cheaper than a full time security tester.
Pen tests also grow security expertise in the organization. Dev and Test resources who participate in the pen test will have a deeper understanding of security issues.
To make the most of a pen test engagement:
- Security contractors are very expensive. Be prepared to spend time before the pen test configuring the software, setting up the test harness and gathering documentation to make the most efficient use of their time.
- Ideally a security test is run on full featured beta code about 8 weeks before the ship date. This will allow time to fix any issues found before shipping. This is not always possible, so run the test when you can.
- Assign severity to the issues; policy should dictate action.
- Many pen testers will want to inspect source code. The steps below outline a black box testing strategy in which source code is not needed. Inspection of source code is generally a longer engagement and should be a supplement to source code analysis used during construction.
- Select an ethical pen tester. Ethical pen testers will obtain permission before engaging, document all steps, judiciously select targets, and report all findings.
To run a pen test, follow these steps.
Step One: Assign a pen test resource
Assign an owner for the pen test. This person will be responsible for writing the RFPs, configuring the test harness, gathering documentation, communicating with the contractor, setting up meetings, and other tasks. Expect these tasks to take roughly one third to one half time during the pen test.
Step Two: Focus on a feature
Select a feature that is exposed and has potential weaknesses that could be exploited. Consider features that have had known vulnerabilities or weaknesses in the past. Also consider features that have known deficiencies in test cases or a long legacy of different engineers. Specify the feature and the constraints in the RFP. Ideally, find a vendor with experience in the subject area.
Step Three: Involve feature engineers
Select engineers from dev and test to be responsible for actions during the pen test. Record roles and responsibilities during the engagement. Dev engineers are responsible for explaining design decisions, asking questions to understand the issues raised, and securing the code. Test engineers are responsible for understanding the source of the issues, writing tickets, and developing test cases around the issue so that no similar issues are found.
Step Four: Set up the scenario
Pre-configure and set up the scenario. If possible, configure VMs with the software and deliver those to the pen tester. Ideally, set up a cul-de-sac network that contains just the software under test and allow the contractor access to that network.
Step Five: Run the pen test
Schedule weekly meetings during the engagement. Involve the dev and test engineers and and make sure they understand the issues and can fix them. Focus on correct mitigations rather than implementation details at this point in time.
Step Six: Followthrough
Classify all vulnerabilities according to vulnerability severity and have them worked according to policy. Write unit tests to ensure that similar vulnerabilities don’t happen. Write test cases to avoid similar issues. Hold a postmortem lessons-learned meeting and spread the security knowledge through your organization.
Following these steps will build a more secure product, grow security knowledge in the organization, and develop better practices around secure development.
