Thursday, February 5, 2015

Software Engineering Security part 6: Running External Penetration Tests

Engineering shops often won’t have the expertise or the time to do deep security testing on all areas of the product. Using external security firms who have specific security knowledge can extend security testing well beyond their own capabilities.

External security testing — often called penetration testing or pen testing — can be effective when the scope is well-defined and narrow. It is better to do four small tests on specific areas rather than one large test across much of the product. Two to three week engagements are relatively inexpensive and should be run once per quarter. Even four engagements a year are generally cheaper than a full time security tester.

Pen tests also grow security expertise in the organization. Dev and Test resources who participate in the pen test will have a deeper understanding of security issues.

To make the most of a pen test engagement:
  • Security contractors are very expensive. Be prepared to spend time before the pen test configuring the software, setting up the test harness and gathering documentation to make the most efficient use of their time.
  • Ideally a security test is run on full featured beta code about 8 weeks before the ship date. This will allow time to fix any issues found before shipping. This is not always possible, so run the test when you can.
  • Assign severity to the issues; policy should dictate action.
  • Many pen testers will want to inspect source code. The steps below outline a black box testing strategy in which source code is not needed. Inspection of source code is generally a longer engagement and should be a supplement to source code analysis used during construction.
  • Select an ethical pen tester. Ethical pen testers will obtain permission before engaging, document all steps, judiciously select targets, and report all findings.

To run a pen test, follow these steps.

Step One: Assign a pen test resource
Assign an owner for the pen test. This person will be responsible for writing the RFPs, configuring the test harness, gathering documentation, communicating with the contractor, setting up meetings, and other tasks. Expect these tasks to take roughly one third to one half time during the pen test.

Step Two: Focus on a feature
Select a feature that is exposed and has potential weaknesses that could be exploited. Consider features that have had known vulnerabilities or weaknesses in the past. Also consider features that have known deficiencies in test cases or a long legacy of different engineers. Specify the feature and the constraints in the RFP. Ideally, find a vendor with experience in the subject area.

Step Three: Involve feature engineers
Select engineers from dev and test to be responsible for actions during the pen test. Record roles and responsibilities during the engagement. Dev engineers are responsible for explaining design decisions, asking questions to understand the issues raised, and securing the code. Test engineers are responsible for understanding the source of the issues, writing tickets, and developing test cases around the issue so that no similar issues are found.

Step Four: Set up the scenario
Pre-configure and set up the scenario. If possible, configure VMs with the software and deliver those to the pen tester. Ideally, set up a cul-de-sac network that contains just the software under test and allow the contractor access to that network.

Step Five: Run the pen test
Schedule weekly meetings during the engagement. Involve the dev and test engineers and and make sure they understand the issues and can fix them. Focus on correct mitigations rather than implementation details at this point in time.

Step Six: Followthrough
Classify all vulnerabilities according to vulnerability severity and have them worked according to policy. Write unit tests to ensure that similar vulnerabilities don’t happen. Write test cases to avoid similar issues. Hold a postmortem lessons-learned meeting and spread the security knowledge through your organization.

Following these steps will build a more secure product, grow security knowledge in the organization, and develop better practices around secure development.

Wednesday, January 28, 2015

CVE-2015-0235: Ghost

The last two days have been taken up with the glibc ghost vulnerability found by Qualys. They have a good description and a good detailed analysis.
Essentially this is the old "parsing untrusted input" problem -- a value comes in from a dubious source and the code needs to be robust against an attacker who can choose arbitrary malicious input.

Vulnerabilities like this are hard to classify. It's almost certainly exploitable by someone who has a local account on a box. But determining if it is remotely exploitable is difficult. This is what is called the "cone of uncertainty"; it takes time to do the research and find out if any code paths are directly vulnerable. Starting at the vulnerable gethostbyname() functions and working out is time consuming.
Many times this week I have wished for a rough equivalent to perl's taint checking -- let me flag a piece of memory as "tainted", and allow me follow it through the system to an arbitrary function call.

Since vulnerability classification is the first step in determining actions, use attack modeling techniques to determine exploit vectors. Enumerate all the ways untrusted input can make it through the system. For example, an attacker may be able to input an IP address into a field in a web app. In this case, an IP address text field could trigger the vulnerability.
Once all vectors are enumerated, determine if mitigations are in place. In the case of ghost, only 1024 or longer inputs may be vulnerable. When that work is done, there should be fewer code paths to analyze.

As Qualys proved, there are many possible attack vectors. If they cannot all be analyzed in a short amount of time, then prepare for the worst case.
Weigh the risks before deciding a plan of action. Often the fix for a new vulnerability is not immediately available and issuing an incomplete fix is worse than taking a few extra days to publish the correct fix.
In this particular scenario, the risk of patching glibc is relatively small -- the code has been patched since 2013 and there is a definitive test case that can be run easily. It is prudent to issue a patch as soon as possible.
Given the classifications published earlier, ghost would qualify as a severe vulnerability.  Actions should be dictated by established policy.

Thursday, January 15, 2015

Software Engineering Security Part 5: Third Party source

To reduce the time to develop products, many software shops include third party source — often open source — into their product. This is a liability when the source has vulnerabilities.

Step One: Set up a license review board. Before including third party source, consider how it will be used and the license. Since many licenses prohibit certain behavior, a license approval board should decide what licenses are acceptable and under what conditions.

Step Two: Set up an inclusion review board. Practice good risk management before including the source. Identify risk attributes and assign a score before inclusion. Quantify the risk by investigating and ranking these factors:
  • History of vulnerabilities. If a project had a high number of vulnerabilities in the past year, it has high risk. OpenSSL had 21 vulnerabilities in 2014.
  • Project maturity. Code that is abandoned by it’s authors carries high risk. If some technology or code is already stale, it is high risk. Even if it is a hot piece of technology, it may be still be abandoned in favor of other technologies. Mature and stable projects will have lower risk.
  • Ownership. If a team wants to include third party source but will not maintain it, it has high risk.
  • Exposure. If the source exposes an external interface, it has more risk.
For example, apache has high exposure, but relatively low risk because of it’s maturity and recent history of vulnerabilities. Third party libraries used to parse untrusted input, e.g. XML or JSON parsers, may have high exposure and high risk due to recent vulnerabilities.

Step Three: Track third party software in use. Appoint an owner and a backup to watch for security updates from each third party package. The build process can be used to track third party source and create tickets to upgrade software. If there is an upstream OS vendor, consider using their updates.

Step Four: Have a maintenance strategy. Consider how source will be maintained and updated by the owners. Consider purchasing an update service from a vendor. If an upstream vendor does not cover the desired packages, consider automating updates from upstream source repositories.
Also consider running security tests aimed specifically at functionality exposed by the third party source. Report the findings back to the vendor or open source project.

Step Four: When there is a vulnerability, assign severity and follow process to update it. Third party source should be treated no differently from proprietary source when there are vulnerabilities.

Following the above steps is an effective risk management process for third party source.

Wednesday, January 14, 2015

Contacting me

If you need to contact me privately, my public key is:

Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools -


And here is some key base info to verify and find me.


I hereby claim:

  * I am an admin of
  * I am costlow ( on keybase.
  * I have a public key with fingerprint 8617 AE0B 3D7E 7706 9F57  D9C2 EA75 ABB2 17B1 40DB

To claim this, I am signing this object:

    "body": {
        "key": {
            "fingerprint": "8617ae0b3d7e77069f57d9c2ea75abb217b140db",
            "host": "",
            "key_id": "ea75abb217b140db",
            "kid": "010110f264e2b3e589aefa9d0c0b13facb62d42dd7bf1597f920f46d0470bd3da2770a",
            "uid": "d44bcf7ca07521dc07ce17dd948da200",
            "username": "costlow"
        "revoke": {
            "sig_ids": [
        "service": {
            "hostname": "",
            "protocol": "https:"
        "type": "web_service_binding",
        "version": 1
    "ctime": 1421248321,
    "expire_in": 157680000,
    "prev": "d02d862793f407df61e3b00c79e6c3ce9ff2720479de2cca39d4689e3ae9d8fc",
    "seqno": 8,
    "tag": "signature"

with the aforementioned key, yielding the PGP signature:

Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools -


And finally, I am proving ownership of this host by posting or
appending to this document.

View my publicly-auditable identity here:


Sunday, January 11, 2015

Software Engineering Security part 4: Definitions and Language

This blog has covered assigning severity to vulnerabilities, but we haven’t covered the definitions of vulnerabilities.

A vulnerability is a weakness in software. For a vulnerability to be exploited there must be a weakness in the system, it must be exposed to an attacker, and it must be exploitable by the attacker.

Security works best in layers, so eliminate or mitigate all three of these possibilities to reduce vulnerabilities in software.

To reduce the number of weaknesses, use secure development practices and a secure development lifecycle.

To reduce exposure, limit the software’s threat surface. Limit the number of services exposed externally, use a firewall, and ensure that untrusted user input goes through a trusted and secured code path.

It’s hard to reduce all exploitability vectors since attacks only get better and future attack methods can’t be reliably predicted. Host-based intrusion protection such as SELinux, stack protection, and ASLR work reliably but take diligence and effort to ensure they are always enabled and kept up to date.

Finally, consider exposure when assigning vulnerability severity. Often a vulnerability has severe consequences but can be mitigated by disabling a service or feature. Calculate the CVSSv2 score for different scenarios and consider how best to mitigate a vulnerability.

Monday, January 5, 2015

Software Engineering Security Part 3: Vulnerability Response

The two previous articles covered vulnerability tracking and secure development. This article will dive deeper into the final phase of the secure development lifecycle — vulnerability response. Responding well to vulnerabilities is a sign of a mature development shop.

The primary objective is to keep all customers as safe as possible. Update software and communicate mitigation information to customers to keep them safe.

Step One: Assign a severity to each vulnerability
See the earlier article on tracking vulnerabilities for example severities. When a vulnerability is critical, a response is required. Often, the cone of uncertainty for a vulnerability will linger for days with new information surfacing regularly. For example, Shellshock contained 6 CVE identifiers spread out over about 5 days. A dedicated team needs to continuously analyze vulnerabilities and watch for new information.

Step Two: Follow process and decide when to update
To keep all customers safe, coordinate the release of software patches so all customers have an upgrade path. Release any mitigation details at the same time as the software patch.
If the vulnerability is in third party source, there may be no pre-arranged coordination. Heartbleed and Shellshock are two examples of third party source that required immediate attention.

Step Three: Update
Release the update to customers.

Step Four: Communicate
Communicate to customers the severity of the issue and give them mitigation strategies if they cannot update immediately. Do not release details that will tell attackers how to attack the vulnerability. Be forthcoming about the vulnerability, do not hide information. Customers would rather patch than be hacked because of a vulnerability that was not well communicated. Finally, use CVE numbers and update CVE/NVD with the details so that customers can find information.

ISO 30111 deals directly with vulnerability response and can be consulted for more information.

Following these steps and rigorously analyzing vulnerabilities will greatly improve your organization’s response to vulnerabilities.

Links to other articles in this series: