Thursday, February 5, 2015

Software Engineering Security part 6: Running External Penetration Tests

Engineering shops often won’t have the expertise or the time to do deep security testing on all areas of the product. Using external security firms who have specific security knowledge can extend security testing well beyond their own capabilities.

External security testing — often called penetration testing or pen testing — can be effective when the scope is well-defined and narrow. It is better to do four small tests on specific areas rather than one large test across much of the product. Two to three week engagements are relatively inexpensive and should be run once per quarter. Even four engagements a year are generally cheaper than a full time security tester.

Pen tests also grow security expertise in the organization. Dev and Test resources who participate in the pen test will have a deeper understanding of security issues.

To make the most of a pen test engagement:
  • Security contractors are very expensive. Be prepared to spend time before the pen test configuring the software, setting up the test harness and gathering documentation to make the most efficient use of their time.
  • Ideally a security test is run on full featured beta code about 8 weeks before the ship date. This will allow time to fix any issues found before shipping. This is not always possible, so run the test when you can.
  • Assign severity to the issues; policy should dictate action.
  • Many pen testers will want to inspect source code. The steps below outline a black box testing strategy in which source code is not needed. Inspection of source code is generally a longer engagement and should be a supplement to source code analysis used during construction.
  • Select an ethical pen tester. Ethical pen testers will obtain permission before engaging, document all steps, judiciously select targets, and report all findings.

To run a pen test, follow these steps.

Step One: Assign a pen test resource
Assign an owner for the pen test. This person will be responsible for writing the RFPs, configuring the test harness, gathering documentation, communicating with the contractor, setting up meetings, and other tasks. Expect these tasks to take roughly one third to one half time during the pen test.

Step Two: Focus on a feature
Select a feature that is exposed and has potential weaknesses that could be exploited. Consider features that have had known vulnerabilities or weaknesses in the past. Also consider features that have known deficiencies in test cases or a long legacy of different engineers. Specify the feature and the constraints in the RFP. Ideally, find a vendor with experience in the subject area.

Step Three: Involve feature engineers
Select engineers from dev and test to be responsible for actions during the pen test. Record roles and responsibilities during the engagement. Dev engineers are responsible for explaining design decisions, asking questions to understand the issues raised, and securing the code. Test engineers are responsible for understanding the source of the issues, writing tickets, and developing test cases around the issue so that no similar issues are found.

Step Four: Set up the scenario
Pre-configure and set up the scenario. If possible, configure VMs with the software and deliver those to the pen tester. Ideally, set up a cul-de-sac network that contains just the software under test and allow the contractor access to that network.

Step Five: Run the pen test
Schedule weekly meetings during the engagement. Involve the dev and test engineers and and make sure they understand the issues and can fix them. Focus on correct mitigations rather than implementation details at this point in time.

Step Six: Followthrough
Classify all vulnerabilities according to vulnerability severity and have them worked according to policy. Write unit tests to ensure that similar vulnerabilities don’t happen. Write test cases to avoid similar issues. Hold a postmortem lessons-learned meeting and spread the security knowledge through your organization.


Following these steps will build a more secure product, grow security knowledge in the organization, and develop better practices around secure development.

Wednesday, January 28, 2015

CVE-2015-0235: Ghost

The last two days have been taken up with the glibc ghost vulnerability found by Qualys. They have a good description and a good detailed analysis.
Essentially this is the old "parsing untrusted input" problem -- a value comes in from a dubious source and the code needs to be robust against an attacker who can choose arbitrary malicious input.

Vulnerabilities like this are hard to classify. It's almost certainly exploitable by someone who has a local account on a box. But determining if it is remotely exploitable is difficult. This is what is called the "cone of uncertainty"; it takes time to do the research and find out if any code paths are directly vulnerable. Starting at the vulnerable gethostbyname() functions and working out is time consuming.
Many times this week I have wished for a rough equivalent to perl's taint checking -- let me flag a piece of memory as "tainted", and allow me follow it through the system to an arbitrary function call.

Since vulnerability classification is the first step in determining actions, use attack modeling techniques to determine exploit vectors. Enumerate all the ways untrusted input can make it through the system. For example, an attacker may be able to input an IP address into a field in a web app. In this case, an IP address text field could trigger the vulnerability.
Once all vectors are enumerated, determine if mitigations are in place. In the case of ghost, only 1024 or longer inputs may be vulnerable. When that work is done, there should be fewer code paths to analyze.

As Qualys proved, there are many possible attack vectors. If they cannot all be analyzed in a short amount of time, then prepare for the worst case.
Weigh the risks before deciding a plan of action. Often the fix for a new vulnerability is not immediately available and issuing an incomplete fix is worse than taking a few extra days to publish the correct fix.
In this particular scenario, the risk of patching glibc is relatively small -- the code has been patched since 2013 and there is a definitive test case that can be run easily. It is prudent to issue a patch as soon as possible.
Given the classifications published earlier, ghost would qualify as a severe vulnerability.  Actions should be dictated by established policy.


Thursday, January 15, 2015

Software Engineering Security Part 5: Third Party source

To reduce the time to develop products, many software shops include third party source — often open source — into their product. This is a liability when the source has vulnerabilities.

Step One: Set up a license review board. Before including third party source, consider how it will be used and the license. Since many licenses prohibit certain behavior, a license approval board should decide what licenses are acceptable and under what conditions.

Step Two: Set up an inclusion review board. Practice good risk management before including the source. Identify risk attributes and assign a score before inclusion. Quantify the risk by investigating and ranking these factors:
  • History of vulnerabilities. If a project had a high number of vulnerabilities in the past year, it has high risk. OpenSSL had 21 vulnerabilities in 2014.
  • Project maturity. Code that is abandoned by it’s authors carries high risk. If some technology or code is already stale, it is high risk. Even if it is a hot piece of technology, it may be still be abandoned in favor of other technologies. Mature and stable projects will have lower risk.
  • Ownership. If a team wants to include third party source but will not maintain it, it has high risk.
  • Exposure. If the source exposes an external interface, it has more risk.
For example, apache has high exposure, but relatively low risk because of it’s maturity and recent history of vulnerabilities. Third party libraries used to parse untrusted input, e.g. XML or JSON parsers, may have high exposure and high risk due to recent vulnerabilities.

Step Three: Track third party software in use. Appoint an owner and a backup to watch for security updates from each third party package. The build process can be used to track third party source and create tickets to upgrade software. If there is an upstream OS vendor, consider using their updates.

Step Four: Have a maintenance strategy. Consider how source will be maintained and updated by the owners. Consider purchasing an update service from a vendor. If an upstream vendor does not cover the desired packages, consider automating updates from upstream source repositories.
Also consider running security tests aimed specifically at functionality exposed by the third party source. Report the findings back to the vendor or open source project.

Step Four: When there is a vulnerability, assign severity and follow process to update it. Third party source should be treated no differently from proprietary source when there are vulnerabilities.


Following the above steps is an effective risk management process for third party source.

Wednesday, January 14, 2015

Contacting me

If you need to contact me privately, my public key is:

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org

mQENBE/89NUBCADDLw1eYccoW50xxniyv4WPTFNIwaEXm25EebioaYS1DOT7LQWS
2xzHxxSayfcJ1cFHhNfH6v03t3oMtKQocT03LN440QPAJaFNj7KXffsobmDqN7uA
xAMzehYtD2Ot1HaQ+NEWR8UpN1+G+EIRzlSI2u/LXTVwN2dL59T7kUvvAPhRHiA4
Diokp43bs+c4DaVPcCrq+Pga0+0DPqsMyjXy8jT816T+rhhE0bZkhplK3GNBQ05+
cOQIXqaMGNmXza/nSazERz/4LMSqbR4ix8ZdIsMoCtqny2/tJQq6offGGN1Sb2s1
v6QQrdSoAOjGYwC+VJsdNp0kTxuJrz7xW5HzABEBAAG0IEplZmYgQ29zdGxvdyA8
Y29zdGxvd0BnbWFpbC5jb20+iQE/BBMBAgApBQJP/PTVAhsvBQkHhh+ABwsJCAcD
AgEGFQgCCQoLBBYCAwECHgECF4AACgkQ6nWrshexQNsWZwgAgZDlEp32l97/3/Km
xuWWIFXJxUlbCK7tnY+0wzCFiZcYSWqXfA4ZwlkQ+OE0A2Pm089+OhGsnBG6VwFq
Ab7MOQBt98PCe1Kky7VxU6BKK8JbvxbLiY3ZBqbV+LUAhYXlRQ+vMw6kYt5REuQU
ry+kYrokwN3gLwQUOsoqyE34wBqVUBHbkYOrozUMdbz7bUmRX7decOsJx+4gVKHN
4+CNFRDHywVojf7aS9IMPppQcaKGeT6a3QqYQLQKlxaOJNPWZK8R6vNoln4WJ5Fm
YB4cK13V8VfNkKC39wqIoWGv0f0EcQmEZruT3e+zwW6m3eF3kjlZAMvob6dJibBW
v8CLZrkBDQRP/PTVAQgAwFQAGbXS/HouHYPrLQsjuXJLERIjBBAuqr4fZ61s34Wx
xPbpMx5HsxBhdyMcntXIH9uCDO5yuxffyJ0QddknWbqRrsTxnD8Bs0+xxwy/TGQ7
M3sQMySCUeea65SuY81Zb6Y2eqIcUhk3j2nXTHhXNvjnjxA7OrAqDUp1pEbZ3zgD
ysry812I8uofdUkI+4CsPaFMyhvAhPWpMJVo7LOmZ7nOxoACT4W/E2gOxvTv7/wH
J4A3pHCQgq2t62Ix7OSXs9S1wXc4Lg8srM5O85vOEdRVfCpbY2drC9bQz+XHYYCQ
W+PyY7RV9mcIjFuXxlaUEx8+pGI4yC2PDAhCzab4DwARAQABiQJEBBgBAgAPBQJP
/PTVAhsuBQkHhh+AASkJEOp1q7IXsUDbwF0gBBkBAgAGBQJP/PTVAAoJEGnuVoT4
8F14ywIIAL0BMe41ydhcYxogS3jyoaXOP67gRbF87tNGswpRw9GsZuHB9oD7eN5C
dD7Guo87ts3QkGkJd/ueA1sB9pXiH2KIwuMf5SMIXZiwQlC4/EpK/2sCq1ru66H7
S58xte3QmESfGtP3rxOS2CjSscR8n3NAO5a2rN81lvK1maU62BRXDSB4xdWHDEZj
HuzsgI4Sau0KrInovybTFhuGtkA4mXfb1Gmz3E2asYC/YN+F0L5FxqEtZ+OMGg5G
sDS5Ki/K77kHhMkuM6zGD3rE1iP6cGUGmBq39M6CrDrQcGlusXNxmy5WE2+n4k5J
4kG64/yDxZ73FCu8ebD6m94KWB1P+u8vJggAhAO18xITKpMg/qZzlu+CrrhjL66x
iCTmtCfzx+o8H7fcRk9i2/+ek49g4GnR0oZGkrMoNladD6SV6tX3eZwbXpq3bHy/
q5tsScJHpxIIeSbqZQsvWu4b1cYwHdFxvmbPea45p1/KQe0NeRgH0xjRV2D8c9/g
kudL4DTtdfVaxROgdEV3f5axPGRkJpNBQRP9uo3NmKahv2LfaT9SMHZ7eK/0LBNe
z0HdJj6M09C7Ucq9uz17+oa6q7ul3nCgAzHMbmrLoAg1eOLeOrgtlTcn0IGLpulT
NEVkQe81IOSWb+K6l+seGOO7kwWwO1pHF1+QDRAaZ5VkaXS9tiRhVfS7CZkBogRC
cFp5EQQAnezbhvgcE5of+PmvMnNzQZScUwq/PxaZemUzOKtUzyPq1af/xk+TT7oC
gT1+WwC8lpicvhMMkBBAMz3We1xhv7svjvENixRXu/I1dnTiwAFj2naqLZpmQEIk
aUZIWVhavGuIqJLuUv7ZTepw1Stdqd13C4BzRLFPbVjg64Wv7X8AoLR7OdZVBEL8
IbATKwKVCKYss8+5A/40UUdxe55tksccjDeeTi2hzELsE0YKTAaxBrbxGYjK223g
xiPvuT4EOr9uiboHkD0nsIEqE9fvz+pgRNLJxVafAGXjY3LGpFCSUxZoo9ntuh+c
/aK2zdfnkVrk0rHnNmCUiXeFz5lepc200u6Rz1kJEqwX9y2GE+hPlnemHFJFpgQA
hzt9uVgAWMPbLvlQfsA+GB93+OM+flEjB12Kfc3QFonjaHbeRTw74YPov5bQYQYh
xdC7uhcumq0x93/EI3uGFoIlVdJ3G8IUHO0qmm/qwZZlTKcnQWLNVgugXEMmjgFv
IZsmBG8OOQNhmBYJrIeMymQKJFwvyhek5Er2VHklkE20IEplZmYgQ29zdGxvdyA8
Y29zdGxvd0BnbWFpbC5jb20+iGEEExECACEFAkJwWnkFCQPCZwAGCwkIBwMCAxUC
AwMWAgECHgECF4AACgkQ9LLXEc64BOMbjwCdGz5hWtG2SzSKKfSJe0kCrDNKXMAA
nAtplHd0M04k4DSWTdChVv4uguRIiGEEExECACEFAkJwWnkFCQPCZwAGCwkIBwMC
AxUCAwMWAgECHgECF4AACgkQ9LLXEc64BOMbjwCffXBkCMqefpeMyvRQc7wgZrub
I1wAoJ+fiNsdCaVRYyEdUU2ABjJ6q/MguQINBEJwWpUQCADV/ujXt+uYqYdDwn+I
eOhlDves/fegIIntMLkCCIjZC5rEWhOz/By0UehXfKhNa+/bpr6Gr0QzEEtolkip
7g3WFZur0ct+CbOKdWtchnjpkkCpBG5L6rjf+mOQrR0px5M+jDA5vDc041ThOzAl
fbaWlIWH3yrRdE3S2OF7e5sQXF/TiXL+S9WjCet02EJ0U5+jFs8TNId+R+aSrByp
ZWL/10Hc8EH38NfwsGuUFzZOhA43Jl9FJADRS/PuKBvCuB6X7iBpA2avML2Wz/1R
ct9lZDeFZ3cjpBozLCa84CQTf5YqwQlw9WmgFBaWdjFJAWt3+Euu7aHvrDkFzI7m
V4bDAAMFB/99/WNE4ItcGT9tEz734K36wTZzjgtsh/pyFadrezRRx1oRROAxTHsD
wLa/zFOjqTexSQbPr8XqHCIuf4aUjFYJW1r/wzkYOXljECLYj+Dy9jSNmOG9RA+Y
z+y//hKOvMJX7xd6DsQkviwA3mOKQZWd/5I2Ke8ldMOVbtPFUQH/64tyoRnKuCUb
3Fr6UAod9lIGrhNpIWFKVH0h43mlNp/ChGY7Uvm6jb2bzgCYLK+s8FDL/idZgD1a
nWPFG+uEPeTMfxU2/Ins8+HJtgWwTlY/fUmzMsVSwKfQT17HicvTGsDu7lEW0DxA
UxAOftYT3M018BvJvjMkphNekAZ/2I6MiEwEGBECAAwFAkJwWpUFCQPCZwAACgkQ
9LLXEc64BOMypACgqvBY1Utuu1sZ3qJcsWxd7vSA53QAn14vVikzrX1t5zxB3gDe
n6MP0qS4
=YCC8
-----END PGP PUBLIC KEY BLOCK-----

And here is some key base info to verify and find me.

==================================================================
https://keybase.io/costlow
--------------------------------------------------------------------

I hereby claim:

  * I am an admin of https://swengineeringsecurity.blogspot.com
  * I am costlow (https://keybase.io/costlow) on keybase.
  * I have a public key with fingerprint 8617 AE0B 3D7E 7706 9F57  D9C2 EA75 ABB2 17B1 40DB

To claim this, I am signing this object:

{
    "body": {
        "key": {
            "fingerprint": "8617ae0b3d7e77069f57d9c2ea75abb217b140db",
            "host": "keybase.io",
            "key_id": "ea75abb217b140db",
            "kid": "010110f264e2b3e589aefa9d0c0b13facb62d42dd7bf1597f920f46d0470bd3da2770a",
            "uid": "d44bcf7ca07521dc07ce17dd948da200",
            "username": "costlow"
        },
        "revoke": {
            "sig_ids": [
                "3c2679a4f1af0d808709852174aa9104618b516dbb0d41815632dd372b577a870f"
            ]
        },
        "service": {
            "hostname": "swengineeringsecurity.blogspot.com",
            "protocol": "https:"
        },
        "type": "web_service_binding",
        "version": 1
    },
    "ctime": 1421248321,
    "expire_in": 157680000,
    "prev": "d02d862793f407df61e3b00c79e6c3ce9ff2720479de2cca39d4689e3ae9d8fc",
    "seqno": 8,
    "tag": "signature"
}

with the aforementioned key, yielding the PGP signature:

-----BEGIN PGP MESSAGE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org

owFtkl9sFEUcx68noDRKpJUaE/RhgtHgpc7Mzu7sXAkG02iaAKJBSSC1zN9juevu
dXfv2kstJvViozUpUYxG8KGkCQIGGv7FEOMT4IOACEZEXwg+GBqoRPwfY51t8M15
mcxvvvP5fX/fzM777srlW4LZl5p/3OodavniushtPDG2fhiISDVAcRiU9fxmgrCk
42ochCkoAt9DlGsoHEU1pdBjxqWKSaw5dbkQGFGBCFQCFMC2KMleWIzgie4MIluz
h75A2er/6MvzFxBBhKDBHtFYONr1GdeGMwUlFMgxXAoPK4KVosIgl1HDMDTEU5BQ
KJSjOLa2uMXV5nGKECENlRxSFyMlIZUaUaUY8a0UwkyY6Djk/dqqpbVciQbBSAHE
uh6VdRZAEpSs5wQUtwBHYo8yTgziBiof+hQy33Ip4ZwhSDzkCxd5SgioCPKR6znW
qUOxcCnlVm1Ar2XbhvVAzsOzkO40TwZ1WApCrW3UpUTLWhykjU5RiUpJNUo7ZdRv
zVbjKI1kVLH6bWlaTYqZ17RRzQCDWvTdQfeJIFQWY1/UdZwEUQiKyCplGmS9EMEI
E9/BqAD0UDWIdV8Q2gmRSz0f2pU10vUsP4iV72HKHEMgVcZD2hEQSsq0Jx2pmTGY
Yhs+UxpLyR2miOcz7XDNlG8kyKYdCCNQ9K1PXsrmDEohT2uxBiOtb7QsX5BryecW
Lcxnny/Xuvj+/35kz3t3z/Hf3q8c3HX1tVWrf5lsWzHx00D7sTfbvz+0vOJdgntv
jU22ziwhz9zYVL702dpr549/N75hB9m6tbZhz7fnQP7iD4367bdHhz9Qpmd838O7
t3f1r5vdc2r6q2+a6z69GT39yIe9r5debMg10w8lE7Xu3Je/dvivPDH+wMr1+NXP
mxfW/tjT7I/z039NjU49fmMuPTLSfXLmzDvo59reVduPfd1+fNfEyf3L/uw4NMCa
+anuT54/vPPRy0cKk6ed86s3PzvbNVN46vqag2RRx9DRy2ejFUs+uvj36XtPCfly
dM+7566MpRsfvL3grR0LVy69UJNtv/eax5bO7j5be+6fM3NtH4++cKXcdWDm2pP/
Ag==
=2Dff
-----END PGP MESSAGE-----

And finally, I am proving ownership of this host by posting or
appending to this document.

View my publicly-auditable identity here: https://keybase.io/costlow

==================================================================

Sunday, January 11, 2015

Software Engineering Security part 4: Definitions and Language

This blog has covered assigning severity to vulnerabilities, but we haven’t covered the definitions of vulnerabilities.

A vulnerability is a weakness in software. For a vulnerability to be exploited there must be a weakness in the system, it must be exposed to an attacker, and it must be exploitable by the attacker.

Security works best in layers, so eliminate or mitigate all three of these possibilities to reduce vulnerabilities in software.

To reduce the number of weaknesses, use secure development practices and a secure development lifecycle.

To reduce exposure, limit the software’s threat surface. Limit the number of services exposed externally, use a firewall, and ensure that untrusted user input goes through a trusted and secured code path.

It’s hard to reduce all exploitability vectors since attacks only get better and future attack methods can’t be reliably predicted. Host-based intrusion protection such as SELinux, stack protection, and ASLR work reliably but take diligence and effort to ensure they are always enabled and kept up to date.

Finally, consider exposure when assigning vulnerability severity. Often a vulnerability has severe consequences but can be mitigated by disabling a service or feature. Calculate the CVSSv2 score for different scenarios and consider how best to mitigate a vulnerability.

Monday, January 5, 2015

Software Engineering Security Part 3: Vulnerability Response

The two previous articles covered vulnerability tracking and secure development. This article will dive deeper into the final phase of the secure development lifecycle — vulnerability response. Responding well to vulnerabilities is a sign of a mature development shop.

The primary objective is to keep all customers as safe as possible. Update software and communicate mitigation information to customers to keep them safe.

Step One: Assign a severity to each vulnerability
See the earlier article on tracking vulnerabilities for example severities. When a vulnerability is critical, a response is required. Often, the cone of uncertainty for a vulnerability will linger for days with new information surfacing regularly. For example, Shellshock contained 6 CVE identifiers spread out over about 5 days. A dedicated team needs to continuously analyze vulnerabilities and watch for new information.

Step Two: Follow process and decide when to update
To keep all customers safe, coordinate the release of software patches so all customers have an upgrade path. Release any mitigation details at the same time as the software patch.
If the vulnerability is in third party source, there may be no pre-arranged coordination. Heartbleed and Shellshock are two examples of third party source that required immediate attention.

Step Three: Update
Release the update to customers.

Step Four: Communicate
Communicate to customers the severity of the issue and give them mitigation strategies if they cannot update immediately. Do not release details that will tell attackers how to attack the vulnerability. Be forthcoming about the vulnerability, do not hide information. Customers would rather patch than be hacked because of a vulnerability that was not well communicated. Finally, use CVE numbers and update CVE/NVD with the details so that customers can find information.

ISO 30111 deals directly with vulnerability response and can be consulted for more information.

Following these steps and rigorously analyzing vulnerabilities will greatly improve your organization’s response to vulnerabilities.

Links to other articles in this series: