Looking at AV Logs More Intelligently

Let's start with the nature of antivirus. Basically, antivirus is intended to catch malware it already knows about. Someone discovers a malware binary, the antivirus vendor grabs it's hash, analyzes the malware, and uses their secret sauce to detect the malware on your machine if it ever arrives. Antivirus vendors have added behavioral and heuristic detection so it's a little more complicated than that, but not much. So what happens when antivirus encounters a malicious file it doesn't recognize? Nothing. That malicious file gets to live happily on your computer and cause whatever trouble it was intended for. But if the malicious file is recognized, it gets quarantined or deleted and you get an entry in your AV logs.

Frequently malware is downloaded in multiple stages and includes multiple files. This is where people start getting confused. A lot of people see that antivirus caught and deleted a malicious file and assume their antivirus is keeping them safe and sound. What about the files AV didn't detect? What about this scenario? The user goes to a hacked website and an exploit kit downloads and executes C:\Users\joeuser\AppData\Local\Temp\stageone.exe. That executable creates the following files:
C:\Users\joeuser\AppData\Roaming\evil.exe
C:\Users\joeuser\AppData\Roaming\evil.dll
C:\Windows\system32\moreevil.exe

Now, antivirus flags on C:\Users\joeuser\AppData\Roaming\evil.dll and quarantines it. Are you safe?
What about the files AV didn't detect, namely C:\Users\joeuser\AppData\Roaming\evil.exe and C:\Windows\system32\moreevil.exe? Your computer is compromised, but a large number of people see AV deleted a file and assume all is well. So how can we tell? Fortunately, we can use some deductive reasoning to make a better assessment based on where the detected file was found. If a file was found in a location where a browser downloads files, it's possible that AV killed it on download before the file ever executed. But if the file was found in a directory it couldn't access without user permissions or local administrator rights, it means another malicious process ran and copied the detected file to that location. That system is infected.

What common ways can a user cause malware to arrive on their computer?

• Download from the Internet/Trojan
• Drive-by download exploit
• Malicious attachment via e-mail
• Infected external drive/USB stick

This means have a finite list of places that malware could land and be caught by malware BEFORE it executes (%username% is the system variable for the logged in user):

• User's Downloads folder
• C:\Users\%username%\Downloads
• Web browser cache (each browser's temp location is different)
• Internet Explorer
• Windows 7/8: C:\Users\%username%\AppData\Local\Microsoft\Windows\Temporary Internet Files\
• Windows XP: C:\Documents and Settings\Username\Local Settings\Temporary Internet Files\Content.IE5\
• Firefox
• Windows 7/8: C:\Users\%username%\AppData\Local\Mozilla\Firefox\Profiles\<random 8 letters>.default\cache
• Windows XP: C:\Documents and Settings\%username%\Local Settings\Application Data\Mozilla\Firefox\Profiles\<8 random letters>.default\Cache2
• Chrome
• Windows Vista: C:\Users\%username%\AppData\Local\Google\Chrome\
• Windows XP: C:\Documents and Settings\%username%\Local Settings\Application Data\Google\Chrome\ 
• Windows 7/8: C:\Users\%username%\AppData\Local\Google\Chrome\User Data\Default\Cache
• Mac OS X: /Users/%username%/Library/Caches/Google/Chrome/
• Temp File Location
• C:\Users\%username%\AppData\Local\Temp
• Files are often downloaded here by regular applications and browser extensions. If you're surfing the web and an add-on like Flash or Java is exploited, the malware downloads to this Temp directory. 
•  Microsoft Outlook temp file location
•  C:\Users\%username%\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.Outlook\
•  USB Stick
•  When you plug in a USB stick, antivirus often runs a full scan immediately. A malicious file here doesn't necessarily mean a system is compromised.

If you see an AV log for malware in one of the directories above, it's possible AV caught the original download and quarantined it before execution. However, it's not guaranteed. Ideally you should check current running processes and forensic indicators showing evidence of execution to determine whether or not the file was executed and whether malware is running.

Also note that plenty of malware executes and creates persistent malicious files in C:\Users\%username%\Local\Temp, so when AV flags from that directory it doesn't mean it was caught on download.

If you see an AV log malware outside the list of directories above, it means another piece of malware has executed on your computer. I cannot stress this strongly enough. If malware is found elsewhere on your computer, a malicious process (stage one) had to have run in order to create it and AV did not catch the original malicious file. Now you have a problem, what else changed on your computer? What other malicious files were downloaded? Unless you have flight recorder style software on your computer, there's no way to know for sure. In those cases, you have a compromised computer and it should be treated as such.

What are the common locations installed malware likes to live that indicate system compromise?

• C:\Users\%username%\AppData\Roaming
• There are no executables that should legitimately run out of this directory. Look at your own system. It's full of vendor directories. Another common tactic is a randomly named (high entropy) directory inside of the Roaming folder. Really obvious malware looks like C:\Users\%username%\AppData\Roaming\slkjdlkj\anoiee.exe. Remember, if you find malicious files here, it means a malicious executable has run elsewhere on your system. It is possible, however, that the malware was run under an account that did not have local administrator rights on the computer.
• C:\ProgramData\
•  Mostly see randomly named directories with malware inside.
• C:\Windows
• There's DLL search order attacks that happen here. Also, executables with the same name as legitimate Windows binaries often get dropped here. Look for masquerading via letter substitution, using an L instead of a capital i because it looks the same at a passing glance.
• C:\Windows\system32
• Malware often tries to hide here. It's hard to tell at a glance what files are illegitimate because few people know them that well. It's also common for attackers to change the Creation Date and Modified Date of malicious files in this directory to ones used by legitimate files so they're harder to identify. 
•  C:\$Recycle.Bin
• Malware authors use this directory because it's a permissions nightmare and is difficult to access and clean from the UI.

If you see malware outside of the user profile, it means malware was able to run with local administrator privileges. We're just seeing pieces of the infection. Malware run with local administrator privileges means total system compromise. Worst case scenario, they stole the passwords from the user account and local administrator account.

But Why Me?
On the Internet I often hear people claim that they won't be targeted and that custom malware that evades antivirus isn't a fear. These people have never heard of polymorphic malware or subscription based malware. Polymorphic malware infects you, then recompiles itself, thus creating fresh malware. I've seen infected USB sticks that infected a computer, then created a new version of the malware (with different hashes and filename), and deleted the original malicious file. Every time an infection happens, it's a brand new piece of malware. It's not that difficult to create and it's an extremely effective attack. AV just can't keep up with malware that changes so quickly. There's no chance to create a signature for it. Then there's criminals who make a living by creating malware and updating it hourly. They test it with their own VirusTotal-style labs to make sure antivirus vendors don't detect it.

In 2014, Sophos said they received 200,000 new pieces of malware per day. Per day! There's no effective way to keep up signature based detection with ever changing malware. That being said, while antivirus isn't a perfect measure it's still an important one. Every time AV flags a malicious file, even if it didn't clean the system, it warned there was an infection and the computer needs to be scrutinized. Instead of thinking of AV as a fully preventive control, use my techniques above to make it a more effective detective control too.

Prevention is Ideal But Detection is a Must

No company wants to get hacked, but the honest truth is that most companies have already been compromised. They just don't know it yet because they aren't looking for it. Businesses have initiatives and antivirus and are following PCI compliance, but compliance isn't security and antivirus won't keep malware off your machines. There's a knee jerk reaction in the corporate world to buy more preventative appliances but that's not a solution. Prevention is ideal, but detection is a must, because prevention often fails. How do we detect? We need people, good people.

First, you need to recognize that your business is going to be compromised. That phishing e-mail that looked like a legitimate customer's purchase order is commonly how attackers get inside. Your employee was trying to do the right thing by working with a customer. Are you willing to stop answering all business e-mail because you could be attacked? Of course not. Attackers know this and use social engineering to the best of their abilities to get you to open their malicious attachments and click their malicious links. Why doesn't antivirus stop it? Attackers have their own labs full of antivirus products and they keep changing their malicious links/files until they get a version that antivirus vendors don't catch. It's essentially VirusTotal.com where their malicious files aren't shared with security vendors. Every time you put up a wall, attackers will try and often succeed in finding a way around it. That doesn't mean you should stop putting up walls making it difficult for the bad guys, you just need to understand that your defenses will be breached.

Now that you understand you've got attackers on the inside, what can you do about it? Detection! You need data, logs, and people. You need to learn what is "normal" on your computers so you can find anomalies. Most malware isn't particularly stealthy. Attackers are lazy, if they can hide in plain sight they will. If you're not looking at your systems, the attackers don't have to use advanced techniques to hide. You also need to learn what normal network traffic looks like. I highly recommend a web proxy. Even if you're not blocking any of your employees' web surfing you still get a record of all the traffic and with those logs comes visible command and control traffic. Once you know how to look for it, I guarantee you'll find it. Most importantly, you need people who can spend the time to understand operating systems, your environment, and learn how to spot the anomalies. This is a skill that is built with experience. It's not something you can learn in a classroom, although once you learn what normal looks like, SANS 511 can teach you great techniques for finding anomalies. I highly recommend the class, but it's not a substitute for getting in the trenches and learning what is normally running on your computers.

There's a common mistake that's made in Security Operation Centers (SOCs) and that's hiring too many inexperienced analysts. Your company security relies on the folks that staff your SOC and watch your operations. If you hire people who haven't worked in IT before, how can you expect them to know what processes are normally running on a Windows system or what botnet traffic looks like? You're essentially sending out a militia and expecting them to do the work of a professional army. If you're going to have inexperienced analysts, they need to be paired with highly experienced analysts who can teach the right instincts. Only then will they become good at detection.

This blog is intended to help bridge some of the gaps in knowledge I've seen in the industry. I've never come across a class that specifically tells you what malware looks like on Windows systems. This kind of work is a blend of operating systems forensics, system administration, and network traffic analysis. In my decade plus in the field I've seen a lot and hopefully I can help move the needle by sharing my war stories.