Automated Threat Hunting Advantages
The Threat Hunting Process
Proactive threat hunting within organizations has never been more critical for cybersecurity professionals. In the age of advanced phishing attacks, ransomware, and other attacks that fly beneath the radar of traditional security tooling, it’s almost required that organizations be proactive in terms of threat hunting in order to find threats in their organizations. The act of threat hunting itself has traditionally been a manual effort, as you tend to follow certain steps for the threat hunting process such as the following:
Create a hypothesis for your threat hunt (e.g. I suspect there are specific living off the land binaries (LOLBINs) being used for malicious purposes on our Windows systems)
Set a timeframe for the threat hunt (e.g. I will hunt for these living off the land binaries for the next 2-4 weeks)
Collect data for your hypothesis (e.g. I will collect 3-6 months of data from our Windows systems)
Manually go through the data to find the threat (e.g. comb through logs to find evidence of the living off the land binaries)
Under ideal circumstances, your organization would be collecting the correct data into your SIEM (in this case, Windows command line logs) and have a threat hunting program where dedicated personnel proactively looking through your SIEM for potential threats. However, if you have ever worked in cybersecurity, you’ll quickly realize the challenges that threat hunting poses.
Threat Hunting Challenges
While traditional threat hunting is easy to understand and perform in theory, it becomes difficult in practice for three reasons:
It requires time and manpower, both of which are scarce in the days of cybersecurity teams trying to do more with less
Vast amounts of data make it near impossible to manually comb through logs to find the threats in question
Buying automated tools is an added cost, and may not be feasible for cybersecurity teams who have competing priorities or don’t have the budget for them
In regards to the first point, the time and manpower required for threat hunting is a roadblock for any cybersecurity team attempting to perform regular threat hunts, let alone creating a dedicated threat hunting program within their organizations. The organizations that are able to have established threat hunting programs tend to be very large, and the smaller organizations and businesses simply don’t have the resources to establish teams like this. This leaves the smaller organizations with a threat hunting sized hole in their cybersecurity teams with a need to be filled.
The second point is a technical challenge that comes inherently with any data analysis activity, including threat hunting. While some hunts may yield smaller datasets that teams can manually threat hunt, other hunts may require looking at thousands of lines of logs that is impossible for any threat hunting team to manually analyze. Going back to the threat hunting example, there’s a large number of LOLBINs that can be executed on Windows systems. If you have hundreds of Windows systems, you’ll have to multiply the number of LOLBINs by the number of Windows systems in your environment. This assumes that you have perfect knowledge of the LOLBINs (which is possible), but you’ll be quickly overwhelmed by the data that you’ll have to analyze. You also have the ability to scope your hunts to specific LOLBINs, but then you risk having to perform multiple threat hunts for the other LOLBINs which can extend your threat hunts to something much longer than was originally intended. In short, data analysis becomes difficult when you start looking across the entire environment.
The last point is one that may not be a challenge for some organizations, but may be for others. If you’re an organization starting fresh and have the budget to buy an automated threat hunting tool that complements your existing cybersecurity tools, then this may be the best route. However, most cybersecurity teams already have established tools that they are already using, and incorporating yet another tool into their toolbox may be a hard ask. When all of your data is already going to a SIEM, the justification to management of buying a completely different tool to perform something that the SIEM should already be doing is quite difficult. This line of reasoning can sometimes be faulty to some cybersecurity professionals (as not all tools are designed for threat hunting), but to people who manage cybersecurity budgets, it’s a legitimate concern.
What is automated threat hunting?
These challenges can all be addressed. Automated threat hunting is the process of having a system perform the manual tasks of data collection and analysis of a threat hunt. In terms of the typical threat hunting process, it removes the human element from steps 3 and 4 of the threat hunting process outlined in the first section of this post. While you still have to create your hypothesis for the threat hunt as well as setting the timeframe for it, the most arduous parts of the actual threat hunt are left to the system to perform. Ultimately, the goal of threat hunting automation is to streamline the hunting process, allowing your team to quickly get to the actionable part of determining actual threats in your environment.
Automated threat hunting advantages
The biggest advantage is that it frees up the time of your hunters to do more productive things with the data collected for your hunt. Regardless of whether your organization is large or small, being able to streamline the log collection and data analysis of any hunt is huge. It allows organizations to begin and finish hunts faster, allowing your team to go back to usual cybersecurity operations or even start more hunts. Large organizations benefit from threat hunting automation because regardless of the size or maturity of their threat hunting program, they’ll be able to analyze large amounts of data for anomalies much more efficiently, which is critical for larger organization with hundreds or thousands of systems to analyze. Smaller organizations benefit from it because you can now perform your own hunts with minimal resources and without a dedicated threat hunting program, as the arduous parts of the threat hunting process are left to the system to handle.
Another advantage of automated threat hunting is that you get to learn what’s normal for your organization much quicker than you can manually. The process of threat hunting is very enlightening, as the data shows you what’s normal and anomalous in your environment. This knowledge can be especially useful when you have to perform an actual incident response process, as you know exactly what’s normal for your environment.
When should you use AI for threat hunting?
Needless to say, the use of automated threat hunting within any threat hunting program hinges on the use of artificial intelligence. That being said, you need to be careful; not all tools are created equal. A lot of vendors claim that their tools utilize artificial intelligence when they may not be using it at all. Unfortunately, artificial intelligence can be defined as “any system that appears to make intelligent decisions”. Because its definition is so broad, it’s easy to slap an AI label onto any product to make it look state of the art. Automated threat hunting tools are no different, and you owe it to yourself to know the difference between one that uses actual artificial intelligence (e.g. statistical analysis, neural networks, etc.) and one that claims it does for marketing purposes. Before buying any tool, here are some questions that you can ask the vendor:
Can you elaborate how this tool utilizes artificial intelligence?
Is there some kind of AI model that this product uses?
How does the AI make its predictions?
Can you describe how the model is trained?
Is the model using basic statistical algorithms or is it using deep learning algorithms?
Can you elaborate further on the AI algorithms being used?
Do the models get better as they’re used over time?
These questions are especially critical for larger organizations with large datasets, as the efficacy of any automated threat detection can vary when you’re dealing with hundreds or thousands of users and systems.
When should you automate threat hunting?
Simply put: you should try to automate threat hunting whenever possible. With the rise of sophisticated cyber attacks, being able to hunt for those hidden threats in an automated fashion is a requirement for any organization. Unfortunately, modern cybersecurity teams are expected to do more with less, leaving cybersecurity teams with no choice but to embrace automation. This isn’t necessarily a bad thing either, as artificial intelligence appears to be one of those things that’s here to stay, so it only makes sense for cybersecurity teams and threat hunting programs to embrace the technology for their own means.
Conclusion
While AI solutions do exist, there tend to be challenges integrating them into existing environments (including money, manpower, and time), as well as configuring them to support your various threat hunting hypotheses. If you’re looking for an AI threat hunting solution that utilizes deep learning methods and integrates into your existing environment, check out how QFunction performs AI-based threat hunting ! And if you’re still on the fence about how the process works, check out the blog post on how AI was integrated into Splunk for threat hunting on domain controllers!
