Hunting down crypto miners on Linux using Microsoft’s Azure Security Center
This year, crypto mining replaced ransomware as the top cybersecurity threat. Malware, in general, accounts for the majority of cybersecurity threats, but crypto mining reigns supreme.
The challenge is detecting these crypto miners, as they get covertly installed on your machines. They might not be doing harm directly (eg: stealing your data) but they are taking away valuable computing power from your environment, leaving no compute cycles for your business app, and potentially increasing your cloud computing bill altogether.
Azure Security Center, which has great support for Linux and container technologies, can help you hunt those miners down.
A third of all malware is crypto mining
General malware, which includes both ransomware and crypto mining, made up 52% of cybersecurity threats in the first half of 2018, according to Webroot’s Mid-Year Threat Report Update. But crypto jacking — the nonconsensual act of crypto mining someone else’s machine — alone accounted for 35% of threats.
“In either case, it may be largely invisible to the end user, who likely won’t notice a small spike in their electric bill. But for an organization, power bills can skyrocket, especially when criminals employ scaling, i.e., keeping the drain on the CPU minimal when a keyboard or mouse is being used but scaling up to 100% at other times.”
How high CPU utilization metrics leads to detection of a crypto miner
We’re running a combination of Linux and Windows systems and servers on Azure, and together they are hosting our business application. The application has been performing slowly and logging into the Azure Portal we notice an Alert in Azure Monitor about high CPU utilization on one of the Linux servers:
DaVinci is not who we think he is
On examining the regular process and daemon list, we see nothing out of the ordinary. However, when we start investigating what else is running on the machine, we find a Docker container under the name “nervous_davinci”. Microsoft provides a solution for Container Monitoring as part of Log Analytics and provides details about the containers running on the machine. The associated CPU time of the container is 132%, and it’s executing a command that start with “cpuminer”, which raises a red flag immediately.
Both Log Analytics and Azure Security Center share the same Log Analytics workspace, thereby centralizing information collection. We head over to the Search function of the workspace and run a query to get more information about the container: when it was created, etc.
Kusto — Microsoft’s new powerful query language
The query language you see above is called Kusto Query Language or KQL in short. The query language was first developed by Microsoft internally, for search through large sets of data. It first surfaced externally when being used in Microsoft’s Operations Management Suite (OMS) which later became Azure Log Analytics. Today it is being used more and more in Azure services, for instance in the newly announced Azure Data Explorer. If you already know SQL, you can use this cheat sheet to quickly start using KQL.
Let’s do some more hunting
We validated that indeed there was a Docker container active on the Linux server, running the CPU miner by logging on the machine itself:
Azure Security Center
Heading over to Azure Security Center, we found that it had already been raising alerts:
Going from reactive to proactive
Because we were being reactive to Azure Security Center, eg: only logging in and viewing the alerts every once in a while, or when an incident occurred, we concluded that we needed to become more proactive about it. We used Azure Security Center’s playbook functionality in combination with Twilio to achieve that.
Playbooks are essentially Azure Logic Apps that you can run when an Alert is raised in Azure Security Center. An easy way to create them is to use the visual designer. You select the “when a response to an Azure Security Center alert is triggered” object as your start object, drag and drop a follow-up action (in this case: sending a text message notification to the engineering team via Twilio), and use the dynamic fields to pass information on to the next item in the ‘chain’.
Linux detections try-it-yourself
Yuri Diogenes, a senior program manager at Microsoft who focuses on Azure and security, wrote a Linux Detections step-by-step workbook for Azure Security Center. If you want to try the Linux detections yourself, Yuri guides you through setting up a Kali Linux and Ubuntu Linux VM on Azure and some samples commands and scripts to trigger alerts.
Happy threat hunting!
— Maarten Goet, MVP & RD