The 2015 National CCDC season started with 100+ teams across 10 regions. Now, there are 10 teams left and they’re headed to the National CCDC event next week. If you’re on one of those student teams, this blog post is for you. I’d like to take you inside the red team room and give you my perspective on what you can expect and some ideas that may help you win.
The stakes at National CCDC are high. Last year’s winning team, University of Central Florida, had their picture posted in Times Square.
http://videostorage.prnewswire.com/storage2/photos/RNCCDC.jpg
They were also personally congratulated by Vice President Joe Biden at the White House in Washington, DC. This is a big honor and it’s evidence of how prominent the CCDC event is becoming.
With all of that motivational stuff out of the way, let’s jump into the event specifics.
The Opening Salvo
Your National CCDC experience will start on a comfortable San Antonio, TX day. During my years at National CCDC, the blue teams always receive a clean network.
This network stays clean for about five seconds. The red team gets permission to attack at the same time you get permission to touch your systems. We sit in the red team room and wait until our Team Captain tells us Go! Once that happens, we press enter and a script attacks and backdoors your networks six ways to Sunday.
Each year, that I’ve participated in National CCDC, it’s happened this way. There’s nothing you can do about it. The fact that the red team “got in” with default credentials and easy vulnerabilities in the first few minutes is not in your control. It also happens to everyone evenly. When I’m the guy running those scripts, I make it my first priority to verify that every team is hooked the same way before I move on to anything else. Keeping things fair is a priority for myself and everyone else I’ve worked with at National CCDC.
Your Network
Each regional event is different, so I’d like to take a moment to describe the network you will defend at National CCDC. [Keep in mind, all of this is subject to change. I have no insider knowledge of the 2015 National CCDC event. I’m giving you what I and return competitors know.]
First, you will defend a network with a variety of operating systems. Expect a 50/50 mix of Windows and UNIX systems. If I had to venture a guess, I suspect you will see an even mix of dated operating systems and modern ones. Likely, these systems will be part of an Active Directory domain.
At your regional, you probably defended one network. At past National CCDC events, your peers had to defend two networks. One network is setup to act as your local site, in accordance with the scenario. The other network is a “cloud” site and you have to control and defend these systems with remote administration tools.
The makeup of operating systems in the two networks is usually similar.
I’m a fan of the cloud network. I see many blue teams use defense tactics that don’t scale well. The local network and cloud networks, combined, are still a small network. Even so, these extra remote systems are enough to stress the blue teams. Each year, I see most teams protect their internal network relatively well, but struggle in big ways with the cloud network. The top teams get both of these networks under control.
The National CCDC Red Team
The National CCDC red team organizes itself into cells of two to three red teamers per blue team. You may think this is unfair and you may have concerns that the red teamers you get will affect your chances of winning. Don’t worry about this. A lot of effort goes into making this construct fair.
First, the assignment of which red teamers will work together is carefully thought out. Each cell lead is someone who proved themselves at a previous year’s National CCDC event. While we each have specialties (I do a lot more Windows); the expectation is that each lead could handle a blue team by themselves, if necessary.
Second, the initial “malware drop” is scripted across all teams. You and your peers will get the same pile of malware [and configuration changes] to deal with.
Different cells will favor different toolsets, but the National CCDC red team is good at handing off accesses. If one cell loses access with their favored toolset, they have the shell sherpas (red teamers who manage the persistent malware) let them back in. No cell is kicked out of a network until you defeat or get rid of all the malware in your network.
Tempo and Timeline
On the first day, the National CCDC red team is expected to stay silent. In past years, the standing order is to do nothing to give away our presence. The goal is to get you to think we’re the worst red team ever and that there is no red team presence in your network. There’s a reason for this plan. The National CCDC red team wants you to action your injects, snapshot your work, and build up an environment that you’re invested into and trust.
On the second day, your red cell will destroy every system they have access to. You may opt to revert to a snapshot. If you do, the red team’s persistent backdoors will call home. The red team will destroy the system again. This dance will go on for about four hours.
What can you do to escape this?
There are a few options. First, you can revert to a base snapshot from before the event. You’ll lose all of your work and you’ll be vulnerable to the red team trying default things again. If it’s not going to cost you accrued points, I would take the risk and try to get your network up and running as quickly as possible. Red Team is fast, but I have yet to work with one that was really good at re-exploitation of default things in a timely manner. We’re too busy.
Your other option is to analyze network traffic, figure out how the red team is communicating with your systems and block it. We don’t typically use pre-implanted time bombs to destroy your systems. If we can’t talk to a system or it can’t talk to us, we probably can’t affect it.
Tools, Techniques, and Procedures
A common question to the red team is, what tools did you use? Each cell will conduct post-exploitation with what they have the most comfort with. Some cells use the Metasploit Framework for post-exploitation. Others take advantage of Cobalt Strike and its Beacon payload. Some red teamers operate with their private tools. This was the case with Brady Bloxham from Silent Break Security last year. I’ve even seen Core Impact make an appearance at National CCDC here and there.
The post-exploitation toolset is usually different from the persistence toolset. Like a real attacker, the National CCDC red team wants to stay in your network, and they take several precautions to limit your opportunity to observe their persistent backdoors.
The red team will use a variety of public and custom tools to persist in your network. These tools will beacon out of your network in a variety of ways. Their purpose isn’t post-exploitation. These tools are a lifeline. If the red team gets kicked out of your network, they will task one of these tools to let them back in with their preferred tool.
It’s common to see blue teams stuck in a whack-a-mole mindset. They see Meterpreter, Poison Ivy, or something else that’s noisy and obvious. They kill it. Five minutes later the attacker’s back. This isn’t because the attacker re-exploited their system. It’s because the defender took action against the attacker before they understood how the attacker was getting back in. If you don’t defeat an attacker’s persistence or their command and control–you accomplish very little by knocking out their loud post-exploitation agent.
I’ll offer one caveat to the above… you don’t know when your red cell is on their last shell in your network. If you see something obvious, feel free to kick it out. Maybe it’s the last thing you need to do send your red cell packing. Some teams reach this nirvana state.
For post-exploitation, the red team will use direct outbound TCP connections if they can get away with it. They will also use HTTP and HTTPS as data channels. In rare instances, we will use DNS, but only if we have to.
Expect that we will have infrastructure for post-exploitation and persistence within the competition environment and out on the internet. I’m known to take heavy advantage of Amazon’s Elastic Computing Cloud during CCDC. I also know one blue team likes to block all of EC2’s IP ranges. I think this is kind of silly, but we do put infrastructure in other places too.
For persistence, expect all kinds of things and expect this to get painful. Expect that the red team will use custom and public tools to slowly call back to them over DNS and HTTP. Again, the red team will have infrastructure locally and in the cloud.
Last year, one of the red team members brought a backdoor that would try multiple ways to get out of your network. It would try a TCP direct connection, HTTP, DNS, and if those failed it would scrape a shared Google Docs file for instructions.
How to Defeat the Red Team
I’m sharing all of this for a reason. I’ve seen some teams get fixated on a specific toolset. They take steps to defeat the Metasploit Framework. Some take steps to defeat Cobalt Strike’s Beacon payload. Others take steps to defeat things we don’t even use.
If you fixate on one specific toolset, you will put yourself at a major disadvantage. If you want to slow down the red team at National CCDC you’re going to have to fall back to generic best practices that defeat whole classes of bad guy activity.
Let’s go through some of the defense practices I see at CCDC…
EMET
I’ve seen many teams install EMET as part of their standard system hardening. I’ve never understood this and I’ve never felt any pain from it. The red team gains most of its accesses in the first five minutes before you have a chance to do much about it. After that, we’re not doing much in the way of memory corruption exploits. Our malware is on your system and we’re just hanging out and doing bad things with it.
Anti-virus Products
Most teams install an anti-virus product or malware scanner. You’re welcome to install one of these products, if you feel there’s nothing more important to do. You may get lucky and catch some off-the-shelf malware with one of these products. In general though, the National CCDC red team uses custom tools and artifacts. I wouldn’t expect much from an anti-virus product.
Hunting for Malware
All blue teams use the Sysinternals Suite from Microsoft to find red team activity. This requires constant vigilance with TCPView and Process Explorer. Sometimes, a team will get lucky and they’ll see something that directs them to the process we live in. If you see SYSTEM processes spawning new processes (especially cmd.exe), that’s probably bad.
I’ll add one caveat to this–a lot of red team post-exploitation activity happens in an asynchronous way. We queue up our tasks, our payload downloads them with one request, executes them, and reports the results. After that, our payload disconnects from our command and control server and there’s nothing for you to see until it wakes up and connects to us again. All of this happens very quickly and if you blink, you might miss it.
Fighting the red team off of your systems, one process at a time, isn’t going to work over the long term. It’s fine to kick the red team out when you see them, but remember–if you don’t get rid of all of the persistence, you didn’t get rid of the red team.
Restrict Egress
So, what’s the winning strategy? If you want to hurt the red team, you need to make it hard for the malware in your network to talk to the red team. Anything you can do to restrict egress will inflict great amounts of pain on the red team.
There’s a balance here. In a real enterprise, you have to make sure your users can do their job and that the business can functions. In the CCDC events, you have to make sure the scorebot can communicate and you have to satisfy the Orange team.
Make sure you whitelist which outbound ports are allowed. If there’s no need for your network to initiate outbound connections on port 4444, you shouldn’t allow it. Ideally, you should figure out what has to connect out and allow only these things. The National CCDC red team shouldn’t be allowed to control your network with an arbitrary outbound TCP connection.
Proxy Servers
Once you restrict outbound connections, you should force all web traffic to go through a proxy server.
In many cases, the Red Team’s HTTP/S persistent agents will run as SYSTEM. If you put in place an authenticated proxy and make it the only way out, you will prevent this malware from ever leaving your network. If our persistent agents can’t communicate with us, we can’t get back into your network and do bad things to it.
I won’t give away any team’s secret sauce here, but I’ve seen blue teams do creative and legitimate things with proxy servers. You can do a lot with a proxy to restrict malware from getting out of your network without impacting users.
DNS Egress
If you do all of the above well, then the next common channel is DNS. There are misconceptions about how this channel works. If your plan is to block port 53 outbound for all systems, except your internal DNS server, you may want to read this blog post.
Your best bet during CCDC is to detect malicious DNS and block it. Detecting isn’t too bad. Configure a sensor to log all DNS queries and sort this log to see which domains have the most activity. The malicious ones in a CCDC event should bubble to the top.
If you have time and want to put in the effort, you can re-architect your network to isolate internal workstations and servers from the global DNS system. If a workstation needs to browse to a website, let the proxy server handle the DNS resolution for it.
Other Egress
All of the above will make life very painful for the red team. I don’t have a good solution to defeat a red team that uses Google Docs or Twitter for command and control. The best I can offer is this: the more you box the red team in and take away options, the harder you make it for them to do things to you. Even if you can’t defeat all things within the realm of possible, you can slow the red team down a lot, and in a competition–this is good enough to get ahead of your peers.
The Dirtiest of Red Team Tricks
You’ll recall that I mentioned the red team is organized into cells. Each cell is focused on one team. In the red team room, no one wants their blue cell to win. You should expect that your red cell will do everything within their power to take points away from you. When a red cell steals a database with lots of PII that costs points–every other red cell drops what they’re doing and goes after the same data.
Once we exhaust the data that we can steal, we do everything we can to make it so the scorebot can not reach your services. We don’t necessarily need access to all of your systems to inflict great pain either. In previous years, we’ve used static ARP entries and ARP poisoning to good effect to create hard to track down layer-2 issues. We’ll also go the old fashioned route and just stop your scored services and delete files that are critical to their operation.
Closing Thoughts
You may read this post and think, wow, those red team folks are mean. That’s really not the case. We’re volunteers who care about your success. If you won a regional, we feel you can handle a great challenge and we want to make sure we give you one.
One of my favorite parts about National CCDC is the interaction I get to have with my blue team. Because the National CCDC red team splits up by team, you’ll find that your red cell has very detailed feedback for you. Once the event is over, I recommend that you track down your red cell and extract as much information from them as you can. You’ll find that your red cell are eager to share what they did, give you advice, AND to learn from you. I participate in CCDC because each year, I see things that impress me and I learn things that make me think about offense and defense in different ways.
I hope you get the most out of your National CCDC experience. I also hope this blog post helps level the playing fields for those of you coming to the event for the first time.
Good luck and do your best!