Loading...
[This is a very vague problem. It is also not a common one. Let's start with clarifying questions we should ask.]
[In fact, this is a mid-level deep dive topic, from the get go.]
Where should we capture network packets?
We assume we want to detect threats to our data center as a whole, instead of to specific parts of the data center, e.g., a data base or a microservice. Therefore, we want to capture packets where all requests get through, for example, a firewall in a network layer, or API Gateway.
There are pros and cons of these two locations:
Both types of attacks are important. Therefore, we should design a system that can capture packets in both locations.
We will not focus on incident response automation, as it is a separate functionality.
Let's assume each packet is 1KB on average.
At 1 million requests per second, it would generate 256MB / second. This is a huge number. Per day it will be: 1KB * 60 * 60 * 24 = 86 TB. It would not be efficient to store and analyze all these data.
As such, we need to apply some rules to reduce the amount of data. The most obvious way is sampling (e.g. 1 out of 100 packets), but this has the risk of missing an important packet. We can have a filtering rule (packets from a known suspicious IP address to specific endpoints). This has a risk of missing new attacks that show behaviors the system is not configured for.
Here again, we should compromise and do both:
With these two mechanisms, let's assume we can the reduce the amount of data by the order of 100.
With this reduction, the data amount would be 0.86TB / day. 314TB / year.
No APIs are invoked explicitly for this system.
Requests that go into the datacenter will be forwarded to our intrusion detection service.
Important data include:
Rules
Rules should be stored in a reliable database. The size will likely be in KBs or MBs. RDB would be a good choice.
Captured Packets
At 314TB / year as estimated, this would require a large database that is suitable for analysis. Google's BigQuery would be a good choice.
Sync vs Async
This is an important tradeoff decision in the high level design.
We will try to implement both synchronous (real-time) detection, and asynchronous detection.
Pro of synchronous detection is that it can respond to a threat immediately. Con is that it is resource intensive. Analyzing packets in real time takes a lot of computational power.
Pro of async detection is that it can do deeper analysis by looking at the content of the packets, looking at multiple, related packets, and so on. Con is that it takes time. The response time would perhaps be in minutes. It cannot stop attack before it happens.
The idea is a hybrid approach:
That's why the High Level Diagram has Sync Analyzer and Async Analyzer.
Requests from client go through Network Layer and API Gateway. Both send (some, not all) packets to Capture Service.
Capture Service writes the messages to Message Queue. Message Queue provides buffer in case a surge in request makes it impossible for Sync Analyzer and Queue Worker to catch up to the requests.
Sync Analyzer takes the message, does fast analysis (e.g. simple matching of IP addresses or header information) and responds (e.g. recommend to block requests).
Async Analyzer runs periodically, runs analysis on the data in Analytics Database, and responds.
[Senior-level Deep Dive Topic]
Sync Analyzer
This service is responsible for applying simple and fast rules to incoming messages to detect a potential threat, very quickly. It is a stream processing system and has to operate in a real time.
It should take simple rules, e.g., simple matching of IP addresses or headers, apply them to incoming messages, and respond quickly.
To do that, it should use caching to store rules. It should partition data. Since Sync Analyzer only looks at message individually, partitioning with a hash of message ID would be a good choice. Consistent Hashing can address the issue of uneven distribution.
Async Analyzer
Async Analyzer is a batch processing system that runs at an interval, e.g., every 5 min, 10 min.
It runs complex queries, e.g., matching packet payload contents to known attacks (XSS, CSRF), or analyzing multiple packets from one IP address to understand that source IP's behavior comprehensively.
As such, this is not a real-time mechanism. There would be some time gap from packet arrival to response.
Hybrid Approach
Ultimately, we should use both Sync and Async Analyzers because we need both characteristics:
[Mid-level Deep Dive Topic]
Rules Based Analysis vs Anomaly Detection
We can analyze packets and detect potential threats in either Rules Based way, or with Anomaly Detection.
Rules Based applies rules pre-defined by security researchers. Pro: Since the rules are created based on research into existing attacks, the false negative rate would be low. Con is that it will not detect unknown, new attacks. Security researcher and system administrator must work hard to keep the rules (attack signatures) up to date.
Anomaly Detection observes the access pattern and learns the normal access pattern. When there's something that deviates from normal, it would flag it as a potential threat. Machine Learning can be used for this purpose. Pro of this approach is that it can be automated more than Rules Based approach. Con is that, unless ML is trained carefully and effectively, it might result in higher false negatives.
Partitioning
Capture Service and Sync Analyzer can be sharded by Packet ID because they treat packets as individual messages.
On the other hand, there should be some groupings in Analytics Database and Async Analyzer because they want to analyze packets that are group by some characteristics.
If this system is serving multiple customers (tenants), customer ID would be a good choice of a partitioning key, as Async queries would typically be on a particular customer.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?