CS6262 Lecture 17 - Data Poisoning and Model Evasion

What is data poisoning in the context of attacks on machine learning?

An attack where adversaries manipulate training data to cause the model to learn incorrect behavior.

What is model evasion in machine learning security?

An attack where adversaries learn the model's decision boundary and craft inputs to bypass detection.

What was a common criticism from security experts regarding early machine-learning-based security models?

They produced too many false positives, making them less useful in practice.

What major change in recent years improved machine learning for security?

A significant increase in available data, including malware samples and logged network data.

Why is it difficult to find standard datasets for machine-learning security research?

Privacy concerns and restrictions surrounding malware and sensitive user data.

What challenge drives the need for machine learning in security?

The need to analyze extremely large amounts of data that cannot be processed manually.

Why is applying machine learning in security more challenging than other fields?

Attackers actively try to manipulate or evade models.

What is adversarial machine learning?

Machine learning applied in environments where attackers attempt to manipulate or evade models.

What is an exploratory attack in adversarial machine learning?

An attack where adversaries probe a model to learn its decision boundary and evade detection.

What is another name for an exploratory attack?

Evasion attack.

What is a causative attack in adversarial machine learning?

An attack that injects malicious training examples to corrupt the learned model.

What is another name for a causative attack?

Data poisoning attack.

Which evasion tactic enables malware to detect its environment before acting?

Environmental awareness.

Which evasion technique involves delaying execution or acting only at specific times?

Timing-based evasion.

Which evasion tactic confuses automated detection tools?

Obfuscating internal data.

What early example demonstrated evasion of anomaly detection models?

Attackers inserted normal system calls into malicious sequences to evade detection.

What technique can corrupt machine-generated worm signatures?

Inserting incorrect network data to produce invalid signatures.

What has recent research in adversarial machine learning focused on?

Developing frameworks and studying limitations of modern models like deep learning.

What is PAYL in the context of intrusion detection?

An anomaly detection system modeling byte frequency distributions in payloads.

What is the assumption underlying PAYL’s detection approach?

Normal traffic exhibits unique byte-frequency traits that differ from malicious traffic.

How does PAYL compute anomaly scores?

By comparing byte frequency distributions to a learned normal profile.

What advantage does PAYL offer?

High efficiency during run-time processing.

What kinds of attacks can PAYL detect?

Zero-day and polymorphic attacks that alter appearance.

What characteristic defines a polymorphic attack?

It changes appearance each time to avoid signature detection.

Why do polymorphic attacks lack predictable signatures?

Each instance is modified, preventing fixed patterns from being used.

What is a polymorphic blending attack?

A polymorphic attack adjusted to resemble normal traffic distributions.

How does a blending attack evade detection?

By matching byte frequency statistics of legitimate traffic.

Why are simple IDS models easier to evade?

They rely on limited features that attackers can mimic.

What makes evasion harder against advanced IDS?

More comprehensive features and sophisticated modeling reduce blending ability.

What is the primary objective of a polymorphic blending attack?

To match legitimate byte distributions and bypass anomaly detection.

What is an effective countermeasure to blending attacks?

Use more complex IDS features incorporating syntactic and semantic information.

What is a benefit of combining multiple randomized IDS models?

It makes it harder for attackers to predict or match detection profiles.

What is the key goal of a successful poisoning attack?

To degrade the model so attacks are not detected.

Why should poisoning attacks be subtle?

To remain undetected over long periods.

Why is permanence important in poisoning attacks?

To make corruption irreversible.

What moral lesson was demonstrated in the navigation app poisoning attempt?

When true signal outweighs noise, poisoning efforts fail.

What is the purpose of clustering in Polygraph’s workflow?

To separate worm flows from innocuous flows before generating signatures.

Why are signatures generated from fake invariants ineffective?

They fail to detect real worms and become useless.

What is a Bayes signature in Polygraph?

A signature based on tokens common within suspicious flows but rare in normal flows.

How is a Bayes signature used for classification?

By summing token scores and comparing to a threshold.

How can an attacker defeat Bayes signatures?

By injecting normal substrings into fake anomalous flows.

What impact does noise injection have on signature generation?

It produces signatures with high false positives or false negatives.

What is needed to mitigate noise-injection attacks?

A precise and reliable flow classifier.

Under what condition can data poisoning attacks be avoided?

When training data is tightly controlled and its integrity is ensured.

Why are poisoning attacks a risk when training in open environments?

Attackers can inject malicious or misleading data.