Secure Operating Systems, Isolation, and Trusted Computing
Operating System Security as an Enforcement Problem
The OS as reference monitor and attack surface
The operating system mediates access to memory, storage, devices, and processes, so it is both a security boundary and a large attack surface. Research questions include how to build complete mediation, minimize TCB size, and preserve isolation despite complex drivers, system calls, and shared resources.
Isolation mechanisms
Process isolation
- Relies on virtual memory and kernel enforcement.
- Fine-grained but vulnerable to kernel compromise.
Virtualization
- Adds a hypervisor layer for separation.
- Useful for consolidation and sandboxing.
What is the primary security benefit of virtualization?
Virtualization can separate tenants and reduce the impact of compromise, though it does not eliminate all risk.
Correct answer: It can create strong isolation between workloads
Why is the kernel typically considered part of the trusted computing base?
If the kernel is compromised, many protection guarantees collapse.
Correct answer: Because it enforces core security decisions such as process isolation, memory protection, and access control.
Memory Safety and Exploit Mitigation
From buffer overflows to control-flow hijacking
Memory corruption remains one of the most consequential classes of vulnerabilities because it can lead to code execution, privilege escalation, and sandbox escape. Research and engineering responses include address space layout randomization, control-flow integrity, sandboxing, hardware-assisted memory tagging, and safer languages.
Typical mitigation stack
-
1
Step 1: Prevent bugs using memory-safe languages where feasible.
-
2
Step 2: Contain failures with sandboxing and least privilege.
-
3
Step 3: Detect exploitation through monitoring and anomaly detection.
-
4
Step 4: Harden control flow and sensitive interfaces.
-
5
Step 5: Reduce exploit value using secrets isolation and rapid patching.
What does control-flow integrity primarily aim to prevent?
CFI constrains indirect branches and returns to legitimate targets.
Correct answer: Unauthorized changes to the program's control-flow graph
Name one limitation of relying only on sandboxing.
Sandboxing reduces risk but does not eliminate the attack surface.
Correct answer: It may still allow escape via kernel bugs, side channels, or overly broad permissions.
Trusted Execution and Attestation
Hardware roots of trust
Trusted execution environments and attestation aim to convince a remote verifier that code runs in a particular protected environment. However, they introduce questions about supply-chain trust, microarchitectural leakage, rollback resistance, and whether attestation meaningfully captures the security properties a user actually needs.
What is attestation used for?
Attestation provides evidence about platform state to an external verifier.
Correct answer: Proving to a remote party what code or environment is running
Why are enclaves not a complete solution to system security?
Protected execution does not automatically solve usability, attestation, or leakage problems.
Correct answer: They can still leak information through side channels and depend on surrounding platform assumptions.