Operating System and Binary Security Concepts
Systems Security Fundamentals
Processes, Privilege, and Isolation
Operating systems enforce security through processes, memory separation, access controls, scheduling, and kernel-mediated resource management. Ethical hacking at this layer studies how privilege boundaries fail under logic errors, unsafe memory handling, misconfigurations, and mitigation gaps.
Memory Safety as a Security Property
Memory safety errors arise when code reads or writes outside intended bounds, uses freed objects, or interprets data with the wrong type or lifetime. These flaws are especially important because they can alter control flow or disclose sensitive state. In abstract terms, a buffer overflow occurs when written bytes exceed allocated bytes such that .
Common Mitigations
| Mitigation | Primary Effect |
|---|---|
| Stack canaries | Detect some stack corruption attempts |
| ASLR | Randomize address layout to raise attack complexity |
| DEP or NX | Prevent execution from non-executable memory |
| Sandboxing | Limit the blast radius of compromise |
Advanced View
A mitigation does not eliminate a class of bugs; it changes the economics, reliability, and observability of exploitation.
Which property is most directly provided by address space layout randomization?
ASLR makes target addresses less predictable, raising attack complexity.
Correct answer: Increased uncertainty about memory addresses
What does a memory-safe language primarily reduce?
It reduces classes such as buffer overflows, use-after-free, and some pointer misuse errors.
Correct answer: memory corruption vulnerabilities
Why is sandboxing valuable in a compromise scenario?
Sandboxing confines the consequences of a successful exploit.
Correct answer: It limits the impact of code execution
Kernel and Privilege Boundaries
Kernel interfaces are high-value because they mediate capabilities across the entire system. Advanced assessments consider not only vulnerabilities in privileged code, but also how userland components can influence kernel attack surface through drivers, syscalls, and policy interactions.