🖥️ Chapter 10 — Operating System Security Cheat Sheet

🗺️ Big Picture

OS Security
│
├── Why OS? → First line of defense; primary attack target
├── Core Concepts → Separation, Protection Levels, Protected Objects
├── Memory Protection → Fence / Base-Bounds / Tagged / Segmentation / Paging
├── Memory Corruption Defense → ASLR, Stack Canary, Buffer Overflow
├── OS Hardening → Install → Remove → Configure → Add Controls → Test
├── Linux/Windows Security → Patches, Permissions, Logging, UAC, BitLocker
├── Defense in Depth → Multiple layers from hardware to data
├── Virtualization Security → Hypervisor, VM Hardening, vSwitch, VMI
└── Hardware Security → TPM + Trusted Boot + TEE (Intel SGX, ARM TrustZone)

1. Why OS Security Matters

• OS is the fundamental controller of all system resources → makes it the primary attack target
• OS is the first line of defense against unwanted behavior:
  • Protects one user from another
  • Protects critical memory/storage from unauthorized processes
  • Performs identification & authentication
  • Ensures fair sharing of hardware resources

🏢 Analogy: The OS is the building's security guard — controls who gets in, who goes where, and stops intruders from damaging things.

OS Must Protect

• Users from each other
• Resources (memory, CPU, I/O, files, networks)
• Itself — including from harm modules, drivers, and add-ons

OS Security Functions

2. OS History → Why Protection Is Needed

Modern OS must: protect one user from another + share resources (CPU, memory, files) safely.

3. OS Layered Structure

┌────────────────────────────────────┐  ← Subprocesses of User Processes
├────────────────────────────────────┤  ← User Processes
├────────────────────────────────────┤  ← Compilers, Database Managers
├────────────────────────────────────┤  ← Utility Functions
├────────────────────────────────────┤  ← File Systems, Device Allocation
├────────────────────────────────────┤  ← Scheduling, Sharing, Memory Mgmt
├────────────────────────────────────┤  ← Synchronization, Allocation
├────────────────────────────────────┤  ← Security Functions
└────────────────────────────────────┘  ← Hardware (rootkits, bootkits live here!)

⚠️ Each layer is vulnerable to attacks from below. If the hardware layer (foundation) is compromised, nothing above it is safe.

4. Types of Separation

Each type has different security strength, implementation complexity, and resource utilization.

5. Levels of Protection (0 → 5)

Level 5 example: Can see department average salary — cannot see John Smith's individual salary.

6. Memory and Address Protection Mechanisms

6.1 Fence Protection

• Hardware divides memory at address n:
  • 0 → n = OS space
  • n+1 → High = User program space
• The fence = a hard boundary

6.2 Fence Registers

• A register holds the fence address (flexible — different OS versions can have different fence values)
• Allows flexible relocation of the OS/user boundary

6.3 Base and Bounds Registers ⭐

$\boxed{\text{Valid address} = \text{Base} \leq \text{address} \leq \text{Bounds}}$

• Base Register = starting address of user's program space
• Bounds Register = ending address of user's program space
• Multiple users (A, B, C) each have their own base/bounds

🌾 Analogy: Like fencing off plots of land — each user owns their plot (base to bounds), nobody can step onto another's.

• Can also have separate base/bounds for program and data — code and data can be non-contiguous.

6.4 Tagged Architecture

• Every memory word has extra bits specifying access rights: R (Read-only), RW (Read/Write), X (Execute-only)
• Bits are set only by privileged (OS) instructions
• Tested every time an instruction accesses that memory location

6.5 Virtual Memory — Segmentation

• Program divided into logical segments (MAIN, DATA_SEG, SUB…)
• A Segment Translation Table maps segment names → physical addresses
• Benefits:
  • Each address reference is checked (not too high or too low)
  • Different data classes can have different protection levels
  • Two users can share a segment with different access rights
  • A user cannot generate an address for an unpermitted segment

6.6 Virtual Memory — Paging

• Program divided into fixed-size pages; memory into equal page frames
• A Page Translation Table maps logical page number → physical frame address
• Page size = power of 2: 2KB, 8KB, 16KB
• Gives efficient memory management (but improperly configured → page fault)

6.7 Combined Paging + Segmentation

• Each segment has its own Page Translation Table
• Address: Segment → Page Table → Physical Address
• Combines: logical structure (segmentation) + efficient memory use (paging)
• Security-wise: better — but performance tradeoff for small programs

🔐 Memory protection implements both separation AND sharing.

7. Memory Corruption Protection

Memory Corruption = interfered with / tampered / no longer genuine

7.1 Address Space Layout Randomization (ASLR)

$\boxed{\text{ASLR: Base address of segment = random value on every execution}}$

• Mitigates exploits that rely on hardcoded stack, heap, code, libc addresses

How ASLR defeats Buffer Overflow:

• Without ASLR: attacker knows exactly where shellcode lands → overwrites return address → jumps there
• With ASLR: stack address changes every run → wrong guess = program crash

How ASLR defeats Return-to-libc:

• Attacker reuses existing functions like system("/bin/sh") instead of injecting code
• With ASLR: libc location randomized → attacker must first leak the base address (much harder)

7.2 Stack and Buffer Overflow

Normal stack layout:

| Return Address          |  ← Points to next instruction
| Saved Frame Pointer     |
| Local Variables         |
| [ char buffer[16] ]     |

Stack after overflow (attacker payload):

| Overwritten Return Addr |  ← Attacker redirects execution here!
| Corrupted Frame Ptr     |
| buffer[16] + Exploit    |  ← Overflowed data writes into upper memory

Unsafe code:

void vulnerable() {
    char buffer[16];
    gets(buffer);  // ⚠️ No bounds checking!
}

Safe code:

void safe() {
    char buffer[16];
    fgets(buffer, sizeof(buffer), stdin);  // ✅ Limits to 15 chars + null terminator
}

💡 C = "You manage memory manually" 💀 — Swift = "System manages memory for you" 🍎

7.3 Stack Canary

A random value placed on the stack just before the return address.

| Local Variables         |
| Canary Value (random)   |  ← Inserted by compiler
| Saved Return Address    |

How it works:

1. Function called → canary inserted on the stack
2. Before function returns → system checks if canary was modified
3. If canary is altered (by buffer overflow) → program aborts immediately before using the corrupted return address

🐤 Like a tamper seal — if the attacker overwrites the buffer to reach the return address, they inevitably overwrite the canary first → program detects tampering and stops.

8. OS Hardening

OS Hardening = configure securely + update + enforce policy + remove unnecessary components → minimize attack surface.

5 Basic Steps

① Install & patch the OS
② Harden & configure (remove unnecessary services/apps/protocols; configure users/groups/permissions)
③ Install additional security controls (AV, host firewall, IDS/IPS, application whitelisting)
④ Test system security (checklists, vulnerability scanners)
⑤ Repeat periodically

Step-by-Step Details

Step 1 — Remove Unnecessary Services, Apps, Protocols

$\boxed{\text{FEWER IS BETTER — reduce attack surface}}$

• System planning identifies what's actually required
• Do NOT use supplied defaults — default config maximizes ease-of-use, not security
• Install additional packages later when needed

Step 2 — Configure Users, Groups, Authentication

• Elevated privileges → only those who need it, only when needed
• Default accounts not required → remove or disable
• Define categories of users, their privileges, accessible data, and authentication methods

Step 3 — Configure Resource Controls

• Set appropriate permissions on data and resources
• Follow security hardening guides for recommended changes

Step 4 — Install Additional Security Controls

• Anti-virus, Host-based firewall, IDS/IPS, Application whitelisting

Step 5 — Test System Security

• Ensure previous steps are correctly implemented
• Identify vulnerabilities
• Use checklists + vulnerability scanning tools
• Repeat after initial hardening and periodically

7 Steps for a Well-Maintained Computer

ASD Top 4 Prevention Strategies

1. Whitelist approved applications
2. Patch third-party applications and OS vulnerabilities
3. Restrict administrative privileges
4. Create a defense-in-depth system

9. Defense in Depth

Multiple layers of defense addressing technical, personnel, and operational issues.

Attack
  ↓
Hardware Broadband Router / Firewall
  ↓
OS / Software Firewall
  ↓
Antivirus / Antimalware
  ↓
Security Patches
  ↓
User Account Controls
  ↓
[Core Data]

Outer to inner: Policies/Procedures/Awareness → Physical → Perimeter → Internal Network → Host → Application → Data

🧅 Analogy: Like the layers of an onion — an attacker must break through each layer to reach the core data.

10. Linux/Unix Security

Local Exploit = vulnerability exploited by attacker with local access → gains elevated privileges Remote Exploit = vulnerability in a network server exploitable by a remote attacker

11. Windows Security

Key Windows Security Features

Biba Integrity Model (Windows Mandatory Integrity Controls)

$\boxed{\text{Subject can write to object only if } \text{integrity}_{subject} \geq \text{integrity}_{object}}$

• Subjects are prevented from writing to objects at higher integrity levels
• Prevents low-integrity processes from corrupting high-integrity data

Windows Logon — Admin vs Standard User

12. OWASP Top 10 (2025) — Web Application Vulnerabilities

13. Virtualization Security

Virtualization Types

Native vs. Hosted Hypervisor

Native (Bare-Metal):             Hosted:
Guest OS 1 | Guest OS 2 ...      Apps | Guest OS 1 ...
         Hypervisor (Type 1)         Hypervisor (Type 2)
         Physical Hardware           Host OS Kernel
                                     Physical Hardware

Type 1 = more secure (no host OS as attack surface beneath hypervisor)

Three Core Virtualization Security Issues

Securing Virtualization — 4 Layers

1. Hypervisor Security

• Use Type 1 (Bare-metal) hypervisors (VMware ESXi, Hyper-V, Xen)
• Minimize attack surface: disable unused services/interfaces/device drivers
• Regular patch management
• Enable TPM + Secure Boot (hardware root-of-trust)

2. VM Hardening

• Start with hardened, minimal OS templates
• Isolate VMs (logical + network isolation)
• Install anti-malware, firewalls, HIDS within each VM

3. Network Security in Virtualization

• Virtual Firewalls between VMs or at vSwitch level
• Microsegmentation via SDN / VMware NSX (per-VM security policies)
• Secure vSwitches — prevent:
  • Promiscuous mode
  • MAC address spoofing
  • Forged transmissions

4. Security Tools and Technologies

• Virtualization-aware AV: Trend Micro Deep Security, Symantec Endpoint Protection for VEs
• VM Introspection (VMI): Inspect VMs from hypervisor level without an agent (KVM + LibVMI, Bitdefender)
• SIEM: Integrate virtualization logs for correlation and alerting

14. TPM — Trusted Platform Module

A hardware security chip embedded in the motherboard (or virtualized as vTPM in cloud).

TPM Functions

TPM Chip
├── Secure Key Storage
├── PCR Registers (boot integrity measurements)
├── Cryptographic Unit
├── Random Number Generator
└── Trusted Boot

🍎 Apple equivalent = T2 Security Chip

15. Trusted Boot (Measured Boot)

Ensures each component loaded during startup is verified and measured against a known-good value.

How It Works

UEFI BIOS/Firmware starts
      ↓
Measures each component (takes SHA hash)
      ↓
Stores hashes in TPM's PCR registers
      ↓
If hash differs from known-good → tampered! (rootkit/malware detected)
      ↓
System refuses to boot / alerts / triggers remediation

🪪 Analogy: Like a tamper-evident seal on each stage of the boot chain — if anything was opened, you'll know.

16. TEE — Trusted Execution Environment

A secure area inside the processor where sensitive code and data are executed in isolation — protected even if OS, hypervisor, or apps are compromised.

CPU
├── Normal World (OS, Applications)
└── Secure World (TEE)
      └── Isolated memory + execution (inaccessible from outside)

Protected Operations in TEE

• Cryptographic key operations
• Authentication, biometric verification
• Payment processing
• Secure AI inference
• Confidential computing

TEE Workflow — 6 Steps

Step 1: Enclave Deployment

$\boxed{eid \leftarrow \text{TEE.Install}(p_{Enc})}$ Server loads program into enclave; gets eid (Enclave ID)

Step 2: Remote Attestation

$\boxed{\rho \leftarrow \text{TEE.Quote}(H(p_{Enc}),\ nonce)}$ $\boxed{\text{TEE.VrfyQuote}(pk_{TEE},\ p_{Enc},\ \rho) = 1}$

• Enclave proves its identity via cryptographic quote
• $H(p_{Enc})$ = hash of enclave program (proves exact code loaded)
• nonce = prevents replay attacks
• Client verifies: returns 1 if genuine → client trusts the enclave

🍽️ Analogy: Like a restaurant showing a health inspection certificate — client checks proof before sending sensitive data.

Step 3: Secure Channel (DH Key Exchange)

$\boxed{K_{sess} \leftarrow \text{KEX}(\text{Client},\ \text{Enclave})}$

Enclave generates (sk_e, pk_e) → sends pk_e to client
Client generates (sk_c, pk_c) → sends pk_c to enclave
Enclave: K_sess = DH(sk_e, pk_c)
Client:  K_sess = DH(sk_c, pk_e)
→ Both arrive at same shared session key ✅

Step 4: Key Provisioning — Two Cases

🏧 Case 2 = generating your ATM PIN inside the machine — even the bank doesn't know it.

Step 5: Client Sends Data

$\boxed{\text{Enc}_{K_{sess}}(m)}$ Enclave receives and decrypts internally.

Step 6: Encrypt Inside Enclave

\boxed{CT = \text{Enc_{AES}}(m,\ k)}

TEE Formal Algorithms

TEE.Seal = when you need to take a key out of the enclave → seal it first. To use it again → unseal it back inside.

Intel SGX — EPC (Enclave Page Cache)

🅿️ Analogy: All apartments share a limited parking lot (EPC) — each tenant has their own locked unit, but parking is shared and limited.

17. TEE Technologies

If CPU does NOT support TEE features → No TEE available.

18. TPM vs. TEE Comparison

How TPM + TEE Work Together

① TPM verifies system integrity during boot (Trusted Boot)
② Secure Boot loads trusted OS
③ TEE runs sensitive operations securely

🏢 Analogy: TPM = security guard at building entrance (checks who enters at boot). TEE = locked executive office inside (where sensitive work actually happens).

When to Use Which?

⚡ Key Facts to Remember