🏗️ Chapter 12 — Requirements for Secure, Multi-Tenant Hadoop (Apache YARN)

Core Goal: Understand the Multi-Tenancy concept + how Apache YARN works and supports it.
"It's Much More Than YARN"

🌍 Part 1 — Hadoop Ecosystem Overview

Hadoop is a massive ecosystem spanning 4 layers:

Big Data Infrastructure Stack (simplified)

Search / BI / ETL / ML
        ↕
NoSQL Databases  |  Analytics Databases
        ↕
      Spark
        ↕
Hadoop MapReduce & HDFS
        ↕
    Data Ingestion
        ↕ (all connected by Data Pipelines)

🐘 It is much more than Hadoop/YARN alone.

🛤️ Part 2 — The Road to Multi-Tenancy

Why Multi-Tenancy Emerged

Hadoop use cases evolved from slow, simple jobs to real-time, complex workloads:

As use cases multiplied, business functions diversified too:
Marketing → Operations → IT → Sales & Marketing
Use cases: Clickstream analysis, customer retention, supply chain, intrusion prevention, real-time decisions.

A multi-purpose platform + multi-functional use cases demand multi-tenant architecture.

❌ Part 3 — The Problem: Status Quo (Separate Clusters)

In most enterprises today, each team runs its own separate cluster:

Marketing:      Prod (v2.2)  |  Test (v2.6)
R&D:            Dev (v2.4)
Manufacturing:  Pre-Prod (v2.2)  |  Dev (v2.5)

Problems with Separate Clusters

🏠 Analogy: Imagine every department in a building having its own separate electricity generator instead of sharing the power grid — wasteful and a nightmare to manage!

Different departments may also need different tools (e.g., Marketing needs Pig/Hive for querying, Manufacturing needs something else) — but they generally share the same physical resources.

Test Environments (Before Production)

Before releasing to production, clusters typically go through:

• Unit Test → Integration Test (SIT) → Load/Stress Test (optional, for huge data volumes) → UAT (User Acceptance Testing)

The Database Consolidation Parallel

This is the same pattern the industry saw with database consolidation — many siloed databases → one shared database.
But Hadoop ≠ a database. It's a distributed platform and ecosystem.

✅ Part 4 — What is Multi-Tenancy?

Definition (Vendor #1)

Multi-tenancy = enabling multiple groups within the same organization to share a common set of resources in a cluster without negatively impacting service levels, violating security, or revealing each other's existence — all via policy, not physical separation.

Definition (Vendor #2)

Multi-tenancy = ability of a single software instance to serve multiple tenants. Creating different instances of Hadoop for various users is not acceptable — it creates silos and makes data sharing harder.

Requirements (What We Actually Need)

• ✅ Shared, common resources
• ✅ Single instance of software
• ✅ Policy-based security & isolation
• ✅ Sharing data without duplication

Vendor Recommendation: Single Cluster

💡 Migrate from many separate clusters → one single, shared cluster.

⚖️ Part 5 — Enterprise Realities (Single Cluster Isn't Always Enough)

Ask yourself — does your org allow:

• Dev on the production Hadoop cluster?
• Testing/evaluating new versions on production?
• All lines of business on one release cycle?
• One and only one centralized storage system?

For most enterprises... NO!

The Real Tension

⚖️ This is the core challenge of enterprise multi-tenancy: flexibility vs. centralization.

Why Enterprises Need Both

• Multiple Clusters → Dev/Test separation, agility to test new tools, LOB independence
• Centralized Management → Shared physical resources, security/compliance, unified data view

🏛️ Part 6 — Comprehensive Multi-Tenancy Requirements

A true multi-tenant solution must support ALL of the following on a shared, centrally managed infrastructure:

Multi-Tenancy Architecture Requirements

• Multiple environments per tenant: Prod, Dev/Test, POC (Proof of Concept — lighter than Test)
• Compute isolation between tenants
• Data isolation by tenant — achieved via access control policies (physically shared, logically isolated)

All running on: Shared, Centrally Managed Compute Infrastructure

💰 Part 7 — Multi-Tenancy Benefits

💰 It's all about agility & cost efficiency.

🔧 Part 8 — On-Premises Multi-Tenancy Options

Four main approaches to achieve multi-tenancy on-premises:

🚦 Part 9 — Apache YARN Deep Dive

What is YARN?

YARN = Yet Another Resource Negotiator

• Introduced in Hadoop 2.0 to overcome limitations of original MapReduce v1 (which had a fixed, centralized JobTracker)
• YARN is the resource management and job scheduling layer of Hadoop
• Provides a central platform for consistent operations, security, and data governance

┌──────────────────────┐
│  MapReduce / Spark   │  ← Distributed Computation
├──────────────────────┤
│  HDFS                │  ← Distributed Storage
├───────────┬──────────┤
│   YARN    │ Common   │  ← Resource Management ← HERE
└───────────┴──────────┘

🚦 Analogy: YARN is like a traffic control system at a busy airport — it decides which planes (applications) get which runways (resources) and when.

Key Functions of YARN

1. Resource Management — Allocates CPU and memory to running applications
2. Job Scheduling — Decides when and where to run tasks based on resource availability + policies
3. Multi-tenancy — Supports multiple processing frameworks beyond MapReduce (Spark, Tez, etc.)

🏗️ Part 10 — YARN Core Architecture

Fundamental Design

YARN splits resource management and job scheduling into separate daemons:

$\text{YARN} = \text{ResourceManager (Global)} + \text{ApplicationMaster (Per-App)}$

An application in YARN = a single job OR a DAG of jobs:

Job1 → Job2 → Job3   (Directed Acyclic Graph)

Core Components

Cluster Structure

Master Node
┌──────────────────┐
│  ResourceManager │ ◄──── Tracks ALL cluster resources
└──────────────────┘
        ▲  ▲  ▲
        │  │  │  (each NodeManager reports in)
Worker 1         Worker 2         Worker N
┌────────────┐  ┌────────────┐  ┌────────────┐
│NodeManager │  │NodeManager │  │NodeManager │
│ CPU | RAM  │  │ CPU | RAM  │  │ CPU | RAM  │
└────────────┘  └────────────┘  └────────────┘

The Scheduler

• Responsible for allocating resources to apps subject to constraints (capacities, queues)
• Pure scheduler only — performs NO monitoring or tracking of application status
• Has pluggable policy: currently supports CapacityScheduler and FairScheduler

🎯 Analogy: The Scheduler = hotel receptionist — assigns rooms (resources) but doesn't check on guests. That's the ApplicationMaster's job.

📊 Part 11 — YARN Resource Monitoring

Resources Tracked

YARN tracks two resources per node:

• v-cores (virtual CPU cores)
• memory (RAM)

How It Works

• Each NodeManager tracks its own local resources and reports to RM
• The ResourceManager maintains a running total of cluster-wide available resources

Example: 100-Worker Cluster

$\boxed{\text{Total} = 100 \times 64 = 6400 \text{ vcores},\quad 100 \times 128 = 12800 \text{ GB RAM}}$

📊 Analogy: The RM is like a bank that knows the total balance across all branches (NodeManagers). Each branch reports its current balance; the bank maintains the total.

📦 Part 12 — YARN Container

• A Container = a request to hold resources (vCores + memory) on the YARN cluster
• Example: { vcore: 1, memory: 8 GB }

Two states:

📦 Analogy: A container = reserved restaurant table (hold) vs. table where people are actually eating(running task).

🔄 Part 13 — YARN Application Lifecycle (Step-by-Step)

Step 1: Client talks to ResourceManager
        Client ──────────────► ResourceManager

Step 2: RM creates ONE container on a NodeManager (for AM)
        ResourceManager ──► NodeManager (allocates container for AM)
        Node: vcores: 60, RAM: 90 GB → 1 vcore + 8 GB allocated

Step 3: ApplicationMaster starts inside that container
        NodeManager ──► [Container: ApplicationMaster running]

Step 4: AM requests MORE containers from RM for actual tasks
        AM ──► ResourceManager ──► NodeManagers (launch task containers)
        Tasks run IN PARALLEL across multiple containers

Step 5: Tasks complete → AM exits → containers de-allocated

Step 6: Client exits

Managed AM = launched inside YARN container
Unmanaged AM = running outside YARN's control

ApplicationMaster Responsibilities

• Accept job submissions
• Negotiate first container with RM (for AM itself)
• Request subsequent resource containers from Scheduler
• Track task status and monitor progress
• Restart the AM container on failure

⚔️ Part 14 — Hadoop v1 (MapReduce) vs Hadoop v2 (YARN)

⚡ Part 15 — Apache Tez

What is Tez?

• "Tez" = "speed" in Hindi
• A framework built on top of YARN that executes complex DAG (Directed Acyclic Graph) workflows
• Default execution engine for Apache Hive and Pig
• Replaces MapReduce for high-performance, interactive workloads

Tez vs. MapReduce

⚡ Analogy: MapReduce = writing your intermediate work on a whiteboard between every step. Tez = passing work directly from person to person without stopping to write it down.

How Tez Works with Hive

SQL Query:

SELECT department, AVG(salary)
FROM employees
WHERE country = 'USA'
GROUP BY department;

MapReduce approach (2–3 stages, data written to disk between each):

1. Map Job → filter rows, emit (dept, salary)
2. Shuffle & Sort
3. Reduce Job → group by dept, calculate AVG

Tez approach (single connected DAG, in-memory):

┌─────────────────────┐
│  Filter + Project   │  ← vertex 1 (filtering country = USA)
└──────────┬──────────┘
           ▼
┌──────────┴──────────┐
│  Group By (dept)    │  ← vertex 2 (group + compute AVG)
│  & Compute AVG      │
└─────────────────────┘

• Each vertex = a task (filter, aggregate, join)
• Edges = data movement (preferably in-memory)

📅 Part 16 — YARN Resource Reservation

• YARN supports resource reservation via the ReservationSystem
• Allows users to:
  • Specify a profile of resources over time
  • Set temporal constraints (e.g. deadlines)
  • Reserve resources to guarantee predictable execution of critical jobs
• The ReservationSystem:
  • Tracks resources over time
  • Performs admission control for reservations
  • Dynamically instructs the scheduler to fulfill the reservation

📅 Analogy: Like booking a conference room in advance — you reserve CPU/RAM for a specific time window so your important job isn't left waiting when resources are full.

Without reservation: if resources are full when a high-priority task arrives → it still has to wait.

🧮 Part 17 — Real Scenario: Retail Hive Query on YARN

Scenario: A retail company wants to analyze customer purchase patterns from terabytes of transaction logs in HDFS.

SELECT region, SUM(amount)
FROM transactions
WHERE date >= '2025-03-01'
GROUP BY region;

Step-by-Step Execution

📋 Part 18 — YARN Scorecard & Limitations

YARN Multi-Tenancy Scorecard

⚠️ Single Hadoop Cluster ≠ Full Multi-tenant Infrastructure

YARN Highlights

• Key Hadoop 2.0 enabler to run multiple computing services on a single cluster simultaneously
• Sophisticated scheduler schemes (e.g. Capacity Scheduler)

YARN Limitations

• ❌ Designed for one cluster — not across multiple clusters
• ❌ Scheduler only — not a full resource manager (Hadoop-centric)
• ❌ Complex queue configuration = admin overhead
• ❌ SQL, NoSQL, and BI tool resources are NOT managed
• ❌ Spark memory consumption poses issues with YARN

YARN Advantages Summary

🏠 Part 19 — On-Premises Options: VM vs Container vs Mesos

Virtual Machines

Docker Containers

VM vs. Container — Key Difference

Virtual Machines                        Containers
┌───────┬───────┬───────┐            ┌───────┬───────┬───────┐
│ App 1 │ App 2 │ App 3 │            │ App 1 │ App 2 │ App 3 │
│Libs   │Libs   │Libs   │            │Libs   │Libs   │Libs   │
│GuestOS│GuestOS│GuestOS│            ├───────────────────────┤
├───────────────────────┤            │    Container Engine   │
│     Hypervisor        │            ├───────────────────────┤
├───────────────────────┤            │    Operating System   │
│    Infrastructure     │            ├───────────────────────┤
└───────────────────────┘            │    Infrastructure     │
                                     └───────────────────────┘

• VMs = virtualize the entire machine down to hardware
• Containers = virtualize only the software layer above the OS (shared kernel)

🏠 Analogy:

• VM = renting a full house (your own kitchen, bathroom, everything — isolated but heavy)
• Container = renting a room in a shared house (shared plumbing/kitchen = OS, your own private space = container)

A Docker container image packages the app + all libraries/dependencies. If Docker engine is installed → containers run reliably on any infrastructure.

Apache Mesos

Mesos vs YARN

🍽️ Analogy:

• Mesos = buffet — the chef puts out food, you choose what to take
• YARN = restaurant — you order what you want, the kitchen decides how to prepare and serve it

Multi-Tenancy Scorecard: VM / Container / Mesos

❓ Part 20 — FAQs

🧩 Part 21 — Quick Reference Summary

🎯 Takeaways

• 🌐 Hadoop ecosystem complexity is increasing
• 🚀 Adoption journey: single workload → multi-workload
• ⚖️ Versatility and complexity require a careful balancing act
• 🎯 The goal: Multi-Tenant Hadoop Infrastructure — shared, policy-driven, centrally managed

Single Hadoop Cluster ≠ Multi-tenant infrastructure.
True multi-tenancy needs: shared compute + policy isolation + multiple app support + SLA guarantees per tenant.