# tuna.cpuset - CPU Set Management for Tuna

## Overview

The `tuna.cpuset` module provides Python interfaces for managing CPU sets (cpusets) using Linux cgroup v2. Cpusets allow you to isolate CPUs by assigning processes to specific CPU groups, complementing the kernel's `isolcpus` boot parameter with userspace-level CPU isolation.

This module was originally prototyped in tuna, implemented in rteval, and has been adapted back to tuna with tuna-specific enhancements.

## Key Features

### Core Functionality (rteval-compatible)
- **Cpuset class**: Create and manipulate individual cpusets
- **CpusetsInit class**: Detect cgroup v2 support and represent the root cgroup
- **TaskMigrate class**: Migrate processes between cpusets
- **Context manager support**: Automatic cleanup with `auto_destroy=True`
- **NUMA awareness**: Multi-NUMA node support
- **CPU partitioning**: Isolated vs member partition types

### Tuna-Specific Enhancements
- **Persistent cpusets** (default): Cpusets remain after program exits, unlike rteval's ephemeral approach
- **Discovery functions**: List and find existing cpusets (including nested hierarchies)
- **Cleanup utilities**: Destroy cpusets by pattern matching
- **Flexible destruction**: Can destroy any cpuset, not just self-created ones
- **Nested cpuset support**: Handles systemd, container, and user-created hierarchies

## Requirements

- **Linux kernel**: cgroup v2 support (kernel 4.5+, recommended 5.0+)
- **Root permissions**: Required for creating/destroying cgroups
- **Python**: 3.6+ (uses f-strings, type hints)

### Checking cgroup v2 Support

```python
from tuna import cpuset

ci = cpuset.CpusetsInit()
if ci.supported:
    print(f"cgroup v2 is supported, {ci.numa_nodes} NUMA nodes")
else:
    print("cgroup v2 is not supported on this system")
```

## Installation

The module is part of tuna and doesn't require separate installation. Just ensure tuna is in your Python path:

```python
from tuna import cpuset
```

## Basic Usage

### Example 1: Creating a Persistent Cpuset (Tuna Style)

```python
from tuna import cpuset

# Create a persistent cpuset (default behavior)
cs = cpuset.Cpuset('tuna_isolated')

# Configure it (must set memory before CPUs in cgroup v2)
cs.write_memnode('0')      # NUMA node 0
cs.assign_cpus('0-3')      # CPUs 0-3

# Optionally set as isolated partition
cs.set_cpu_exclusive()

# The cpuset persists after the program exits
# Other programs can use it later
```

### Example 2: Using Context Manager for Automatic Cleanup (rteval Style)

```python
from tuna import cpuset

# Create with auto_destroy=True for automatic cleanup
with cpuset.Cpuset('temp_cpuset', auto_destroy=True) as cs:
    cs.write_memnode('0')
    cs.assign_cpus('4-7')

    # Do work with the cpuset...

# Cpuset is automatically destroyed when exiting the with block
```

### Example 3: Moving Processes to a Cpuset

```python
from tuna import cpuset
import os

# Create and configure cpuset
cs = cpuset.Cpuset('workload_cpus')
cs.write_memnode('0')
cs.assign_cpus('8-15')

# Move current process
cs.write_pid(os.getpid())

# Move another process
cs.write_pid(12345)
```

### Example 4: Migrating All Tasks from Root to Cpuset

```python
from tuna import cpuset

# Get root cgroup
root = cpuset.CpusetsInit()

# Create isolated cpuset
isolated = cpuset.Cpuset('isolated_cpus')
isolated.write_memnode('0')
isolated.assign_cpus('0-7')

# Migrate most tasks to isolated cpuset
tm = cpuset.TaskMigrate(root, isolated)
migrated, failed = tm.migrate()
print(f"Migrated {migrated} tasks, {failed} failed")
```

### Example 5: Discovering Existing Cpusets

```python
from tuna import cpuset

# List all cpusets
all_cpusets = cpuset.list_cpusets()
print(f"Found cpusets: {all_cpusets}")

# List only tuna-created cpusets
tuna_cpusets = cpuset.list_cpusets(pattern='tuna_*')
print(f"Tuna cpusets: {tuna_cpusets}")

# List with recursive search (finds nested cpusets)
nested = cpuset.list_cpusets(pattern='systemd*', recursive=True)
print(f"Systemd cpusets: {nested}")
```

### Example 6: Cleaning Up Cpusets

```python
from tuna import cpuset

# Destroy a single cpuset
cpuset.destroy_cpuset('tuna_isolated', force=False)

# Destroy with force (migrates tasks to root first)
cpuset.destroy_cpuset('busy_cpuset', force=True)

# Bulk cleanup with pattern matching
cpuset.cleanup_cpusets('tuna_*', force=True)

# Cleanup nested hierarchies (destroys children before parents)
cpuset.cleanup_cpusets('systemd/user*', force=True, recursive=True)
```

## API Reference

### Classes

#### `Cpuset(name, auto_destroy=False)`

Core class for managing a single cpuset.

**Parameters:**
- `name` (str): Name of the cpuset (required)
- `auto_destroy` (bool): If True, destroy on context manager exit (default: False)

**Methods:**
- `assign_cpus(cpu_str)`: Assign CPUs (e.g., "0-3,5,7")
- `write_memnode(memnode)`: Set NUMA memory node (e.g., "0")
- `write_pid(pid)`: Move a process to this cpuset (returns True/False)
- `get_tasks()`: Get list of PIDs in this cpuset
- `set_cpu_exclusive()`: Set as isolated partition
- `unset_cpu_exclusive()`: Set as member partition
- `destroy()`: Remove this cpuset (must be empty)

**Properties:**
- `cpuset_path`: Full path to cpuset (e.g., "/sys/fs/cgroup/tuna_0")
- `cpuset_name`: Name of cpuset
- `auto_destroy`: Whether to auto-destroy on context exit

#### `CpusetsInit()`

Represents the root cgroup and provides system information.

**Properties:**
- `supported`: True if cgroup v2 is available
- `numa_nodes`: Number of NUMA nodes in the system
- `cpuset_path`: Path to root cgroup ("/sys/fs/cgroup")

**Methods:**
- Inherits all methods from `Cpuset` class
- Use to migrate tasks to/from root cgroup

#### `TaskMigrate(cpuset_from, cpuset_to)`

Helper class for bulk task migration.

**Methods:**
- `migrate()`: Migrate all tasks, returns (migrated_count, failed_count)

**Attributes:**
- `migrated`: Number of successfully migrated tasks
- `failed`: Number of tasks that couldn't be migrated

### Functions

#### `list_cpusets(pattern=None, recursive=True)`

List cpusets under `/sys/fs/cgroup`.

**Parameters:**
- `pattern` (str): Glob pattern to filter (e.g., "tuna_*")
- `recursive` (bool): If True, search nested cpusets

**Returns:** List of cpuset paths relative to /sys/fs/cgroup

**Example:**
```python
# All cpusets
all_cpusets = cpuset.list_cpusets()

# Only tuna cpusets
tuna_cpusets = cpuset.list_cpusets(pattern='tuna_*')

# Top-level only
top_level = cpuset.list_cpusets(recursive=False)
```

#### `destroy_cpuset(name_or_path, force=False)`

Destroy a single cpuset.

**Parameters:**
- `name_or_path` (str): Cpuset name or full path
- `force` (bool): If True, migrate tasks to root first

**Raises:**
- `OSError`: If cpuset doesn't exist or can't be removed
- `ValueError`: If trying to destroy root cgroup

**Example:**
```python
# Destroy empty cpuset
cpuset.destroy_cpuset('tuna_isolated')

# Destroy with tasks (migrates to root first)
cpuset.destroy_cpuset('busy_cpuset', force=True)
```

#### `cleanup_cpusets(pattern, force=True, recursive=True)`

Bulk cleanup of multiple cpusets.

**Parameters:**
- `pattern` (str): Glob pattern (e.g., "tuna_*")
- `force` (bool): If True, migrate tasks before destroying
- `recursive` (bool): If True, find and clean nested cpusets

**Example:**
```python
# Clean all tuna cpusets
cpuset.cleanup_cpusets('tuna_*', force=True)

# Clean nested hierarchies (children destroyed before parents)
cpuset.cleanup_cpusets('user/*', force=True, recursive=True)
```

## Testing

### Running Tests

The test suite uses Python's unittest framework:

```bash
# Run basic tests (no root required)
python3 -m unittest tests.test_cpuset

# Run all tests (requires root)
sudo python3 -m unittest tests.test_cpuset -v

# Run specific test class
sudo python3 -m unittest tests.test_cpuset.TestCpusetCreation -v

# Run basic functionality test script
sudo python3 test_cpuset_basic.py
```

### Test Coverage

- **TestCpusetsInit**: Initialization and cgroup v2 detection (no root needed)
- **TestCpusetCreation**: Creating and destroying cpusets (requires root)
- **TestCpusetConfiguration**: CPU/memory assignment (requires root)
- **TestContextManager**: auto_destroy behavior (requires root)
- **TestTaskMigration**: Process migration (requires root)
- **TestDiscoveryFunctions**: list_cpusets() functionality (requires root)
- **TestCleanupFunctions**: destroy_cpuset() and cleanup_cpusets() (requires root)

## Key Differences from rteval

| Feature | rteval | tuna |
|---------|--------|------|
| Default behavior | Ephemeral (auto-destroyed) | Persistent |
| auto_destroy default | True (via context manager) | False |
| Cleanup after exit | Automatic | Manual (or opt-in auto_destroy) |
| Discovery functions | No | Yes (list_cpusets) |
| Arbitrary destroy | No | Yes (destroy_cpuset, cleanup_cpusets) |
| Nested cpuset support | Basic | Enhanced (recursive search) |
| Primary use case | Testing (temporary isolation) | Environment setup (persistent) |

## Architecture

### Persistent vs Ephemeral Cpusets

**rteval approach** (ephemeral):
- Creates cpusets at start of measurement run
- Uses them during testing
- Destroys them automatically when run finishes
- Cpusets don't persist after rteval exits

**tuna approach** (persistent):
- Creates cpusets that persist after tuna exits
- Other programs use these cpusets after tuna has finished
- Provides utilities to list and manage existing cpusets
- Can destroy any cpuset (not just ones tuna created)

### Context Manager Behavior

```python
# rteval style: auto-cleanup
with cpuset.Cpuset('temp', auto_destroy=True) as cs:
    # Use cpuset
    pass
# Cpuset is destroyed here

# tuna style: persistent (default)
with cpuset.Cpuset('persistent') as cs:
    # Use cpuset
    pass
# Cpuset still exists here
```

## Nested Cpusets

The module supports nested cpuset hierarchies created by systemd, containers, or users:

```python
# Create parent
parent = cpuset.Cpuset('parent')
parent.write_memnode('0')

# Create child manually
child_path = os.path.join(parent.cpuset_path, 'child')
os.mkdir(child_path)

# Discovery finds both
all_cpusets = cpuset.list_cpusets(recursive=True)
# Returns: ['parent', 'parent/child']

# Cleanup handles nesting (children destroyed first)
cpuset.cleanup_cpusets('parent*', recursive=True)
```

## Common Use Cases

### 1. Isolate CPUs for Real-Time Workload

```python
from tuna import cpuset

# Create isolated cpuset for RT tasks
rt_cpuset = cpuset.Cpuset('realtime_cpus')
rt_cpuset.write_memnode('0')
rt_cpuset.assign_cpus('0-3')
rt_cpuset.set_cpu_exclusive()

# Move RT processes to isolated CPUs
for pid in [1234, 5678]:
    rt_cpuset.write_pid(pid)
```

### 2. Move Housekeeping Tasks Away from Critical CPUs

```python
from tuna import cpuset

# Create cpuset for housekeeping
housekeeping = cpuset.Cpuset('housekeeping')
housekeeping.write_memnode('0')
housekeeping.assign_cpus('4-7')

# Migrate most tasks from root to housekeeping
root = cpuset.CpusetsInit()
tm = cpuset.TaskMigrate(root, housekeeping)
migrated, failed = tm.migrate()
```

### 3. Setup and Teardown for Testing

```python
from tuna import cpuset

# Setup: Create test environment
def setup_test_cpusets():
    # Housekeeping CPUs
    hk = cpuset.Cpuset('test_housekeeping')
    hk.write_memnode('0')
    hk.assign_cpus('4-7')

    # Isolated CPUs
    iso = cpuset.Cpuset('test_isolated')
    iso.write_memnode('0')
    iso.assign_cpus('0-3')
    iso.set_cpu_exclusive()

    return hk, iso

# Teardown: Clean up
def teardown_test_cpusets():
    cpuset.cleanup_cpusets('test_*', force=True)
```

### 4. Discover and Report Existing Cpusets

```python
from tuna import cpuset

def report_cpusets():
    """Generate a report of all active cpusets"""
    all_cpusets = cpuset.list_cpusets(recursive=True)

    for cs_name in all_cpusets:
        cs_path = f"/sys/fs/cgroup/{cs_name}"

        # Read configuration
        with open(f"{cs_path}/cpuset.cpus") as f:
            cpus = f.read().strip()
        with open(f"{cs_path}/cgroup.procs") as f:
            tasks = f.read().strip().split('\n')

        print(f"{cs_name}:")
        print(f"  CPUs: {cpus}")
        print(f"  Tasks: {len(tasks)}")
```

### 5. High-Level Workflow vs Manual Primitives

The `tuna isolate --cpuset` command provides a high-level workflow that automatically creates both isolated and housekeeping cpusets, migrates processes, and sets up kernel-enforced CPU isolation. For advanced users or custom setups, you can replicate this manually using the primitive cpuset commands.

#### High-Level Workflow (Recommended)

```bash
# Automatic: Creates both cpusets, sets partition=isolated, migrates processes
$ tuna isolate -c 0-1 --cpuset --cpuset-housekeeping 2-3

# What happens automatically:
# 1. Creates tuna_isolated cpuset with CPUs 0-1 (partition=isolated, empty)
# 2. Creates tuna_housekeeping cpuset with CPUs 2-3 (partition=isolated)
# 3. Auto-detects NUMA nodes for both cpusets
# 4. Migrates all processes from root to tuna_housekeeping
# 5. Leaves tuna_isolated empty and ready for RT workload
```

#### Manual Equivalent Using Primitives

```bash
# Step 1: Create isolated cpuset (partition=isolated, empty)
$ tuna cpuset create -c 0-1 -n tuna_isolated --isolated

# Step 2: Create housekeeping cpuset (partition=isolated)
$ tuna cpuset create -c 2-3 -n tuna_housekeeping --isolated

# Step 3: Migrate all processes from root to housekeeping
# (This requires reading PIDs from root and moving them one by one)
$ for pid in $(cat /sys/fs/cgroup/cgroup.procs); do
    echo $pid > /sys/fs/cgroup/tuna_housekeeping/cgroup.procs 2>/dev/null || true
done

# Or use Python with the cpuset module:
# >>> from tuna import cpuset
# >>> root = cpuset.CpusetsInit()
# >>> hk = cpuset.Cpuset('tuna_housekeeping', existing_ok=True)
# >>> tm = cpuset.TaskMigrate(root, hk)
# >>> migrated, failed = tm.migrate()
```

#### When to Use Each Approach

**Use the high-level workflow** (`tuna isolate --cpuset`):
- When you want complete CPU isolation setup in one command
- For standard RT workload scenarios (isolated CPUs + housekeeping CPUs)
- When you want automatic NUMA-aware configuration
- For quick setup with best practices built-in

**Use manual primitives** (`tuna cpuset create`, `modify`, `move`):
- When you need fine-grained control over each cpuset
- For non-standard configurations (e.g., multiple isolated cpusets)
- When you want to create cpusets without immediate process migration
- For scripting custom workflows or integration with other tools
- When you need to modify existing cpusets incrementally

#### Design Philosophy

The primitive commands (`tuna cpuset create`, `modify`, `move`, etc.) are **building blocks** that follow the Unix philosophy: do one thing well. They give you complete control.

The workflow commands (`tuna isolate --cpuset`, `tuna move --cpuset`) combine multiple primitives to accomplish common high-level tasks with sensible defaults.

Both approaches are valid and coexist peacefully - choose based on your use case and level of control needed.

## Troubleshooting

### "Permission denied" errors

Most operations require root permissions. Run with `sudo`:

```bash
sudo python3 your_script.py
```

### "cgroup v2 not supported"

Check if cgroup v2 is enabled:

```bash
# Check mount
mount | grep cgroup2

# Should show:
# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

# Enable cgroup v2 (if needed)
# Add to kernel boot parameters: systemd.unified_cgroup_hierarchy=1
```

### "Invalid argument" when assigning CPUs

In cgroup v2, you must set memory nodes before CPUs:

```python
# Correct order
cs.write_memnode('0')    # First
cs.assign_cpus('0-3')    # Then

# Wrong order (will fail)
cs.assign_cpus('0-3')    # Error!
cs.write_memnode('0')
```

### Can't destroy cpuset: "Device or resource busy"

The cpuset still has processes. Use `force=True` to migrate them first:

```python
cpuset.destroy_cpuset('busy_cpuset', force=True)
```

### Can't destroy parent cpuset: "Directory not empty"

Destroy children before parents:

```python
# Manual approach
os.rmdir('/sys/fs/cgroup/parent/child')
os.rmdir('/sys/fs/cgroup/parent')

# Or use recursive cleanup (handles ordering automatically)
cpuset.cleanup_cpusets('parent*', recursive=True)
```

## Contributing

When contributing to this module, please:

1. Maintain compatibility with rteval's core API
2. Keep tuna-specific enhancements clearly documented
3. Add tests for new functionality
4. Ensure tests can run both with and without root (use `@unittest.skipUnless`)
5. Update this documentation

## References

- [Linux cgroup v2 documentation](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html)
- [CPU management in cgroup v2](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#cpu)
- [rteval cpuset implementation](https://git.kernel.org/pub/scm/utils/rteval/rteval.git/)

## License

SPDX-License-Identifier: GPL-2.0-only

Copyright 2026 John Kacur <jkacur@redhat.com>

Originally prototyped in tuna, implemented in rteval, and adapted back to tuna with enhancements.
