Add Cloud Native AI Scheduling Challenges Whitepaper

[rajaskakodkar]

Adds the Cloud Native Scheduling Challenges Whitepaper

[andreyvelich]

Thanks for this effort @rajaskakodkar!
Overall, looks great, I left few thoughts.

[andreyvelich]

Maybe we can also mention unstructured data?

Suggested change

	From a scheduling perspective, data preparation is typically CPU and I/O intensive rather than GPU-intensive. That said, GPU-accelerated frameworks can significantly speed up large-scale data processing tasks such as filtering, joining, and aggregating datasets. Jobs are often parallelizable—you can clean different partitions of a dataset independently. Event-driven scheduling is common: new data arriving triggers a preparation pipeline.
	From a scheduling perspective, data preparation is typically CPU and I/O intensive rather than GPU-intensive. That said, GPU-accelerated frameworks can significantly speed up large-scale data processing tasks such as filtering, joining, and aggregating datasets. Additionally, GPUs work well for unstructured data like images because image processing involves massive parallel math operations.
	Jobs are often parallelizable—you can clean different partitions of a dataset independently. Event-driven scheduling is common: new data arriving triggers a preparation pipeline.

[andreyvelich]

What about Spark here?

Suggested change

	Kubernetes resources like Jobs and CronJobs handle these workloads reasonably well. Workflow orchestrators (Airflow, Argo Workflows, Flyte) coordinate multi-step pipelines.
	Kubernetes resources like Jobs, CronJobs, and SparkApplications handle these workloads reasonably well. Workflow orchestrators (Airflow, Argo Workflows, Flyte) coordinate multi-step pipelines.

[andreyvelich]

Would it make sense to add topic around HPO?

Suggested change

	* **Model architecture** involves selecting the type of model (linear regression, decision tree, neural network, transformer) and designing its structure. For deep learning, this means defining layers, attention mechanisms, and other architectural choices. This work is often interactive—a data scientist experimenting in a notebook—and does not require significant compute resources until training begins.
	* **Model architecture** involves selecting the type of model (linear regression, decision tree, neural network, transformer) and designing its structure. For deep learning, this means defining layers, attention mechanisms, and other architectural choices. This work is often interactive—a data scientist experimenting in a notebook—and does not require significant compute resources until training begins.
	* **Hyperparameter tuning** optimizes how the model learns rather than the structure of the model itself. This includes adjusting parameters such as learning rate, batch size, optimizer choice, number of epochs, and dropout rates. Unlike architecture design, hyperparameter tuning is compute-intensive because it requires repeatedly training and evaluating many model variants. These tuning jobs are highly parallelizable and are commonly distributed across GPUs or clusters.

[andreyvelich]

With the efforts around WAS, I wouldn't mention this, maybe we can say:
cc @helayoty @kannon92 @mm4tt

Suggested change

	The default Kubernetes scheduler cannot handle these requirements. It will start pods as resources become available, potentially leaving a job stuck with partial resources indefinitely.
	These characteristics require additional Kubernetes scheduler capabilities to support efficient all-or-nothing placement and topology-aware scheduling.

[andreyvelich]

Suggested change

* **Tightly coupled distribution.** Distributed training uses collective communication patterns where all workers must participate. You cannot start with 7 of 8 workers and add the 8th later. You cannot lose one worker and continue with the remaining 7\. Either all workers are running, or the job cannot proceed. This is fundamentally different from web services, where losing one replica just shifts load to the others.

* **Tightly coupled distribution.** Distributed training uses collective communication patterns where all workers must participate. You cannot start with 7 of 8 workers and add the 8th later. You cannot lose one worker and continue with the remaining 7. Either all workers are running, or the job cannot proceed. This is fundamentally different from web services, where losing one replica just shifts load to the others.

[andreyvelich]

Suggested change

	These tools provide higher-level abstractions for ML workflows:
	These tools provide higher-level abstractions for AI workloads:

[andreyvelich]

Ref: https://github.com/kubeflow/trainer#overview

Suggested change

	* **Kubeflow Trainer** supports distributed training across frameworks (PyTorch, TensorFlow, PaddlePaddle, XGBoost). Provides job abstractions that handle worker coordination, including gang scheduling requirements.
	* **Kubeflow Trainer** is a Kubernetes-native distributed AI platform for scalable LLM fine-tuning and training of AI models across a wide range of frameworks, including PyTorch, MLX, HuggingFace, DeepSpeed, JAX, XGBoost, and more. Provides job abstractions that handle worker coordination, including gang scheduling requirements and HPC workloads orchestration such as MPI and Flux.

[andreyvelich]

Suggested change

	| Fault Tolerance | \- | Slinky, | Kubeflow (elastic training) | Training | Framework-dependent |
	| Fault Tolerance | \- | Slinky, | Kubeflow Trainer | Training | Framework-dependent |

[andreyvelich]

I think for Preemption and TAS all platforms that integrate with Kueue should support it including KubeRay, Trainer, and k8s
WDYT @rajaskakodkar ?

Suggested change

	| Preemption | PriorityClass (pod-level) | KAI, Kueue, Slinky, Volcano | \- | Both | Job-level preemption needs external tools |
	| Priority Scheduling | PriorityClass | All batch schedulers | \- | Both | Job-level priority in batch schedulers |
	| Reservation & Backfill | \- | Slinky, Volcano, YuniKorn | \- | Training | Advanced feature in some schedulers |
	| Topology Awareness (Node) | Topology Manager (NUMA), DRA CPU Driver (CPU topology) | KAI, Kueue, Slinky, Volcano | \- | Both | GPU interconnect awareness varies |
	| Topology Awareness (Cluster) | Topology Spread Constraints, DRANET (network DRA Driver) (limited) | KAI, Kueue, Slinky, Volcano | \- | Both | Network topology awareness is emerging |
	| Preemption | PriorityClass (pod-level) | KAI, Kueue, Slinky, Volcano | KubeRay, Kubeflow Trainer, Kubernetes | Both | Job-level preemption needs external tools |
	| Priority Scheduling | PriorityClass | All batch schedulers | KubeRay, Kubeflow Trainer, Kubernetes | Both | Job-level priority in batch schedulers |
	| Reservation & Backfill | \- | Slinky, Volcano, YuniKorn | KubeRay, Kubeflow Trainer, Kubernetes | Training | Advanced feature in some schedulers |
	| Topology Awareness (Node) | Topology Manager (NUMA), DRA CPU Driver (CPU topology) | KAI, Kueue, Slinky, Volcano | KubeRay, Kubeflow Trainer, Kubernetes | Both | GPU interconnect awareness varies |
	| Topology Awareness (Cluster) | Topology Spread Constraints, DRANET (network DRA Driver) (limited) | KAI, Kueue, Slinky, Volcano | KubeRay, Kubeflow Trainer, Kubernetes | Both | Network topology awareness is emerging |

[andreyvelich]

Suggested change

	2. Use the appropriate job abstractions (PyTorchJob, MPIJob, etc.) rather than raw pods.
	2. Use the appropriate job abstractions (TrainJob, MPIJob, etc.) rather than raw pods.