v1beta2 migration script - support large scale

[mimowo]

What would you like to be added:

We have an issue using the mitigation script at large scale. https://raw.githubusercontent.com/kubernetes-sigs/kueue/main/hack/migrate-to-v1beta2.sh

The error we were getting:

Migrating workloads.kueue.x-k8s.io...
Error from server: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=Workload failed: Post "[https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s](https://kueue-webhook-service.kueue-system.svc/convert?timeout=30s)": EOF

We were successful to migrate by splitting running the script namespace-by-namespace.

We may also need to do paging better, but maybe namespace-by-namespace is enough.

Why is this needed:

To support migrating to v1beta2 for large evironments.

[mimowo]

@mwielgus @mbobrovskyi

[mimowo]

/cc @mbobrovskyi
tentatively, who is already familiar with that. We will sync on the next version of the script. Our patched version namespace-by-namespace must be cleaned up

[mimowo]

/assign @mbobrovskyi

[tenzen-y]

I can share my production cluster migration step in the following:

1. Update Kueue Controller Manager with kustomize build "$KUEUE_V016_MANIFESTS_DIR" | kubectl apply --server-side --force-conflicts -f -
2. Wait for all kueue-controller-managers ready (All replicas must be ready)
3. Execute ./migrate-to-v1beta2.sh

I'm not sure if this is the expected approach.
But, as I check your migration error (Error from server: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=Workload failed: Post "[https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s](https://kueue-webhook-service.kueue-system.svc/convert?timeout=30s)": EOF), 2 possible errors are considered: (1 one of the kueue replicas is unavailable, or (2 a lack of replicas. In case of a large cluster, they should increase manager replicas as much as possible in advance because the concurrent webhook execution should be delegated to multiple kueue managers.

[mbobrovskyi]

I can share my production cluster migration step in the following:

If it is not a secret how many workloads do you have?

[mbobrovskyi]

I think we need to add pagination to our script to allow batch updates instead of fetching all objects.

[tenzen-y]

I can share my production cluster migration step in the following:

If it is not a secret how many workloads do you have?

I shared my env in another space, then @mbobrovskyi confirmed that the error facing the env is far from my case.
So, I believe that batch processing would be worth it.

[mwielgus]

The underlying problem is that conversion webhook needs to process the whole batch in 150 seconds. If the processing takes more, the list is dropped. How much Kueue can process and send back within these 150 seconds depends on the network and node performance but with workloads counted in XX k it definitely fails.