Kubernetes quotas queue

Question

I need to queue Kubernetes resources, basing on the Kubernetes quotas.

Sample expected scenario:

• a user creates Kubernetes resource (let's say a simple X pod)
• quora object resource count reached, pod X goes to the Pending state
• resources are released (other pod Y removed), our X pod starts creating

For, now this scenario will not work, due to the quota behavior which returns 403 FORBIDDEN, when there are no free resources in quota:

If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.

Question: Is there a way to achieve this via native Kubernetes mechanisms?

I was trying to execute pods over Kubernetes Jobs, but each job starts independently and I'm unable to control the execution order. I would like to execute them in First In First Out method.

Answer 1

IMO, if k8s hasn't accepted the resource, how come it manage its lifecycle or execution order.

If I understood your question correctly, then its the same pod trying to be scheduled then, your job should be designed in such a way that order of job execution should not matter because there could be scenarios where one execution is not completed and next one comes up or previous one failed to due some error or dependent service being unavailable. So, the next execution should be able to start from where the last one left.

You can also look at the work queue pattern in case it suits your requirements as explained https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/

In case you just want one job to be in execution at one time.

Answer 2

I think, running jobs in predefined order must be managed by external logic. We use Jenkins Pipeline for that.