0

I am trying to use task spooler to schedule CI/CD testing tasks for a CUDA project. The system contains multiple GPUs, and a single task uses most of the memory of one GPU, so I want to avoid giving a single GPU multiple tasks. tsp allows "MULTI-SLOT" mode, where it can have multiple active slots that can pull from the queue, rather than just one. However, I so far see no way for the task itself to deduce which slot it has been assigned, which means that I don't know which GPU to run the task on. I could check with something like nvidia-smi which GPUs are being utilized, but that might cause race conditions if multiple items are pulled from the queue in a small amount of time.

Is there a way to deduce this slot number, or send it to the process somehow?

Jan Heemstra
  • 1
  • What about one task spooler server per GPU? – Marco Jul 19 '23 at 10:06
  • @Marco that would have no race conditions, but it would mean that I would have to "queue-balance" the input queues in order to get similar performance. Since tsp is a per-user queue, it would also mean that I would have to add a user for every GPU in the system. In my case I think it wouldn't be worth the headache over just using a single GPU. – Jan Heemstra Jul 19 '23 at 11:03
  • No, taskspooler is not a "per-user queue", it is a "per-socket queue", which is configured via "TS_SOCKET" environment variable. – Marco Jul 20 '23 at 15:10

0 Answers0