Model build generates several artifacts (e.g., several .mat files) for a model which are used in the code generation process. So if you are using one of these commands to build a single model multiple times via multiple threads, then these threads will run into resource contention for these artifacts. But from your explanation I assume you are building multiple models which are independent of each other. In this case, you can use rtwbuild with parallel computing features.
Now, there could be multiple factors affecting the speed of the build in workers. Are you running the workers locally or in a remote machine? - If they are running locally in the machine same as the client, then the increased overload on the machine's memory and CPU load could be slowing down the build. - If they are running remotely, then the remote machine's ability (memory and CPU load) to handle memory (or any resource) contention by parallel processes efficiently matters.
Having said all these, the one important information that you should consider is "How slower are these build?". So before diagnosing the effect of all these factors, please note that, in programming world, parallel computation is not always guaranteed to be faster than sequential computations. There would be scenarios where the parallel computations could be a bit slower than sequential.
In general, rtwbuild is designed for single-threaded application and hence there is no difference in the way it works on client and on workers.