overcommitted kvm performance issue - c

Question

I'm having performance problem when I run a benchmark program on a virtual machine with 1 pCPU and 4 vCPUs. I used mpstat to investigate about what might be causing the problem and how it is causing the problem. it shows an increase in steal time, idle time and sys time.
here is output of mpstat on my virtual machine when I run my program:
CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
all 3.14 0.00 8.00 0.00 0.00 0.00 58.29 0.00 30.57
0 4.60 0.00 8.05 0.00 0.00 0.00 64.37 0.00 22.99
1 2.27 0.00 5.68 0.00 0.00 0.00 53.41 0.00 38.64
2 3.37 0.00 8.99 0.00 0.00 0.00 61.80 0.00 25.84
3 3.45 0.00 8.05 0.00 0.00 0.00 54.02 0.00 34.48
and here is its output on a host machine (when I run the program):
CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
all 91.80 0.00 0.50 0.00 0.00 0.00 0.00 0.00 7.70
0 91.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 8.50
1 93.03 0.00 0.50 0.00 0.00 0.00 0.00 0.00 6.47
2 90.59 0.00 0.50 0.00 0.00 0.00 0.00 0.00 8.91
3 93.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.00
I know that (from ibm.com):
Steal time is the percentage of time a virtual CPU waits for a real
CPU while the hypervisor is servicing another virtual processor.
I know this increase in steal time is maybe because my virtual machine is over-committed, but I wonder why I did not have this problem with other programs that I tested. and I don't know why the idle time increases as well.
the source code for my test program is quite large and I posted it here in github. I don't know if it is important or not but this code does some busy waiting. here is the code where busy waiting happens:
int parsec_barrier_wait(parsec_barrier_t *barrier) {
int master;
int rv;
if(barrier==NULL) return EINVAL;
rv = pthread_mutex_lock(&barrier->mutex);
if(rv != 0) return rv;
//A (necessarily) unsynchronized polling loop followed by fall-back blocking
if(!barrier->is_arrival_phase) {
pthread_mutex_unlock(&barrier->mutex);
volatile spin_counter_t i=0;
while(!barrier->is_arrival_phase && i<SPIN_COUNTER_MAX) i++;
while((rv=pthread_mutex_trylock(&barrier->mutex)) == EBUSY);
if(rv != 0) return rv;
//Fall back to normal waiting on condition variable if necessary
while(!barrier->is_arrival_phase) {
rv = pthread_cond_wait(&barrier->cond, &barrier->mutex);
if(rv != 0) {
pthread_mutex_unlock(&barrier->mutex);
return rv;
}
}
}
//We are guaranteed to be in an arrival phase, proceed with barrier synchronization
master = (barrier->n == 0); //Make first thread at barrier the master
barrier->n++;
if(barrier->n >= barrier->max) {
//This is the last thread to arrive, don't wait instead
//start a new departure phase and wake up all other threads
barrier->is_arrival_phase = 0;
pthread_cond_broadcast(&barrier->cond);
} else {
//wait for last thread to arrive (which will end arrival phase)
//we use again an unsynchronized polling loop followed by synchronized fall-back blocking
if(barrier->is_arrival_phase) {
pthread_mutex_unlock(&barrier->mutex);
volatile spin_counter_t i=0;
while(barrier->is_arrival_phase && i<SPIN_COUNTER_MAX) i++;
while((rv=pthread_mutex_trylock(&barrier->mutex)) == EBUSY);
if(rv != 0) return rv;
//Fall back to normal waiting on condition variable if necessary
while(barrier->is_arrival_phase) {
rv = pthread_cond_wait(&barrier->cond, &barrier->mutex);
if(rv != 0) {
pthread_mutex_unlock(&barrier->mutex);
return rv;
}
}
}
}
barrier->n--;
//last thread to leave barrier starts a new arrival phase
if(barrier->n == 0) {
barrier->is_arrival_phase = 1;
pthread_cond_broadcast(&barrier->cond);
}
pthread_mutex_unlock(&barrier->mutex);
return (master ? PARSEC_BARRIER_SERIAL_THREAD : 0);
}
any idea would be very much appreciated.