General mistakes in parallel computing

Posted 2020-07-23

这是2013年写的一篇旧文，放在gegahost.net上面  http://raison.gegahost.net/?p=97

March 11, 2013

General mistakes in parallel computing

Filed under: concurrency,software — Tags: atomic, cocurrency, data race, dead lock, parellel — Raison @ 2:51 am

(Original Work by Peixu Zhu)

In parallel computing environment, some general mistakes are frequent and difficult to shoot, caused by random CPU sequence in different thread contexts. Most of them are atomic violation, order violation, and dead lock. Studies show that some famous software also have such mistakes, like mysql, Apache, Mozilla, and OpenOffice.

1. Atomic violation

In sequent programming, we seldom care the atomic operation, however, in parallel programming, we must remember atomic operations at first. for example:

[Thread 1]


if (_ptr)         // A
*_ptr = 0; // B


[Thread 2]

_ptr = NULL;        // C

For above code, there’s one statement to be executed in thread 1 and thread 2 respectively, it seems that it should be running the statement in thread 1 or thread 2, they should not be interlaced. But, in fact, statement in thread 1 is not atomic, at least, it can be divided into step A and B, thus, if it is arranged to execute in order of A-B-C, it is okay, however, it is also possible be scheduled to run as A-C-B, this will bring an unexpected memory access error.

We assume that the statement region in thread 1 is atomic, but it is not true. This is the root of the atomic violation. In many cases, the problem is caused by code modification, for above example, the statement in thread 1 may be a simple assignment statement at first:
_ptr = &_val;
And later, the code is modified, and the implicit atomicity is broken.

For systems with multiple cores, the problem will be more complicated, since each core may cache a block of memory respectively. For example, core 1 runs thread 1, and core 2 runs thread 2:
[Thread 1]
_ptr = &_val;

[Thread 2]
_ptr = NULL;

Are they atomic ? No, they are not in fact. the `_ptr` may be optimized to be register value in one core locally, or it is cached in different core. Thus, the we can not determine the value of `_ptr`.

To avoid atomic violation, we must make the code region atomic, by locking or atomic operations. Explicit atomic operations on a shared variable is a good habit, since we are noticed by the statement that it is atomicity demanded when we try to modify the code.

2. Order violation

Considering below example:
[Thread 1]
_ptr = allocate_memory(); // A

[Thread 2]
_ptr[1] = "right"; // B

If the code is not synchronized, execution order of A-B or B-A are all possible. In such cases, we must synchronize the code block to ensure the order of execution.

3. Dead lock

Locking is elemental in concurrent programming. If there’s more than one threads working with more than with one shared resource, such as memory block, it is possible that each thread owning a resource is waiting for each others resource.
[Thread 1]

lock_a.lock();
a = 0; // A
lock_b.lock();
b = 0; // B
lock_b.unlock();
lock_a.unlock();


[Thread 2]

lock_b.lock();
b = 1; // C
lock_a.lock();
a = 1; // D
lock_a.unlock();
lock_b.unlock();


if the code is running as A-B-C-D, there’s no problem, however, if it is running as A-C-B-D, there’s dead lock. Dead locking requires four conditions:
a. mutex exclusion
b. hold and wait
c. no preemption
d. circular waiting

Breaking at least one of above four condition will break the dead locking.