One of the methods of exchanging data between processes with the multiprocessing module is directly shared memory via multiprocessing.Value. As any method that's very general, it can sometimes be tricky to use. I've seen a variation of this question asked a couple of times on StackOverflow:
I have some processes that do work, and I want them to increment some shared counter because [... some irrelevant reason ...] - how can this be done?
The wrong way
And surprisingly enough, some answers given to this question are wrong, since they usemultiprocessing.Value incorrectly, as follows:
import time
from multiprocessing import Process, Value
def func(val):
for i in range(50):
time.sleep(0.01)
val.value += 1
if __name__ == '__main__':
v = Value('i', 0)
procs = [Process(target=func, args=(v,)) for i in range(10)]
for p in procs: p.start()
for p in procs: p.join()
print v.value
This code is a demonstration of the problem, distilling only the usage of the shared counter. A "pool" of 10 processes is created to run the func function. All processes share a Value and increment it 50 times. You would expect this code to eventually print 500, but in all likeness it won't. Here's some output taken from 10 runs of that code:
> for i in {1..10}; do python sync_nolock_wrong.py; done
435
464
484
448
491
481
490
471
497
494
Why does this happen?
I must admit that the documentation of multiprocessing.Value can be a bit confusing here, especially for beginners. It states that by default, a lock is created to synchronize access to the value, so one may be falsely led to believe that it would be OK to modify this value in any way imaginable from multiple processes. But it's not.
Explanation - the default locking done by Value
This section is advanced and isn't strictly required for the overall flow of the post. If you just want to understand how to synchronize the counter correctly, feel free to skip it.
The locking done by multiprocessing.Value is very fine-grained. Value is a wrapper around a ctypesobject, which has an underlying value attribute representing the actual object in memory. All Value does is ensure that only a single process or thread may read or write this value attribute simultaneously. This is important, since (for some types, on some architectures) writes and reads may not be atomic. I.e. to actually fill up the object's memory, the CPU may need several instructions, and another process reading the same (shared) memory at the same time could see some intermediate, invalid state. The built-in lock of Value prevents this from happening.
However, when we do this:
val.value +=1
What Python actually performs is the following (disassembled bytecode with the dis module). I've annotated the locking done by Value in #<-- comments:
0 LOAD_FAST 0 (val)
3 DUP_TOP
#<--- Value lock acquired
4 LOAD_ATTR 0 (value)
#<--- Value lock released
7 LOAD_CONST 1 (1)
10 INPLACE_ADD
11 ROT_TWO
#<--- Value lock acquired
12 STORE_ATTR 0 (value)
#<--- Value lock released
So it's obvious that while process #1 is now at instruction 7 (LOAD_CONST), nothing prevents process #2 from also loading the (old) value attribute and be on instruction 7 too. Both processes will proceed incrementing their private copy and writing it back. The result: the actual value got incremented only once, not twice.
The right way
Fortunately, this problem is very easy to fix. A separate Lock is needed to guarantee the atomicity of modifications to the Value:
import time
from multiprocessing import Process, Value, Lock
def func(val, lock):
for i in range(50):
time.sleep(0.01)
with lock:
val.value += 1
if __name__ == '__main__':
v = Value('i', 0)
lock = Lock()
procs = [Process(target=func, args=(v, lock)) for i in range(10)]
for p in procs: p.start()
for p in procs: p.join()
print v.value
Now we get the expected result:
> for i in {1..10}; do python sync_lock_right.py; done
500
500
500
500
500
500
500
500
500
500
A value and a lock may appear like too much baggage to carry around at all times. So, we can create a simple "synchronized shared counter" object to encapsulate this functionality:
import time
from multiprocessing import Process, Value, Lock
class Counter(object):
def __init__(self, initval=0):
self.val = Value('i', initval)
self.lock = Lock()
def increment(self):
with self.lock:
self.val.value += 1
def value(self):
with self.lock:
return self.val.value
def func(counter):
for i in range(50):
time.sleep(0.01)
counter.increment()
if __name__ == '__main__':
counter = Counter(0)
procs = [Process(target=func, args=(counter,)) for i in range(10)]
for p in procs: p.start()
for p in procs: p.join()
print counter.value()
Bonus: since we've now placed a more coarse-grained lock on the modification of the value, we may throw away Value with its fine-grained lock altogether, and just use multiprocessing.RawValue, that simply wraps a shared object without any locking.
转自:http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing
--end
相关推荐
Python_multiprocessing_6_共享内存_shared_memory_(多进程_多核运算_教学教程tutori
python-multiprocessing-enginemap_jobs 并行化作业,返回一个DataFrame或Series indicators = map_jobs ( func = handle_task , molecules = ( 'jobs' , jobs ), threads = 8 , batches = 4)map_reduce_jobs 并行...
主要介绍了Python进程Multiprocessing模块原理解析,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的参考学习价值,需要的朋友可以参考下
Python_multiprocessing_4_效率对比_multithreading,_multiprocessing_co
Python的multiprocessing模块不但支持多进程,其中managers子模块还支持把多进程分布到多台机器上。一个服务进程可以作为调度者,将任务分布到其他多个机器的多个进程中,依靠网络通信。想到这,就在想是不是可以...
主要介绍了python基于multiprocessing的多进程创建方法,实例分析了multiprocessing模块操作进程的相关技巧,需要的朋友可以参考下
Python的multiprocessing模块不但支持多进程,其中managers子模块还支持把多进程分布到多台机器上。一个服务进程可以作为调度者,将任务分布到其他多个机器的多个进程中,依靠网络通信。 想到这,就在想是不是可以...
Python_multiprocessing_7_lock_锁__(多进程_多核运算_教学教程tutorial)
本文实例讲述了python使用multiprocessing模块实现带回调函数的异步调用方法。分享给大家供大家参考。具体分析如下: multipressing模块是python 2.6版本加入的,通过这个模块可以轻松实现异步调用 from ...
Python_multiprocessing_3_queue_进程输出_(多进程_多核运算_教学教程tutorial)
Python_multiprocessing_2_创建进程_(多进程_多核运算_教学教程tutorial)
主要介绍了Python使用multiprocessing创建进程的方法,实例分析了multiprocessing模块操作进程的相关技巧,需要的朋友可以参考下
Python_multiprocessing_5_进程池_pool_(多进程_多核运算_教学教程tutorial)
python基础_32_Python_multiprocessing_1_什么是多进程_(多进程_多核运算_教学教程tutori
Python multiprocessing.pool创建非守护进程的方法