parallel: BackgroundTaskRunner: pass down args/kwargs to tasks
In order to use fun multiprocessing objects like Value and Queue, you have
to serialize the objects sooner rather than later. That means you can't
do something perfectly reasonable like:
results = multiprocessing.Queue()
with parallel.BackgroundTaskRunner(myfunc) as queue:
for x in (1, 2, 3):
queue.put((results, x))
Python will crap itself like so:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
File "/usr/lib64/python2.7/multiprocessing/queues.py", line 77, in __getstate__
assert_spawning(self)
File "/usr/lib64/python2.7/multiprocessing/forking.py", line 51, in assert_spawning
' through inheritance' % type(self).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance
In order to pass that results queue down to myfunc, you have to include it
in the multiprocess.Process() initialization. Unfortunately, our current
framework does not support passing args/kwargs at that phase.
So let's throw in the plumbing to support arbitrary passing of args and
kwargs down to the function at task creation time. Not only does this
solve the above problem, but it makes implementations cleaner when you
have a set of args/kwargs that are common to all calls as you no longer
need to manually add them to the queue.
Thus the aforementioned snippet now becomes:
results = multiprocessing.Queue()
with parallel.BackgroundTaskRunner(myfunc, results) as queue:
for x in (1, 2, 3):
queue.put((x,))
BUG=chromium:209442
BUG=chromium:213204
TEST=`lib/parallel_unittest.py` passes
TEST=my new code that needs this functionality works
Change-Id: I83ac1aa73f98f72c6a8d3a909f4b649059faf98e
Reviewed-on: https://gerrit.chromium.org/gerrit/57182
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
2 files changed
