python - Progress in a itertools.combinations pool() map , result passed in list -
i run multipocessing script this:
def intersterer(i): somestuff return x if __name__ == '__main__': pool = pool() list = pool.map(intersterer, itertools.combinations(xrange(great_number), 2))
i want know how can see progress of work. around shared counter, ugly , seems not telling truth. (perhaps wrong code in it) can me great pythonic way?
edit: exemple, want store results in list, answers @ question : show progress of python multiprocessing pool map call? doesn't me this. thks
edit2:
here soluced way. thks every helpers!
def intersterer(i): somestuff return x numtot = binomial(number,2) pool = pool() list=[] tpsb=time.time() in pool.imap_unordered(intersterer, itertools.combinations(xrange(number),2)): list.append(i); print "{:3.2f}%, approximatly {:3.2f} seconds left \r".format(100.*len(list)/numtot, (time.time()-tpsb)*100./(100.*len(list)/numtot)-(time.time()-tpsb)),
use imap_unordered
instead of map
:
>>> pathos.multiprocessing import pool >>> pool = pool() >>> import itertools >>> >>> def interstellar(i): ... return sum(i)**2 ... >>> result = [] >>> in pool.imap_unordered(interstellar, itertools.combinations(xrange(10),2)): ... result.append(i); print "{:3.2f}%".format(100.*len(result)/45) ... 2.22% 4.44% 6.67% 8.89% 11.11% 13.33% 15.56% 17.78% 20.00% 22.22% 24.44% 26.67% 28.89% 31.11% 33.33% 35.56% 37.78% 40.00% 42.22% 44.44% 46.67% 48.89% 51.11% 53.33% 55.56% 57.78% 60.00% 62.22% 64.44% 66.67% 68.89% 71.11% 73.33% 75.56% 77.78% 80.00% 82.22% 84.44% 86.67% 88.89% 91.11% 93.33% 95.56% 97.78% 100.00% >>> result [1, 4, 9, 16, 25, 36, 49, 64, 81, 9, 16, 25, 36, 49, 64, 81, 100, 25, 36, 49, 64, 81, 100, 121, 49, 64, 81, 100, 121, 144, 81, 100, 121, 144, 169, 121, 144, 169, 196, 169, 196, 225, 225, 256, 289]
i'm being lazy, , using pathos.multiprocessing
can work interpreter… should work standard multiprocessing
if done file , in __main__
. in solution, have know there 45
combinations, otherwise can't percent w/o knowing final length.
since don't care order , care speed, i've used imap_unordered
. if care order, use imap
.
edit: aha… duplicate question.
Comments
Post a Comment