jAndersonTech
12/29/2017 - 5:44 AM

Multi/Sub processing of files

Given a group of files, use multiprocessing and subprocessing to do something to them.This snippet includes throttle where the user can determine how many files are process at one time.

multiprocess_files.py
content_copyfile_download
import multiprocessing as mp
import subprocess
import glob

file_paths = glob.glob('some/wildcard/file/path/template')
processes = []
count = 1

for inpath in file_paths:
    # Create and start process to act on each file
    command = ['some', 'iterable', 'array', 'for', 'subprocess', 'using', 'inpath']
    process = mp.Process(target=lambda cmd: subproccess.Popen(cmd).communicate(), args=(command,))
    processes.append(process)
    process.start()

    # Wait for started processes to finish when throttle is reached
    if count % throttle == 0:
        for p in processes:
            p.join()
        processes = []

    count += 1

# Wait for any leftover processes to finish
for p in processes:
    p.join()