jgoenetxea
8/20/2018 - 8:32 AM

How to read folders with too many files

Sometimes linux is not able to read the content of folders with thousands or millions of files. In those cases, we have to take other alternatives. An example is to write the content of the folder in a file using python (for examlpe) avoiding inbetween buffers and other elements used by the terminal or window based file mangers.

import os
from os import listdir
from os.path import isfile, join

if __name__ == '__main__':
    # load file list name
    folderPath = "db_original/"

    totalExtension = '.json'

    # loop and mv train files
    outfile = open("json_file_list.log", "w")
    for f in listdir(folderPath):
        if isfile(join(folderPath, f)) and f.endswith(totalExtension):
            outfile.write(f)
            outfile.write('\n');

    outfile.close()
    print("Finished")