sepehr
2/10/2015 - 12:58 PM

Shell: Analyze huge files for repeating text portions

Shell: Analyze huge files for repeating text portions

# Sorts the file by duplicate line count
sort /path/to/filename | uniq -c | sort -nr > ./_aggregated.tmp

# Just read the head as it's probably a huge file
head -n 1000 ./_aggregated.tmp | less