Command to log training performance for caffe
Usually I train with Caffe from Python, since it allows me to initiate layers with just a bit of Python code. python solve.py <solverstate file> 2>&1 | tee -a solve.log Caffe prints everything out to stderr instead of stdout, this is why I redirect it using 2>&1. Tee splits stdout to both a file (which is appended thanks to -a) and also displays it to the screen. If I didn't use tee I would just end up with another terminal for tail -f.
1.Command to parse training log $python /path/to/caffe/tools/extra/parse_log.py /logfile_path/logfile.log /output_dir Ouput example: # my_model.log.train NumIters,Seconds,LearningRate,loss 6000.0,10.468114,1e-06,0.0476156 6020.0,17.372427,1e-06,0.0195639 6040.0,24.237645,1e-06,0.0556274 6060.0,31.084703,1e-06,0.0244656 6080.0,37.927866,1e-06,0.0325582 6100.0,44.778659,1e-06,0.0131274 6120.0,51.62342,1e-06,0.0607449 2. Command to visualize log python ~/caffe/tools/extra/parse_log.py my_model.log . gnuplot -persist gnuplot_commands where gnuplot_commands is a file that stores a set of gnuplot commands. # gnuplot_commands set datafile separator ',' set term x11 0 plot '../my_model.log.train' using 1:4 with line,\ '../my_model.log.test' using 1:5 with line set term x11 1 plot '../my_model.log.test' using 1:4 with line