Chapter 1

Environment Variables and Variables

print the env variables within a given process cat /proc/$PID/environ

Variable cheatsheet

find the length of variable ${#var} UID is a env variable used to identify the current user, the root's UID is 0. So,

[ $UID -ne 0 ] && echo "not root" || echo "I am root" 

Doing Math

We use (( )), [], and let to do simple math in shell, also use commands like bc.

1. let sets up a local environment, inside which $ is not necessary to prefix the variables. e.g.

let i++
let i--
let a = a + 4

1. [ ] and (( )) are similar to let. e.g.

a1=$[ a2 + 3 ]
a1=$(( a2 + 3 )) 

Arrays

arr=(1 2 3)
## OR, assign per slot 
arr[0]=2
arr[1]=5 
...

print all elements in an array: echo ${arr[@]}

print the length of array: echo ${#arr[*]}

define associative array: declare -A ass_array

init key-value pairs in the associative array:

ass_array=([key1]=value1 [key2]=value2)

list the indexes of an array: echo ${!arr[*]}

Dates

date command is used to display current date. The date string formatters are used to show datetime in various formats. For example, day: %d, month: %b/B , year:%y/Y.

prefixt the formatter with + to format date output:

> date '+%b %d %y'
>>>> Jan 03 16

### seconds since epoch
> date +%s 
>>> 12829389238

use: time a code snippet

start=$(date +%s)
do_something 
end=$(date +%s)
diff=$(( end - start  ))
echo "used ${diff} seconds" 

produce delay

Use sleep [seconds] to produce delays

Debugging

1. Debug the whole script: use bash -x script.sh
2. Debug only a section inside the script: use set -x and set +x to enclose the debug area, which prints the commands and arguments during execution.
3. Debug inputs, to print inputs when they are read: set -v and disable: set +v
4. customized debug output format using env variable _DEBUG

function DEBUG {
[ $_DEBUG == 'on' ] && $@ || : 

}

### debug a single line example

for i in {1..5}; do
DEBUG echo $i 
done

Read Inputs read

read -n [number_of_chars] var  ### read n characters into variable var 

read -d [delim] var  ### use delim as input delimiter 

read -t [timeout] var ### set timeout for input reading

read -s var ### silent-mode for reading passwords 

Comparisons

use [[ ]] for string comparisons and [ ] or (( )) for number comparisons

[[ -n $str1 ]] ### test if non-empty
[[ -z $str2 ]] ### test if empty 

Chapter 2. Common commands

'cat'

cat to concatenate files. use cat -s <file> to squeeze contiguous blank lines.

use cat -T <file> to highlight tabs as ^I in file.

use cat -n <file> to show line numbers along

'find'

### find both txt and pdf files 
find \( -name *.txt -o -name *.pdf \) -print
### use -path to match full-path names 
find . -path *synx* -print

### match by regex 
find . -regex/-iregex "<grep-style-pattern>" -print 

### negating patterns

find . ! -name <pattern> -print 

### restrict search depth by -maxdepth and -mindepth
find . -maxdepth 2 -name <pattern> -print 

### timestamp options -atime -mtime -ctime (access, modify, change) 
### use + and - to denote more and less than XXX days 

find -name <pattern> -atime +4 -print ## look for files of pattern that are more than 4 days old 

delete files based on file matches

find -name <pattern> -type f -delete 

executing commands or actions with found files

use the find ... -exec <command> {} \; to execute some command using found files as args. {} is in-place placeholder for each matched file. -exec only takes a single command

Example use: format file name output with printf

find -name <pattern> -type f -exec printf "Filename: %s \n" {} \;

skip directories for find using '-prune' option

### print files but not .git files 

find path/to/src \( -name "*.git" -prune \) -o \( -type f -print \)

'xargs'

xargs takes stdin data stream and parse them as command arguments separated by default delimiter (white-space). For instance, to convert a multi-line file to a single-line file: cat file.txt | xargs

To use a given delimiter, use -d option.

echo 'splitXsplitXsplit' | xargs -d X 

>>> split split split

to run a command with X args per each execution, use `-n'

INPUT | xargs -n X 

When we also need some constant parameters/options along with the args from the INPUT, we need '-I' to set the replacement string for the inbound argument. e.g., -I {} sets {} as placeholder for args. When used with -I, the command is run like a loop; each time {} is replaced by the argument one by one.

cat args.txt | xargs -I {} <command> -o1 -o2 {} 

NOTICE: when use xargs with find , use the find option -print0 and xargs option -0 to set '\0' as delimiter and resolve filenames containig blanks.

remove all txt files

find . -name '*.txt' -type f -print0 | xargs -0 rm -f

count total number of lines in C source files

find path/to/src/-name '*.c' -type f -print0 | xargs -0 wc -l 

'tr' for character set mapping

'tr' only takes stdin and prints to stdout the result. format: tr [options] set1 set2

convert tabs to spaces: cat file | tr '\t' ' '

delete characeters using -d cat file | tr -d '[set1]'

squeeze character set (remove continuous repeating characters) by -s

### squeeze spaces in a line
> echo 'i have     many       spaces  x' | tr -s ' ' 
>>> i have many spaces x 

trick to add a list of numbers in a file (one number per line)

>cat file
1
2
3
4
>cat file | echo $[ $(tr '\n' '+')  0 ]
>>> 10 

'sort' and 'uniq'

sort can take multiple file inputs and sort them together. sort <file1> <file2> ... -o <outfile>

various uses of sort:

## sort by number
sort -n file
## reverse sort
sort -r file
## check if sorted
sort -C file ; [ $? -eq 0 ] && echo 'sorted' || echo 'not sorted' 

## show only unique lines 
sort -u unsorted 

## specify key column number 

sort -nk 2 file 

## merge two sorted files
sort -m sorted1 sorted2 

## ignore leading blanks and use dictionary order

sort -b -d unsorted.txt 

sometimes, we need to use a character range as key when keys are not columns. e.g.

file:
1010akdjfklajd
2030alkdjfkjgu
3212iuelkjkd

## start-pos,end-pos = 1,2
sort -nk 1,2 file 

NOTE: use '-z (zero)' option when used along 'xargs -0'

uniq only works for sorted input

Useful options of uniq are '-u' (unique), '-d' (duplicates) '-c' (count)

## show only unique lines
uniq -u sorted.txt 
## show duplicate lines
uniq -d sorted.txt

to specify compare-keys, use -s and -w

-s N' to skip first N characters '-w Width gives maximum number of characters to compare

temporary file names

use process ID as suffix of a temp file. temp_file="/tmp/file_.$$" '.$$' expands to current script process ID.

generate a zero-filled binary file of given size

dd if=/dev/zero bs=100k count=1 of=file.data

above generates a file of all zeros of size 100kb.

_Rename and move files in batch

substring removal: prefix-remove #, suffix removal % substring replacement: ${s/match/repl} or greedy form: ${s//match/repl}

Automate interactive user input using 'expect'

refer to using expect scripts to automate

Chapter 3 File In and Out

Use 'pushd' and 'popd' to navigate directories

pushd <dir> pushes the dir onto stack and switch to it.

dirs lists the indexed-content of the stack

pushd +<no> rotate the stack and switch to the indexed directory

popd pop the stack and switch to the popped directory

popd +<no> remove given dir from stack and switch to it.

---

Make a file immutable

(sudo) chattr +i <file> ## make immutable
(sudo) chattr -i <file> ## reverse, mutable again

Finding differences between files and patching

## produce unified patch for readability

diff -u version1.c version2.c > file.patch 

## apply patch 

patch -p1 version1.c < file.patch

## generating diff against directories (recursive)
diff -Naur dir1 dir2