jcanfield
11/21/2016 - 8:55 AM

Unix Command Line Notes

Unix Command Line Notes

Unix Command Line Notes

  • Command settings

    • .bashrc
      • Aliases - place the following on a new line in the .bashrc file
        • alias [alias-name]='[alias command(s)]
        • Ex:alias ls='ls -lh'
    • .profile or .bash_profile
      • Add folders to you path
      • Use source .profile to reload the profile
    • history
      • By default, remembers the previous 100 commands you've typed
      • history will list the stored comands
      • ![number] will repeat one of the commands storied in the history
      • history | grep [search] will display only the commands in the history that match your search query
    • man [command] to get documentation for a particular command
      • q will exit out of manual pages
  • Manipulating files

    • pwd - print working directory
    • df - report on disk usage
    • ls /[directory-path] - list files within a directory
      • Specifying a directory is optional; will list the current directory by default
      • -a show hidden files and directories (all files)
      • -d gets directory information (in addition to the files in that directory)
      • -l long listing
      • -R list files recursively
      • -F classify files with special characters: display a / after each directory, @ after each link, and * after each executable (normal files have nothing special)
      • -s include the number of blocks the file takes up (for more accurate space indication of what the file actually uses on the disk)
      • -h for human readable file sizes
      • -e show ACLs, if present
      • -t shows files from most recently edited
      • −1 lists files in a single column
      • -r reverse the sort order
    • cp [filename/folder] [destination] - copies files
      • Automatically overwrites destination
      • -i asks for confirmation before overwriting existing files
      • -a preserve symbolic links (-d), recusively (-R), and reserve attributes (preserve=all)
    • rm [file/folder] - remove a file or directory
      • Normally, a directory must be empty to remove it
      • -f force removal
      • -r remove recursively (everything inside)
      • -i ask for confirmation
      • -d removes directories
      • -v for verbose output
    • mv [file] [new-file-name] or mv [file] [/new-destination] - renames or moves files or folders
    • du - space a file/directory takes up
    • touch [filename]
      • For existing files, updates the accessed time
      • If the file doesn't exist, it creates a new, blank file
    • cat [filename] - displays the contents of a file
    • Replace or add to files
      • "[contents]" > [filename] will create a new file or replace the existing file with the contents you specify
      • "[contents]" >> [filename] will append the contents to an existing file (or create a new file with those content)
    • Symlinks (for soft links, which can span partitions)
      • ln -s [/path/destination] [sym-link name]
    • rsync [options] [source] [destination]
      • Copies only the folders/files that have been changed from one location to another

      • Can be used over network connections (local <-> remote)

      • End the path with / to indicate copy just the contents of a folder, and not the folder itself

      • Remote destinations can be indicated through ssh user@server:/path/to/files

      • rsync to backup a Mac system volume

          sudo rsync -avPE --exclude "/Volumes" --exclude "/dev" --exclude ".MobileBackups.trash" / /destination
        
      • Options

        • -a archive mode: recursive, preserves symbolic links, preserves permissions, preserves timestamps, preserves owner and groups (generally a good idea to use this)
        • -E copy extended attributes, including resource forks and ACLs (mac only)
        • -e ssh indicates use a remote shell and use ssh for transfer
        • -h human readable output
        • -i view the differences (without modifying files) between destination and source
          • f indicates a file
          • s size change
          • t timestamp change
          • o owner change
          • g group change
        • -n or --dry-run to run through changes without modifying any files
        • -r recursive through folders
        • -u don't overwrite modified files in the destination
        • -v verbose, output each file
        • -W transfer the whole file that's been changed, instead of just the block(s) that have been changed
        • -z use compression (most useful for local to/from remote syncs)
        • Backups
          • --backup
          • --backup-dir
          • --suffix
        • --delete deletes files in the destination that are not in the source directory
        • --existing copy/update only the existing files in source to the destination
        • --include "[pattern]" or --exclude "[pattern]" to include/exclude certain files
          • Can be used multiple times to specify multiple files
          • --exclude-list will specify a text file with a list of exclude patterns, one on each line
        • --max-size="[size]" to set the max size of files to be transfered
        • --progress or -P to see the overall progress of the operation
  • Processes

    • ps show currently running processes
      • -a show processes from all users (with open terminals)
      • -f display extra information, like uid, pid, CPU usage, etc
      • -l long listing
      • -m sort by memory usage, rather than name/id
      • -r sort by current CPU usage
      • -u display processes belonging to the specified user names
      • -x display all processes, even if they don't have a controlling terminal
      • Use with | grep [process-name] to show only the processes you are looking for
    • top displays a dynamic updated list of running processes (default sorted by pid)
      • -o [key] changes the order of the process list; keys include
        • cpu cpu usage
        • uid user id
        • user user name
    • kill [process-id] will stop a process by its id
      • −9 will force kill it by sending KILL
      • -s to send a specific signal other than TERM
    • killall [process-name] kills all instances of a process by name
  • Packages

    • lsbom lists the files installed by a package
      • -f lists files
      • -l lists symbolic links
      • -s prints only the path of each file
      • -h prints the full usage of each file
    • tar [options] [source] [destination]
      • Types
        • tar is just a collection of files within a single tar file
        • tar.gz or .tgz is a compressed tar
          • Use -z option
          • Use tar then compress with gzip
      • tar -cjvf [file] [destination] creates a compressed tarfile
      • tar -tvf [file] lists contents without extracting the file
      • tar -xvf [file] extracts a file in the current directory
      • Options
        • -c create
        • -r replace
        • -x extract/restore
        • -w ask confirmation for each file
        • -v verbose
        • -j compress using bzip
        • -z compress using gzip
    • mpkg
      • xar -xvf [package] extracts a package file
      • xar -c [source] -vf [destination] compresses a package
      • Options
        • -c create/compress
        • -x extract
        • -v verbose
        • -f files
    • unzip [options] [zip-file-to-unzip] [list-of-specific-files-to-unzip] [-x files to exclude] [-d where to unzip files]
      • -f freshens existing files, extracting only files that already exist on disk and are newer
    • open [filename.zip] will unzip zip files as well
  • Directories

    • cd
    • cd will take you to the previous working directory
    • cd ~
    • pushd
    • popd
  • Permissions and ACLs

    • Permissions
      • d rwx rwx rwx
        • d directory, - file, l link, s socket file
        • r read (4)
        • w write (2)
        • x execute (1)
        • user (u)
        • group (g)
        • others/world (o)
      • Number of hardlinks
      • Owner and group
    • chmod [options] [permission] [filename] (to set permissions) or chmod [filename] (to see permissions)
      • Example: g+w adds write permissions to group
    • chown [user]:[group] [files]
      • Can enter user, group, or both, depending on which side of the : you specify
      • Options
        • -R recursive for all files and folders within a directory
        • -H will effect symbolic links if -R is set (not effected by default)
        • -h change the mode of the symbolic links, but not follow them
        • -L follow symbolic links (change all files and folders within links)
        • -P symbolic links are uneffected (this is the default)
        • -f do not display an error message
        • -v verbose
    • Change permissions of files or folders
      • Files only find . -type f -print0 | xargs -0 chmod 644
      • Folders only find . -type d -print0 | xargs -0 chmod 755
  • Clipboard

    • pbcopy - copies contents of file into your clipboard
      • Ex: pbcopy < path/file.txt
    • pbpaste - allows use of the clipboard in terminal
      • Ex: pbpaste > path/file.txt - writes the contents of the clipboard into a file
  • Searching files

    • grep -[flags] [search-pattern] [target-directory/file] (global regular expression print)
      • Used to search files with matching patterns
      • -R - search recursively
    • egrep
    • find [path] -[option] [query]
      • Time flags
        • By days
          • -mtime searches by modification time
            • find . -mtime 1 searches for all files modified exactly 1 day ago
            • find . -mtime -1 searches for all files modified within the last day
            • find . -mtime +1 searches for all files modified more than a day ago
          • -atime searches by last accessed time
          • -ctime searches by last changed time
        • By minutes
          • -cmin searches by last changed time in minutes
      • -perm searches by permission
        • find . -perm 644 finds all files with 644 permissions
      • -name
        • -iname is case insensitive
      • -delete will delete the files found using the find command
      • -type
        • find [path to install] -type d -exec chmod 755 {} \; will find all directories and change their permissions to 755
        • find [path to install] -type f -exec chmod 644 {} \; will find all files and change their permissions to 644
      • xargs
        • Used to execute other file manipulation commands on the results of a find query
        • Place after a find command and separate with |
        • find . -name "*.tmp" | xargs rm -f to remove .tmp files
        • find . -name "*.txt" | xargs grep "text" will find all text files that contain a certain text
  • Search and replace

    • sed 's/[regex-to-replace]/replacement/ oldfile > newfile' - search and replace
      • Default behavior is to change only the first matching occurrence of the regex on each line
      • / is the default delimiter; any character can be used, and it's best to use a character that's not in the search or replace string (: and | are popular)
      • & can be used in the replacement string; it represents what is found in the search string
      • Add g after the replacement delimiter to replace every occurrence of the matched expression (only replaces the first occurance on each line, by default)
      • -i 'filename' to output directlyi to a file - use -i '' on osx if you want to overwrite the original file
    • perl
      • From the command line: perl -i -pe 's/search/replace/g' [filename]
        • -i read from file; use -i.bak or similar to create a backup
        • -e define perl code to be executed (in '')
        • -p adds loops around -e code
        • Search and replace options (after the s/search/replace/)
          • g - replace all occurrences of the pattern (not just the first)
          • m - search across line breaks
          • s - include line breaks as any other letter
          • i - case insensitive search
        • Search across multiple lines: perl -i -pe 'undef $/; s/search/replace/smg' [filename]
  • File comparison

    • diff [file1] [file2] to compare files line by line
      • -u unified output (use this unless you have a reason not to)
      • -i ignore case
      • -E ignore changes due to tab extension
      • -b ignore whitespace changes
      • -w ignore all white space
      • -B ignore all lines that are blank
    • Creating patches for files
      • Use diff to create a patch: diff -u [original-file] [new-file] > file.patch
      • Use patch to revert the new file to the original file: patch -R < file.patch
        • -R will reverse the patch
        • -b makes backups
  • Nano text editor - nano [filename]

    • -w start nano with line wrapping off (important for configuration files)
    • Enable mouse services - Alt+M
    • Search
      • Ctrl + w search within a file
      • Alt + w find next
    • Cut/past
      • Ctrl + k cut line
      • Ctrl + u paste line
      • To cut and paste several lines, cut them successively with Ctrl + k then paste all with Ctrl + u
  • Networking

    • hostname - find host/domain name and IP address
      • Displays the local hostname if not given any options
      • -d displays the domain name the machine belongs to
      • -f displays the fully qualified host and domain name
      • -i displays the IP address for the current machine
    • ping [hostname] - test network connection by sending packets of information to the defined source (also used for testing speed of connection)
      • Prefix with do to begin continuous test; use Ctrl + C to stop the test
    • ifconfig - get current network adapter configuration
    • netstat - network connections, including routing tables, interface statistics
      • -g find all the multicast groups (network) subscribed to this host
      • netstat -nap | grep [port-number] - display process id of application which is using the specified port
      • -a or -all will display all connections including TCP and UDP
      • -r for routing tables
      • --tcp or -t will display only TCP connections
      • --udp or -us will display only UCP connections
    • nslookup [hostname or ip address] - query DNS lookup name
      • Gets hostname or ip address from given ip address or hostname
    • telnet [hostname] [port] - check status of destination host
    • traceroute - view number of hops and response time to get to a remote system or web site
    • finger - view user information (user name, real name, terminal name, and write status); old command, rarely used
    • route -v add -net 10.208 -interface ppp0
      • Changes routing tables to route any ip addresses beginning with 10.208 through the VPN connection (PPP0)
    • curl - get remote files
      • -O [url] downloads a file and writes it with the same name
      • -o [filename] [url] downloads a file and lets you specify the name
    • lsof -i scan active ports for local machine
    • iptables
  • Checksums

    • md5 [filename]
    • shasum [filename] (for SHA1)
      • -a to change to 224, 256, 384, or 512
  • Efficiency tips

    • Use ![command] to quickly redo the previous command (including flags and arguments)
    • Use !! for executing the previous command exactly as typed
    • Ctrl + R + [search] will search for the last command that contains whatever words you type
    • history | grep "[keyword]" searches the UNIX command history
    • Use | (pipe symbol) to separate multiple commands on the same line
    • Set up aliases in .bashrc or .profile
    • Suspending processes
      • Ctrl + Z to suspend a process
      • fg [number] or bg [number] to bring suspended processes to the foreground or background
  • Users

    • adduser [username] creates new users and prompts you for the details
    • useradd creates new users and lets you manually specify each option
      • -d /home/username specifies the location of the home directory
      • -m forces the creation of the home directory
      • -g/--gid [group] lets you specify which group to add the user to (check adduser.conf to see what the default is)
      • -G/--groups [group1,group2,etc] for supplementary groups
      • --disabled-password prevents password logins (as for when only using SSH keys)
    • passwd [user] sets the password for a user -l changes the user's password to an untypable string so it can't be used to login
    • /etc/adduser.conf
      • Stores user configuration settings
      • Ubuntu defaults (change if desired)
        • All users added to their own group
        • Home user directory permissions set to 755
    • su [username] will switch to a particular user (useful to become the root user)
    • Get a list of users - cat /etc/passwd
    • Groups
      • usermod [options] [username] modifies user information
        • -g [group] change the user's initial login group
        • -a -G [group1,group2,etc] [user] will add a user to supplementary groups (the -a adds the user to those additional groups; without it, the user will be removed from any groups not specified
        • -c "Desired Name" [user] to change a users full name
      • deluser [user] [group] will delete that user from the group
      • groups [user] will display a list of groups for that user
      • groupadd and groupdel to add and remove groups
    • Sudo
      • Use sudo visudo to configure which users can use sudo
    • User shells
      • The file /etc/passwd lists each user and determines which shell they run when logged in