a tour of command line tools not installed by default but available in the package manager


general | editors | version control | programming | debugging | images | documents | scientific | web | cloud | cryptography | compression | databases | appendix

Installation on Mac

for 10.13 (High Sierra)

Use the App Store to install Xcode.

Install Homebrew.

$ brew install ant apache-spark asciidoc aspell automake awscli bash bison check cmake colordiff \
  coreutils cppcheck ctags cvs ditaa elasticsearch emacs flex ffmpeg gawk global glpk \
  gnu-sed gnupg gnuplot gradle graphviz hexedit hive htop imagemagick ivy jenkins jq librsvg lzop m4 makedepend mariadb \
  markdown maven mercurial mongodb nmap openssl packer pandoc parallel pari pig pkg-config \
  postgresql potrace presto pssh pup pv python3 qpdf rcs redis rlwrap screenfetch shellcheck splint sqlite tcpflow \
  the_silver_searcher tig tmux tree webp w3m xpdf xz

Some of the above formulas install servers. If you want to have any of these servers run at login, follow the instructions at:

$ brew info elasticsearch jenkins mariadb mongodb postgresql presto redis

Some Mac apps can also be installed with brew:

$ brew tap caskroom/cask
$ brew cask install 1password blender chromedriver divvy docker emacs firefox flip4mac google-chrome java java8 \
  kindle mactex r-app rstudio screenflow slack wireshark xquartz

After Xcode is installed, gcc is actually clang. Notes on installing the real GCC on Mac?

To get java use brew cask install java.

The brew formula for node will install the current version. To get LTS node and npm, download the installer or build from source. The installer will run as root, but building from source takes 15 minutes. node and npm are the only executables the installer puts in /usr/local/bin.

$ pip3 install mathics virtualenv

Installation on Ubuntu

To install on Ubuntu 18.04 LTS:

$ sudo apt-get install -y ant apt-file asciidoc aspell automake awscli bison check chromium-chromedriver \
  cmake colordiff cppcheck cssc curl cvs ditaa emacs exuberant-ctags flex ffmpeg gcc gdb git global glpk-utils gnuplot5 \
  gradle graphviz hexedit htop imagemagick ivy jq libcppunit-dev librsvg2-bin libtool m4 make markdown maven mercurial \
  mongodb-clients mysql-client nano nmap nmon openjdk-8-jdk openssl pandoc parallel pari-gp pkg-config \
  postgresql-client potrace pssh pv python3.6-dev qpdf rcs redis-tools rlwrap screenfetch shellcheck silversearcher-ag \
  splint sqlite3 strace subversion tcpflow tig tmux tree valgrind vim virtualenv w3m webp xpdf xutils-dev

To install and run servers:

$ sudo apt-get install -y elasticsearch mariadb-server mongodb-server postgresql redis-server

GUI apps are also managed with apt:

$ sudo apt-get install -y chromium-browser
$ sudo apt-get remove -y openoffice*.* thunderbird unity-webapps-common libreoffice* software-center rhythmbox totem ubuntu-software

To get node and npm I've always downloaded and built from source. There are Linux binaries available for download, however.


brew package apt-get package pip package command line
ant ant ant
apache-spark pyspark
apt-file apt-file
asciidoc asciidoc asciidoc
aspell aspell aspell -a < PATH
aspell -c PATH
automake automake autoconf
awscli awscli aws
bash bash
bison bison bison
check check
chromedriver chromium-chromedriver chromedriver
cmake cmake cmake
checkstyle checkstyle checkstyle
colordiff colordiff colordiff
cppcheck cppcheck cppcheck
coreutils gfactor
cppunit libcppunit-dev
cssc sccs
ctags exuberant-ctags ctags
curl curl
cvs cvs cvs
dash dash dash
ditaa ditaa ditaa
docker-machine docker
emacs emacs emacs
flex flex flex
ffmpeg ffmpeg ffmpeg
gawk gawk
gcc gcc
gdb gdb gdb
git git
global global global
glpk glpk-utils glpsol
gnu-sed gsed
gnupg gnupg gpg2
gnuplot gnuplot5 gnuplot
gradle gradle gradle
graphviz graphviz dot
hexedit hexedit hexedit
hive hive
htop htop htop
imagemagick imagemagick composite
ivy ivy ivy
openjdk-8-jdk jar
jenkins jenkins jenkins
jq jq jq
librsvg rsvg-convert
libtool libtool
m4 m4 m4
make make
makedepend xutils-dev makedepend
markdown markdown markdown
mathics mathics
maven maven mvn
mercurial mercurial hg
mariadb mariadb-cient mysql
mongodb mongodb-clients mongoexport
nano nano
nmap nmap nmap
nmon nmon
openssl openssl openssl
packer packer
pandoc pandoc pandoc
parallel parallel parallel
pari pari-gp gp
pig pig
pkg-config pkg-config pkg-config
postgresql postgresql-client psql
potrace potrace potrace
pmd pmd
presto presto
pssh pssh pssh
pup pup
pv pv pv
qpdf qpdf qpdf
rcs rcs rcs
redis redis-tools redis-cli
rlwrap rlwrap rlwrap
screenfetch screenfetch screenfetch
shellcheck shellcheck shellcheck
splint splint splint
sqlite sqlite3 sqlite3
strace strace
subversion svn
tcpflow tcpflow tcpflow
the_silver_searcher silversearcher-ag ag
tig tig tig
tmux tmux tmux
tree tree tree
valgrind valgrind
vim vim
w3m w3m w3m
webp webp webp
xpdf xpdf pdftotext
xz xz-utils unxz


ag | bash | colordiff | coreutils | docker | emacs | git | jenkins | m4 | make | nano | parallel | pv | rlwrap | screenfetch | tree


ag is a C implementation of the Perl script ack.

ag is similar to grep -r PATTERN $PWD. However, it will also ignore files listed in any .gitignore or .hgignore files it finds. It will ignore binary files and hidden files; i.e. leading . in the name.

An {{.agignore}} file can be created to ignore additional files. Also, the {{--ignore flag can be given a file name pattern.

Use the -a flag to disable all of the above rules for ignoring files.

The ag flags -A, -B, -c, -C, -F, -i, -l, -L, -o, -v, -w have the same effect as the corresponding flags for grep.


Many of the command line tools on Mac are BSD versions. If you are used to the GNU/Linux versions, some flags you expect might be missing.

Installing coreutils on Mac will install command line tools with a g prefix to avoid conflicts: e.g. gcut, gecho, gjoin, ...

The {{gsort}} command has some features which the {{sort}} installed on Mac does not. See the {{-R}} flag for randomly shuffling a file, and the {{--parallel flag for using multiple cores when sorting.

todo: what about gawk and gnu-sed






m4 is a language neutral solution for substituting values into templates. Here is an example:

$ cat ls-home.m4
cd HOME && ls

$ m4 -DHOME=$HOME ls-home.m4 > ls-home

$ chmod +x ls-home

You might think that sed could do the same thing. However this doesn't work:

$ sed s/HOME/$HOME/ < ls-home.m4

The problem is after the shell expands $HOME, it probably has slashes in it, which are special to sed.

It is possible to prevent a string from getting expanded:

$ cat ls-home.m4
echo `HOME' is HOME
cd HOME && ls

$ m4 -DHOME=$HOME ls-home.m4 > ls-home

$ chmod +x ls-home

You might be tempted to use the C preprocessor, i.e. cpp or gcc -E, but the C preprocessor tends to raise errors for text which doesn't conform to the rules of C code.

The ` and ' are the string delimiting characters in M4. You can escape them so by doubling them, e.g.


will expand to


Escaping just one of the characters is harder. You must temporarily change the string delimiting characters:


will expand to


Note how "foo" must be quoted so the changequote builtin will be recognized.

Instead of passing a substitution value on the command line, one could define it in the file:

$ cat ls-clark.m4
define(`dir', `/Users/clark')
ls dir

$ m4 ls-clark.m4 > ls-clark
$ chmod +x ls-clark
$ ./ls-clark

macros with arguments and recursion

Built-in macros:

  • define
  • eval
  • format
  • ifdef
  • ifeq

loops, including files, debugging


make is not installed by default on Ubuntu.


The version of make that one gets on a Mac when installing the developer tools is 3.81, which is over ten years old. Installing make with brew puts an executable named gmake in the the search path.

Version 3.82 of GNU make added the .SHELLFLAGS and .ONESHELL special variables.

Version 4.0 of GNU make added $(guile ...) variable function for executing Guile and the $(file ...) variable function for writing a value to a file. The Guile extension language is a compilation option, and Homebrew does not install a version of make with it enabled.


Generate 16 million random numbers:

$ yes | head -16000000 | awk '{print rand()}' > rand.txt

Generate 16 million numbers more quickly by forking off 16 separate pipelines:

$ for i in $(seq 0 15); do yes | head -1000000 | awk '{print rand()}' > rand.$i.txt & ; done

The example executes a process for every word in a string. It is also possible to execute a process for every line in a file:

$ while read line; do echo $line & ; done < /etc/passwd

With xargs we can invoke 10 processes, but never execute more than 3 processes at the same time:

$ seq 11 20 | xargs -n 1 -P 3 sleep

The -P flag limits the number of concurrent processes. The -n flag controls the number of arguments passed to each command.

The -I flag can be used to defined a string in the command which is replaced by the arguments. This can be used to illustrate the -n flag:

$ echo foo bar baz | xargs -n 1 -I {} echo wombat {} wumpus
wombat foo wumpus
wombat bar wumpus
wombat baz wumpus

$ echo foo bar baz | xargs -I {} echo wombat {} wumpus
wombat foo bar baz wumpus

If -n is not specified, xargs passes as many arguments to the command as it can without exceeded ARG_MAX.

parallel is a Perl script which acts as a drop in replacement for xargs. It has many of the same flags as xargs, but with better default values. For example the default value for -n is 1:

$ echo $'foo\nbar\nbaz' | parallel echo wombat {} wumpus
wombat foo wumpus
wombat bar wumpus
wombat baz wumpus

$ echo $'foo\nbar\nbaz' | parallel -n 3 echo wombat {} wumpus
wombat foo bar baz wumpus

Also, the default value for {{-P}} is the number of available cores, whereas the default value of {{-P}} for {{xargs}} is 1. Also, it is not necessary to use the {{-I}} flag with {{parallel}} to define {{{} as the argument replacement string.

parallel has many options and is perhaps a good choice for parallelizing work, but don't forget about make -j NUM which is analogous to xargs -P NUM and parallel -P NUM.


A way to check up on the throughput one is seeing on a pipe:

$ yes | pv | cat > /dev/null


Here are a few tools without readline built in:

$ rlwrap ocaml
$ rlwrap racket
$ rlwrap sbcl


$ screenfetch
                 -/+:.          clark@ClarkMac
                :++++.          OS: 64bit Mac OS X 10.12.5 16F73
               /+++/.           Kernel: x86_64 Darwin 16.6.0
       .:-::- .+/:-``.::-       Uptime: 22d 19h 21m
    .:/++++++/::::/++++++/:`    Packages: 217
  .:///////////////////////:`   Shell: zsh 5.2
  ////////////////////////`     Resolution: 2880x1800
 -+++++++++++++++++++++++`      DE: Aqua
 /++++++++++++++++++++++/       WM: Quartz Compositor
 /sssssssssssssssssssssss.      WM Theme: Blue
 :ssssssssssssssssssssssss-     Font: SFMono-Regular
  osssssssssssssssssssssssso/`  CPU: Intel Core i7-6920HQ @ 2.90GHz
  `syyyyyyyyyyyyyyyyyyyyyyyy+`  GPU: AMD Radeon Pro 460 / Intel HD Graphics 530
   `ossssssssssssssssssssss/    RAM: 10092MiB / 16384MiB
$ screenfetch
                          ./+o+-       clark@ubuntu
                  yyyyy- -yyyyyy+      OS: Ubuntu 16.04 xenial
               ://+//////-yyyyyyo      Kernel: x86_64 Linux 4.8.0-51-generic
           .++ .:/++++++/-.+sss/`      Uptime: 13d 14h 51m
         .:++o:  /++++++++/:--:/-      Packages: 1855
        o:+o+:++.`..```.-/oo+++++/     Shell: bash 4.3.48
       .:+o:+o/.          `+sssoo+/    Resolution: 1680x1050
  .++/+:+oo+o:`             /sssooo.   DE: Unity 7.4.0
 /+++//+:`oo+o               /::--:.   WM: Compiz
 \+/+o+++`o++o               ++////.   WM Theme: Ambiance
  .++.o+++oo+:`             /dddhhh.   GTK Theme: Ambiance [GTK2/3]
       .+.o+oo:.          `oddhhhh+    Icon Theme: ubuntu-mono-dark
        \+.++o+o``-````.:ohdhhhhh+     Font: Ubuntu 11
         `:o+++ `ohhhhhhhhyo++os:      CPU: 2x Intel Core i7-6920HQ CPU @ 2.903GHz
           .o:`.syhhhhhhh/.oo++o`      RAM: 2316MiB / 3932MiB
                   ````` +oo+++o\:


Try this to get an HTML page for browsing Python code:

$ tree -I '*.pyc' -H $(pwd) . > ~/python_file.html

Selected options:

$ tree --help
  ------- Listing options -------
  -d            List directories only.
  -l            Follow symbolic links like directories.
  -f            Print the full path prefix for each file.
  -x            Stay on current filesystem only.
  -L level      Descend only level directories deep.
  -I pattern    Do not list files that match the given pattern.
  -------- File options ---------
  -p            Print the protections for each file.
  -u            Displays file owner or UID number.
  -g            Displays file group owner or GID number.
  -s            Print the size in bytes of each file.
  -h            Print the size in a more human readable way.
  -D            Print the date of last modification or (-c) status change.
  -F            Appends '/', '=', '*', '@', '|' or '>' as per ls -F.
  --inodes      Print inode number of each file.
  --device      Print device ID number to which each file belongs.
  ------- Sorting options -------
  -t            Sort files by last modification time.
  -r            Reverse the order of the sort.
  ------- Graphics options ------
  -i            Don't print indentation lines.
  ------- XML/HTML/JSON options -------
  -X            Prints out an XML representation of the tree.
  -J            Prints out an JSON representation of the tree.
  -H baseHREF   Prints out HTML format with baseHREF as top directory.


emacs | nano | hexedit | vim

"text mode" editors which can be run inside a terminal or on the far side of an ssh connection



Mac OS ships with Emacs, but it is an old version.


A binary file editor.

F1 for help.


Nano is easier to learn than emacs or vim because it displays the available commands at the bottom of the screen.


A "multi-modal" editor.

Version Control


A colorized wrapper to diff. It uses the same flags as diff.

$ sed 's/root/ROOT/' < /etc/passwd > /tmp/passwd
$ colordiff /etc/passwd /tmp/passwd



git is not installed by default on Ubuntu.








c/c++ | java | javascript | python | shell



  • autoconf
  • automake
  • bison
  • check
  • cmake
  • cppcheck
  • cppunit
  • ctags
  • flex
  • gcc
  • libtool
  • makedepend
  • pkg-config
  • splint
  • valgrind


$ find . -name '*.[ch]' | gtags -f -
$ global -x main
$ global -xr main

pkg-config is used like this:

$ gcc -o test test.c $(pkg-config --libs --cflags libpng)

The package manager is supposed to put .pc files in a location that pkg-config knows about, either /usr/lib/pkgconfig or /usr/local/lib/pkgconfig.





$ npm install -g grunt
$ npm install -g gulp




shellcheck is a static analyzer for shell scripts. Implemented in Haskell. It uses the shebang line to determine the flavor

$ shellcheck foo.sh

The version of bash that ships with the Mac is old. So we use brew to install an up-to-date version, and we rely on the fact that /usr/local/bin is ahead of /bin in the PATH. Bash scripts should have a

#!/usr/bin/env bash


If you want a shell script to be portable, you can run {{bash}} with the {{--posix flag, but it is better to run the script with dash.

Only use external programs in this list:


Also, only use POSIX mandated flags for thos external programs:



gdb | htop | nmon | strace


$ gcc -g -o foo foo.c

Three ways to start gdb:

$ gdb CMD
$ gdb CMD PID
$ gdb CMD CORE

On Mac OS X, use lldb instead of gdb.

cmd gdb
help h
list l [first, last]
next statement n
step into function s
set breakpoint b [file:]line
list breakpoints i b
delete breakpoint d num
continue c
show backtrace bt
move up stack u
move down stack do
print expression p expr
(re)run r [arg1[, arg2 ...]]
quit debugger q


Some features require htop to be started as root.

Config file at ~/.config/htop/htoprc.

keys notes
help h ? F1
setup S
cursor control ↓ ↑ → ← F / 0-9 F: stay on process 0-9: search by pid
filter processes \ u \: by command name u: by user
tag processes <SPC> c U c: tag process and children
view toggles H t p + - H: threads p: path t: process tree: + -: show/hide tree parts
sort processes by resource P M T > I P: CPU
inspect process e s l e: environment s: strace l: lsof
control process k [ ] i a k: kill [ ]: nice i: i/o priority a: cpu affinity



Syscalls for a command to be launched; for a running process identified by a PID; also strace children; set size string arguments to be displayed:

$ strace ls
$ strace -p 7191
$ strace -f
$ strace -s120

Timestamp each syscall; give elapsed time for each syscall:

$ strace -ttt
$ strace -T

Summarize counts of syscalls at exit; summarize counts and also suppress normal syscall output:

$ strace -C ls
$ strace -c ls

There is a tool called dtruss, part of dtrace, which is similar in function to strace. To use it, System Integrity Protection must be disabled on the Mac, which involves rebooting in recovery mode (hold down ⌘R), type csrutil disable in the terminal, then reboot again.


For examples of using the image processing tools, see http://clarkgrubb.com/image-tools


A tool for editing video and audio.


The package provides composite, convert, montage, and identify, which do edits like what Photoshop can do.


A tool which can convert bitmap images to SVG.


A tool which can convert SVG images to a bitmap format.


Convert a webp file to a png.

$ dwebp foo.webp -o foo.png


asciidoc | aspell | markdown | pandoc | xpdf



A lightweight markup frontend for DocBook.

Create foo.html:

$ asciidoc foo.txt


Using aspell to interactively fix spelling errors. The original file is modified:

$ echo 'Four score and seven yeers ago' > /tmp/gettysburg.txt
$ aspell -c /tmp/gettysburg.txt

Using aspell non-interactively on a file stream:

$ echo 'Four score and seven yeers ago' | aspell list

$ echo 'Four score and seven yeers ago' | aspell -a
& yeers 85 21: years, yews, yes, year's, yew's, yeas, beers, jeers, leers, peers, seers, veers, yea's, yeahs, yes's, yours, eyes, yeses, Meyers, tees, Ayers, Byers, Myers, dyers, eye's, yaws, yer, queers, yrs, Yeats, yearns, beer's, deer's, jeer's, leer's, peer's, seer's, tee's, veer's, seer, yaw's, yeah's, year, yens, yeps, Yves, bees, ears, fees, gees, hers, lees, pees, sees, wees, yous, cheers, sheers, yeggs, yells, yen's, yep's, yetis, Lee's, bee's, fee's, lee's, pee's, see's, wee's, Meyer's, you's, Dyer's, Yeager's, dyer's, yore's, Er's, queer's, Meier's, ear's, her's, Cheer's, cheer's, sheer's, yegg's



A Perl script which converts Markdown to HTML.

$ echo '* foo' | markdown


A tool implemented in Haskell for converting Markdown to a variety of formats, including HTML:

$ echo '* foo' | pandoc


$ echo '* foo' | pandoc -o output.pdf

To man page:

$ cat > foo.1.md
% FOO(1) Foo User Manual
% John Smith
% November 8, 2012


foo - a command line tool


foo [*file*]...


Foo performs an operation on files specified on the command line.  If no files
are specified the operation is performed on *stdin*.


`bar` (1).

$ pandoc -s -w man foo.1.md -o foo.1

To HTML slideshow:

$ cat > gnomes.txt
# Business Plan

* collect underpants
* ?
* profit

# Phase 1: Collect Underpants

3:30 AM is the optimal time.

# Phase 2: ?

Details TBD

# Phase 3: Profit

Financial results will be discussed in the earnings call.

$ pandoc gnomes.txt -s --webtex -i -t slidy -o gnomes.html


$ qpdf --help
$ qpdf --stream-data=uncompress input.pdf output.pdf
$ qpdf --linearize input.pdf output.pdf


xpdf is an X Windows PDF viewer. Installing it also installs pdftotext which extracts plain text from a PDF.

$ pdftotext foo.pdf foo.txt



ditaa converts ASCII art to box-and-arrow diagrams such as produced by Visio and Omnigraffle.



gpsol solves linear programming problems.



gnuplot creates graphics such as histograms, box plots, scatterplots, and run charts.



dot converts DOT files, which are a notation for graphs, into PNG files.



mathematica comparied

Mathics supports a subset of the Mathematica language.



This package install a command line calculator with arbitrary length integers, rationals, arbitrary precision decimals, complex numbers, vectors, matrices, polynomials. Lots of built-in functions for number theory.

To use it interactively:

$ gp


$ echo '1 + 1' | gp -q



curl -d KEY=VALUE [-d KEY=VALUE ...]
curl -d @DATA_FILE
curl -e REFERER
curl -G      (send data as GET)
curl -H HEADER [-H HEADER ...]
curl -i      (put HTTP header in output)
curl -I       (HEAD request)
curl -l      (follow redirect)
curl --limit-rate BPS
curl -O      (name local file after path of remote file)
curl -s      (no progress bar or error messages)
curl -sS     (no progress bar, but error messages are shown)
curl -T FILE  (use PUT to upload FILE)


A port scanner.


Use CSS selectors to extract elements from HTML. Implemented in Go.


$ curl http://google.com | pup body


To dump the IP packets:

$ sudo tcpdump [-i INTERFACES] -w dump.pcap [PCAP_FILTER]
$ tcpdump [-v|-A] -r dump.pcap

The first command continues listening until interrupted with ^C.

Run ifconfig to see the available interfaces. INTERFACES can be a comma separated list. If none are specified, the default choice usually includes the main interface.

A filter can be used to reduce the number of packets captured. See man pcap-filter. The basic predicates of the filter language are:

[dst|src] host IP_ADDRESS
[dst|src] port PORT
not PRED
( PRED )

The file dump.pcap is in the pcap next generation capture file format which tcpflow can also read.

The-v flag prints two lines are printed per packet instead of one and includes information about the protocol (UDP, TCP).

The -A flag prints the payload of the protocol in ASCII.

tcpflow does a better job of assembling in data being transmitted in the protocols. It is invoked as:

$ tcpflow [-r PCAP_FILE | -i INTERFACE] [-o DIR] [-a]

tcpflow can read from a PCAP_FILE or an INTERFACE. If no DIR is specified, it creates files in the working directory.

tcpflow creates files with names of the form


identifying the TCP connection whose data it contains.


Start w3m:

$ w3m http://google.com

Use arrows to position the cursor. Type {{<ENTER>}} to click on the link below the cursor. If a form input element is below the cursor, click {{<ENTER> and then type text which will appear at the bottom of the screen. Type q to exit w3m.

w3m can be use to render an HTML document in plain text:

$ curl https://www.google.com > google.html
$ w3m -dump google.html


awsclil | packer | pssh | tmux


Much of what can be done in the AWS console can also be done at the command line.

To get set up, go to IAM in the AWS console and create your key pair. Then use this command to write them to your ~/.aws/credentials file.

$ aws configure

Copy an S3 bucket and its contents:

$ aws s3 cp --recursive s3://MY_BUCKET/images/ .

Getting help:

$ aws help
$ aws s3 help
$ aws s3 cp help


$ cat example.json
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "branch": "master"
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "us-west-2",
    "source_ami": "ami-efd0428f",
    "instance_type": "t2.micro",
    "ssh_username": "ubuntu",
    "ami_name": "ubuntu.16.04 python3.5 openmail {{timestamp}}"
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sleep 30",
      "cat /etc/*elease",
      "sudo apt-get update",
      "sudo apt-get install -y gcc g++ git make python3.5-dev virtualenv",
      "ssh-keyscan github.com >> ~/.ssh/known_hosts",
      "mkdir ~/.aws",
      "echo '[default]' > ~/.aws/credentials",
      "echo 'aws_access_key_id = {{user `aws_access_key`}}' >> ~/.aws/credentials",
      "echo 'aws_secret_access_key = {{user `aws_secret_key`}}' >> ~/.aws/credentials",
      "git clone git@github.com:THE_USER/THE_PROJECT.git",
      "cd THE_PROJECT",
      "git checkout {{user `branch`}}",
      "virtualenv -p python3.5 ../ve",
      ". ../ve/bin/activate",
      "pip install -r /tmp/requirements.txt",
      "make -k test"

$ packer validate example.json

$ packer build --var 'aws_access_key=YOUR ACCESS KEY' \
  -var 'aws_secret_key=YOUR SECRET KEY' \


Execute a command on multiple remote hosts:

pssh -i -A -H joe@host1 -H joe@host2 echo "hello world"

The -i flag causes stdtout and stderr to be displayed. The -A flag prompts for a password.


tmux and screen compared

When using ssh to connect to a remote host, if the network connection is lost, the shell on the remote host will be sent a SIGHUP. The shell will in turn send a SIGHUP to any jobs that are running. The default behavior is for the jobs to exit.

To prevent loss of network connection from interrupting a long running batch job, there a couple of things the user can do:

$ nohup sleep 1000 &

$ sleep 1000 &
[1]  + running    sleep 1000
$ disown %1

nohup writes stdout to the file nohup.out.

A tmux session will survive loss a network connection. Start a tmux session with:

$ tmux

If disconnected, log back in to the remote server and attach to the session with:

$ tmux attach




$ gpg2 --generate-key
$ gpg2 --list-keys
$ cp /etc/passwd passwd.txt
$ gpg2 --clear-sign passwd.txt
$ gpg2 --verify passwd.txt.asc


$ openssl dgst -sha256 < /etc/passwd
$ openssl base64 < /bin/ls



suffix compress paths uncompress paths uncompress stdin
.gz gzip gunzip gzcat
.bz2 bzip2 bunzip2 bzcat
.xz xz unxz xzcat
$ tar xf linux-3.17.1.tar.xz
$ find linux-3.17.1 -name '*.[ch]' | xargs cat > /tmp/big.txt
$ wc -c /tmp/big.txt
 482314125 /tmp/big.txt
$ time gzip /tmp/big.txt
gzip /tmp/big.txt  15.19s user 0.19s system 99% cpu 15.425 total
$ wc -c /tmp/big.txt.gz
 105127193 /tmp/big.txt.gz
$ time gunzip /tmp/big.txt.gz
gunzip /tmp/big.txt.gz  1.45s user 0.23s system 98% cpu 1.701 total
size ratio time untime
gzip 105127193 21.79% 15.425 1.701
bzip2 82311952 17.07% 43.773 16.617
xz 70984840 14.72% 3:42.77 6.308
zip 105138420 21.80% 15.632 3.294


and map/reduce tools





jq language syntax










$ brew tap dart-lang/dart
$ brew install cabal-install coq dart elixir erlang gforth ghc gnu-apl go groovy guile idris kotlin leiningen \
  lua maxima ocaml opam racket rust sbcl sbt scala scalastyle swi-prolog typescript

$ brew tap caskroom/cask
$ brew cask install pharo sage
brew package apt package command line
cabal-install cabal-install cabal install
clojure1.6 clojure
coq coq coqtop
dart dart
elixir elixir
erlang erlang erl
fish fish fish
gforth gforth gforth
ghc ghc ghc
gnu-apl apl
go golang go
groovy groovy2 groovysh
guile guile-2.0 guile
idris idris
kotlin kotlinc
leiningen lein repl
lua lua5.3 lua
maxima maxima maxima
ocaml ocaml ocaml
opam opam opam
perl perl
php php
racket racket racket
ruby ruby
rust rustc rustc
sbcl sbcl sbcl
sbt sbt
scala scala scala
scalastyle scalastyle
swi-prolog swi-prolog swipl
tcl tclsh
typescript node-typescript tsc