Installed Command Line Tools

a tour of command line tools not installed by default but available in the package manager

http://clarkgrubb.com/package-managers

general | editors | version control | programming | debugging | images | documents | scientific | web | cloud | cryptography | compression | databases | appendix

Installation on Mac

for 10.12 (Sierra)

Use the App Store to install Xcode.

Install Homebrew.

$ brew install ant apache-spark asciidoc aspell automake awscli bash bison check chromedriver cmake colordiff \
  coreutils cppcheck ctags cvs ditaa elasticsearch emacs flex ffmpeg gawk global homebrew/science/glpk \
  gnu-sed gnupg gnuplot gradle graphviz hexedit hive htop imagemagick ivy jenkins jq librsvg m4 makedepend mariadb \
  markdown maven mercurial mongodb nmap nmon openssl packer pandoc parallel pari pig pkg-config \
  postgresql potrace presto pssh pup pv python3 qpdf rcs redis rlwrap screenfetch shellcheck splint sqlite tcpflow \
  the_silver_searcher tig tmux valgrind w3m xpdf xz

Some of the above formulas install servers. If you want to have any of these servers run at login, follow the instructions at:

$ brew info elasticsearch jenkins mariadb mongodb postgresql presto redis

Some Mac apps can also be installed with brew:

$ brew tap caskroom/cask
$ brew cask install 1password amazon-music divvy docker dropbox emacs firefox flip4mac google-chrome java \
  kindle mactex r-app rstudio screenflow skype slack wireshark xquartz xscope

After Xcode is installed, gcc is actually clang. Notes on installing the real GCC on Mac?

To get java use brew cask install java.

The brew formula for node will install the current version. To get LTS node and npm, download the installer or build from source. The installer will run as root, but building from source takes 15 minutes. node and npm are the only executables the installer puts in /usr/local/bin.

$ pip3 install mathics virtualenv

Installation on Ubuntu

To install on a Debian-based system such as Ubuntu 16.04 LTS:

$ sudo apt-get install -y ant apt-file asciidoc aspell automake awscli bison check chromium-chromedriver \
  cmake colordiff cppcheck cssc curl cvs ditaa emacs exuberant-ctags flex ffmpeg gcc gdb git global glpk-utils gnuplot5 \
  gradle graphviz hexedit htop imagemagick ivy jq libcppunit-dev librsvg2-bin libtool m4 make markdown maven mercurial \
  mongodb-clients mysql-client nano nmap nmon openjdk-8-jdk openssl pandoc parallel pari-gp pkg-config \
  postgresql-client potrace pssh pv python3.5-dev qpdf rcs redis-tools rlwrap screenfetch shellcheck silversearcher-ag \
  splint sqlite3 strace subversion tcpflow tig tmux valgrind vim virtualenv w3m xpdf xutils-dev

To install and run servers:

$ sudo apt-get install -y elasticsearch mariadb-server mongodb-server postgresql redis-server

GUI apps are also managed with apt:

$ sudo apt-get install -y chromium-browser
$ sudo apt-get remove -y openoffice*.* thunderbird unity-webapps-common libreoffice* software-center rhythmbox totem ubuntu-software

To get node and npm I've always downloaded and built from source. There are Linux binaries available for download, however.

Summary

brew package apt-get package pip package command line
ant ant ant
apache-spark pyspark
spark-shell
spark-submit
sparkr
apt-file apt-file
asciidoc asciidoc asciidoc
aspell aspell aspell -a < PATH
aspell -c PATH
automake automake autoconf
automake
awscli awscli aws
bash bash
bison bison bison
check check
chromedriver chromium-chromedriver chromedriver
cmake cmake cmake
checkstyle checkstyle checkstyle
colordiff colordiff colordiff
cppcheck cppcheck cppcheck
coreutils gfactor
gsort
cppunit libcppunit-dev
cssc sccs
ctags exuberant-ctags ctags
curl curl
cvs cvs cvs
dash dash dash
ditaa ditaa ditaa
docker-machine docker
elasticsearch
emacs emacs emacs
flex flex flex
ffmpeg ffmpeg ffmpeg
gawk gawk
gcc gcc
gdb gdb gdb
git git
global global global
gtags
glpk glpk-utils glpsol
gnu-sed gsed
gnupg gnupg gpg2
gnuplot gnuplot5 gnuplot
gradle gradle gradle
graphviz graphviz dot
hexedit hexedit hexedit
hive hive
htop htop htop
imagemagick imagemagick composite
convert
identify
montage
ivy ivy ivy
openjdk-8-jdk jar
java
javac
jenkins jenkins jenkins
jq jq jq
librsvg rsvg-convert
libtool libtool
m4 m4 m4
make make
makedepend xutils-dev makedepend
markdown markdown markdown
mathics mathics
maven maven mvn
mercurial mercurial hg
mariadb mariadb-cient mysql
mongodb mongodb-clients mongoexport
mongoimport
mongo
nano nano
nmap nmap nmap
nmon nmon nmon
node
npm
openssl openssl openssl
packer packer
pandoc pandoc pandoc
parallel parallel parallel
pari pari-gp gp
pig pig
pkg-config pkg-config pkg-config
postgresql postgresql-client psql
potrace potrace potrace
pmd pmd
presto presto
pssh pssh pssh
pup pup
pv pv pv
python
python3
python3.5-dev
virtualenv
pip
python
virtualenv
qpdf qpdf qpdf
rcs rcs rcs
redis redis-tools redis-cli
rlwrap rlwrap rlwrap
screenfetch screenfetch screenfetch
shellcheck shellcheck shellcheck
splint splint splint
sqlite sqlite3 sqlite3
strace strace
dtruss
subversion svn
tcpflow tcpflow tcpflow
the_silver_searcher silversearcher-ag ag
tig tig tig
tmux tmux tmux
valgrind valgrind valgrind
vim vim
w3m w3m w3m
xpdf xpdf pdftotext
xz xz-utils unxz
xz
xzcat

General

ag | bash | colordiff | coreutils | docker | emacs | git | jenkins | m4 | make | nano | parallel | pv | rlwrap | screenfetch | vim

ag

ag is a C implementation of the Perl script ack.

ag is similar to grep -r PATTERN $PWD. However, it will also ignore files ilisted n any .gitignore or .hgignore files it finds. It will ignore binary files and hidden files; i.e. leading . in the name.

An .agignore file can be created to ignore additional files. Also, the --ignore flag can be given a file name pattern.

Use the -a flag to disable all of the above rules for ignoring files.

The ag flags -A, -B, -c, -C, -F, -i, -l, -L, -o, -v, -w have the same effect as the corresponding flags for grep.

coreutils

Many of the command line tools on Mac are BSD versions. If you are used to the GNU/Linux versions, some flags you expect might be missing.

Installing coreutils on Mac will install command line tools with a g prefix to avoid conflicts: e.g. gcut, gecho, gjoin, …

The gsort command has some features which the sort installed on Mac does not. See the -R flag for randomly shuffling a file, and the --parallel flag for using multiple cores when sorting.

todo: what about gawk and gnu-sed

docker

http://clarkgrubb.com/docker

jenkins

m4

https://www.gnu.org/software/m4/manual/m4.html

m4 is a language neutral solution for substituting values into templates. Here is an example:

$ cat ls-home.m4 
#!/bin/sh
cd HOME && ls

$ m4 -DHOME=$HOME ls-home.m4 > ls-home

$ chmod +x ls-home

You might think that sed could do the same thing. However this doesn't work:

$ sed s/HOME/$HOME/ < ls-home.m4

The problem is after the shell expands $HOME, it probably has slashes in it, which are special to sed.

It is possible to prevent a string from getting expanded:

$ cat ls-home.m4 
#!/bin/sh
echo `HOME' is HOME
cd HOME && ls

$ m4 -DHOME=$HOME ls-home.m4 > ls-home

$ chmod +x ls-home

You might be tempted to use the C preprocessor, i.e. cpp or gcc -E, but the C preprocessor tends to raise errors for text which doesn't conform to the rules of C code.

macros with arguments and recursion

Built-in macros:

  • define
  • eval
  • format
  • ifdef
  • ifeq

loops, including files, debugging

make

make is not installed by default on Ubuntu.

http://clarkgrubb.com/makefile-style-guide

The version of make that one gets on a Mac when installing the developer tools is 3.81, which is over ten years old. Installing make with brew puts an executable named gmake in the the search path.

Version 3.82 of GNU make added the .SHELLFLAGS and .ONESHELL special variables.

Version 4.0 of GNU make added $(guile …) variable function for executing Guile and the $(file …) variable function for writing a value to a file. The Guile extension language is a compilation option, and Homebrew does not install a version of make with it enabled.

parallel

Generate 16 million random numbers:

$ yes | head -16000000 | awk '{print rand()}' > rand.txt

Generate 16 million numbers more quickly by forking off 16 separate pipelines:

$ for i in $(seq 0 15); do yes | head -1000000 | awk '{print rand()}' > rand.$i.txt & ; done

The example executes a process for every word in a string. It is also possible to execute a process for every line in a file:

$ while read line; do echo $line & ; done < /etc/passwd

With xargs we can invoke 10 processes, but never execute more than 3 processes at the same time:

$ seq 11 20 | xargs -n 1 -P 3 sleep

The -P flag limits the number of concurrent processes. The -n flag controls the number of arguments passed to each command.

The -I flag can be used to defined a string in the command which is replaced by the arguments. This can be used to illustrate the -n flag:

$ echo foo bar baz | xargs -n 1 -I {} echo wombat {} wumpus
wombat foo wumpus
wombat bar wumpus
wombat baz wumpus

$ echo foo bar baz | xargs -I {} echo wombat {} wumpus 
wombat foo bar baz wumpus

If -n is not specified, xargs passes as many arguments to the command as it can without exceeded ARG_MAX.

parallel is a Perl script which acts as a drop in replacement for xargs. It has many of the same flags as xargs, but with better default values. For example the default value for -n is 1:

$ echo $'foo\nbar\nbaz' | parallel echo wombat {} wumpus
wombat foo wumpus
wombat bar wumpus
wombat baz wumpus

$ echo $'foo\nbar\nbaz' | parallel -n 3 echo wombat {} wumpus
wombat foo bar baz wumpus

Also, the default value for -P is the number of available cores, whereas the default value of -P for xargs is 1. Also, it is not necessary to use the -I flag with parallel to define {} as the argument replacement string.

parallel has many options and is perhaps a good choice for parallelizing work, but don't forget about make -j NUM which is analogous to xargs -P NUM and parallel -P NUM.

pv

A way to check up on the throughput one is seeing on a pipe:

$ yes | pv | cat > /dev/null

rlwrap

Here are a few tools without readline built in:

$ rlwrap ocaml
$ rlwrap racket
$ rlwrap sbcl

screenfetch

$ screenfetch
                 -/+:.          clark@ClarkMac
                :++++.          OS: 64bit Mac OS X 10.12.5 16F73
               /+++/.           Kernel: x86_64 Darwin 16.6.0
       .:-::- .+/:-``.::-       Uptime: 22d 19h 21m
    .:/++++++/::::/++++++/:`    Packages: 217
  .:///////////////////////:`   Shell: zsh 5.2
  ////////////////////////`     Resolution: 2880x1800
 -+++++++++++++++++++++++`      DE: Aqua
 /++++++++++++++++++++++/       WM: Quartz Compositor
 /sssssssssssssssssssssss.      WM Theme: Blue
 :ssssssssssssssssssssssss-     Font: SFMono-Regular
  osssssssssssssssssssssssso/`  CPU: Intel Core i7-6920HQ @ 2.90GHz
  `syyyyyyyyyyyyyyyyyyyyyyyy+`  GPU: AMD Radeon Pro 460 / Intel HD Graphics 530 
   `ossssssssssssssssssssss/    RAM: 10092MiB / 16384MiB
     :ooooooooooooooooooo+.    
      `:+oo+/:-..-:/+o+/-
$ screenfetch
                          ./+o+-       clark@ubuntu
                  yyyyy- -yyyyyy+      OS: Ubuntu 16.04 xenial
               ://+//////-yyyyyyo      Kernel: x86_64 Linux 4.8.0-51-generic
           .++ .:/++++++/-.+sss/`      Uptime: 13d 14h 51m
         .:++o:  /++++++++/:--:/-      Packages: 1855
        o:+o+:++.`..```.-/oo+++++/     Shell: bash 4.3.48
       .:+o:+o/.          `+sssoo+/    Resolution: 1680x1050
  .++/+:+oo+o:`             /sssooo.   DE: Unity 7.4.0
 /+++//+:`oo+o               /::--:.   WM: Compiz
 \+/+o+++`o++o               ++////.   WM Theme: Ambiance
  .++.o+++oo+:`             /dddhhh.   GTK Theme: Ambiance [GTK2/3]
       .+.o+oo:.          `oddhhhh+    Icon Theme: ubuntu-mono-dark
        \+.++o+o``-````.:ohdhhhhh+     Font: Ubuntu 11
         `:o+++ `ohhhhhhhhyo++os:      CPU: 2x Intel Core i7-6920HQ CPU @ 2.903GHz
           .o:`.syhhhhhhh/.oo++o`      RAM: 2316MiB / 3932MiB
               /osyyyyyyo++ooo+++/    
                   ````` +oo+++o\:    
                          `oo++.

Editors

emacs | | nano | hexedit | vim

"text mode" editors which can be run inside a terminal or on the far side of an ssh connection

http://hyperpolyglot.org/text-mode-editors

emacs

Mac OS ships with Emacs, but it is an old version.

hexedit

A binary file editor.

F1 for help.

nano

Nano is easier to learn than emacs or vim because it displays the available commands at the bottom of the screen.

vim

A "multi-modal" editor.

Version Control

colordiff

A colorized wrapper to diff. It uses the same flags as diff.

$ sed 's/root/ROOT/' < /etc/passwd > /tmp/passwd
$ colordiff /etc/passwd /tmp/passwd

cvs

git

git is not installed by default on Ubuntu.

http://hyperpolyglot.org/version-control

hg

rcs

sccs

svn

tig

Programming

c/c++ | java | javascript | python | shell

c/c++

https://github.com/clarkgrubb/sample-c-project

  • autoconf
  • automake
  • bison
  • check
  • cmake
  • cppcheck
  • cppunit
  • ctags
  • flex
  • gcc
  • libtool
  • makedepend
  • pkg-config
  • splint
  • valgrind

bison

$ find . -name '*.[ch]' | gtags -f -
$ global -x main
$ global -xr main

pkg-config is used like this:

$ gcc -o test test.c $(pkg-config --libs --cflags libpng)

The package manager is supposed to put .pc files in a location that pkg-config knows about, either /usr/lib/pkgconfig or /usr/local/lib/pkgconfig.

java

https://github.com/clarkgrubb/sample-java-project

javascript

https://github.com/clarkgrubb/sample-javascript-project

python

https://github.com/clarkgrubb/sample-python-project

shell

shellcheck is a static analyzer for shell scripts. Implemented in Haskell. It uses the shebang line to determine the flavor

$ shellcheck foo.sh

The version of bash that ships with the Mac is old. So we use brew to install an up-to-date version, and we rely on the fact that /usr/local/bin is ahead of /bin in the PATH. Bash scripts should have a

#!/usr/bin/env bash

shebang.

If you want a shell script to be portable, you can run bash with the --posix flag, but it is better to run the script with dash.

Only use external programs in this list:

https://www.gnu.org/software/make/manual/make.html#Utilities-in-Makefiles

Also, only use POSIX mandated flags for thos external programs:

http://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html

Debugging

gdb | htop | nmon | strace

gdb

$ gcc -g -o foo foo.c

Three ways to start gdb:

$ gdb CMD
$ gdb CMD PID
$ gdb CMD CORE

On Mac OS X, use lldb instead of gdb.

cmd gdb
help h
list l [first, last]
next statement n
step into function s
set breakpoint b [file:]line
list breakpoints i b
delete breakpoint d num
continue c
show backtrace bt
move up stack u
move down stack do
print expression p expr
(re)run r [arg1[, arg2 …]]
quit debugger q

htop

Some features require htop to be started as root.

Config file at ~/.config/htop/htoprc.

keys notes
help h ? F1
setup S
cursor control ↓ ↑ → ← F / 0-9 F: stay on process 0-9: search by pid
filter processes \ u \: by command name u: by user
tag processes <SPC> c U c: tag process and children
view toggles H t p + - H: threads p: path t: process tree: + -: show/hide tree parts
sort processes by resource P M T > I P: CPU
inspect process e s l e: environment s: strace l: lsof
control process k [ ] i a k: kill [ ]: nice i: i/o priority a: cpu affinity

nmon

strace

Syscalls for a command to be launched; for a running process identified by a PID; also strace children; set size string arguments to be displayed:

$ strace ls
$ strace -p 7191
$ strace -f
$ strace -s120

Timestamp each syscall; give elapsed time for each syscall:

$ strace -ttt
$ strace -T

Summarize counts of syscalls at exit; summarize counts and also suppress normal syscall output:

$ strace -C ls
$ strace -c ls

There is a tool called dtruss, part of dtrace, which is similar in function to strace. To use it, System Integrity Protection must be disabled on the Mac, which involves rebooting in recovery mode (hold down ⌘R), type csrutil disable in the terminal, then reboot again.

Images

For examples of using the image processing tools, see http://clarkgrubb.com/image-tools

ffmpeg

A tool for editing video and audio.

imagemagick

The package provides composite, convert, montage, and identify, which do edits like what Photoshop can do.

potrace

A tool which can convert bitmap images to SVG.

rsvg-convert

A tool which can convert SVG images to a bitmap format.

Documents

asciidoc | aspell | markdown | pandoc | xpdf

asciidoc

http://hyperpolyglot.org/lightweight-markup

A lightweight markup frontend for DocBook.

Create foo.html:

$ asciidoc foo.txt

aspell

Using aspell to interactively fix spelling errors. The original file is modified:

$ echo 'Four score and seven yeers ago' > /tmp/gettysburg.txt
$ aspell -c /tmp/gettysburg.txt

Using aspell non-interactively on a file stream:

$ echo 'Four score and seven yeers ago' | aspell list
yeers

$ echo 'Four score and seven yeers ago' | aspell -a
*
*
*
*
& yeers 85 21: years, yews, yes, year's, yew's, yeas, beers, jeers, leers, peers, seers, veers, yea's, yeahs, yes's, yours, eyes, yeses, Meyers, tees, Ayers, Byers, Myers, dyers, eye's, yaws, yer, queers, yrs, Yeats, yearns, beer's, deer's, jeer's, leer's, peer's, seer's, tee's, veer's, seer, yaw's, yeah's, year, yens, yeps, Yves, bees, ears, fees, gees, hers, lees, pees, sees, wees, yous, cheers, sheers, yeggs, yells, yen's, yep's, yetis, Lee's, bee's, fee's, lee's, pee's, see's, wee's, Meyer's, you's, Dyer's, Yeager's, dyer's, yore's, Er's, queer's, Meier's, ear's, her's, Cheer's, cheer's, sheer's, yegg's

markdown

http://hyperpolyglot.org/lightweight-markup

A Perl script which converts Markdown to HTML.

$ echo '* foo' | markdown
<ul>
<li>foo</li>
</ul>

pandoc

A tool implemented in Haskell for converting Markdown to a variety of formats, including HTML:

$ echo '* foo' | pandoc
<ul>
<li>foo</li>
</ul>

To PDF:

$ echo '* foo' | pandoc -o output.pdf

To man page:

$ cat > foo.1.md
% FOO(1) Foo User Manual
% John Smith
% November 8, 2012

# NAME

foo - a command line tool

# SYNOPSIS

foo [*file*]...

# DESCRIPTION

Foo performs an operation on files specified on the command line.  If no files
are specified the operation is performed on *stdin*.

# SEE ALSO

`bar` (1).

$ pandoc -s -w man foo.1.md -o foo.1

To HTML slideshow:

$ cat > gnomes.txt
# Business Plan

* collect underpants
* ?
* profit

# Phase 1: Collect Underpants

3:30 AM is the optimal time.

# Phase 2: ?

Details TBD

# Phase 3: Profit

Financial results will be discussed in the earnings call.

$ pandoc gnomes.txt -s --webtex -i -t slidy -o gnomes.html

qpdf

$ qpdf --help
$ qpdf --stream-data=uncompress input.pdf output.pdf
$ qpdf --linearize input.pdf output.pdf

xpdf

xpdf is an X Windows PDF viewer. Installing it also installs pdftotext which extracts plain text from a PDF.

$ pdftotext foo.pdf foo.txt

Scientific

ditaa

ditaa converts ASCII art to box-and-arrow diagrams such as produced by Visio and Omnigraffle.

http://clarkgrubb.com/image-tools#ditaa

glpk

gpsol solves linear programming problems.

http://hyperpolyglot.org/misc-math
http://clarkgrubb.com/optimization

gnuplot

gnuplot creates graphics such as histograms, box plots, scatterplots, and run charts.

http://hyperpolyglot.org/misc-math
http://clarkgrubb.com/image-tools#gnuplot

graphviz

dot converts DOT files, which are a notation for graphs, into PNG files.

http://clarkgrubb.com/image-tools#graphviz

mathics

mathematica comparied

Mathics supports a subset of the Mathematica language.

pari

http://hyperpolyglot.org/more-computer-algebra

This package install a command line calculator with arbitrary length integers, rationals, arbitrary precision decimals, complex numbers, vectors, matrices, polynomials. Lots of built-in functions for number theory.

To use it interactively:

$ gp

Non-interactively:

$ echo '1 + 1' | gp -q
2

Web

curl

curl -A USER_AGENT
curl -b COOKIE_NAME=DATA
curl -c COOKIE_JAR_FILE
curl -d KEY=VALUE [-d KEY=VALUE ...]
curl -d @DATA_FILE
curl -D HEADER_DUMP_FILE
curl -e REFERER
curl -F FORM_KEY=VALUE
curl -F FORM_KEY=@FILE
curl -G      (send data as GET)
curl -H HEADER [-H HEADER ...]
curl -i      (put HTTP header in output)
curl -I       (HEAD request)
curl -K CONFIG_FILE
curl -l      (follow redirect)
curl --limit-rate BPS
curl -o OUTPUT_FILE
curl -O      (name local file after path of remote file)
curl -s      (no progress bar or error messages)
curl -sS     (no progress bar, but error messages are shown)
curl -T FILE  (use PUT to upload FILE)
curl -u USER:PASSWORD

nmap

A port scanner.

pup

Use CSS selectors to extract elements from HTML. Implemented in Go.

https://github.com/EricChiang/pup

$ curl http://google.com | pup body

tcpflow

To dump the IP packets:

$ sudo tcpdump [-i INTERFACES] -w dump.pcap [PCAP_FILTER]
$ tcpdump [-v|-A] -r dump.pcap

The first command continues listening until interrupted with ^C.

Run ifconfig to see the available interfaces. INTERFACES can be a comma separated list. If none are specified, the default choice usually includes the main interface.

A filter can be used to reduce the number of packets captured. See man pcap-filter. The basic predicates of the filter language are:

tcp|udp|icmp
[dst|src] host IP_ADDRESS
[dst|src] port PORT
not PRED
PRED and PRED
PRED or PRED
( PRED )

The file dump.pcap is in the pcap next generation capture file format which tcpflow can also read.

The-v flag prints two lines are printed per packet instead of one and includes information about the protocol (UDP, TCP).

The -A flag prints the payload of the protocol in ASCII.

tcpflow does a better job of assembling in data being transmitted in the protocols. It is invoked as:

$ tcpflow [-r PCAP_FILE | -i INTERFACE] [-o DIR] [-a]

tcpflow can read from a PCAP_FILE or an INTERFACE. If no DIR is specified, it creates files in the working directory.

tcpflow creates files with names of the form

SRC_IP_ADDRESS.PORT-DST_IP_ADDRESS.PORT

identifying the TCP connection whose data it contains.

w3m

Start w3m:

$ w3m http://google.com

Use arrows to position the cursor. Type <ENTER> to click on the link below the cursor. If a form input element is below the cursor, click <ENTER> and then type text which will appear at the bottom of the screen. Type q to exit w3m.

w3m can be use to render an HTML document in plain text:

$ curl https://www.google.com > google.html
$ w3m -dump google.html

Cloud

awsclil | packer | pssh | tmux

awscli

Much of what can be done in the AWS console can also be done at the command line.

To get set up, go to IAM in the AWS console and create your key pair. Then use this command to write them to your ~/.aws/credentials file.

$ aws configure

Copy an S3 bucket and its contents:

$ aws s3 cp --recursive s3://MY_BUCKET/images/ .

Getting help:

$ aws help
$ aws s3 help
$ aws s3 cp help

packer

$ cat example.json
{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "branch": "master"
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "us-west-2",
    "source_ami": "ami-efd0428f",
    "instance_type": "t2.micro",
    "ssh_username": "ubuntu",
    "ami_name": "ubuntu.16.04 python3.5 openmail {{timestamp}}"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sleep 30",
      "cat /etc/*elease",
      "sudo apt-get update",
      "sudo apt-get install -y gcc g++ git make python3.5-dev virtualenv",
      "ssh-keyscan github.com >> ~/.ssh/known_hosts",
      "mkdir ~/.aws",
      "echo '[default]' > ~/.aws/credentials",
      "echo 'aws_access_key_id = {{user `aws_access_key`}}' >> ~/.aws/credentials",
      "echo 'aws_secret_access_key = {{user `aws_secret_key`}}' >> ~/.aws/credentials",
      "git clone git@github.com:THE_USER/THE_PROJECT.git",
      "cd THE_PROJECT",
      "git checkout {{user `branch`}}",
      "virtualenv -p python3.5 ../ve",
      ". ../ve/bin/activate",
      "pip install -r /tmp/requirements.txt",
      "make -k test"
    ]
  }]
}

$ packer validate example.json

$ packer build --var 'aws_access_key=YOUR ACCESS KEY' \
  -var 'aws_secret_key=YOUR SECRET KEY' \
  example.json

pssh

Execute a command on multiple remote hosts:

pssh -i -A -H joe@host1 -H joe@host2 echo "hello world"

The -i flag causes stdtout and stderr to be displayed. The -A flag prompts for a password.

tmux

tmux and screen compared

When using ssh to connect to a remote host, if the network connection is lost, the shell on the remote host will be sent a SIGHUP. The shell will in turn send a SIGHUP to any jobs that are running. The default behavior is for the jobs to exit.

To prevent loss of network connection from interrupting a long running batch job, there a couple of things the user can do:

$ nohup sleep 1000 &

$ sleep 1000 &
[1]  + running    sleep 1000
$ disown %1

nohup writes stdout to the file nohup.out.

A tmux session will survive loss a network connection. Start a tmux session with:

$ tmux

If disconnected, log back in to the remote server and attach to the session with:

$ tmux attach

Cryptography

gnupg

Setup

$ gpg2 --generate-key
$ gpg2 --list-keys
$ cp /etc/passwd passwd.txt
$ gpg2 --clear-sign passwd.txt
$ gpg2 --verify passwd.txt.asc

openssl

$ openssl dgst -sha256 < /etc/passwd
$ openssl base64 < /bin/ls

Compression

xz

suffix compress paths uncompress paths uncompress stdin
.gz gzip gunzip gzcat
.bz2 bzip2 bunzip2 bzcat
.xz xz unxz xzcat
$ tar xf linux-3.17.1.tar.xz
$ find linux-3.17.1 -name '*.[ch]' | xargs cat > /tmp/big.txt
$ wc -c /tmp/big.txt
 482314125 /tmp/big.txt
$ time gzip /tmp/big.txt
gzip /tmp/big.txt  15.19s user 0.19s system 99% cpu 15.425 total
$ wc -c /tmp/big.txt.gz 
 105127193 /tmp/big.txt.gz
$ time gunzip /tmp/big.txt.gz
gunzip /tmp/big.txt.gz  1.45s user 0.23s system 98% cpu 1.701 total
size ratio time untime
gzip 105127193 21.79% 15.425 1.701
bzip2 82311952 17.07% 43.773 16.617
xz 70984840 14.72% 3:42.77 6.308
zip 105138420 21.80% 15.632 3.294

Databases

and map/reduce tools

elasticsearch

http://clarkgrubb.com/elasticsearch

hive

jq

jq language syntax

mongo

mysql

pig

postgresql

presto

redis

spark

sqlite

Appendix

$ brew tap dart-lang/dart
$ brew install cabal-install coq dart elixir erlang gforth ghc gnu-apl go groovy guile idris kotlin leiningen \
  lua maxima ocaml opam racket rust sbcl sbt scala scalastyle swi-prolog typescript

$ brew tap caskroom/cask
$ brew cask install pharo sage
brew package apt package command line
cabal-install cabal-install cabal install
clojure1.6 clojure
coq coq coqtop
dart dart
elixir elixir
erlang erlang erl
erlc
fish fish fish
gforth gforth gforth
ghc ghc ghc
ghci
gnu-apl apl
go golang go
groovy groovy2 groovysh
guile guile-2.0 guile
idris idris
kotlin kotlinc
leiningen lein repl
lua lua5.3 lua
maxima maxima maxima
ocaml ocaml ocaml
opam opam opam
perl perl
cpan
php php
racket racket racket
ruby ruby
gem
rake
irb
erb
rust rustc rustc
sbcl sbcl sbcl
sbt sbt
scala scala scala
scalastyle scalastyle
swi-prolog swi-prolog swipl
tcl tclsh
typescript node-typescript tsc
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License