trouble shooting from the command line and the browser

system configuration | logs and online help | cpu and memory usage | disk and files | network connections and i/o | processes, tracing, and debugging | networking | firewalls | binaries | web debugging | gui tools | administrative privileges | environment variables | file system | file permissions | directory layout | installlation notes | tool index

Tools for Ubuntu 14.04 Linux, Mac OS 10.10, and Windows 7.

System Configuration

32-bit or 64-bit

AMD introduced a 64-bit extension to x86 called x86_64 in 2003. Although Intel already had the 64-bit Itanium architecture, it began making chips using the x86_64 instruction set in 2004.

Linux and Windows are available in 32-bit and 64-bit versions. 32-bit versions can run on x86 and x86_64 but 64-bit versions can only run on x86_64. Applications are sometimes specific to the 32-bit or 64-bit version of the operating system. The emulation system which 64-bit Windows uses to run 32-bit appications is called Wow64. A version of Windows XP for x86_64 was released in 2005.

uname -m returns i686 when running 32-bit Linux on a x86_64 chip. It returns x86_64 when running 64-bit Linux on a x86_64 chip.

To determine whether Windows is 32-bit or 64-bit, go to:

Start | Computer | Properties

Windows Registry

Windows provides a hierarchical key-value store called the Registry for system and application configuration information. Linux and Mac OS X store configuration information on the file system.

Each value in the registry has a type and an ACL. The Windows API provides library functions which applications can use to manipulate the registry. A user with Administrator privilege can inspect and edit the registry with the regedit command.

Online Help and Logs

linux:

There is no standard way to get the usage for a command line executable. To get the usage in a safe way (i.e. without the risk of performing a destructive action), one should read the man page. The {{--help option is a GNU tool convention, but not all tools support long options. Some, such as javac use the -help option to print usage, a practice which is not POSIX compliant.

On a UNIX system, packages which install command line tools should also install man pages for them. Section 1 of the man pages is for command line tools, and 3rd party tools would normally install their documentation in /usr/local/share/man/man1.

Shell built-ins are documented in the man page for the shell. Since these man pages are long, some shells provide a shortcut. In Bash:

$ help cd

In Zsh, type the name of the built-in and then use the M-h keystroke.

windows:

To get a list of Windows provided command line tools, use

$ help

To get documentation for a specific command.

$ help set

Although the help command can be run from a Cygwin shell, some of the commands are built in to cmd.exe and can't be run from a Cygwin shell.

Here is a list of Windows Commands.

Cygwin installs Unix style man pages for commands which it provides.

Windows GUI applications by convention provide help under the F1 key. There are at least four different formats that one might encounter on a modern Windows system:

eventvwr:

Event Viewer can be used to export log data to a file in .evt, .txt, or .csv format.

.evt is the native format for Event Viewer.

Selecting .txt produces a tab delimited format.

CPU and Memory Usage

htop differences

• show threads
• can use arrows to scroll through all processes
• search for process by pid or cmd name
• strace or lsof for process
• tag and signal (or nice) multiple processes
• per core cpu meter

The version of top on Darwin has meters for disk i/o and network i/o.

threads

How to get the number of threads for each process?

Use htop.

interrupts

CPU interrupts are managed by the interrupt descriptor table (IDT) which provides the memory address of the code to handle the interrupt. On x86 the int instruction generates a CPU interrupt and takes a byte as an argument. Values 0x00 through 0x1F are reserved for architecture use.

Each device has an interrupt request (IRQ) line for signaling the CPU. On Linux one can see which devices are using which IRQ lines with

$ cat /proc/interrupts

On Windows the information is available in System Information, which can be launched from the command line as msinfo32. Navigate to this location:

System Summary | Hardware Resources | IRQs

On a multiprocessor machine IRQ lines can be used for inter-processor communication.

context switch

A context switch means changing the currently executing process. If the current process is not exiting, the operating system must save the current register values so the process can resume execution later.

page faults

The operating systems described on this sheet use virtual memory and organize system memory and virtual memory in pages. On Linux and Mac, the following command shows the page size in bytes:

$ getconf PAGESIZE

The operating system maps the pages used by each process to system pages. It is possible that the page is not even in system memory but instead saved to disk. When the process accesses such a page, a page fault occurs and the kernel loads the page. On Windows this is also called a hard fault.

Implementing virtual memory requires hardware support. The part of the CPU which implements this is called the memory management unit (MMU). On x86_64 a page fault has number 0x0E in the interrupt descriptor table.

top and ps on Windows

Cygwin provides top and ps as part of the procps package, but they only report on Cygwin processes.

PowerShell has a cmdlet called get-process which is similar to ps.

Disk and Files

mac:

I'm not aware of a way to break down Network I/O by interface.

lsof:

lsof will usually return incomplete information unless run as sudo.

The -n flag prevents replacing IP address with DNS names. The -P flag prevents replacing port numbers with names in /etc/services.

On Linux, lsof is implemented by reading from the directories /proc/PID/fd and proc/PID/fdinfo. These directories are only readable by the owner of PID.

Here are the columns that lsof normally outputs:

$ lsof | head -1 | tr -s ' ' '\n' | nl
     1	COMMAND
     2	PID
     3	USER
     4	FD
     5	TYPE
     6	DEVICE
     7	SIZE/OFF
     8	NODE
     9	NAME

With the -t flag, only the PID column will be output.

Network Connections and I/O

Processes, Tracing, and Debugging

The ps command is complicated. It supports both POSIX style flags and BSD style flags with no hyphens. The FreeBSD version of ps used on Mac OS X does not have the same options as the Linux version of ps, and neither seems to be completely described by its man page.

The BSD u flag and the POSIX -v flag show CPU usage and memory usage. On Linux, the -v flag includes total CPU time, but not %CPU. On Linux, ps -ef and ps aux show kernel processes, but ps -evx does not.

When adding columns, the POSIX -o flag tends to work better than the BSD O flag. The O flag cannot be used with the v flag on Linux, and it gives less control over the placement of the added columns. Note that comm is just the command name, whereas command is the command name and its arguments. On Linux comm takes the basename and truncates it to 15 characters.

There are flags for filtering and sorting the processes, but this are usually redundant, because awk works well on the output. The only column with internal whitspace is the command with arguments, which is usually written last.

system calls

linux:

Traditionally on the x86 architecture, a process would request a system call via the int instruction and the 0x80 interrupt handler. A code would be placed in the eax register indicating the desired system call.

Up to five parameters could be passed using the ebx, ecx, edx, esi, and edi registers. If more paramaters are needed, a single register with an address into user space is used.

The return value is placed in the eax register. For most system calls, 0 indicates success and -1 indicates failure. On failure the system call will write to the global errno variable of the process. Symbolic constants for the possible errno values are defined in errno.h, and the man page for errno describes their use.

The Pentium II processor (1997) introduced the sysenter and sysexit instructions, which provide a faster mechanism than int for implementing a system call.

windows:

Tables of system calls for NT-based versions of Windows:

• http://j00ru.vexillium.org/ntapi/
• http://j00ru.vexillium.org/ntapi_64/

[TODO: taskmgr has a show processes from all users button. Relaunches with a different window with a checkbox]

Process Monitor (Procmon) show file access, networking, registry access, thread and process creation.

[TODO: how to install handles on windows and use it]

[TODO: debugger threads, set a variable, dump an object, inspect a class (methods)]

Networking

mac:

The Setup Assistant, which runs when a Macintosh is booted up for the first time, generates a computer name from the name of the initial account created on the system. An example is Fred's MacBook Pro. The name used for networking will be Freds-Macbook-Pro.local. The computer name can be changed at

System Preferences... | Sharing

The .local domain is a special TLD which is not managed by a DNS server. One does a DNS lookup on a .local hostname by broadcasting on the local subnet, and the host itself responds. The protocol is called mDNS and is described in RFC 6762. Bonjour, which is Apple's zero configuration services discovery protocol, is built on top of mDNS.

windows:

Windows requests a name for the computer during the installation process. The name can be changed at

Control Panel | System | Computer name, domain, and workgroup settings

To be on the network, a Windows machine must belong to a workgroup or a domain. The default workgroup name in my experience is WORKGROUP. One can change the workgroup at the same place one can change the computer name. If the workgroup exists, it is joined. If it doesn't exist, it is created.

If a workgroup contains a windows server, DNS services can be run on it. This allows the machines in the workgroup to have fully qualified domain names.

Permissions in workgroups are managed using Windows security identifiers (SIDs). Windows 7 and Windows 8 have a feature called homegroups. Unlike workgroups, homegroups are password protected.

To join an Active Directory domain, the user must have an account in the domain. When the computer is joined to the domain, all users in the domain are given accounts on the computer, and the domain administrators acquire administrative rights. The joining computer gets registered in the domain DNS.

Workgroups and Active Directory use the SMB protocol, which in modern versions of Windows is a protocol on top of TCP/IP. Samba is a reverse engineering of the SMB protocol which is available for Linux and Mac. It can be used to access Windows file servers.

Bonjour is available for Windows. It gets installed when iTunes is installed on Windows.

tcpflow

Show what is happening on a specific device and port:

$ tcpflow -i lo0 -C port 6007

The -C makes the output go to stdout. Otherwise it goes to a file in the current directory.

tshark

I find tshark difficult to use. If possible use Wireshark. On Mac OS X, this unfortunately means installing and running XQuartz.

Without arguments the columns in the output are

• seconds elapsed since tool started
• local ip address
• direction of packet (-> or <-)
• remote ip adresss
• protocol (TCP HTTP DNS etc...)
• more

How to put output to a file and then later read it. Note that the file is not human readable. The format is libpcap, also used by tcpdump.

$ tshark -w /tmp/tshark.out
$ tshark -r /tmp/tshark.out

tshark has two filter languages for reducing the output. The capture filter language is faster, but the read filter language is more expressive. Can the capture language be used with the -r option?

The CAPTURE_FILTER syntax is documented under man pcap-filter or man tcpdump.

The READ_FILTER syntax is documented under man wireshark-filter.

$ tshark -R READ_FILTER [-f] CAPTURE_FILTER

A dump of fields and protocols that tshark is aware of?

$ tshark -G fields
$ tshark -G protocols

Output control:

$ tshark -e FIELD -T fields
$ tshark -T (pdml|psml|ps|text|fields)

For importing into a spreadsheet:

$ tshark -T fields -E separator=, -E quote=d

TCP packets with IP addresses resolved

$ tshark -N n tcp

When displaying TCP packets, these are the output fields:

• seconds.milliseconds since start
• source ip address or dns name
• -
• destination ip address or dns name
• TCP
• a size?
• source port
• destination port
• [ACK,SYN,FIN]
• Seq= ,Ack= , Win= , Len= TSval= TSecr

curl

curl can be used for FTP, SMTP, IMAP, or LDAP, but we only describe HTTP(S)

brace {} and [BAR-FOO] notation for multiple URLs

curl -A USER_AGENT
curl -b COOKIE_NAME=DATA
curl -c COOKIE_JAR_FILE
curl -d KEY=VALUE [-d KEY=VALUE ...]
curl -d @DATA_FILE
curl -D HEADER_DUMP_FILE
curl -e REFERER
curl -F FORM_KEY=VALUE
curl -F FORM_KEY=@FILE
curl -G      (send data as GET)
curl -H HEADER [-H HEADER ...]
curl -i      (put HTTP header in output)
curl -I       (HEAD request)
curl -K CONFIG_FILE
curl -l      (follow redirect)
curl --limit-rate BPS
curl -o OUTPUT_FILE
curl -o and # notation for multiple URLs
curl -O      (name local file after path of remote file)
curl -s      (no progress bar or error messages)
curl -sS     (no progress bar, but error messages are shown)
curl -T FILE  (use PUT to upload FILE)
curl -u USER:PASSWORD

dig

dig vs nslookup?

dig example.com +nostats +nocomments +nocmd
# Look up DNS record for example.com
# ;example.com.                    IN      A
# example.com.             3259    IN      A       204.232.175.78

dig www.example.com +nostats +nocomments +nocmd
# ;www.example.com.               IN      A
# www.example.com.        3592    IN      CNAME   username.github.io.
# username.github.io.      43192   IN      A       204.232.175.78

nc

When used as a UDP client, nc -u does not exit after it encounters EOF on standard input. It is necessary to use a ^C. Some versions of nc -u (not the Mac OS X version) can be given a -q 1 to make it exit?

Use -U FILE to make nc communicate over a Unix domain socket.

Use -w SECONDS to specify a timeout limit.

Use -c to convert newlines to CRLF when sending.

telnet

The telnet protocol is implemented on top of the TCP protocol. It defines ways to negotiate options with the remote server, but when connecting to a port other than 23, it does not do any initial option negotiating. The upshot is that telnet can sometimes be used as a generic TCP client.

[TODO: node name vs local hostname]

[TODO: examples of how to use tshark: IP, TCP]

[TODO: run tshark on windows; Cygwin too slow, use ps or cmd.exe?]

[TODO: lots of documentation about curl]

Firewalls

linux

• iptables
• ip6tables
• arptables
• ebtables
• nftables
• ufw

darwin

System Preferences | Security & Privacy | Firewall

windows

Binaries

bc:

The default input and output bases are 10.

hexedit

hexedit patch for mask and alignment search

[TODO: Explain how to enter F1 on a Mac]

[TODO: hexedit commands]

Web Debugging

In Safari, one must enable the Develop menu. Go to the Advanced pane of Preferences and select the "Show Develop menu in menu bar" checkbox.

how to spoof user agent; how to simulate mobile screen size

GUI Tools

Resource Monitor (Win) also performs much of the same functionality as Activity Monitor (Mac).

Administrative Privileges

linux:

One can execute a command with administrative privileges by becoming the root user. If one knows the the root password, one can use su to launch a root shell.

A user can also execute a command with administrative privileges if they are listed in the /etc/sudoers file. This is done with the sudo command. The user will be challenged to enter their own password instead of the root password.

The visudo command is used to edit /etc/sudoers.

mac:

A user can execute a command with administrative privilege using sudo if they are in the admin group. A user in the admin group can add or remove another user from the admin group by going to:

System Preferences | Users & Groups | Password | Allow user to administer this computer

The admin group is listed in /etc/sudoers.

windows:

When a user is created on Windows, the user can be designated as an administrator:

Control Panel | User Accounts | Manage another account | Create a new account | Administrator

When a user who is an administrator runs a command which requires administrator privileges, the user is presented with the User Account Control dialog to confirm the user wants to run the command. There is no password challenge like on Linux and Mac.

A shell can be "Run as Administrator", but by default it is not. Commands which require administrator privileges will fail if run from a shell which was not run as administrator.

To run a shell as administrator, right click its icon and select "Run as Administrator". Alternatively one can type Ctrl+Shift+Enter when launching the shell with Search.

[TODO: is it possible to su to Administrator? What about runas?]

Environment Variables

linux:

At login the system will set the environment variables USER and HOME according to the first and sixth fields of /etc/passwd. A POSIX compliant system will set LOGNAME as well. The system is usually expected to set LANG, TERM, and TMPDIR as well.

[TODO: What environment variables bash and zsh set. The startup files .bashrc and .zshenv]

Shell variables vs environment variables. How to set shell variable which isn't exported. How to access a shell variable.

printenv

env command. Also this style of setting a variable: FOO=7 my-command.rb

su - and what it does.

zsh: vared

mac:

windows:

PATH can be set in two places; will be merged.

PATH has ; as separator.

PATHEXT has a list of ; separated suffixes. Used by operating system to find executables.

Editing PATHs is difficult in windows because both regedit and Control Panel provide a text box which is too small.

pathed

Ways to set env variable: regedit,

Computer | Properties | Change settings | Advanced | Environment Variables...

set and setx. setx can be called from Cygwin.

Cygwin is case sensitive. It sees all environment variables from Windows as upper case. Lower case versions can be used as local shell variables and they won't conflict.

For Cygwin processes, Cygwin replaces the C:\Windows style paths with /cygdrive/c/Windows style paths. It prepends /usr/local/bin and /usr/bin to the PATH.

File System

EXT4, HFS Plus, NTFS

[TODO: linux mounting]

symlinks and hard links

Mac and lack of case sensitivity. But it is optional.

Path separators.

Newline vs \r\n.

text and binary mode (Windows)

/cygdrive/c instead of C:\

cygpath

Windows symlinks and Windows shortcuts

File Permissions

Access Control List

Windows SID

[TODO]

linux:

read/write/execute for user/group/other.

meaning of execute for directories.

what about the

chattr

mac:

chflags

windows:

attrib

acl [also used by registry]

Directory Layout

linux:

[file system hierarchy standard]

/home/fred

HOME initialized at login using /etc/passwd. Becomes the working directory of the shell.

mac:

/Users/fred

windows:

/Users/fred

Consoles are launched by shortcuts. The working directory can be specified in the shortcut properties. If not set, it usually defaults to the directory containing the executable.

cygwin:

Installation Notes

linux:

Missing tools can be installed using apt-get.

mac:

To have all the tools listed on this page, install these:

• Chrome
• Firefox
• Xcode
• Homebrew (or MacPorts)
• Wireshark [TODO: can tshark be installed with brew?]

When XCode is installed, go to

XCode | Preferences... | Downloads | Components

and install Command Line Tools.

Missing tools are command line tools which can be installed using brew or port.

windows:

To have all the tools listed on this page, install these:

• Chrome
• Firefox
• Cygwin
• Visual Studio
• Process Monitor (Sysinternals)
• Handle (Sysinternals)
• Wireshark

Missing tools are command line tools which can be installed using Cygwin Setup.

[TODO: Procmon and handle are in ~/Source. Put in PATH]

Tool Index

• /proc
• blktrace
• cifsiostat
• dstat
• dtrace
• free
• gperftools
• gprof
• htop
• iostat
• iotop
• ip
• kcachegrind
• ktap
• latencytop
• lsof
• lttng
• mpstat
• netstat
• nfsiostat
• nicstat
• ntop
• oprofile
• perf
• pidstat
• ping
• pmap
• powertop
• ps
• sar
• slabtop
• ss
• stap
• strace
• sysctl
• tcpdump
• top
• traceroute
• uptime
• valgrind
• vmstat