Shell Programming

From the “Learn Python the Hard Way” crash course:

You are not going to destroy your computer. You are not going to be thrown into some jail at the bottom of Microsoft’s Redmond campus. Your friends won’t laugh at you for being a nerd. Simply ignore any stupid weird reasons you have for fearing the command line… Why? Because if you want to learn to code, then you must learn this. Programming languages are advanced ways to control your computer with language. The command line is the baby little brother of programming languages. Learning the command line teaches you to control the computer using language. Once you get past that, you can then move on to writing code and feeling like you actually own the hunk of metal you just bought.

Below are commands for Linux Shell, Mac Terminal, and Windows PowerShell. Unix (Linux/Mac) commands are usually preceded by a file path ending with a $ symbol; while Windows commands are usually preceded by a file path ending with an > symbol. When looking up commands online, you may find the $ or > symbol preceding the command. Do not enter the symbol. It is only there to indicate the operating system.

IMPORTANT: Never enter the command rm -rf \ in your computer. Some unsavory sites and forums try to trick users into typing this command which will wipe out your system. The “command, if run by a superuser on the root directory, would cause the contents of nearly every writable mounted filesystem on the computer to be deleted, up to the point the system itself crashes from missing some crucial file, directory, or the like.”

Linux/Mac	Windows	Definition
`pwd`	`pwd`	print working directory
`hostname`	`hostname`	my computer’s network name
`mkdir`	`mkdir`	make directory
`cd`	`cd`	change directory
`ls`	`ls`	list directory
`rmdir`	`rmdir`	remove directory
`pushd`	`pushd`	push directory
`popd`	`popd`	pop directory
`cp`	`cp`	copy a file or directory
N/A	`robocopy`	robust copy
`mv`	`mv`	move a file or directory
`less`	`more`	page through a file
`cat`	`type`	print the whole file
N/A	`forfiles`	run a command on lots of files
`xargs`	N/A	execute arguments
`find`	`gci`	find files
`grep`	`select-string`	find things inside files
`man`	`help`	read a manual page
`apropos`	`helpctr`	find what man page is appropriate
`env`	`ls env:\`	look at your environment
`echo`	`echo`	print some arguments
`export`	`set`	export/set a new environment variable
`exit`	`exit`	exit the shell
`sudo`	`sudo`	DANGER! become super user root DANGER!

Command Line Crash Course

Paths, Folders, Directories

pwd

## /home/rstudio

If You Get Lost

Return to home directory.

cd ~

Make A Directory

mkdir temp
mkdir temp/stuff
mkdir temp/stuff/things
mkdir -p temp/stuff/things/orange/apple/pear/grape

Change Directory

cd temp
pwd
cd stuff/things/orange/apple/pear/grape
pwd
cd ..
pwd
cd ../../../../../..
pwd

## /home/rstudio/temp
## /home/rstudio/temp/stuff/things/orange/apple/pear/grape
## /home/rstudio/temp/stuff/things/orange/apple/pear
## /home/rstudio

List Directory

cd temp
ls
cd stuff
ls
cd things
ls
cd orange
ls
cd apple
ls
cd pear
ls
pwd

## stuff
## things
## orange
## apple
## pear
## grape
## /home/rstudio/temp/stuff/things/orange/apple/pear

cd ~
cd temp
ls -lR

## .:
## total 4
## drwxr-xr-x 3 rstudio rstudio 4096 Oct 27 11:37 stuff
## 
## ./stuff:
## total 4
## drwxr-xr-x 3 rstudio rstudio 4096 Oct 27 11:37 things
## 
## ./stuff/things:
## total 4
## drwxr-xr-x 3 rstudio rstudio 4096 Oct 27 11:37 orange
## 
## ./stuff/things/orange:
## total 4
## drwxr-xr-x 3 rstudio rstudio 4096 Oct 27 11:37 apple
## 
## ./stuff/things/orange/apple:
## total 4
## drwxr-xr-x 3 rstudio rstudio 4096 Oct 27 11:37 pear
## 
## ./stuff/things/orange/apple/pear:
## total 4
## drwxr-xr-x 2 rstudio rstudio 4096 Oct 27 11:37 grape
## 
## ./stuff/things/orange/apple/pear/grape:
## total 0

Remove Directory

cd temp/stuff/things/orange/apple/pear/grape
pwd
cd ..
rmdir grape
pwd
cd ../../../../..
rmdir -p stuff/things/orange/apple/pear
pwd

## /home/rstudio/temp/stuff/things/orange/apple/pear/grape
## /home/rstudio/temp/stuff/things/orange/apple/pear
## /home/rstudio/temp

Moving Around

cd temp
mkdir -p i/like/icecream
pushd i/like/icecream
pwd
popd
pwd
rmdir -p i/like/icecream

## ~/temp/i/like/icecream ~/temp
## /home/rstudio/temp/i/like/icecream
## ~/temp
## /home/rstudio/temp

Making Empty Files

cd temp
touch iamcool.txt

Copy a File

cd temp
cp iamcool.txt neat.txt
cp neat.txt awesome.txt
cp awesome.txt thefourthfile.txt
ls
mkdir something
cp awesome.txt something/
ls something/
cp -r something newplace
ls newplace/

## awesome.txt
## iamcool.txt
## neat.txt
## thefourthfile.txt
## awesome.txt
## awesome.txt

Moving a File

cd temp
mv awesome.txt uncool.txt
ls
mv newplace oldplace
ls
mv oldplace newplace
ls

## iamcool.txt
## neat.txt
## newplace
## something
## thefourthfile.txt
## uncool.txt
## iamcool.txt
## neat.txt
## oldplace
## something
## thefourthfile.txt
## uncool.txt
## iamcool.txt
## neat.txt
## newplace
## something
## thefourthfile.txt
## uncool.txt

Write to File

Using the >> operator will append data at the end of the file, while using the > will overwrite the contents of the file if already existing.

cd temp
touch test.txt
echo "Hello World!" > test.txt

View a File

On Unix you use the spacebar and the letter w to go down and up. Arrow keys also work. On Windows just hit the spacebar to page through.

cd temp
less test.txt

## Hello World!

Stream a File

On Unix you use the spacebar and the letter w to go down and up. Arrow keys also work. On Windows just hit the spacebar to page through.

cd temp
echo "Bash" > test.txt
touch test2.txt
echo "Shell" > test2.txt
echo "Scripting" > test2.txt
echo "on" >> test2.txt
echo "Unix" >> test2.txt
cat test.txt
cat test2.txt

## Bash
## Scripting
## on
## Unix

cd temp
cat test.txt test2.txt

## Bash
## Scripting
## on
## Unix

Removing a File

cd temp
ls
rm iamcool.txt neat.txt uncool.txt thefourthfile.txt test.txt test2.txt

## iamcool.txt
## neat.txt
## newplace
## something
## test2.txt
## test.txt
## thefourthfile.txt
## uncool.txt

Clean up.

cd temp
rm -rf something newplace
cd ~
rmdir temp

Command Line Data Science

Import Data

mkdir practice
cd practice
wget https://www.data36.com/demo1.csv --no-check-certificate
wget https://www.data36.com/demo2.csv --no-check-certificate

## --2017-10-27 11:37:36--  https://www.data36.com/demo1.csv
## Resolving www.data36.com (www.data36.com)... 46.101.253.204
## Connecting to www.data36.com (www.data36.com)|46.101.253.204|:443... connected.
## WARNING: no certificate subject alternative name matches
##  requested host name ‘www.data36.com’.
## HTTP request sent, awaiting response... 200 OK
## Length: 39 [text/csv]
## Saving to: ‘demo1.csv’
## 
##      0K                                                       100% 4.21M=0s
## 
## 2017-10-27 11:37:36 (4.21 MB/s) - ‘demo1.csv’ saved [39/39]
## 
## --2017-10-27 11:37:36--  https://www.data36.com/demo2.csv
## Resolving www.data36.com (www.data36.com)... 46.101.253.204
## Connecting to www.data36.com (www.data36.com)|46.101.253.204|:443... connected.
## WARNING: no certificate subject alternative name matches
##  requested host name ‘www.data36.com’.
## HTTP request sent, awaiting response... 200 OK
## Length: 356 [text/csv]
## Saving to: ‘demo2.csv’
## 
##      0K                                                       100% 36.7M=0s
## 
## 2017-10-27 11:37:37 (36.7 MB/s) - ‘demo2.csv’ saved [356/356]

View Data

cd practice
cat demo1.csv

## 4
## 7
## 12
## 1
## 15
## 1
## 7
## 8
## 5
## 4
## 13
## 12
## 9
## 7
## 2
## 10
## 5

cd practice
cat demo2.csv

## Helen Powell
## Jean Turner
## Barbara Diaz
## Doris Brown
## Roger Robinson
## Timothy Moore
## Paula Gonzalez
## Ronald Foster
## George Coleman
## Jesse Thompson
## Nancy Jones
## Jeffrey Cooper
## Ralph Gonzales
## Eric Richardson
## Christopher Howard
## Joshua Thomas
## Jimmy Phillips
## Kathy Sanchez
## Michael Bell
## Diana Hughes
## Thomas Wilson
## James Young
## Stephanie Harris
## Craig Carter
## Terry Henderson

Sort Data

Alphabetical order.

cd practice
sort demo2.csv

## Barbara Diaz
## Christopher Howard
## Craig Carter
## Diana Hughes
## Doris Brown
## Eric Richardson
## George Coleman
## Helen Powell
## James Young
## Jean Turner
## Jeffrey Cooper
## Jesse Thompson
## Jimmy Phillips
## Joshua Thomas
## Kathy Sanchez
## Michael Bell
## Nancy Jones
## Paula Gonzalez
## Ralph Gonzales
## Roger Robinson
## Ronald Foster
## Stephanie Harris
## Terry Henderson
## Thomas Wilson
## Timothy Moore

cd practice
sort demo1.csv

## 1
## 1
## 10
## 12
## 12
## 13
## 15
## 2
## 4
## 4
## 5
## 5
## 7
## 7
## 7
## 8
## 9

Numerical order.

cd practice
sort -n demo1.csv

## 1
## 1
## 2
## 4
## 4
## 5
## 5
## 7
## 7
## 7
## 8
## 9
## 10
## 12
## 12
## 13
## 15

Reverse order.

cd practice
sort -r demo2.csv

## Timothy Moore
## Thomas Wilson
## Terry Henderson
## Stephanie Harris
## Ronald Foster
## Roger Robinson
## Ralph Gonzales
## Paula Gonzalez
## Nancy Jones
## Michael Bell
## Kathy Sanchez
## Joshua Thomas
## Jimmy Phillips
## Jesse Thompson
## Jeffrey Cooper
## Jean Turner
## James Young
## Helen Powell
## George Coleman
## Eric Richardson
## Doris Brown
## Diana Hughes
## Craig Carter
## Christopher Howard
## Barbara Diaz

cd practice
sort -r -n demo1.csv

## 15
## 13
## 12
## 12
## 10
## 9
## 8
## 7
## 7
## 7
## 5
## 5
## 4
## 4
## 2
## 1
## 1

Remove duplicates.

cd practice
sort -n -u demo1.csv

## 1
## 2
## 4
## 5
## 7
## 8
## 9
## 10
## 12
## 13
## 15

Minimum

cd practice
sort -n demo1.csv | head -1

## 1

Maximum

cd practice
sort -n demo1.csv | tail -1

## 15

Count

Number of rows.

cd practice
wc -l < demo1.csv

## 17

Median

cd practice
sort -n demo1.csv | head -9 | tail -1

## 7

Frequencies

cd practice
sort demo1.csv | uniq -c

##       2 1
##       1 10
##       2 12
##       1 13
##       1 15
##       1 2
##       2 4
##       2 5
##       3 7
##       1 8
##       1 9