Unix-like shells

• Bash (Bourne again shell)
• sh
• zsh
• tcsh

Windows

• Powershell
• DOS (CMD)

We will focus on Bash which is the most common unix-like shell

Key commands

• cd : Change directory
• ls : List files in a directory
• pwd: Print working directory
• mkdir : Create directories
• cp / mv : Copy / Rename files

A Path specifies the location of a directory in the filesystem.

A forward slash / is used to separate folders in the hierarchy

For example this shows the path to your HOME directory

pwd

## /Users/iracooke/Dropbox/Teaching/Workshops/bcc2020CLE

Notice how it starts with a /. This means it is an absolute path. An absolute path specifies the location from the very top of the hierarchy.

If a path doesn’t start with / it is relative. A relative path specifies a location in relation to the current working directory.

For example in this series of the commands the second call to cd is using a relative path

cd /
cd home/

One of the main functions of the shell is that it allows you run commands. With very few exceptions commands are like any other computer programs. Each command exists as a file with computer readable instructions somewhere on your computer. By typing the name of the command you are asking your shell to run it.

Lets work with a command called fortune

fortune

## Humor in the Court:
## Q.  Mrs. Jones, is your appearance this morning pursuant to a deposition 
##     notice which I sent to your attorney?
## A.  No.  This is how I dress when I go to work.

We can use the which command to find out where it is on our computer (ie its path)

which fortune

## /usr/local/bin/fortune

The behaviour of commands can be changed by providing options, or by providing arguments (eg input files). For almost all commands the syntax is;

command [options] [arguments]

The fortune command has a few options. You can find out all about them by using the man command

man fortune

For example the -m option can be used to print out all fortunes attributed to Mark Twain

fortune -m 'Mark Twain'

The ls command also has options. For example the -l option will list files in long format.

ls -l

The ls command can also take an argument. If you don’t provide an argument (eg above) it will list files in the current working directory. This behaviour can be changed by providing a path to a different directory.

For example

ls -l /

The echo command simply evaluates its arguments and prints them back to screen. I will use it often in examples because it is a good way to demonstrate other bash commands.

echo "Hello"

## Hello

Echo can also be used to print the value of environment variables (more about those later)

echo $HOME

## /Users/iracooke

Bash provides several special characters that can be combined together in glob patterns to allow quite precise matching.

• * Matches any string
• ? Matches any single character

echo demo/s?_r1.fastq

## demo/s1_r1.fastq demo/s2_r1.fastq

• [...] Matches any one of the enclosed characters

echo demo/s[12]_r1.fastq

## demo/s1_r1.fastq demo/s2_r1.fastq

When you run a command in bash it usually writes its output to one or more of the following locations

• Standard output (by default this just gets printed to your terminal screen)
• Standard error (also gets printed to your terminal screen. Usually reserved for errors or warnings)
• Files (eg the program FastQC will generate an html file for each fastq file it processes)

Often we want to capture the output of a command into a file. This can be done using the output redirection operator, >.

For example we could capture all quotes attributed to Darth Vader

fortune -m "Darth Vader" > vader_quotes.txt

Try entering this into your terminal. It should create a file called vader_quotes.txt. Use the head command to look at the first few lines of this file

head -n 3 vader_quotes.txt

## There is much Obi-Wan did not tell you.
##      -- Darth Vader
## %

Notice how when we used > to redirect output there were still quite a few lines of text printed. This is the content of Standard Error.

Sometimes we might want to capture standard error as well as standard output, but in this case Standard Error is just getting in the way. We can modify the redirection to specifically redirect Standard Output to our file and Standard Error to nowhere (/dev/null). We use the numerical designator 2 to refer to Standard Error

fortune -m "Darth Vader" 2>/dev/null > vader_quotes.txt

Another thing to note about > is that it always overwrites its output. To append output to an existing file use >>.

For example if we wanted quotes by Darth Vader and Yoda we could do

fortune -m "Darth Vader"  > vader_yoda_quotes.txt
fortune -m "Yoda"  >> vader_yoda_quotes.txt

The grep program is a very flexible and fast way to search text for patterns. We can use it to count the number of quotes from the previous example.

cat vader_quotes.txt

grep '^%$' vader_quotes.txt

## %
## %
## %
## %

grep -c '^%$' vader_quotes.txt

## 4

So far we have only used single commands on their own. Bash provides various ways to combine multiple commands together to accomplish larger tasks. One such method is pipes. The pipe operator is |. It sends output from one command to become the input for another.

For example, in order to count the number of Darth Vader quotes in the fortune database we first ran fortune and then grep. Up to now we used an intermediate file vader_quotes.txt to allow us to combine these two programs. Instead of doing this we can accomplish the entire task in one line with a pipe

fortune -m "Darth Vader" 2>/dev/null | grep -c '^%$'

## 6

This is passing data as shown the diagram below

Once you have learned the cowsay and fortune commands it is very natural to want to combine them with a pipe.

Just to make things complete we first need to format our vader_quotes.txt database so it can be used by fortune

strfile vader_quotes.txt

fortune vader_quotes.txt | cowsay -f vader 

##  _______________________________________ 
## / When I left you, I was but the pupil. \
## \ Now, I am the master. - Darth Vader   /
##  --------------------------------------- 
##         \    ,-^-.
##          \   !oYo!
##           \ /./=\.\______
##                ##        )\/\
##                 ||-----w||
##                 ||      ||
## 
##                Cowth Vader

Imagine that you have new sequencing data available from your service provider.

They tell you that there are 24 separate files available for download.

The filenames follow a pattern as illustrated in the figure below.

This corresponds to 12 samples (labelled EM1, EM2, … EM12).

Each sample has two files where the R1 contains forward reads and the R2 contains reverse reads.

Variables are placeholders for information. They occur in all programming languages including bash.

sample="EM1"

sample="EM1"
echo $sample

## EM1

sample="EM1"
echo $sample
echo "${sample}_R1.fastq.gz"

## EM1
## EM1_R1.fastq.gz

You might be wondering what the point of this is?

Why would we assign data to a name and then retrieve the same data back?

The reason this is useful is because it allows us to represent specific instances of the data (eg “EM1”, “EM2”) using the same abstract identifier (eg sample).

In practical terms, when we reference sample we know that we will get whatever the current value of sample is even if it changes. For example;

sample="EM1"

echo $sample

## EM1

sample="EM2"

echo $sample

## EM2

But this still isn’t very useful is it?

We will see a truly useful use for variables when we come to loops. Before we get to that we need to cover one more topic.

Another way to capture the output of a command or to combine multiple commands together is called Command Substitution.

To illustrate this let’s use the seq command which creates a sequence of numbers. For example we can generate numbers 1 to 12 as follows;

seq 1 12

We can capture the output of this by putting our whole command inside an expression like $(command).

sample_numbers=$(seq 1 12)
echo ${sample_numbers}

## 1 2 3 4 5 6 7 8 9 10 11 12

echo $(seq 1 12)

## 1 2 3 4 5 6 7 8 9 10 11 12

Loops allow you to perform the same task repeatedly in different contexts (eg on different files, different names in a list, columns in a table etc etc)

Most programming languages support looping in some form or another. The syntax for a for loop in bash is like this

for placeholder in <list>
do
  <commands>
done

This will take each item in list and temporarily assign it to a variable called placeholder, The part between do and done can include as many commands as we want and we can refer to the current value of placeholder when running commands.

Now let’s try using loops to generate a list of sample names so that we can automate downloading our 24 fastq files

Our goal is to generate filenames like

EM[number]_R[12].fastq.gz

for sample in EM1 EM2
do
  echo $sample
done

## EM1
## EM2

for sample_num in $(seq 1 2)
do
  echo "EM${sample_num}"
done

## EM1
## EM2

We are getting close.

for sample_num in $(seq 1 2)
do
  for r in R1 R2
    do
    echo "EM${sample_num}_${r}.fastq.gz"
  done
done

## EM1_R1.fastq.gz
## EM1_R2.fastq.gz
## EM2_R1.fastq.gz
## EM2_R2.fastq.gz

base_url="https://s3-ap-southeast-2.amazonaws.com/bc3203/EM_10k/"
for sample_num in $(seq 1 2)
do
  for r in R1 R2
    do
    echo "${base_url}EM${sample_num}_${r}.fastq.gz"
  done
done

## https://s3-ap-southeast-2.amazonaws.com/bc3203/EM_10k/EM1_R1.fastq.gz
## https://s3-ap-southeast-2.amazonaws.com/bc3203/EM_10k/EM1_R2.fastq.gz
## https://s3-ap-southeast-2.amazonaws.com/bc3203/EM_10k/EM2_R1.fastq.gz
## https://s3-ap-southeast-2.amazonaws.com/bc3203/EM_10k/EM2_R2.fastq.gz

To finish this off we only need to substitute the wget command for echo. wget is a command that downloads files. We also need to change our seq command to include all 12 samples not just the first 2.

Bruce Barnett’s Grymoire https://www.grymoire.com/Unix/index.html . This page looks old and unmaintained but it is a gold mine of useful information. It is also quite witty and well written

The bash manual is the definitive reference https://www.gnu.org/software/bash/manual/bash.html