Linux quickies | 2/5 — Shell Scripting

The ultimate goal of a Software Engineer is to automate themselves out of the task.
— John Rofrano, DevOps Champion @ IBM T.J. Watson Research Center

This statement profoundly influenced my perspective as an IBM Software Engineering student.

Welcome!

Hey everyone, I’m Saman, a passionate Cloud Engineer, and I’m here to document my knowledge on a wide range of stuff, from Cloud computing and networking, to Linux machines and automation tools!

Today, we’re going to discover one of the essentials tools and skill that you need in your arsenal. Shell Scripting.
In This Article we’ll have:

• What is Shell Scripting
• Basics overview
• Examples
• Use case scenarios

** This Story is a part of a series, so please make sure to read the previous articles to get the most out of all **

What is Shell Scripting

Shell scripting, also known as shell programming, is writing scripts or code using a command line shell, e.g Bash, to automate tasks and execute a series of commands. It’s widely used in many workflow for various purposes including System administration, Deployment, and Automating tasks. A Shell script is an executable text file containing a sequence of commands.

Shell scripting is like spelling magic words. you cast your spell, and some magic happens automatically. You just need to learn the spells.

— From Myself

Shell Script Example — 1

Above, you can see a .sh file containing scripts for setting up a python web application development environment. We’ll break down each line in a few minutes, but before, let’s talk about some basics of a shell script file.

Basics Overview and Examples

• Shebang #!: The shebang is placed at the beginning of a shell script and indicates the interpreter that should be used to execute the script. For example, #!/bin/bash specefies that the script should be interpreted by the Bash shell, which is located in /bin/bash.

• Comments #: They are lines within a shell that serve as the documentation for the shell script. Anything written after # will be ignored and won’t be executed. e.g # This is a comment not a script.
• Variables: Shell scripts use variables to store and use data. Variables can be declared using assignment operator =. For example, name=saman assigns the value saman to the variable name. They later can be referenced by preficing the variable name with a dollar sign $. For instance echo $name will print the value stored in name, which is saman in our example.
• Command Substitution: It allows you to execute a command and use its output as part of another command or assignment. It is denoted by enclosing the command within $( and ) or backticks ( ). For example, date=$(date +%Y-%m-%d) assigns the current date to the date variable.

• Control structures: Conditionals like (if, else statements) and loops (for loop, while loop). Take a look at example below and analyze each line patiently:

Shell script example — 2

Try rewriting the code above in a file.sh, make it executable and run it.
e.g chmod +x file.sh to give execute permission and then ./file.sh to execute the file. We will have a comprehensive separate story for permissions.

• Functions: Functions in shell scripts allow you to define reusable blocks of code, and modularize scripts to improve code organizations. The function below will take two arguments and will print a simple welcome message. you can run it with ./<file-name>.sh first_name last_name

Function Example

• Input/Output (I/O): Shell scripts can interact with users by reading input from the keyboard, or accepting command line arguments.

You can consider a shell script a series of commands and tasks that you can do one by one manually in the terminal, but written one after other in order in a text file which can be executed. Let’s take a look at the first example(Shell script example — 1) we saw earlier:

1. It prints a message to indicate the start of the script. (sleep 1 waits for 1 second before running the next command. It’s used to avoid fast execution of commands and to provide a human interactive experience)
2. Checks, if the virtualenv command is installed. If not, it installs virtualenv using pip, and uses echo to print the message in the terminal.
3. Creates a python virtual environment named .venv
4. Activates the virtual environment using the source command.
5. Upgrades pip to the latest version.
6. Installs the dependencies listed in the requirements.txt file.
7. Updates the requirements.txt file with the current dependencies.
8. Prints a message to indicate that the setup process is completed
9. Finally, it provides instructions on how to activate the virtual environment.
   All with just running one single command ./<file-name>.sh

Use case scenarios

Shell scripting is widely used for automating processes. They are used in test case automation, system provisioning, CI/CD pipelines, Dockerfiles and virtually every where that you might need to run a shell command.

Here, I’m going to share some of the ways I benefit Shell Scripting.

1. CI/CD Pipelines:
   This is one of the places you scripts the most. Whether you’re using GitHub Actions, Jenkins, or other CI/CD pipeline tools, you will definetly use some shell scripts to help you faciliate your pipeline automation. Let’s take a look at example below:

Above, you can see four steps in my GitHub action workflow. The first step just Checkouts the source code, and as you see in the 2nd to 4th step, all the tasks in the steps are performed by shell scripts:
step 2: It builds the containers and run them via docker-compose —
step 3: It retrieves the running container’s ID, and executes the pytest command inside them to run the automated test cases available in the container
step 4: runs the code linting with flake8 to the all the the files with .py extensions in the app/ and tests/ directory.

Remember just like that, you can create a text file starting with #!/usr/bin/python3 or the correct path to the python interpreter to your system, fill it with powerful python script, and yet again leverage just with a single ./pythonfile.py command. Shell Scripting is not limited to Bash Shell.

2. In Dockerfiles:
Shell scripting also can be used in Dockerfile to perform several tasks while building the container image. For example, you can modify config files, copy files and directories, create users, manage permissions, and lots of other administrative tasks with scripts written in the Dockerfile, so that you won’t need to execute commands inside the docker container manually.

Dockerfile

Above, you can see how scripts are run inside a Dockerfile. All the RUN commands are scripts that perform:
1. updating packages, and installing httpd apache webserver
2. echoing the hostname (in this case container ID) to the index.html file
3. Creating a user names saman and assigning necessary ownerships to the user.
4. Modifying the permission of the log file an giving a root level privilege to the user to be able to bind special ports (ports below 1000, will discuss later).

Brief on the Dockerfile:
FROM, RUN, EXPOSE, USER, and CMD are Dockerfile commands.
- FROM: will choose the base image.
- EXPOSE: Does not execute anything and works more like a documentation indicating it should be run on port 80.
- USER: switches the user to saman as a non-root user.
- CMD: is the final command in the Docker, and is responsible for running the command inside the []. This command will run when you start the container.

3. Log Analysis: Another example that shell scripting can be used is in Log analysis. For example the script below searches for keywords “ERROR” and “WARNING” and generates a report:

#!/bin/bash

LOG_FILE="application.log"
KEYWORDS=("ERROR" "WARNING")

for keyword in "${KEYWORDS[@]}"
do
    echo "Occurrences of '$keyword' in $LOG_FILE:"
    grep -c "$keyword" "$LOG_FILE"
done

4. Task Automation: This is perhaps the ultimate use case of shell scripts. You can virtually automate any task with scripts. Below you can see a simple example of automating a backup process of a directory:

#!/bin/bash

# Example: Task Automation
# This script automates the backup of a directory

SOURCE_DIR="/path/to/source"
BACKUP_DIR="/path/to/backup"
BACKUP_FILE="backup_$(date +%Y%m%d).tar.gz"

echo "Starting backup of $SOURCE_DIR to $BACKUP_DIR/$BACKUP_FILE..."

tar -czf "$BACKUP_DIR/$BACKUP_FILE" "$SOURCE_DIR"

echo "Backup completed!"

The script above can easily be scheduled with Cronjob to be executed in a certain period of time, e.g each day, or each week:

1. Open the terminal
2. type cronetab -e
3. choose your editor if its your first time
4. add the line below to the editor:

0 0 * * 0 /path/to/backup_script.sh

5. save and exit.

Now your your desired directory will have backup each Sunday midnight.

Conclusion

There are lots of places where you can benefit Shell Scripts, such as Data processing, Application development, etc. But I think it was enough for realizing its enormous power in automation and get familiar with some of its use cases.

I hope you enjoyed this article, and hopefully you’ve learned something new! Make sure to stay updated and read the next article on Linux quickies | 3/6 — File System as its an important piece of the Linux puzzle!