When dealing with scripting and automation, file manipulation is an essential skill. Whether you’re analyzing log files, processing data sets, or parsing configuration files, the ability to read and interpret file content is critical.

In this blog post, we’ll learn how to read a file line by line in Bash using while and for loops. Let’s get started!

Prerequisites

To try out the scripts in this blog post, you need access to a Bash shell. You also need a text editor, such as "nano" or "vim", which come pre-installed by default in many Unix-like operating systems.

For the purpose of this blog post, I'll be using KodeKloud’s Ubuntu playground, which lets you access a pre-installed Ubuntu operating system in just one click. Best of all, you won't need to go through the hassle of installing any additional software— everything you need is already set up and ready to use.

Creating a Sample File

In this blog post, we'll be working with a sample file called names.txt, which will contain five names. Here's how to create that file and fill it with the necessary data:

Open a terminal window and navigate to the directory where you want to create the file. Then, run the following command:

cat << EOF >> names.txt
Jane Smith
John Doe
Alice Johnson
Bob Brown
Charlie Davis
EOF

In this command:

• << EOF: Starts a here document (also referred to as a "heredoc").  It's a way to pass multiline input to a command. The here document starts with << followed by a delimiter. In our example, EOF (End Of File) is used as the delimiter, but it's really just a convention, you could use any word or phrase as long as it's the same at the beginning and the end of the here document.
• >> names.txt: Tells the shell to append the contents of the here document to names.txt.
• The lines between << EOF and the second EOF are the contents of the here document. These lines will be appended to names.txt.

To confirm that the file has been created, and the names have been written to it, run the following command to display its contents:

cat names.txt

You should see the file content printed on your terminal like this:

We now have a file called names.txt, filled with five names, that will be used in the examples throughout the post.

Next, let’s get the full path of the file (we’ll need this for a later section) by running the following command:

realpath names.txt

You should see the full path displayed on your terminal, as shown below:

Reading a File Line By Line in Bash With a While Loop

A while loop is a commonly used method for reading a file line by line. In this section, we'll write a Bash script that leverages a while loop to read the contents of our names.txt file, one line at a time.

First, let’s create a script file named demo.sh in the /usr/local/bin directory and open it using the nano editor.

Note: While you're free to create the demo.sh file in any directory of your choice, we're placing it in the /usr/local/bin directory for a specific reason. In most Linux distributions, this directory is included in the system's command path. This means we can run our script without making it executable.

To confirm whether the /usr/local/bin directory is part of your command search path, run the following command:

$PATH

As you can see, the /usr/local/bin directory is indeed included in the PATH, which is essentially a list of directories that your system searches when attempting to execute a command.

Now, to create and open the demo.sh file in the nano editor, run the following command:

nano /usr/local/bin/demo.sh

This will launch the nano editor as shown below:

Now, add the following script to the editor:

#!/bin/bash

# Define the input file
INFILE=/root/names.txt

# Read the input file line by line
while read -r LINE
do
    printf '%s\n' "$LINE"
done < "$INFILE"

Once you've added the script, you'll need to save your work and exit the nano editor. To save, press ctrl + o. You'll see a message asking for the File Name to Write. Just press Enter to agree with the name that's already there. After your file is saved, you can leave nano by pressing ctrl + x.

Before we run this file, let's take a look at the script we've written:

• #!/bin/bash: This is a shebang (#!), used to indicate that the script should be executed using the bash shell.
• INFILE=/root/names.txt: Here, we're creating a variable called INFILE and assigning it the full path of the names.txt file. We need to provide the full path, as the file is not in the same directory as our script.
• while read -r LINE: This begins a while loop. The read -r LINE command reads a single line from the input file. The -r option prevents backslashes in the input from being interpreted as escape characters. So if you have input like Hello\nWorld, without the -r option, read would interpret this as two lines: Hello and World. If you use the -r option (read -r), read treats backslashes as normal characters. So Hello\nWorld would be interpreted as the literal string Hello\nWorld, not as two lines. The LINE is a variable that holds the content of the line that's been read.
• printf '%s\n' "$LINE": This is what happens inside the loop. For each line that's read from the file, the script executes the printf command. The printf '%s\n' "$LINE" command prints the content of the LINE variable followed by a newline (\n). %s is a placeholder that gets replaced with the value of LINE.
• done < "$INFILE": This marks the end of the while loop. The < "$INFILE" part tells the shell that the read command inside the loop should take its input from the file named names.txt (the value of the "INFILE" variable).

Now that we've gone through what the script does, it's time to run it. To do this, run the following command:

bash demo.sh

You should see each line from the names.txt file printed in the terminal on its own line, like this:

Reading a File Line By Line in Bash With a For Loop

You can also use a for loop to read the names.txt file line by line.

Replace the previous script in the demo.sh file with the following script. This new script uses a for loop to read each line from the names.txt file and print it to the terminal window:

#!/bin/bash

# Define the input file
INFILE=/root/names.txt

# Read the input file line by line using a for loop
IFS=$'\n' # set the Internal Field Separator to newline
for LINE in $(cat "$INFILE")
do
    echo "$LINE"
done

Before we run this, let's talk about what the script is doing:

• IFS=$'\n': Here, we're setting the Internal Field Separator (IFS) to a newline ($'\n'). The IFS is a special variable that tells the for loop what to split on. By setting IFS to a newline, we're making the for loop treat each line in the names.txt file as a separate item, even if there are spaces in the line.
• for LINE in $(cat "$INFILE"): This starts the for loop. The $(cat "$INFILE") command reads the INFILE and gives its contents to the for loop. The for loop goes through each line, treating each line as its own item and putting it in the LINE variable.

An important point to note here is that the $(cat "$INFILE") part of the command is known as command substitution. What it does is, it runs the cat "$INFILE" command, which reads all the content of the file that INFILE points to, and then replaces $(cat "$INFILE") with that content. This means that before the for loop even starts, the whole file is already loaded into memory as a list for the for loop to process. This is why using a for loop can cause issues with very large files

• echo "$LINE": This is what the for loop does for each line. It uses the echo command to show the contents of the LINE variable, which is the current line from the file.

Now, let's run the script with the following command:

bash demo.sh

You should see the lines from the names.txt file in the terminal, each on its own line, like this:

Conclusion

In this post, we learned how to write Bash scripts that use while and for loops to read a file line by line.

If your file is small, you can use either a while loop or a for loop. But if your file is big, it's better to use a while loop. The while loop reads the file one line at a time, so it doesn't use a lot of memory. A for loop, however, reads the whole file at once, which can make things slower if the file is very big.

Looking to build a solid foundation in shell scripting or taking your existing skills to the next level? Check out these courses from KodeKloud:

• Shell Scripts for Beginners: In this course, you'll dive into the practical world of Linux shell scripting. Regardless of your programming experience, you'll master fundamental scripting concepts such as variables, loops, and control logic. Throughout the course, you'll get plenty of hands-on experience using our comprehensive labs. Not only that, you'll also receive immediate feedback on your scripts, which will help you improve and refine them.
• Advanced Bash Scripting: In this course, you'll start with fundamentals like variables, functions, and parameter expansions and then dive deeper into streams, input/output redirection, and command-line utilities like awk and sed. You'll master arrays for data manipulation and storage and learn best practices to create robust scripts.