When dealing with scripting and automation, file manipulation is an essential skill. Whether you’re analyzing log files, processing data sets, or parsing configuration files, the ability to read and interpret file content is critical.
In this blog post, we’ll learn how to read a file line by line in Bash using while and for loops. Let’s get started!
To try out the scripts in this blog post, you need access to a Bash shell. You also need a text editor, such as "nano" or "vim", which come pre-installed by default in many Unix-like operating systems.
For the purpose of this blog post, I'll be using KodeKloud’s Ubuntu playground, which lets you access a pre-installed Ubuntu operating system in just one click. Best of all, you won't need to go through the hassle of installing any additional software— everything you need is already set up and ready to use.
Creating a Sample File
In this blog post, we'll be working with a sample file called
names.txt, which will contain five names. Here's how to create that file and fill it with the necessary data:
Open a terminal window and navigate to the directory where you want to create the file. Then, run the following command:
cat << EOF >> names.txt
In this command:
<< EOF: Starts a
heredocument (also referred to as a "heredoc"). It's a way to pass multiline input to a command. The
heredocument starts with
<<followed by a delimiter. In our example,
EOF(End Of File) is used as the delimiter, but it's really just a convention, you could use any word or phrase as long as it's the same at the beginning and the end of the
>> names.txt: Tells the shell to append the contents of the
- The lines between
<< EOFand the second
EOFare the contents of the
heredocument. These lines will be appended to
To confirm that the file has been created, and the names have been written to it, run the following command to display its contents:
You should see the file content printed on your terminal like this:
We now have a file called
names.txt, filled with five names, that will be used in the examples throughout the post.
Next, let’s get the full path of the file (we’ll need this for a later section) by running the following command:
You should see the full path displayed on your terminal, as shown below:
Reading a File Line By Line in Bash With a While Loop
A while loop is a commonly used method for reading a file line by line. In this section, we'll write a Bash script that leverages a while loop to read the contents of our
names.txt file, one line at a time.
First, let’s create a script file named
demo.sh in the
/usr/local/bin directory and open it using the nano editor.
Note: While you're free to create the
demo.sh file in any directory of your choice, we're placing it in the
/usr/local/bin directory for a specific reason. In most Linux distributions, this directory is included in the system's command path. This means we can run our script without making it executable.
To confirm whether the
/usr/local/bin directory is part of your command search path, run the following command:
As you can see, the
/usr/local/bin directory is indeed included in the
PATH, which is essentially a list of directories that your system searches when attempting to execute a command.
Now, to create and open the
demo.sh file in the nano editor, run the following command:
This will launch the nano editor as shown below:
Now, add the following script to the editor:
# Define the input file
# Read the input file line by line
while read -r LINE
printf '%s\n' "$LINE"
done < "$INFILE"
Once you've added the script, you'll need to save your work and exit the nano editor. To save, press
ctrl + o. You'll see a message asking for the
File Name to Write. Just press
Enter to agree with the name that's already there. After your file is saved, you can leave nano by pressing
ctrl + x.
Before we run this file, let's take a look at the script we've written:
#!/bin/bash: This is a shebang (#!), used to indicate that the script should be executed using the bash shell.
INFILE=/root/names.txt: Here, we're creating a variable called
INFILEand assigning it the full path of the
names.txtfile. We need to provide the full path, as the file is not in the same directory as our script.
while read -r LINE: This begins a while loop. The
read -r LINEcommand reads a single line from the input file. The
-roption prevents backslashes in the input from being interpreted as escape characters. So if you have input like
Hello\nWorld, without the
readwould interpret this as two lines:
World. If you use the
-roption (read -r),
readtreats backslashes as normal characters. So
Hello\nWorldwould be interpreted as the literal string
Hello\nWorld, not as two lines. The
LINEis a variable that holds the content of the line that's been read.
printf '%s\n' "$LINE": This is what happens inside the loop. For each line that's read from the file, the script executes the
printf '%s\n' "$LINE"command prints the content of the
LINEvariable followed by a newline (\n).
%sis a placeholder that gets replaced with the value of
done < "$INFILE": This marks the end of the while loop. The
< "$INFILE"part tells the shell that the read command inside the loop should take its input from the file named
names.txt(the value of the "INFILE" variable).
Now that we've gone through what the script does, it's time to run it. To do this, run the following command:
You should see each line from the
names.txt file printed in the terminal on its own line, like this:
Reading a File Line By Line in Bash With a For Loop
You can also use a for loop to read the
names.txt file line by line.
Replace the previous script in the
demo.sh file with the following script. This new script uses a for loop to read each line from the
names.txt file and print it to the terminal window:
# Define the input file
# Read the input file line by line using a for loop
IFS=$'\n' # set the Internal Field Separator to newline
for LINE in $(cat "$INFILE")
Before we run this, let's talk about what the script is doing:
IFS=$'\n': Here, we're setting the Internal Field Separator (IFS) to a newline ($'\n'). The
IFSis a special variable that tells the for loop what to split on. By setting
IFSto a newline, we're making the for loop treat each line in the
names.txtfile as a separate item, even if there are spaces in the line.
for LINE in $(cat "$INFILE"): This starts the for loop. The
$(cat "$INFILE")command reads the
INFILEand gives its contents to the for loop. The for loop goes through each line, treating each line as its own item and putting it in the
An important point to note here is that the
$(cat "$INFILE") part of the command is known as command substitution. What it does is, it runs the
cat "$INFILE" command, which reads all the content of the file that
INFILE points to, and then replaces
$(cat "$INFILE") with that content. This means that before the for loop even starts, the whole file is already loaded into memory as a list for the for loop to process. This is why using a for loop can cause issues with very large files
echo "$LINE": This is what the for loop does for each line. It uses the
echocommand to show the contents of the
LINEvariable, which is the current line from the file.
Now, let's run the script with the following command:
You should see the lines from the
names.txt file in the terminal, each on its own line, like this:
In this post, we learned how to write Bash scripts that use while and for loops to read a file line by line.
If your file is small, you can use either a while loop or a for loop. But if your file is big, it's better to use a while loop. The while loop reads the file one line at a time, so it doesn't use a lot of memory. A for loop, however, reads the whole file at once, which can make things slower if the file is very big.
Looking to build a solid foundation in shell scripting or taking your existing skills to the next level? Check out these courses from KodeKloud:
- Shell Scripts for Beginners: In this course, you'll dive into the practical world of Linux shell scripting. Regardless of your programming experience, you'll master fundamental scripting concepts such as variables, loops, and control logic. Throughout the course, you'll get plenty of hands-on experience using our comprehensive labs. Not only that, you'll also receive immediate feedback on your scripts, which will help you improve and refine them.
- Advanced Bash Scripting: In this course, you'll start with fundamentals like variables, functions, and parameter expansions and then dive deeper into streams, input/output redirection, and command-line utilities like
sed. You'll master arrays for data manipulation and storage and learn best practices to create robust scripts.