Day 50: Expanding EC2 Instance Storage for Development Needs

Hello everyone,

I’m facing an issue with Day 50: Expanding EC2 Instance Storage for Development Needs in the 100 Days of Cloud (AWS) path.

I believe my configuration is correct, but unfortunately I failed the challenge :sweat_smile:. I’m not sure whether this is a platform bug or if I made a mistake (though I don’t think so).

Here’s the process I followed:

  1. Increased the volume size using the AWS Console (waited until the volume state changed to “in-use”).

  2. Connected to the EC2 instance via SSH.

  3. Ran the following commands:

    • First, expanded the partition:
      sudo growpart /dev/xvda 1
    • Then, expanded the filesystem:
      sudo xfs_growfs -d /

wait 5min

[ec2-user@ip-172-31-16-140 ~]$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  4.0M     0  4.0M   0% /dev
tmpfs          tmpfs     475M     0  475M   0% /dev/shm
tmpfs          tmpfs     190M  2.9M  188M   2% /run
/dev/xvda1     xfs        12G  1.6G   11G  13% / <--- this
tmpfs          tmpfs     475M     0  475M   0% /tmp
/dev/xvda128   vfat       10M  1.3M  8.7M  13% /boot/efi
tmpfs          tmpfs      95M     0   95M   0% /run/user/1000

wait another 5min :broken_heart:

============================= test session starts ==============================
platform linux -- Python 3.10.17, pytest-8.3.5, pluggy-1.6.0
rootdir: /usr/share
plugins: testinfra-10.2.2
collected 1 item

../usr/share/test.py F                                                   [100%]

=================================== FAILURES ===================================
____________________________ test_volume_expansion _____________________________

    def test_volume_expansion():
        host = testinfra.get_host("local://")

        # Get the instance ID and public IP
        instance_info = host.run("aws ec2 describe-instances --filters Name=tag:Name,Values=devops-ec2 --query 'Reservations[0].Instances[0].[InstanceId,PublicIpAddress]' --output text")
        assert instance_info.rc == 0, "Failed to get the instance ID and public IP"
        instance_id, instance_ip = instance_info.stdout.strip().split()

        # Check the volume size in AWS
        volume_size = host.run(f"aws ec2 describe-volumes --filters Name=attachment.instance-id,Values={instance_id} --query 'Volumes[0].Size' --output text")
        assert volume_size.rc == 0, f"Failed to get the volume size: {volume_size.stderr}"
        assert volume_size.stdout.strip() == "12", "Volume size is not 12 GiB."

        # SSH into the instance to check the root partition size
        ssh_host = testinfra.get_host(f"ssh://ec2-user@{instance_ip}", ssh_identity_file="/root/devops-keypair.pem")
>       partition_size = ssh_host.check_output("df -h / | grep /dev/xvda1 | awk '{print $2}'")

/usr/share/test.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.10/site-packages/testinfra/host.py:98: in run
    return self.backend.run(command, *args, **kwargs)
/usr/local/lib/python3.10/site-packages/testinfra/backend/ssh.py:46: in run
    return self.run_ssh(self.get_command(command, *args))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testinfra.backend.ssh.SshBackend object at 0x79f5fc2d2a40>
command = "df -h / | grep /dev/xvda1 | awk '{print $2}'"

    def run_ssh(self, command: str) -> base.CommandResult:
        cmd, cmd_args = self._build_ssh_command(command)
        out = self.run_local(" ".join(cmd), *cmd_args)
        out.command = self.encode(command)
        if out.rc == 255:
            # ssh exits with the exit status of the remote command or with 255
            # if an error occurred.
>           raise RuntimeError(out)
E           RuntimeError: CommandResult(backend=<testinfra.backend.ssh.SshBackend object at 0x79f5fc2d2a40>, exit_status=255, command=b"df -h / | grep /dev/xvda1 | awk '{print $2}'", _stdout=b'', _stderr=b'Host key verification failed.\r\n')

/usr/local/lib/python3.10/site-packages/testinfra/backend/ssh.py:95: RuntimeError
=========================== short test summary info ============================
FAILED ../usr/share/test.py::test_volume_expansion - RuntimeError: CommandRes...
============================== 1 failed in 2.50s ===============================

Has anyone encountered this issue before or knows what might be going wrong?

thanks in advance!
Kev$n

Finally, after several hours of debugging, I found the root cause by analyzing the error thrown after clicking the check button:

_stderr=b'Host key verification failed.\r\n'

The jumphost was failing to connect to the EC2 instance via SSH because the EC2 host key was not present in the known_hosts file.

Solution: run the following command from the machine executing the tests to register the EC2 host key:

ssh-keyscan -H "EC2_PUBLIC_IP" >> ~/.ssh/known_hosts

After that, the SSH connection was established successfully and the tests passed.

known_hosts is populated when you try to connect to a remote server for the first time with a public key. Without initiating an SSH connection, there won’t be any entry for that server in known_hosts.

I was able to SSH to the remote server without any issues.

Regarding the Python test script you shared, is that output appearing after the terminal has been idle for some time? I tried reproducing the behavior on my end, but wasn’t able to observe the same issue.

Hello @Santosh_KodeKloud ,

Step 4 says to access the instance via SSH from the AWS client host, but my mistake was connecting to the instance through EC2 Instance Connect (via the AWS console) instead.