Managing CI/CD Pipelines Using Terraform

Dear Team,

Course Name: KodeKloud Engineer

Level 4: Managing CI/CD Pipelines Using Terraform

I recently completed the “Managing CI/CD Pipelines Using Terraform” task, but it was marked as failed due to “main.tf symlink is incorrect in dev”. I belive everything was executed correctly. Attached Screenshot for reference. Kindly review and let me know if there’s anything I may have missed.

My Soultion:

script.sh

#!/bin/bash
set -e

mkdir -p /home/bob/terraform/env/dev
mkdir -p /home/bob/terraform/env/prod
mkdir -p /home/bob/terraform/modules/dynamodb
mkdir -p /home/bob/terraform/modules/secretsmanager
mkdir -p /home/bob/terraform/modules/elasticsearch
mkdir -p /home/bob/terraform/shared

Relative symlinks for shared files

Absolute symlinks for shared files in dev

ln -sf /home/bob/terraform/shared/main.tf /home/bob/terraform/env/dev/main.tf
ln -sf /home/bob/terraform/shared/variables.tf /home/bob/terraform/env/dev/variables.tf
ln -sf /home/bob/terraform/shared/outputs.tf /home/bob/terraform/env/dev/outputs.tf

Absolute symlinks for shared files in prod

ln -sf /home/bob/terraform/shared/main.tf /home/bob/terraform/env/prod/main.tf
ln -sf /home/bob/terraform/shared/variables.tf /home/bob/terraform/env/prod/variables.tf
ln -sf /home/bob/terraform/shared/outputs.tf /home/bob/terraform/env/prod/outputs.tf

Root symlinks (absolute is fine here)

ln -sf /home/bob/terraform/shared/main.tf /home/bob/terraform/main.tf
ln -sf /home/bob/terraform/shared/variables.tf /home/bob/terraform/variables.tf
ln -sf /home/bob/terraform/shared/outputs.tf /home/bob/terraform/outputs.tf

Create dev.tfvars and prod.tfvars if missing

cat > /home/bob/terraform/env/dev/dev.tfvars <<EOF
KKE_ENV = “dev”
KKE_DYNAMODB_TABLE_NAME = “datacenter-dev-table”
KKE_SECRET_NAME = “datacenter-dev-secret”
KKE_SECRET_VALUE = “datacenter-dev-value”
KKE_ELASTICSEARCH_DOMAIN = “datacenter-dev-es”
EOF

cat > /home/bob/terraform/env/prod/prod.tfvars <<EOF
KKE_ENV = “prod”
KKE_DYNAMODB_TABLE_NAME = “datacenter-prod-table”
KKE_SECRET_NAME = “datacenter-prod-secret”
KKE_SECRET_VALUE = “datacenter-prod-value”
KKE_ELASTICSEARCH_DOMAIN = “datacenter-prod-es”
EOF

Create LocalStack buckets for state backend

aws --endpoint-url http://aws:4566 s3 mb s3://datacenter-dev-terraform-state || true
aws --endpoint-url http://aws:4566 s3 mb s3://datacenter-prod-terraform-state || true


modules/dynamodb/main.tf
resource “aws_dynamodb_table” “this” {
name = var.KKE_DYNAMODB_TABLE_NAME
billing_mode = “PAY_PER_REQUEST”

hash_key = “id”

attribute {
name = “id”
type = “S”
}
}

modules/dynamodb/variables.tf
variable “KKE_DYNAMODB_TABLE_NAME” {
type = string
}

modules/dynamodb/outputs.tf
output “table_name” {
value = aws_dynamodb_table.this.name
}


modules/secretsmanager/main.tf

resource “aws_secretsmanager_secret” “this” {
name = var.KKE_SECRET_NAME
}

resource “aws_secretsmanager_secret_version” “this” {
secret_id = aws_secretsmanager_secret.this.id
secret_string = var.KKE_SECRET_VALUE
}

modules/secretsmanager/variables.tf
variable “KKE_SECRET_NAME” {
type = string
}

variable “KKE_SECRET_VALUE” {
type = string
}

modules/secretsmanager/outputs.tf

output “secret_arn” {
value = aws_secretsmanager_secret.this.arn
}

modules/elasticsearch/main.tf

resource “aws_elasticsearch_domain” “this” {
domain_name = var.KKE_ELASTICSEARCH_DOMAIN
elasticsearch_version = “7.10”

cluster_config {
instance_type = “t3.small.elasticsearch”
}

ebs_options {
ebs_enabled = true
volume_size = 10
}
}

modules/elasticsearch/variables.tf

variable “KKE_ELASTICSEARCH_DOMAIN” {
type = string
}

modules/elasticsearch/outputs.tf
output “endpoint” {
value = aws_elasticsearch_domain.this.endpoint
}

shared/variables.tf

variable “KKE_ENV” {}
variable “KKE_DYNAMODB_TABLE_NAME” {}
variable “KKE_SECRET_NAME” {}
variable “KKE_SECRET_VALUE” {}
variable “KKE_ELASTICSEARCH_DOMAIN” {}

shared/main.tf

module “dynamodb” {
source = “/home/bob/terraform/modules/dynamodb”
KKE_DYNAMODB_TABLE_NAME = var.KKE_DYNAMODB_TABLE_NAME
}

module “secretsmanager” {
source = “/home/bob/terraform/modules/secretsmanager”
KKE_SECRET_NAME = var.KKE_SECRET_NAME
KKE_SECRET_VALUE = var.KKE_SECRET_VALUE
}

module “elasticsearch” {
source = “/home/bob/terraform/modules/elasticsearch”
KKE_ELASTICSEARCH_DOMAIN = var.KKE_ELASTICSEARCH_DOMAIN
}

shared/outputs.tf

output “kke_dynamodb_table_name” {
value = module.dynamodb.table_name
}

output “kke_secret_arn” {
value = module.secretsmanager.secret_arn
}

output “kke_elasticsearch_domain_endpoint” {
value = module.elasticsearch.endpoint
}


env/dev/terraform_config.tf
terraform {
backend “local” {
path = “terraform.tfstate”
}
}

env/prod/terraform_config.tf

terraform {
backend “local” {
path = “terraform.tfstate”
}
}

Hi @tamilzhnationalist

I noticed you created the Terraform files in the shared folder. Please try again and follow the structure shown below.

image

Link command:

ln -s /home/bob/terraform/main.tf main.tf
ln -s /home/bob/terraform/variables.tf variables.tf
ln -s /home/bob/terraform/outputs.tf outputs.tf
ln -s /home/bob/terraform/modules/ modules

Hi @raymond.baoly, I am not succeeding in keeping the terraform state for both dev and prod environments when I apply the configuration after the other.

In the terraform_config.tf, I have this like @tamilzhnationalist mentioned above

env/dev/terraform_config.tf

terraform {
backend “local” {
path = “terraform.tfstate”
}
}

env/prod/terraform_config.tf

terraform {
backend “local” {
path = “terraform.tfstate”
}
}

But it is not working. I am quite lost to be honest.

I thank you in advance for your support.

Best regards,
Nathan

Hi @Nathan92

Here is the code I used to complete the task. Please refer to it and try again.

Many thanks @raymond.baoly for your time and support. However, unfornutalety after 2 attempts with absolute and relativ paths, I am still NOT able to complete the task. I checked the existence of the terraform.tfstate file for both environments. They contain the right information. But after the apply command for prod, Terraform quite lost its state even with the terraform.tfstate. I tried to reapply the dev but the prod lost its state this time. I made some search on the web and copilot without any success.

Did you check and use my code?

Sorry for the delay to answer you. But yes I used your code and followed exactly your written steps.

Hi @raymond.baoly,
Like even after using the solution you provided by updating the parameters only. It is giving the same error

I really can’t figure out why the AssertionError is coming again and again. Can someone help on this…?

I’m not getting this one to work either, but to make it easier to try out Raymond’s files, I’ve pushed them to a github repo here. Use the repo to access his files (this will be faster) and see if his “steps.sh” script works for you. That should give you a bit more to work with.

1 Like

Thanks @rob_kodekloud for uploading the solution to Git.

Hi @Yash-Varma

I’ve just checked and confirmed the same issue as you. This is a new issue, and I’ve forwarded it to the team. I’ll keep you updated.

Hi @Yash-Varma

The reason is that we haven’t updated the contents of env/dev/dev.tfvars and env/prod/prod.tfvars to match the requirements from the task. Please try again and make sure these files are updated correctly. For example:
sed -i ‘s/xfusion-/nautilus-/g’ env/dev/dev.tfvars env/prod/prod.tfvars

Hi @raymond.baoly,
Can you please share the updated dev.tfvars and prod.tfvars file which is according to the current lab setup

The task generates requirements randomly, so I can’t provide a fixed solution for it. You need to read the task carefully and update the configuration accordingly.

Hi @raymond.baoly,
I actually tried again but getting the same error.



as I feel the Output coming is also fine:

Outputs:

kke_dynamodb_table_name = “xfusion-prod-table”
kke_elasticsearch_domain_endpoint = “xfusion-prod-es.us-east-1.es.localhost.localstack.cloud:4566”
kke_secret_arn = “arn:aws:secretsmanager:us-east-1:000000000000:secret:xfusion-prod-secret-ERhiaP”

the task requirements generates randomly, but some parameters that you are using and I am not. Your solution can help me with that will get the understanding and make the changes accordingly.

As far as I am understanding that the solution I am doing is fine. It is just the strict validation of the Lab is causing the same error again and again

Hi @Yash-Varma

Here is the requirement for the outputs.tf file. I have also updated the GitHub solution. Please check it and try again.