Default eks cluster security group ammendment

Terraform
#1

Hi,

I created cluster for learning terraform on aws and noticed that eks create the security group itself…even though I have created security group for cluster and nodegroup both …eks cluster created sg has allow rule for ALL.
What if I want to ammend this rule using terraform ?
I don;t want all ports and protocol and only suppose specific port like 8080 (except ports needed by k8s itself for its working)

I tried adding in vpc_config as

cluster_additional_security_group_ids = [aws_security_group.additional_sg.id]
but this doesn;t work

my cluster tf is as below :

resource “aws_eks_cluster” “testcluster” {
name = “${var.cluster_name}”
role_arn = aws_iam_role.tfiamrole.arn
vpc_config {
security_group_ids = [“${var.addsg}”]
cluster_additional_security_group_id = “${var.addsg}”
subnet_ids = [
“${var.private-1a}”,
“${var.private-1b}”,
“${var.private-1c}”,

]

}

depends_on = [aws_iam_role_policy_attachment.eks-AmazonEKSClusterPolicy]
}

Can some1 suggest the fix please?

#2

I don’t think you can do it as you create the EKS Cluster. You will have to use a second run to make the modifications to the security group.

#3

what if

  1. in first run we get the ID of the security group created by EKS, we store it somewhere in variable file or output file , can be used in step 3
  2. we will implement the sleep timewait suppose 5 minutes
  3. After sleep time gets over another tf will get executed example with below content

resource “aws_security_group_rule” “eks_default_sg_rule” {
security_group_id = “sg-029a25ca856aa96bb”
type = “ingress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“100.100.212.12/32”]
}

So can you guide me for point 1 ?

Regards
Nitish Patni

#4

Once the terraform has been run, run terraform with the output parameter and assign it to a variable in bash:

MY_NSG=$(terraform output -raw nsg)
#5

but how this NSG can be used in point 3 ?

#6

Also I am validating terraform cloud so i don;t think we have bash access on free trial account.
My requirement I can edit the security group after creation in single run

#7

Something like this:

terraform apply -var nsg=$MY_NSG
#8

How can you run something after, in a single run? Your logic makes no sense.

#9

so there is a typo in my text

what i wanted is

  1. I have multiple tf script like one creates security group , one create nodegroup , one create cluster
  2. so once the cluster.tf gets executed it will create the default security group

Now at this moment I want to get output of default security group created by EKS and run another tf script securitygroup.tf (this security group.tf is depends on cluster.tf and time_wait sleep of 5 minutes)

  1. this securitygroup.tf edit the default security group like below

resource “aws_security_group_rule” “eks_default_sg_rule” {
security_group_id = “sg-029a25ca856aa96bb”
type = “ingress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“100.100.212.12/32”]
}

  1. nodegroup will get attached to the cluster once the nodegroup.tf gets executed

and this way the complete execution gets finish.

so I just want to know if we can achive this what I wrote after point 2 and before point 3

#10

You could try it this way.

Have a data source that refers to the EKS cluster that is being deployed, like this.

From this data source you should be able to read the security group that was created by the cluster and use it in an aws_security_group_rule to add new rules. Ensure the rule resource has a depends_on to the cluster resource to ensure it is fully created first.

#11

I tried below :

data “aws_eks_cluster” “test” {
name = aws_eks_cluster.test.name
}

output “security” {
value = “${data.aws_eks_cluster.test[*].vpc_config[0].cluster_security_group_id}”
}

resource “aws_security_group_rule” “eks_default_sg_rule” {
depends_on = [aws_eks_node_group.private-nodes]
security_group_id = “data.aws_eks_cluster.test[*].vpc_config[0].cluster_security_group_id”
type = “ingress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“100.100.2.1/32”]

above tf file is from securitygroup.tf

i made depends on on nodegroup creation complete as I want first complete cluster including nodegroup gets up and running then only security group gets change.

Can you help in correcting the above or guide me as right now with that I am getting following error :

Error: InvalidGroupId.Malformed: Invalid id: “data.aws_eks_cluster.test[*].vpc_config[0].cluster_security_group_id” (expecting “sg-…”)
│ status code: 400, request id: 5c98158a-950c-437d-902d-d2ec3ed313ec

#12

Comment out the security group role for now.

What does this output print? It’s clearly not a single security group ID, which will be why you get the error, meaning there is something wrong wit your expression.

output "security" {
“${data.aws_eks_cluster.test[*].vpc_config[0].cluster_security_group_id}”
}
#13

aws_security_group_rule.eks_default_sg_rule will be created

  • resource “aws_security_group_rule” “eks_default_sg_rule” {
    • cidr_blocks = [
      • “100.100.2.1/32”,
        ]
    • from_port = 0
    • id = (known after apply)
    • protocol = “-1”
    • security_group_id = “data.aws_eks_cluster.capiods[*].vpc_config[0].cluster_security_group_id”
    • self = false
    • source_security_group_id = (known after apply)
    • to_port = 0
    • type = “ingress”
      }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
** + security = [**
** + “sg-029a25ca856aa96bb”,**
** ]**

after applying getting below

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:

  • security = [
    • “sg-029a25ca856aa96bb”,
      ]

Do you want to perform these actions?
Terraform will perform the actions described above.
Only ‘yes’ will be accepted to approve.

Enter a value: yes

aws_security_group.tfsg1: Creating…
aws_security_group_rule.eks_default_sg_rule: Creating…
╷
│ Error: InvalidGroupId.Malformed: Invalid id: “data.aws_eks_cluster.capiods[*].vpc_config[0].cluster_security_group_id” (expecting “sg-…”)
│ status code: 400, request id: 46a763f9-a81d-43b6-8240-dda1c56b4420
│
│ with aws_security_group_rule.eks_default_sg_rule,
│ on ammendmentsg.tf line 17, in resource “aws_security_group_rule” “eks_default_sg_rule”:
│ 17: resource “aws_security_group_rule” “eks_default_sg_rule” {

kindly suggest please

#14

The output is showing you that the value you have retrieved is a list of security group IDs.
The rule resource expects a single security group.

Before you make further changes, inspect sg-029a25ca856aa96bb in the console and determine it is the SG you expect. If it is, then you should be able to pick it off with a final [0] on the end of the expression.

#15

Not sure where is the problem but yes sg-029a25ca856aa96bb this is the required sg group

[root@ip-172-31-1-56 eksdeploy]# cat ammendmentsg.tf
#data “aws_security_group” “cluster_security_group” {

name = “aws_eks_cluster.test[].vpc_config[].cluster_security_group_id”

#}

data “aws_eks_cluster” “test” {
name = aws_eks_cluster.test.name
}

output “security” {
value = “${data.aws_eks_cluster.test[].vpc_config[].cluster_security_group_id}”
}

resource “aws_security_group_rule” “eks_default_sg_rule” {
depends_on = [aws_eks_node_group.private-nodes]
security_group_id = “data.aws_eks_cluster.test[].vpc_config[].cluster_security_group_id[0]”
type = “ingress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“100.100.2.1/32”]
}

when doing terrafor apply getting below error

aws_security_group_rule.eks_default_sg_rule: Creating…
╷
│ Error: InvalidGroupId.Malformed: Invalid id: “data.aws_eks_cluster.test[].vpc_config[].cluster_security_group_id[0]” (expecting “sg-…”)
│ status code: 400, request id: 747cb385-6ea3-4cff-a52a-c9e262fd1cb2
│
│ with aws_security_group_rule.eks_default_sg_rule,
│ on ammendmentsg.tf line 17, in resource “aws_security_group_rule” “eks_default_sg_rule”:
│ 17: resource “aws_security_group_rule” “eks_default_sg_rule” {
│
╵

#16

I even tried using node_group_resource

[root@ip-172-31-1-56 eksdeploy]# cat ammendmentsg.tf
data “aws_security_group” “cluster_security_group” {
name = “aws_eks_cluster.capiods.node_group_resources[0].security_group_id}”
}

#data “aws_eks_cluster” “capiods” {

name = aws_eks_cluster.capiods.name

#}

#output “security” {

value = “${data.aws_eks_cluster.capiods[].vpc_config[].cluster_security_group_id}”

#}

resource “aws_security_group_rule” “eks_default_sg_rule” {
depends_on = [aws_eks_node_group.private-nodes]
security_group_id = “data.aws_security_group.cluster_security_group.name”
type = “ingress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“100.100.2.1/32”]
}

and with this got error as below :

Error: no matching SecurityGroup found
│
│ with data.aws_security_group.cluster_security_group,
│ on ammendmentsg.tf line 1, in data “aws_security_group” “cluster_security_group”:
│ 1: data “aws_security_group” “cluster_security_group” {

#17

This worked

data “aws_eks_cluster” “test” {
name = “test”
}

resource “aws_security_group_rule” “eks_default_sg_rule” {
depends_on = [aws_eks_node_group.private-nodes]
security_group_id = data.aws_eks_cluster.test.vpc_config[0].cluster_security_group_id
type = “ingress”
from_port = 0
to_port = 0
protocol = “-1”
cidr_blocks = [“100.100.100.1/32”, “100.100.100.0/24”]
}

#18

I knew you’d get there in the end :wink:

It was just a case of understanding the schema of data.aws_eks_cluster to correctly target the security group ID.

You will come across this kind of problem a lot when working with terraform, especially when using resources you may not have used before. It’s simply a case of “work it out”. Putting things as outputs can help you understand what certain attributes look like - e.g. did you get a list when you expected a single item.

#19

Thank you so much for your guidence…This helps in our learning path.