I am certain that I saved all the firewall rules on all the app servers but the task came back as failed and claimed that I had not saved them on the app1 server. Also I was going to go back to the app1 server to take a screenshot as it claimed that I had 5 minutes and I could send this to prove my case but the environment went offline right away.
I just reviewed your answer and found that you saved the rules in /etc/iptables.conf file however you never restored/saved them to the original file i.e /etc/sysconfig/iptables so rules were not considered permanent.