Note to myself: do not forget iptables on AWS!

Last week I spent a few hours struggling against an Oracle database connectivity problem on a RedHat machine on AWS that almost made me jump from the window.

The database seemed OK but I was not able to remote connect to it.

To start with, the usual verifications:

 - Are Oracle env variables correctly configured?

 env | grep -i ora 

 - Is the DB up and running?

 ps -ef | grep -i pmon 

 - Are the listeners up and running?

 lsnrctl status
 netstat -ln

 - Can I connect to the DB?

 sqlplus <user>/<passwd>@<host>:<port>/<service>

Everything seemed fine but I was still not able to remote connect to the database. I could connect locally but not remotely.

So I checked my tnsnames.ora. I've had this problem before, the listener which listens only on localhost (127.0.0.1). That was not the problema now, though. The listener was correctly configured on tnsnames.ora.

Then I turn my attention to AWS security groups. Could it be it was blocking my incoming connections? I checked everything and it looked fine. But I still couldn't connect remotely. I changed a few permissions. No luck. Then I completely open the machine to the world. Still no luck.

I was almost filling a bug report on AWS security groups when I decided to look for local firewalls. Bingo! Iptables was the culprit. Problem solved simply flushing all iptables rules:

 sudo iptables -F

To make it persistent:

 sudo service iptables save

Lesson learned: even though it seems completely illogical to me to have iptables blocking connections on a AWS machine (which is already behind the security groups rules) some people do it and there are even some AMIs which come bundled that way. So, always check iptables (and/or other local firewalls) if your security groups rules seem correct but you can't connect to your instance!