I encountered a new scenario that I hadn’t in the past.
The Problem
I was trying to access Elasticsearch on a Linux VM and it wasn’t allowing me to connect to the port 9200.
Check Ports and Processes
I checked if the process was running and the correct ports were open –
$ ps aux | grep elastic
elastic+ 18403 7.3 11.6 3094568 239536 ? Sl 15:08 0:32 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -d -p /var/run/elasticsearch/elasticsearch.pid --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.conf=/etc/elasticsearch
akalyana 18506 0.0 0.0 14040 916 pts/4 S+ 15:15 0:00 grep elastic
$ netstat -ntl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:34221 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:53015 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:52230 0.0.0.0:* LISTEN
tcp6 0 0 :::45196 :::* LISTEN
tcp6 0 0 :::111 :::* LISTEN
tcp6 0 0 :::9200 :::* LISTEN
tcp6 0 0 :::9300 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 ::1:631 :::* LISTEN
tcp6 0 0 :::44580 :::* LISTEN
This sorted, the challenge was finding out why it never worked. Digging deeper and talking to a colleague, I found out the problem was in the iptables.
Enter iptables
iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores.
Source: Wikipedia
Basically although there were no firewall rules for the network, iptables had to be explicitly configured for the port to be opened. Here is what my iptables really contained –
$ sudo iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -f -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m udp --dport 123 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 11111 -j ACCEPT
-A INPUT -p udp -m udp --dport 11111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2050 -j ACCEPT
-A INPUT -p udp -m udp --dport 111 -j ACCEPT
-A INPUT -p udp -m udp --dport 2049 -j ACCEPT
-A INPUT -p udp -m udp --dport 2050 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m addrtype --dst-type BROADCAST -j DROP
-A INPUT -m addrtype --dst-type MULTICAST -j DROP
Configure iptables
I added the rule to open port 9200 on iptables and verified the same.
$ sudo iptables -A INPUT -i eth0 -p tcp -m tcp --dport 9200 -j ACCEPT
$ sudo iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -f -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m udp --dport 123 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 11111 -j ACCEPT
-A INPUT -p udp -m udp --dport 11111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2050 -j ACCEPT
-A INPUT -p udp -m udp --dport 111 -j ACCEPT
-A INPUT -p udp -m udp --dport 2049 -j ACCEPT
-A INPUT -p udp -m udp --dport 2050 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m addrtype --dst-type BROADCAST -j DROP
-A INPUT -m addrtype --dst-type MULTICAST -j DROP
-A INPUT -i eth0 -p tcp -m tcp --dport 9200 -j ACCEPT
Reloading the web page fixed the issue
Gotchas and Useful Information
Do not restart iptables service once you insert a new rule because it flushes out your entry.
Interestingly, when you add a new rule on iptables, it ends up storing it temporarily. Here is the output after restart –
$ sudo service iptables restart
Starting iptables: iptables.
$ sudo iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -f -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m udp --dport 123 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 11111 -j ACCEPT
-A INPUT -p udp -m udp --dport 11111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2050 -j ACCEPT
-A INPUT -p udp -m udp --dport 111 -j ACCEPT
-A INPUT -p udp -m udp --dport 2049 -j ACCEPT
-A INPUT -p udp -m udp --dport 2050 -j ACCEPT
-A INPUT -s 172.17.80.50/32 -i eth0 -j ACCEPT
-A INPUT -s 172.17.80.51/32 -i eth0 -j ACCEPT
-A INPUT -s 172.22.186.23/32 -i eth0 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m addrtype --dst-type BROADCAST -j DROP
-A INPUT -m addrtype --dst-type MULTICAST -j DROP
You can ensure the new rules you added persist by backing up current iptables rules. You need to install a utility called iptables-persistent using the apt package installer.
This command needs root access. It will not work with sudo.
# apt-get install iptables-persistent
# iptables-save > /etc/iptables/rules.v4
