Debugging ERR_CONNECTION_TIMED_OUT on Linux VM

I encountered a new scenario that I hadn’t in the past.

The Problem

I was trying to access Elasticsearch on a Linux VM and it wasn’t allowing me to connect to the port 9200.

ERR_CONNECTION_TIMED_OUT from Virtual Machine

Check Ports and Processes

I checked if the process was running and the correct ports were open –

$ ps aux | grep elastic
elastic+ 18403  7.3 11.6 3094568 239536 ?      Sl   15:08   0:32 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -d -p /var/run/elasticsearch/elasticsearch.pid --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.conf=/etc/elasticsearch
akalyana 18506  0.0  0.0  14040   916 pts/4    S+   15:15   0:00 grep elastic

$ netstat -ntl  
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:34221           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:53015           0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:52230           0.0.0.0:*               LISTEN     
tcp6       0      0 :::45196                :::*                    LISTEN     
tcp6       0      0 :::111                  :::*                    LISTEN     
tcp6       0      0 :::9200                 :::*                    LISTEN     
tcp6       0      0 :::9300                 :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:631                 :::*                    LISTEN     
tcp6       0      0 :::44580                :::*                    LISTEN   

This sorted, the challenge was finding out why it never worked. Digging deeper and talking to a colleague, I found out the problem was in the iptables.

Enter iptables

iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores.

Source: Wikipedia

Basically although there were no firewall rules for the network, iptables had to be explicitly configured for the port to be opened. Here is what my iptables really contained –

$ sudo iptables -S  
-P INPUT DROP  
-P FORWARD DROP  
-P OUTPUT ACCEPT  
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT  
-A INPUT -i lo -j ACCEPT  
-A INPUT -f -j ACCEPT  
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT  
-A INPUT -p udp -m udp --dport 123 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 11111 -j ACCEPT  
-A INPUT -p udp -m udp --dport 11111 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 2050 -j ACCEPT  
-A INPUT -p udp -m udp --dport 111 -j ACCEPT  
-A INPUT -p udp -m udp --dport 2049 -j ACCEPT  
-A INPUT -p udp -m udp --dport 2050 -j ACCEPT 
-A INPUT -p icmp -j ACCEPT  
-A INPUT -m addrtype --dst-type BROADCAST -j DROP  
-A INPUT -m addrtype --dst-type MULTICAST -j DROP

Configure iptables

I added the rule to open port 9200 on iptables and verified the same.

$ sudo iptables -A INPUT -i eth0 -p tcp -m tcp --dport 9200 -j ACCEPT  
$ sudo iptables -S  
-P INPUT DROP  
-P FORWARD DROP  
-P OUTPUT ACCEPT  
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT  
-A INPUT -i lo -j ACCEPT  
-A INPUT -f -j ACCEPT  
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT  
-A INPUT -p udp -m udp --dport 123 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 11111 -j ACCEPT  
-A INPUT -p udp -m udp --dport 11111 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 2050 -j ACCEPT  
-A INPUT -p udp -m udp --dport 111 -j ACCEPT  
-A INPUT -p udp -m udp --dport 2049 -j ACCEPT  
-A INPUT -p udp -m udp --dport 2050 -j ACCEPT  
-A INPUT -p icmp -j ACCEPT  
-A INPUT -m addrtype --dst-type BROADCAST -j DROP  
-A INPUT -m addrtype --dst-type MULTICAST -j DROP  
-A INPUT -i eth0 -p tcp -m tcp --dport 9200 -j ACCEPT

Reloading the web page fixed the issue

Accessing Elasticsearch Health from Virtual Machine

Gotchas and Useful Information

Do not restart iptables service once you insert a new rule because it flushes out your entry.

Interestingly, when you add a new rule on iptables, it ends up storing it temporarily. Here is the output after restart –

$ sudo service iptables restart   
Starting iptables: iptables.  

$ sudo iptables -S  
-P INPUT DROP  
-P FORWARD DROP  
-P OUTPUT ACCEPT  
-A INPUT -i lo -j ACCEPT  
-A INPUT -f -j ACCEPT  
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT  
-A INPUT -p udp -m udp --dport 123 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 11111 -j ACCEPT  
-A INPUT -p udp -m udp --dport 11111 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT  
-A INPUT -p tcp -m tcp --dport 2050 -j ACCEPT  
-A INPUT -p udp -m udp --dport 111 -j ACCEPT  
-A INPUT -p udp -m udp --dport 2049 -j ACCEPT  
-A INPUT -p udp -m udp --dport 2050 -j ACCEPT  
-A INPUT -s 172.17.80.50/32 -i eth0 -j ACCEPT  
-A INPUT -s 172.17.80.51/32 -i eth0 -j ACCEPT  
-A INPUT -s 172.22.186.23/32 -i eth0 -j ACCEPT  
-A INPUT -p icmp -j ACCEPT  
-A INPUT -m addrtype --dst-type BROADCAST -j DROP  
-A INPUT -m addrtype --dst-type MULTICAST -j DROP  

You can ensure the new rules you added persist by backing up current iptables rules. You need to install a utility called iptables-persistent using the apt package installer.

This command needs root access. It will not work with sudo.

# apt-get install iptables-persistent  
# iptables-save > /etc/iptables/rules.v4

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s