1

I am stuggeling with port 9000: Connection refused error.

I work on Ubuntu 14.04 and faced the problem when trying to run Hadoop in a non-distributed mode, as a single Java process (compare Hadoop 2.4.1 documentation). I tried to follow Hadoop Wiki suggestions on this error (hadoop/ConnectionRefused) but I did not succed (I am a beginner Ubuntu user and find it difficult even to 100% understand the suggestions given). I posted a stackoverflow question from which I conclude that I have some general problem with port 9000 Connection.

telnet output:

martakarass@marta-komputer:~$ telnet localhost 9000
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

nmap output: 

martakarass@marta-komputer:~$ nmap localhost

Starting Nmap 6.40 ( http://nmap.org ) at 2015-04-27 11:09 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00022s latency).
Not shown: 995 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
139/tcp open  netbios-ssn
445/tcp open  microsoft-ds
631/tcp open  ipp
902/tcp open  iss-realsecure

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

Netcat configuration:

I tried to make use of the following command to force the 9000 port to be open:

nc -k -l 9000

but it did not work well (I was still not able to perform the standlone operation mentioned and linked above).

Judging from my google research results, I see that the problem is quite common and poses a huge struggle especially for those who are not good at "admin-job-related issues". As I belong to those, I kindly ask for answers to the following questions:

  • Q1: What is the origin of such problem in general? (Some for a layman introductory words / references about basic issues connected to ports / connections etc. would be very very welome).

  • Q2: How to deal with this problem?

Update.

sudo netstat -nlp | grep :9000

returns nothing.

Community
  • 1
Marta Karas
  • 19
  • 1
  • 1
  • 7
  • 2
    Are you sure the hadoop process is actually running / listening on port 9000? please add the output of sudo netstat -nlp | grep :9000 to your question. – steeldriver Apr 27 '15 at 12:10
  • Hi @steeldriver , thank you for your attention! I updated my post. – Marta Karas Apr 28 '15 at 19:15
  • 2
    Based on the netstat output, the most likely explanation is that the hadoop process has either failed to start, or is not listening on port 9000. – steeldriver Apr 28 '15 at 19:32
  • Please note it is a Hadoop operation in standalone mode. According to the documentation hadoop-modes-explained-standalone, I am not expected to perform any configuration which makes Hadoop listening on port 9000 (unless I misunderstood something). – Marta Karas Apr 29 '15 at 09:55
  • 1
    dupe: http://askubuntu.com/questions/352868/i-cant-connect-to-hadoop-port-9000 Please see if the accepted answer gets you on the right track – Rinzwind Apr 30 '15 at 08:19
  • 1
    Hi @Rinzwind , thank you for your attention. I had visited this post (and tried the solutions provided) before posting this one. A careful reader would even notice my comment there, dated Apr 26 at 20:42 :) Unfortunately, the solution that solved an issue in that case ("i got overlapping ip ports with some apps on my host machine (...) sudo kill $(sudo lsof -t -i:9000)afterwards everything was ok") does not work in mine case (in my case sudo lsof -t -i:9000 reports nothing). – Marta Karas Apr 30 '15 at 11:04
  • it really looks like the hadoop process is just not running. you can take a look at ps -aux if you can find the process... – Ben Apr 30 '15 at 18:58

3 Answers3

1

TL;DR first, more details following:

"Connection refused" is the error you get when you try to connect to a service on a computer or server (or locally on your own computer) when the said service is either not listening on the specified TCP port (wrong port or service not started), or when a firewall explicitly rejects a connection instead of ignoring the request (not very common behavior).

And now a little more detail:

It can only happen with TCP (UDP services are sessionless) and this error is indeed very common for sys admins.

When a client application connects to a TCP service, it'll send a first packet with the SYN flag set. If we simplify, there are two possible answers to that:

  • The server actualy listens on the required TCP port and replies with a SYN-ACK packet to acknowledge the SYN packet, after which the clients rends an ACK packet to "confirm" to the server that the SYN-ACK was received. That's the moment where you have established a TCP session, and when you can start "talking" with the server.
  • The server receives your SYN packet, but isn't listening on the requested port. The connection is rejected with a packet that has the RST (reset) flag set. This is 99% of the time when you get the infamous "connection refused".

How do I fix this ?

Well, there are a few things you can check: Are you querying the right port on the client side ? Is your service started ? Is it listening on the right port ?

Those three questions will generally help you solve your issue.

Quick generic command to be entered on the server (the machine that runs the service)

Checking that the service is listening on the expected port

ss -nat | grep <enter port number here> | grep LISTEN

Some people on older systems might also use this

netstat -an | grep <enter port number here> | grep LISTEN

If you don't see anything here that looks like your port number, your service is either not started or not listening on the port number you specified.

Checking that the service is running

service <service name> status
Marc-Olivier Barre
  • 71
  • 5
  • I should also add that this question (and its answer) is very different from http://askubuntu.com/questions/352868/i-cant-connect-to-hadoop-port-9000.

    The other question is hadoop related, while the OP here wanted information that is generic to all "Connection refused" errors. Hadoop here is out of the scope of the question.

     – Marc-Olivier Barre Apr 30 '15 at 18:55
  • Hi @Marc-Olivier Barre, thank you a lot for this answer. I assume "my service" is, in this case, the same as "the client" and equals my single machine (please correct me if I am wrong). Do I understand well that I "myself" (my machine) want to connect with "myself" and it is not possible because it, most likely, is not listening on the port 9000? (I do not understand how " Is your service started ?" could be my case.) In my case ss -nat | grep 9000 | grep LISTEN returns nothing. Could you please provide some guide on *how to make my service listen on the port 9000*? – Marta Karas Apr 30 '15 at 19:18
  • Little explanation for the comment above: I do not understand how " Is your service started ?" could be my case because, as I understand, "my service" is my machine and when I try to perform the operation on the machine it means that the machine is, obvioulsly, running. – Marta Karas Apr 30 '15 at 19:21
  • ok, vocabulary needs a little explanation: service is hadoop in the OP case. Client is generally a browser, or the telnet program for instance. Server is generally the machine running the service. Client program and service can be located on the same machine (i.e. your laptop). In the case the server is just your laptop. – Marc-Olivier Barre May 01 '15 at 09:10
  • "In your hadoop configuration files, does fs.default.name and mapred.job.tracker refer to localhost? If so, then Hadoop will only listen to port 9000 and 9001 on the loopback interface, which is inaccessible from any other host. Make sure fs.default.name and mapred.job.tracker refer to your machine's externally accessible host name" -- source http://stackoverflow.com/questions/4855808/hadoop-job-tracker-only-accessible-from-localhost – Aravinda May 01 '15 at 09:50
  • Hi @Aravinda , thank you for your comment. I am not sure if I am missing domething, but I understand from the post you sourced that "Hadoop will only listen to port 9000 and 9001 on the loopback interface, which is inaccessible from any other host" and, as "The loopback device is a special, virtual network interface that your computer uses to communicate with itself" it sould be sufficient for what I am in need for when it comes to runing Hadoop in a non-distributed mode, on a single machine - isn't it? – Marta Karas May 01 '15 at 15:07
  • According to stackoverflow answer, you may check the config files !! – Aravinda May 01 '15 at 15:10
  • I am having similar problem with with remote xdebug with netbeans. I can see with netstat that netbeans is listening on port 9000. Nevertheless from the remote server I can not telnet my machine with netbeans. What should be done on this case? – Christoforos Jan 05 '20 at 09:35
1

Try do disable iptables:

sudo service iptables stop && sudo service ip6tables stop

Then restart hadoop. If this helped you need make proper configuration for your firewall.

UNIm95
  • 696
  • 7
  • 21
  • Hi @UNIm95 , thank you for your answer. I forgot to mention in the post that this is not my case (I investigated this following suggestions in other stackoverflow question that I linked in my question here). Please note I have eventually found a solution which I describe in the answer I markeds as "accepted" here. – Marta Karas May 05 '15 at 12:28
0

Eventually, I managed to make my service listen to the port 9000 by adding to the /etc/ssh/sshd_config file the following line:

Port 9000

I followed this serverguide/openssh-server (it contains also some important remarks about making a copy of the original file, restarting the sshd server application etc.)

After this I can see:

telnet output:

martakarass@marta-komputer:~$ telnet localhost 9000
Trying 127.0.0.1...
Connected to localhost.

nmap output:

martakarass@marta-komputer:~$ nmap localhost

Starting Nmap 6.40 ( http://nmap.org ) at 2015-05-01 18:28 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00023s latency).
Not shown: 994 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
139/tcp  open  netbios-ssn
445/tcp  open  microsoft-ds
631/tcp  open  ipp
902/tcp  open  iss-realsecure
9000/tcp open  cslistener

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

netstat output:

martakarass@marta-komputer:~$ sudo netstat -nlp | grep :9000
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      16397/sshd      
tcp6       0      0 :::9000                 :::*                    LISTEN      16397/sshd
Marta Karas
  • 19
  • 1
  • 1
  • 7