Jan 30, 2015

Problems in YARN: Why only SecondaryNameNode started? [Solved]

I was configuring a YARN cluster and after all set I tried to start services.
./start-dfs.sh
./start-yarn.sh

To my surprise, I only see SecondaryNameNode service got started. What? I didn't configure the SecondaryNameNode service, also where is my NameNode service which I really want?

I also checked the log and it says that:


The log shows that port 50070 already in use. But I run a command:
lsof -i:50070
There is nothing there using the port.

Also the host address is 0.0.0.0, this seems very wrong.
I checked many posts and they all didn't guide me to the root cause. Finally I found out the root cause

[Solution]

in hdfs-site.xml, there is one entry that's necessary:


dfs.http.address
myHostNodename:50070


If you want your secondary namenode, please add another one


dfs.secondary.http.address
my2ndHostNodename:50090

Happy World!

If you find this blog is useful, please kindly click the ads on this page to help. Thank you very much.

Jan 20, 2015

Storm Supervisor Error - kill ***: No such process

My Storm cluster has 4 nodes: 1 nimbus node and 3 supervisor nodes. When I was trying to start supervisor using the command:

storm supervisor

It's strange that 2 supervisors got started successfully but 1 supervisor not. When I check the log:

It is saying lots of kill ***: No such process errors. After some search I find it's an error caused by workers not get started.

When checking the log of one of the worker  $STORM_HOME/logs/worker-****.log


It shows that worker died, because there are missing information from zookeeper, which comes from unexpected shutdown of the storm nodes.

The solution is this:
1. Kill all the running topology (I did this step but not sure whether this is an mandatory step)
2. Clear the supervisor data information in storm data folder, for example my storm data is /opt/mount/data/storm-data, and there is a folder called supervisor and maybe one called worker, please delete both of them.

The storm data location is defined in the storm.yaml file,


After all these, start storm again, and it's running happily!

If you find this blog is useful, please kindly click the ads on this page to help. Thank you very much.

Jan 6, 2015

[Solved]HBase Java program Joda time problem - NoSuchMethodError: org.joda.time.DateTime.now()

I am writing a HBase program and use Joda time as my time type. I have a simple api call

System.out.println(DateTime.now())  which DateTime was imported as

In Eclipse the code is compiled successfully, but when I tried to execute it it poped up a strange error:  NoSuchMethodError: org.joda.time.DateTime.now()

What? DateTime.now() is a well-known API and I check the source code, it's available since Joda jar 2.0, in my maven dependency it clearly shows that my joda jar library is 2.0. Then I even manually added a latest version which is version 2.6 and clearly see that the 2.6 version is picked up.

But, the problem still exists. What's wrong with Joda?
Then I see that in my project, I also have storm-core api, which also has a joda-time dependency, so I exclude it. Now I see that
in Eclipse the DateTime also has library, and it points to jruby-complete-1.6.8.jar, then it tells me whether the problem is.


Root cause:
1. my HBase program imported external jars, one of them is jruby-complete-1.6.8.jar, which internally has a joda-time reference, but it's a very low version.

2. somehow this version got picked up and makes DateTime.now() has a problem as it's only available after 2.0

Solution
1. remove the external library jruby-complete-1.6.8.jar in Java Build Path (as the screenshot above, choose the jruby-complete-1.6.8.jar and click Remove)
2. Use maven to add jruby-complete-1.6.8 dependency


org.jruby
jruby-complete
1.6.8
        provided



3. Add Joda-time dependency and make sure it comes first as the screenshot below shows

joda-time
joda-time
2.6



DateTime.now() is a wonderful world!

If you find this blog is useful, please kindly click the ads on this page to help. Thank you very much.