IntelliJ "Provided" Scope Problem

IntelliJ is such a famous tool that lots of users saying that once you get used to it, you can never come back. To a Eclipse user, there is some learning curves to use IntelliJ. Today I met a interesting problem that exist only in IntelliJ.

The background is that I moved my project from Eclipse to IntelliJ, the project runs very well in eclipse but I used the same way to run in IntelliJ (short as idea), it throws the error like below:

I searched a lot and finally find that it's because the storm package in pom.xml is specified as provided.

 org.apache.storm
 storm-core
 0.10.0-beta1
 provided

The reason the package must be specified as provided is that the storm cluster already contains the storm jar file, if our package contains the storm jar file again, it will cause conflicts.

But according to the definition of provided from maven website: https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html

• provided
  This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

This is why there is no problem when compile but has problems during runtime.

Here we come to a dilemma: when run it in IDE (intelliJ) we need to scope to be compile, but when package it to deploy on cluster, we need it to be package scope. One way to solve this is each time we change the scope, but this is very annoying. 

So here is the solution:

1. Set the scope to be a parameter, such as ${myscope}  

 org.apache.storm
 storm-core
 0.10.0-beta1
 ${myscope}

2. create a mvn task
expand Lifecycle, right click an item for example install, and choose create Practise...

then specify command line: clean install -Dmyscope=provided

With this, it's running very well and also can package with provided scope!

If you think this article is useful, please click the ads on this page to help. Thank you very much.

Storm slf4j multiple binding problem

We are using Storm 0.9.3 and when we try to start supervisor node we found it reported the error below:

Clearly this is a slf4j multiple binding problem. Basically this famous library is used in different jars, when there are multiple slf4j find in the path only one will be picked up so that some jars called in a different version of slf4j will cause these compatibility problems. 

For the log above it clear states that there are 3 slf4j bindings under lib

• slf4j-jdk14-1.5.6.jar
• slf4j-nop-1.5.3.jar
• logback-classic-1.0.13.jar

And it needs 1.6 above as it said

SLF4J: slf4j-api 1.6.x (or later) is incompatible with this binding.

SLF4J: Your binding is version 1.5.5 or earlier.

SLF4J: Upgrade your binding to version 1.6.x.

Storm is compiled and tested, why it will have multiple slf4j* jars? The actual result is Storm vanilla package is good, but we add some jars to its lib folder:

So that when we package our Storm jar, we don't need each time to include all the packages it required, this is to save time: we develop on windows and upload developed package jar into cluster, if we can have a smaller generated jar file we save our time. 

Let's not judge whether this method is good or not, but the problem here is to solve the binding problem.

The solution is simple: 

From eclipse dependency 

Let's remove these two jars which are  slf4j 1.5.*

• slf4j-jdk14-1.5.6.jar
• slf4j-nop-1.5.3.jar

It can successfully launch now.

Then here comes another question: when we package Storm application, we want solve the slf4j multiple binding problem?

There is a nice pom.xml for you to reference (partial of pom.xml), this is what I searched and copied from internet, honor and thanks go to the original author.

 org.codehaus.gmaven
 gmaven-plugin
 1.5
 
  
   package
   
    execute
   
   
    
    File targetDir = new
    File("${project.basedir.path}/target".toString())
    println "dir is ${targetDir.path}"
    String jarBaseName = "${project.artifactId}-${project.version}"
    File jarWithUnwantedStuff = new File(targetDir,
    "${jarBaseName}-jar-with-dependencies.jar".toString())

    def explodedJarDir = new File(targetDir, "explodedJar".toString())
    def ant = new AntBuilder() // create an antbuilder
    ant.unzip(src: "${jarWithUnwantedStuff.path}",
    dest: explodedJarDir.path,
    overwrite: "false")
    File finalJar = new File(targetDir, "${jarBaseName}-deployable.jar")
    unwantedClassesDir = new File(explodedJarDir,
    "/org/slf4j/impl".toString())
    unwantedClassesDir.deleteDir()
    ant.zip(basedir: explodedJarDir.path, destFile: finalJar.path)
   
  
 

If you find this blog is useful, please kindly click the ads on this page to help. Thank you very much.

Storm Supervisor Error - kill ***: No such process

My Storm cluster has 4 nodes: 1 nimbus node and 3 supervisor nodes. When I was trying to start supervisor using the command:

storm supervisor

It's strange that 2 supervisors got started successfully but 1 supervisor not. When I check the log:

It is saying lots of kill ***: No such process errors. After some search I find it's an error caused by workers not get started.

When checking the log of one of the worker  $STORM_HOME/logs/worker-****.log

It shows that worker died, because there are missing information from zookeeper, which comes from unexpected shutdown of the storm nodes.

The solution is this:
1. Kill all the running topology (I did this step but not sure whether this is an mandatory step)
2. Clear the supervisor data information in storm data folder, for example my storm data is /opt/mount/data/storm-data, and there is a folder called supervisor and maybe one called worker, please delete both of them.

The storm data location is defined in the storm.yaml file,

After all these, start storm again, and it's running happily!

If you find this blog is useful, please kindly click the ads on this page to help. Thank you very much.