IntelliJ "Provided" Scope Problem

IntelliJ is such a famous tool that lots of users saying that once you get used to it, you can never come back. To a Eclipse user, there is some learning curves to use IntelliJ. Today I met a interesting problem that exist only in IntelliJ.

The background is that I moved my project from Eclipse to IntelliJ, the project runs very well in eclipse but I used the same way to run in IntelliJ (short as idea), it throws the error like below:

I searched a lot and finally find that it's because the storm package in pom.xml is specified as provided.

 org.apache.storm
 storm-core
 0.10.0-beta1
 provided

The reason the package must be specified as provided is that the storm cluster already contains the storm jar file, if our package contains the storm jar file again, it will cause conflicts.

But according to the definition of provided from maven website: https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html

• provided
  This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

This is why there is no problem when compile but has problems during runtime.

Here we come to a dilemma: when run it in IDE (intelliJ) we need to scope to be compile, but when package it to deploy on cluster, we need it to be package scope. One way to solve this is each time we change the scope, but this is very annoying. 

So here is the solution:

1. Set the scope to be a parameter, such as ${myscope}  

 org.apache.storm
 storm-core
 0.10.0-beta1
 ${myscope}

2. create a mvn task
expand Lifecycle, right click an item for example install, and choose create Practise...

then specify command line: clean install -Dmyscope=provided

With this, it's running very well and also can package with provided scope!

If you think this article is useful, please click the ads on this page to help. Thank you very much.

Good tool Java Decompiler - View source codes from jar or .class file

One problem of using SVN is that you might meet conflicts and end up losing your own codes. I met the problem once: I did an update, and there are conflicts. I tried to revert the update but found out that the revert is reverting my local change but keep the update! Now I lost my local development work...

Luckily I have a compiled jar file which I build last time, but they are all .class files. How to view the original files?

Java Decompiler comes to the resue!
http://jd.benow.ca/

Use this tool to open a jar file or .class file, you will see all the contents:
It's structured and clear! Thanks to the author!

If you think this article is useful, please click the ads on this page to help. Thank you very much.

Hello Spark - How to configure Spark development environment on Windows with IntelliJ

1. Install prerequisites

a) JDK 1.7

Download it here

b) Scala 2.10.5

Download it here

3) Maven latest version

Download it here

4) SBT

Download it here

After install or unzip these components, configure them in the system variables, for example:
JAVA_HOME -->  C:\Program Files\Java\jdk1.7.0_60
MAVEN_HOME --> D:\Program Files\Dev\apache-maven-3.2.3
SCALA_HOME --> C:\Program Files (x86)\scala
SBT_HOME --> C:\Program Files (x86)\sbt\

and append the following items in PATH
%JAVA_HOME%\bin; %MAVEN_HOME%\bin;%SCALA_HOME%\bin;%SBT_HOME%\bin;

2. Download and setup IntelliJ IDE

a) Download IntelliJ here

b) Download Scala plugin

Note: it's mandatory to enable HTTP proxy if you are in a company firewall.

c) Configure Maven settings if under proxy

We can copy a settings.xml file from %MAVEN_HOME%\conf\settings.xml, and when it's under proxy, please copy it to C:\Users\*****\.m2 , and enable proxy

After modification:

3. Create project

Let's create a sample project called HelloSpark

1) File --> New --> Project..., please choose Scala

After click Finish, it will pop up a dialog, we can choose "New Window"

2). Add maven support

Right click project name and choose "Add Framework Support...", please scroll down and select "Maven"

Double click pom.xml and add the following content with existing content of pom.xml

    
        
        
            maven-compiler-plugin
            3.1
            
                1.7
                1.7
            
        
        
    

       
        
        
            org.apache.spark
            spark-core_2.10
            1.4.0
            provided
        
        
        
            org.apache.spark
            spark-mllib_2.10
            1.4.0
            provided
        

        
        
            org.apache.spark
            spark-sql_2.10
            1.4.0
        

        
        
            org.apache.spark
            spark-hive_2.10
            1.4.0
        

        
        
            org.apache.spark
            spark-streaming_2.10
            1.4.0
            provided
        
        
            org.apache.kafka
            kafka_2.10
            0.8.1
            provided
        
        
            org.apache.spark
            spark-streaming-kafka_2.10
            1.4.0
        
    
 
    
        
            Maven
            http://repo1.maven.org/maven2
        
        
            clojars
            http://clojars.org/repo/
        
        
            m2.java.net
            Java.net Maven 2 Repository
            http://download.java.net/maven/2
            default
        
    

Hello
A sample pom.xml can be viewed here

After pasted the content, on the top right it will pop up a dialog, please choose Enable Auto-Import and maven will start downloading specified dependencies.

Or you can do it via
right click project name--> Maven --> Reimport

3) Create a folder for scala

expand project file structure,  src--> main, right click main, New--> Directory,
name it as Scala

Then add this new folder "Scala" to project source
File--> Project Structure (shortcut Ctrl+Alt+Shift+S)

Modules--> scala -->Source , and as the screenshot shows, Click 1, 2 and 3, the result will display 4.

4) Create a scala class

Right click scala folder, new Scala class

Add modify the content as the screenshot.
Also please be noted
1) org.apache.spark.SparkContext need be imported.
2) create a file called pagecounts,
3) This program is to read the content from a file named pagecounts, and then print out the first 10 lines, and also print out the total line counts of this file.
You can put arbitrary content in pagecounts, a sample file can be viewed here. If you place in another folder, please modify the file path accordingly.

5) Add Spark jar file

We need to download and Spark latest package and unzip it
Go to: https://spark.apache.org/downloads.html
Downoad spark package, you can choose 2.4 or 2.6 based on your requirement. For example, a sampe spark-1.4.0-bin-hadoop2.4.tgz can be downloaded here.

After unzip, we can add the package in our project, click OK with the popup.

6) Set run configuration

in the IntelliJ menu, Run-->Edit Configuration, please choose Application and set up the content as the screenshot below

Final: Run it!

Click the run button on the toolbar, and the result is good!

Please note that in the beginning it will display SLF4J multiple binding problem and Winutil problem like java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. These can be ignored for now.

Happy Spark!


If you think this article is useful, please click the ads on this page to help. Thank you very much.

Nexus 5 OTA after rooted and TWRP installed

I have been using Nexus 5 for quite a long time, and it's great to frequently receive update notification from google to get me the latest version of Android OS. We often joke that as a Nexus user, we paid for the system and get the device for free.

The update is called OTA (Over The Air). But since google changed its policy of verifying whether an OTA can perform or not, our lives become hard. In short, most of Nexus users will root the device, unlock bootloader and flash third party recovery for instance TWRP. (With this, you have most of controls of the Android OS and install lots of applications.) After done these, the system files will be changed. Google now changed the OTA policy to not only check some of the files are original version but check the whole system partition, which means in our case the OTA check will definitely fail and cannot run the update.

Below is a sample error:
expects build fingerprint of google/hammerhead/hammerhead 5.0.1/LRX22C/160258:user/release-keys or google/hammerhead/hammerhead 5.1 LMY47D/1743759   user/release-keys 
this device has Android/omni_hammerhead/hammerhead:4.4.4/KTU84P/4ef4c299f5:eng/test-keys

When what to do?

I had a lot of pain trying different methods, each time failed by the phone cannot boot and I have to reinstall the whole factory image, which means I have a complete fresh OS and have to install all the applications! So each update I need to spend days to get my phone ready, this is really hard.

During this Labors' Day vacation, I decide to study this problem and luckily I found a solution.

First of all,
Don't try the sideload, don't try to use TWRP to install... I have tried all these and they failed. :)

Solution:

1. Download Nexus Root Toolkit and update it to the latest version
2. click

 3. Very importantly, very importantly, very importantly CHECK don't flash userdata

Then follow all the instructions to flash, after this is done, phone will boot and updates all the application. Last but not least, flash the latest custom recovery and the latest Super

Enjoy the latest Android!

If you find this article is useful, please click the ads on this page to help. Thank you very much.

Apache Commons CLI not working? No, it's a junk character problem

I am using Apache command line interface to accept run time parameters.

While in my code, I am using -t to specify run time topology name.
But to my surprise, it didn't work. I took 2 hours to debug this problem and finally by chance I compare the result and find our that

It's complaining about the - character. What is wrong with it?

I used Notepad++ and there is good plugin called HexEditor and it gives our the answer

This is causing strange problem.

You may ask why. This is because when someone sent me the command from Outlook, it was already junked. 

OK. the lesson and learn here is to be careful of the characters you pasted from Outlook, especially when it's in Chinese environment. We may try to type the command again to avoid problems.

If you find this article is useful, please click the ads on this page to help. Thank you very much.