Storm Supervisor Error - kill ***: No such process

My Storm cluster has 4 nodes: 1 nimbus node and 3 supervisor nodes. When I was trying to start supervisor using the command:

storm supervisor

It's strange that 2 supervisors got started successfully but 1 supervisor not. When I check the log:

It is saying lots of kill ***: No such process errors. After some search I find it's an error caused by workers not get started.

When checking the log of one of the worker  $STORM_HOME/logs/worker-****.log

It shows that worker died, because there are missing information from zookeeper, which comes from unexpected shutdown of the storm nodes.

The solution is this:
1. Kill all the running topology (I did this step but not sure whether this is an mandatory step)
2. Clear the supervisor data information in storm data folder, for example my storm data is /opt/mount/data/storm-data, and there is a folder called supervisor and maybe one called worker, please delete both of them.

The storm data location is defined in the storm.yaml file,

After all these, start storm again, and it's running happily!

If you find this blog is useful, please kindly click the ads on this page to help. Thank you very much.