How to run this java program that is using spark?

Question

First off let me start by saying I really have no experience with spark, but I need to add some logic to a program and test it locally. Everything I believe is already functioning and set up as this program is in a production environment. I may leave out some details code wise if it contains confidential information.

What I THINK i've figured out is that i need to run the program passing command line arguments, but to start the program i'm really at a loss. When i do the command

--verbose result: Using properties file: null Error: Must specify a primary resource (JAR or Python or R file) Run with --help for usage help or --verbose for debug output

The properties file is located in src/main/resources, as for the primary resource (assuming a jar) not really sure.

If anybody could help point step through this it would be highly appreciated, i don't really have the time to learn/study up a bunch on Spark so i apologize for the i'm assuming rather trivial question. Thank you so much in advance for any help.

Answer 1

Just incase anybody ever stumbles across this thread. I had to run the file that held the main class as a java application and give it two params in run configurations that the program was designed to take at start up.

Answer 2

Keep your MAVEN-GENERATED-JAR-FILE-0.0.1-SNAPSHOT.jar in the folder where you are doing spark-submit on the edge node

Then try this

spark-submit --class com.xxxx.xxx.xxxxx.xxxxxx.yourProgramName MAVEN-GENERATED-JAR-FILE-0.0.1-SNAPSHOT.jar

Once this works, you may add other required parameter. And do advance stuff.