tomer-ben-david
1/8/2015 - 9:34 AM

Run hadoop job in intellij or as main

Run hadoop job in intellij or as main

run.hadoop.job.in.intellij.or.as.main.scala
content_copyfile_download
Main class: com.twitter.scalding.Tool
VM options: -Xmx1024m
Program arguments: WordCountJob --local --input input.txt --output output.txt
Working Directory: C:\workspace\ScaldingExample

make sure your dependencies have:

<dependency><groupId>com.twitter</groupId><artifactId>scalding-core_2.10</artifactId><version>0.9.1</version></dependency>
<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-common</artifactId><version>2.0.0-cdh4.3.0</version></dependency>
<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-core</artifactId><version>2.0.0-mr1-cdh4.3.0</version></dependency>