How to install Apache Spark

Mac OS/X

Installing Spark on Mac OS/X will involve downloading the required
archive files into the final location where we want to run Spark and also Scala from. On my machine that directory location is:
/Users/myuser/local

  1. cd to destination directory:
    cd /Users/myuser/local
  2. Download Spark from http://spark.apache.org/downloads.html
    wget http://apache.arvixe.com/spark/spark-1.4.1/spark-1.4.1.tgz
  3. Download scala from http://www.scala-lang.org/download/
    wget http://downloads.typesafe.com/scala/2.11.7/scala-2.11.7.tgz
    
  4. Uncompress the spark and scala archive files you downloaded:
    tar -xvzf spark-1.4.1.tgz
    tar -xvzf scala-2.11.7.tgz
    

    The above step uncompresses the file contents into these directories: spark-1.4.1, scala-2.11.7

  5. Set the following environment variables – SCALA_HOME, SPARK_HOME, PATH:

    export SCALA_HOME=/Users/myuser/local/scala-2.11.7
    export SPARK_HOME=/Users/myuser/local/spark-1.4.1
    export PATH=$PATH:$SCALA_HOME/bin:$SPARK_HOME/bin
    
  6. Run sbt – Scala build tool
    cd $SPARK_HOME
    build/sbt clean assembly
    
  7. Run spark shell
     ./bin/spark-shell 

    Sometimes you may get an error about an invalid sbt-launch file.
    To resolve this do the following:

    cd build
    wget http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.7/sbt-launch.jar
    mv sbt-launch.jar sbt-launch.0.13.7.jar
    

Reference Links

Leave a Reply

Your email address will not be published. Required fields are marked *