How to install Apache Spark

Mac OS/X

Installing Spark on Mac OS/X will involve downloading the required
archive files into the final location where we want to run Spark and also Scala from. On my machine that directory location is:

  1. cd to destination directory:
    cd /Users/myuser/local
  2. Download Spark from
  3. Download scala from
  4. Uncompress the spark and scala archive files you downloaded:
    tar -xvzf spark-1.4.1.tgz
    tar -xvzf scala-2.11.7.tgz

    The above step uncompresses the file contents into these directories: spark-1.4.1, scala-2.11.7

  5. Set the following environment variables – SCALA_HOME, SPARK_HOME, PATH:

    export SCALA_HOME=/Users/myuser/local/scala-2.11.7
    export SPARK_HOME=/Users/myuser/local/spark-1.4.1
    export PATH=$PATH:$SCALA_HOME/bin:$SPARK_HOME/bin
  6. Run sbt – Scala build tool
    cd $SPARK_HOME
    build/sbt clean assembly
  7. Run spark shell

    Sometimes you may get an error about an invalid sbt-launch file.
    To resolve this do the following:

    cd build
    mv sbt-launch.jar sbt-launch.0.13.7.jar

Reference Links

Leave a Reply

Your email address will not be published. Required fields are marked *