Apache Spark Cluster On Local Machines Setup

Before setting up an apache spark cluster on your server environment, you might want to test it by setting up a similar configuration on your local environment and play around with it.

Spark Installation 

Download Apache Spark from the link or you can install it from the command line from here

• Next verify Java installation with the command below:

$java -version

If it has been installed you should see a similar response below:

java version "1.7.0_71" 

Java(TM) SE Runtime Environment (build 1.7.0_71-b13) 

Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode)

If not then you have to install it

• Next verify that Scala has been installed 

$scala -version

That should also give a response to the below mesage

Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL

If not then you have to install it.

Continuing with Apache Spark installation 

• Extract the tar file

$ tar xvf spark-1.3.1-bin-hadoop2.6.tgz 

• Move the spark folder to the desired directory e.g (/usr/local/spark)

$ su – 

Password:  

# cd /home/Hadoop/Downloads/ 

# mv spark-1.3.1-bin-hadoop2.6 /usr/local/spark 

# exit 

• Add the following line to the ~/.bashrc file. The essence of this is to add the location where the spark source file are located to the path variable.

export PATH = $PATH:/usr/local/spark/bin

Use the following command for sourcing the ~/.bashrc file.

$ source ~/.bashrc

• Verify the successful spark installation on the desired system 

$spark-shell

If its successful you should see the following response below:

Spark assembly has been built with Hive, including Datanucleus jars on classpath 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 

15/06/04 15:25:22 INFO SecurityManager: Changing modify acls to: hadoop
15/06/04 15:25:22 INFO SecurityManager: Changing view acls to: hadoop 

ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop) 
15/06/04 15:25:22 INFO SecurityManager: SecurityManager: authentication disabled;
15/06/04 15:25:22 INFO HttpServer: Starting HTTP Server 

/ __/__  ___ _____/ /__ 
15/06/04 15:25:23 INFO Utils: Successfully started service 'HTTP class server' on port 43292. 
Welcome to 
      ____              __ 
    _\ \/ _ \/ _ `/ __/  '_/ 
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0 
      /_/  
  

scala> 
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_71) 
Type in expressions to have them evaluated. 

Spark context available as sc  

All systems that would be on the cluster must first have spark running on them, hence you would have to perform the above operation on all the systems.

Setting-Up The Local Cluster

The way the spark cluster works is one system is the master and the rest are the slaves:

• Go to SPARK_HOME/conf and create a file with the name spark-env.sh.

There will be spark-env.sh.template in same folder and this file gives you details on nhowto declare various environment variables. 

• Enter the master ip address on the masters system

• Open the slaves file in the same folder i.e SPARK_HOME/conf and if there is none then create it. Note the slave file does not have any extension.

All these files must be saved on all systems with the same data i.e the master system ip address entered in the spark-env.sh file on all systems and the slave ip address of all other systems entered in the slaves folder of all systems as well. This is very important.

• Navigate to the spark folder sbin directory (/usr/local/spark/sbin) and enter the following command

$sudo ./start-all.sh

• Enter the password on the prompt
• Go to your browser and enter IP_ADDRESS_OF_YOUR_MASTER_SYSTEM:8080 in the URL and press enter

The sudo command is very important or you will get and error message of permission denied.

Note:

The ./start-all.sh command is preferred to starting the master first and then subsequently starting all the slaves afterwards which you can see how that works in the references I added but if you prefer that route then you can do that. I prefer this where one command can be used to start all the systems at once and then stop them afterwards.

you might get an error message of access during after running the start all command but I have actually forgotten how I overcame that access error. If you do get that error. Please contact me and I would help resolve it. Enjoy.

Reference

http://paxcel.net/blog/how-to-setup-apache-spark-standalone-cluster-on-multiple-machine/

https://www.tutorialspoint.com/apache_spark/apache_spark_installation.htm

Unable To Locate Package Skype and None Responsive Install button Errors While Installing Skype On Ubuntu

This is an error message that occurs many times while trying to install Skype on Ubuntu 16.04.3 LTS by using the terminal interface.
This post has been updated and the link to skype for linux can be found at Skype for linux so, I suggest you try this first before you continue trying to fix this error with the solutions provided below.

Link to official skype for linux Skype

unable to locate package skype

Un-clickable Install button for skype

This error can be unclear and many of the solutions you will see online point to the same method which at times is what results in this particular error message. I will list out some possible routes to solve this error during installation.

Installing Skype alternatively From the website: 

• A direct Download of the Deb package for Skype for Linux with your favorite web browser or HTTP client.

Running the commands on the terminal:
Open the terminal and type in the following commands:

• sudo add-apt-repository "deb http://archive.canonical.com/ $(lsb_release -sc) partner"
• sudo apt-get update
• sudo apt-get install skype

Installing a 32 bit version of Skype:

For 11.10 and above

Starting in Ubuntu 11.10 multi-arch is used for applications that formerly used ia32-libs. Here is the official work around for Skype on AMD64.

Skype is a 32 bit binary and under Ubuntu 11.10 you must install the 32 bit version of Skype for both Ubuntu 11.10 i386 and amd64 versions.

You can get this from the Ubuntu Partner Repository.

• To get it from the partner repository first enable it in your software sources, and then get it from the Software Center:
• Click the ubuntu button, then search for "Software Sources", click on it, and go to the "Other Software Tab":

The UI way

The Command line way

• Open the sources.list file: gksudo gedit /etc/apt/sources.list (or with the command line editor of your choice, sudo nano /etc/apt/sources.list would use nano on the command line without a GUI)
• Add the partner repositories by removing the # in front of the following lines (maverick is the version of your Ubuntu installation, it may differ, so use the codename of the release you are using instead of 'maverick'. If you're not sure run lsb_release -c to find out.)

         # deb http://archive.canonical.com/ubuntu maverick partner

         # deb-src http://archive.canonical.com/ubuntu maverick partner

When you click close run the software center, and search for Skype.

If skype still does not appear in software-center, then close the application.

In a terminal type:

  sudo apt-get update

  sudo apt-get --reinstall install software-center

This will force the package management to be updated, followed by the software-catalog for software-center.

To install the Skype for Linux application (v5 Beta):

Note: The legacy Skype for Linux application (v4.*) is now discontinued and was removed from Canonical’s partner repositories1, so most of the answers here are obsolete.

All Ubuntu releases as of July 2017

1. Download the Deb package for Skype for Linux with your favourite web browser or HTTP client. 
• For a terminal-based solution use:
• wget https://repo.skype.com/latest/skypeforlinux-64.deb
2. Install the Deb package with your favourite package manager, e. g. Software Center or GDebi.
• For a terminal-based solution or to debug the package installation use:
• sudo dpkg -i skypeforlinux-64.deb
• sudo apt install -f
3. You’re done! When you installed the package it automatically added a repository […] to keep your installation of Skype up to date via Ubuntu's package update process.
4. Sometimes like in my case after all this the Skype does not still get installed and so you might have to return to the downloaded "Deb package for Skype for Linux" and re-lunch it. This time on clicking of the install button a password prompt will be shown to the user and on entering the password, skype will be successfully installed.