Saturday, August 4, 2018

Assignment on graph processing using GraphX in Apache Spark

  1. Generate a un-directional graph RDD from a given graph data.
  2. Compute listed vertex-based similarity measures for all the pairs of nodes in label data file. These similarity measures are computed between two nodes by utilizing neighborhood and/or node information of both nodes. Common neighbors Jaccard coefficient Adamic/Adar Preferential Attachment
  3. Bonus Question: Link Prediction Model Using the measures generated from the graph and labels from the labeled data to predict the possibility of new link formation. Please use the following steps. 1. Create a dataset by combining measures ( as features) and class labels from the labeled data. 2. Use decision tree based algorithm in SparkML to train the prediction model. 3. Split the dataset to generate the training data and testing data. 4. Use training data to build model and testing data to evaluate the model. 5. Present the model performance metrics: Accuracy, Recall, and Precision.
  1. Please follow the program submission instructions ( same as the previous assignments)
  2. Must use spark and GraphX for generating measures and use SparkML for bonus questions.
  3. More explanation on above graph measures here (
Graph data:
Use this link ( dl=0) to download graph data and file is formatted as shown below:

Label data:
click here (h ?dl=0) to download label data

Friday, December 15, 2017

Apache Spark Cluster On Local Machines Setup

Before setting up an apache spark cluster on your server environment, you might want to test it by setting up a similar configuration on your local environment and play around with it.

Spark Installation 
Download Apache Spark from the link or you can install it from the command line from here
  • Next verify Java installation with the command below:

$java -version
If it has been installed you should see a similar response below:
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b13)
Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode)
If not then you have to install it
  • Next verify that Scala has been installed 

$scala -version
That should also give a response to the below mesage
Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL
If not then you have to install it.

Continuing with Apache Spark installation 
  • Extract the tar file

$ tar xvf spark-1.3.1-bin-hadoop2.6.tgz 
  • Move the spark folder to the desired directory e.g (/usr/local/spark)

$ su –
# cd /home/Hadoop/Downloads/
# mv spark-1.3.1-bin-hadoop2.6 /usr/local/spark
# exit
  • Add the following line to the ~/.bashrc file. The essence of this is to add the location where the spark source file are located to the path variable.

export PATH = $PATH:/usr/local/spark/bin
Use the following command for sourcing the ~/.bashrc file.
$ source ~/.bashrc
  • Verify the successful spark installation on the desired system 

If its successful you should see the following response below:
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Using Spark's default log4j profile: org/apache/spark/
15/06/04 15:25:22 INFO SecurityManager: Changing modify acls to: hadoop
15/06/04 15:25:22 INFO SecurityManager: Changing view acls to: hadoop
ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/06/04 15:25:22 INFO SecurityManager: SecurityManager: authentication disabled; 15/06/04 15:25:22 INFO HttpServer: Starting HTTP Server
/ __/__ ___ _____/ /__
15/06/04 15:25:23 INFO Utils: Successfully started service 'HTTP class server' on port 43292. Welcome to ____ __ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.4.0 /_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_71) Type in expressions to have them evaluated.
Spark context available as sc

All systems that would be on the cluster must first have spark running on them, hence you would have to perform the above operation on all the systems.

Setting-Up The Local Cluster
The way the spark cluster works is one system is the master and the rest are the slaves:
  • Go to SPARK_HOME/conf and create a file with the name

There will be in same folder and this file gives you details on nhowto declare various environment variables. 
  • Enter the master ip address on the masters system

  • Open the slaves file in the same folder i.e SPARK_HOME/conf and if there is none then create it. Note the slave file does not have any extension.

All these files must be saved on all systems with the same data i.e the master system ip address entered in the file on all systems and the slave ip address of all other systems entered in the slaves folder of all systems as well. This is very important.
  • Navigate to the spark folder sbin directory (/usr/local/spark/sbin) and enter the following command

$sudo ./
  • Enter the password on the prompt
  • Go to your browser and enter IP_ADDRESS_OF_YOUR_MASTER_SYSTEM:8080 in the URL and press enter
The sudo command is very important or you will get and error message of permission denied.
The ./ command is preferred to starting the master first and then subsequently starting all the slaves afterwards which you can see how that works in the references I added but if you prefer that route then you can do that. I prefer this where one command can be used to start all the systems at once and then stop them afterwards.

you might get an error message of access during after running the start all command but I have actually forgotten how I overcame that access error. If you do get that error. Please contact me and I would help resolve it. Enjoy.


Friday, August 25, 2017

Unable To Locate Package Skype and None Responsive Install button Errors While Installing Skype On Ubuntu

This is an error message that occurs many times while trying to install Skype on Ubuntu 16.04.3 LTS by using the terminal interface.
This post has been updated and the link to skype for linux can be found at Skype for linux so, I suggest you try this first before you continue trying to fix this error with the solutions provided below.

Link to official skype for linux Skype

unable to locate package skype
Un-clickable Install button for skype

This error can be unclear and many of the solutions you will see online point to the same method which at times is what results in this particular error message. I will list out some possible routes to solve this error during installation.

Installing Skype alternatively From the website: 

Running the commands on the terminal:
Open the terminal and type in the following commands:
  • sudo add-apt-repository "deb $(lsb_release -sc) partner"
  • sudo apt-get update
  • sudo apt-get install skype
Installing a 32 bit version of Skype:
For 11.10 and above
Starting in Ubuntu 11.10 multi-arch is used for applications that formerly used ia32-libs. Here is the official work around for Skype on AMD64.
Skype is a 32 bit binary and under Ubuntu 11.10 you must install the 32 bit version of Skype for both Ubuntu 11.10 i386 and amd64 versions.
You can get this from the Ubuntu Partner Repository.
  • To get it from the partner repository first enable it in your software sources, and then get it from the Software Center:
  • Click the ubuntu button, then search for "Software Sources", click on it, and go to the "Other Software Tab":

The UI way

The Command line way
  • Open the sources.list file: gksudo gedit /etc/apt/sources.list (or with the command line editor of your choice, sudo nano /etc/apt/sources.list would use nano on the command line without a GUI)
  • Add the partner repositories by removing the # in front of the following lines (maverick is the version of your Ubuntu installation, it may differ, so use the codename of the release you are using instead of 'maverick'. If you're not sure run lsb_release -c to find out.)
         # deb maverick partner
         # deb-src maverick partner

When you click close run the software center, and search for Skype.

If skype still does not appear in software-center, then close the application.
In a terminal type:

  sudo apt-get update
  sudo apt-get --reinstall install software-center

This will force the package management to be updated, followed by the software-catalog for software-center.

To install the Skype for Linux application (v5 Beta):

Note: The legacy Skype for Linux application (v4.*) is now discontinued and was removed from Canonical’s partner repositories1, so most of the answers here are obsolete.

All Ubuntu releases as of July 2017

  1. Download the Deb package for Skype for Linux with your favourite web browser or HTTP client. 
    • For a terminal-based solution use:
      • wget
  2. Install the Deb package with your favourite package manager, e. g. Software Center or GDebi.
    • For a terminal-based solution or to debug the package installation use:
      • sudo dpkg -i skypeforlinux-64.deb
      • sudo apt install -f
  3. You’re done! When you installed the package it automatically added a repository […] to keep your installation of Skype up to date via Ubuntu's package update process.
  4. Sometimes like in my case after all this the Skype does not still get installed and so you might have to return to the downloaded "Deb package for Skype for Linux" and re-lunch it. This time on clicking of the install button a password prompt will be shown to the user and on entering the password, skype will be successfully installed.

Sunday, August 14, 2016

The type 'Microsoft.Reporting.WebForms.ReportViewer' exists in both

Resolving the error message " The type 'Microsoft.Reporting.WebForms.ReportViewer' exists in both 'c:\Windows\assembly\GAC_MSIL\Microsoft.ReportViewer.WebForms\\Microsoft.ReportViewer.WebForms.dll' and 'c:\Windows\assembly\GAC_MSIL\Microsoft.ReportViewer.WebForms\\Microsoft.ReportViewer.WebForms.dll' "

First Solution:
The first solution to this type of error is to download and install the  Microsoft Report Viewer 2010 Redistributable Package called the ReportViewer.exe which can be found here. Once this file has been downloaded and installed the issue will be resolved

Second Solution:
This second solution mainly depends on which version of the Microsoft Report Viewer you want in the solution. The first step will be to navigate to the assembly folder under windows folder and copy out the required version dll to either the bin or package folder and reference it directly from there into the required class library in the solution. The dll's in the assembly can not be accessed through the GUI on the windows operating system but can be accessed through the command line interface.
For the Webforms dll navigate to :

C:\Windows\assembly\GAC_MSIL>cd Microsoft.ReportViewer.WinForms
For the Common dll navigate to :
C:\Windows\assembly\GAC_MSIL>cd Microsoft.ReportViewer.Common

Based on the dll required you will make a final navigation into the dll's folder for the version 8.0.0.
you will type cd and press enter. They execute a copy command from the command line interface to copy the dll from the folder to an accessible directory like the document or download folder in the operating system.
Wikihow has an article on how you can do that here
Third Solution:
Find all the references the these dll files inside the web.config file in solution and manually change the numbers from one to another. So if its the version 8 you want then you must change all the places where exist to Also ensure you check inside the bin folder for web.config files there as well and if such files exist then you should change the referenced versions in this files to the desired one as well.

Tuesday, August 9, 2016

The Art Of Debugging (Applications In The Localhost/Live Environment)

Bugs lie in hidden places. In programming we get to create as we architecture and create ingenuously by coding a application from ideas but more often than not there is a bug or two hiding withing out architecture waiting to reveal it self at the most inconvenient of times. Debugging is the art of finding out those bugs and eliminating them from the code perfecting the application. making it a masterpiece if you will. There are many ways to spot and error and many tools to aid you as a developer. It behooves you as a developer to take advantage of this tools to make the debugging process as effective and efficient as possible. Often we have a local environment to run, test and debug the application before pushing it to the staging or live environment but sometimes and most especially for legacy solutions, we have applications that for one reason or the other one might not be running on the local environment (I know it might sound incredible nevertheless its true with SITECORE kind of solutions) or bugs that don't real veal themselves in the localhost only to rear their ugly head in the live environment.

This article will focused on the two scenarios:

  1.  An application you can run and debug on the localhost environment.
  2. An application you have access to the codes but must debugged on the live environment.

Localhost Debugging (Debugging In a local environment):
As it is with all architectures we begin with the presentation layer i.e the view and proceed to the back-end before ending in the database layers. Each of these layers have tool where a software developer can quickly check to find error messages in the application which can give an alert of the problem and more importantly a hint to the required solution.


Setting A JQuery Break-Point:
Stepping through the JQuery code to see the input field values in real-time, one can find out which
variables are throwing an undefined value or giving an error message when reached. Sometimes the back-end runs smoothly nevertheless the code keeps breaking in the view because one input field or the other. Using inspect element and clicking on the source tab in the lower part of the view and putting a break-point on the Javascript or JQuery codes to see what is actually being returned to the view and implementing a solution based on that. It is prudent to solve a view problem in the view with either css, JQuery/Javascript or Html and not pass the wrong value to the code-behind and then attempt to solve the problem there, for one thing it prevents having to reload the page unnecessarily many times when you can simply check the value with Html or Jquery and give the user a feedback/error message if its an input problem thereby making you application faster and more user friendly.

Using The Debugger Syntax:
Its possible that a javascript/Jquery code might not be able to set a break-point at a position in the code you would like or there might actually be a break-point there but still the break-point is being skipped then manually inserting the debugger syntax in the JQuery code to force the code to stop on the JQuery part at run-time is a good way around this problem. Always keep in mind there a many ways to skin a cat.

Effect: Being able to step through the JQuery codes without needing to set breakpoints in the view as the code will automatically stop at these points when hit. Remember to remove them as soon as you are done as they can equally be a bug on the live environment.

Setting A Break-Point In A Method Inside The Code:
Debugging with an IDE (Integrated Development Environment) e.g Visual Studio. We are all familiar with this so I need not dwell on the process. It simply involves stepping through the code till the exception or error message is thrown. At times it might require using a try catch to get the error message in details this is useful when dealing with imported dll's created by others an example is EntityFrameWorks.

Changing The parameter Value At Run-Time:
Another great way to know what is happening where inside the code. Setting a fixed value for a property field or a Boolean controlling a conditional statement might be a great way to see what happens when a particular area of the code is reached.

Using A Watch In Visual Studio IDE:
If you have a variable in the code that constantly changes values like a counter in a for or foreach loop statement, then setting a Watch can be used to see all the various values that the variable changes into. Also you might not want to see all the values but might only be interested in a situation where the variable value is e.g 2 and in that case a watch can be set for when the value is two and when you run the application it will get into the for loop and halt at the point where the variable is exactly the value you want and you can carry on your debugging from there.


Debugging SQL Stored Procedures:
Just the same as setting a breaking point in Visual Studio but this is done in the Microsoft SQL Server Management Tool or whatever Database management tool you are using.  Used mainly for stored procedures and SQL queries in general you set the break-point and when the code hits you begin to step through to find-out where the error is coming from. Very useful then the error has nothing to do with both the presentation(view) or business logic (code-behind) layers but is with when the value is either being gotten from or returned into the database.

Executing The Stored procedure:
In this technique you add the values directly into the stored procedure through the SQL server management studio to see the out puts. Really comes in handy when what you just want to do is work with the SQL query alone and not have to run the entire application and add values from the view in order to see the effect in the database.
Once the stored procedure values and the execute button clicked it triggers the specified procedure and outputs the value which the developer can check to confirm if its selecting or inserting the correct value from/into the database.

Using SQL Profiler Tool:
The SQL profiler is a powerful tool for debugging the data layer as sometimes you might not know which particular stored procedure is being called in a particular method or at what point exactly, the SQL profiler can help here and save you the trouble of running the application and then religiously following each method to found out which stored procedure is called as the SQL profiler displays the queries called making it easier to know what is called and when.

Debugging In A live environment (When the application does not run on the localhost):
Many are times you might not be able to debug the application locally but must run it either in a staging or live environment and troubleshoot it there to discover the error and resolve the issue.
This can seem a monumental task and be really frustrating is one doesn't know how to go about fixing the error. Below are few good ways of debugging the application and finding and fixing the error.

Decompiling The DLL File:
No Code is the grail. It might come in handy to decompile the dll file (if and only if one is not breaking any copywriter violation law) in-order to see what is really going on and either insert a personal code that handles it, add a save method in the code that saves the exception, error message or value at that point in the database or ought right implement the same method in ones code and avoid calling the dll in that instance but implement your own method that also solves the error based on the method on the decompiled code of the dll file. One might even learn a thing or two in the process.

Logging The Error In The Database
Using third party dll's like Elmah a software developer can log run-time errors with the dates/time and then go through the errors to know what is breaking in the application. This could also be useful even when the application doesn't stop working or give and error to the user but breaks internally and continues working due to the fact that the exception is caught in a try catch and the finally kicks in. An error is and error and the objective is to eliminate all bug.

Using Fiddler:
Fiddler is another great product from the Telerik team and this can be used to test API's with POST and GET requests or monitor the error message from a browser when an event is triggered from the view. The same bug might have different error messages depending on the medium used to view it. It might give a partial/summarized version in the view but with fiddler a more detailed error message can be seen when the log of the action is clicked on and examined closely. can equally be used to test an API. An example of this is during a post of the required data to a payment gateway that doesn't save the response message in a place where the developer can see the status like sandbox, a site like this is the best bet where the generated link will be assigned as the call back URL and then the returned data can be checked.

Mobile View Of The Web App In Chrome:
Some errors are peculiar to mobile versions i.e they might render well in the desktop version  of the application and even in one or two mobile devices but eventually fail on another mobile device (a solution I once worked on ran smoothly on a Windows and an Android phone but failed on an IOS) and in this case the best way to go about it is to render the application in the desired mobile version and troubleshoot it from there. By inspecting the element and selecting the various versions of mobile devices using the google chrome browser one can simulate with as many devices as possible and debug accordingly.

Checking The JQuery Console For Error Messages:
The Colsole tab in the browser after an inspect element has been activated is another place where a developer can see certain errors and carve out a solution from there.

Using Different Browsers:
Just like the difference between the the desktop and the mobile device or to be more concise, even among various mobile devices it is a good procedure to debug the application/solution with different browsers as well not only for how the view renders in the browser but also if an error occurs in a certain browser that doesn't occur in the others. Checking older versions of a browser is not an overkill as its difficult to know which browser the user will be accessing the site with hence due diligence on all fronts is considered a prudent and responsible practice in this case.

Minimizing the size of your browser is not the same thing as effectively checking a page's rendering and functionality in the various mobile views. One should either check it inside the device or use the chrome browser tool to debug that. using notepad can also help to clearly look throught the error message because some times the message is compressed together but when copied and pasted in a notepad file one can easily look through it and separate it out to check the line numbers which are sometimes correctly included in the error message. Setting fixed values to variables to see what breaks is another good way to troubleshoot/debug. Using alerts in the view can sometimes also be helpful when debugging through JQuery codes as this can be used to return values to the developer to see what occurs at what point in the code. Removing the detailed things (number lines, name of solution and methods) that are peculiar to your error message and dumping the rest in google to search for a similar problem and the proposed solution.
Finally when all else fails reaching out by sourcing for help from a colleague or posting the error in a company or general developers problems/solutions forum e.g stack overflow e.t.c can be the last step of the debugging process.

Sunday, January 24, 2016

Error installing Visual Studio 2015 followed by error installing Windows update kb2919355

Installing an version of Visual Studio is a time consuming task that is very well worth the wait as its a debugging tool second to none. Nevertheless when faced with some installation challenges it could be a challenging task that could take several days of online and offline research, consultation, experimentation and what have you to resolve it. I would like to share my experience of an error of installing Visual studio 2015 requiring the installation of windows update KB2919355 (this version of Visual Studio requires the April 2014 update to windows 8.1 and windows server 2012 R2 known as kb2919355).

which was made more gruesome by the error installing kb2919355 (this package is not applicable to this computer)

So to resolve it (both errors). This was what I did.
This link will take you to the Microsoft site where you will download Windows Update Diagnostic. After downloading, run it and select Background Intelligence Transfer Service and click the next button.
  • Open your system command line in elevated mode i.e (by selecting Run as Administrator)
  • Scan Windows system components: sfc /scannow

  • Remove KB2919355 package:
    dism /online /remove-package /packagename:Package_for_KB2919355~31bf3856ad364e35~amd64~~    (this is for 64 bit systems.) for a (32 bit system use dism /online /remove-package /packagename:Package_for_KB2919355~31bf3856ad364e35~x86~~

  • Clean up the WinSxS folder: dism /online /cleanup-image /startcomponentcleanup /resetbase

  •  Restart computer again.
  • Run offline installer: Windows8.1-KB2919355-x86.msu
  • Restart computer (You have to do this several times, but keep checking Windows Update each time until the update is listed as "successful") .Tiring yes but we are developers and vigilance is our watch word.
Once this is done, its one error down and one more to go.
  • Well little else is required but to Run Visual Studio 2015 setup as an administrator.

With that done you are certainly good to go.
As with try and catch code block, finally if you system should prove to be too stubborn and still not install the downloaded kb2919355 package, you can always reconnect your antivirus, switch back on your internet and check for windows update through your control panel.
i.e Control Panel -> System and Security -> Check for updates (ensure you also chick both " Give me the recommended updates the same way I receive important updates "  and " Give me the updates for other Microsoft products when I update Windows." which are in Windows update change settings)

You might need to switch off your antivirus cause they have been various instructions, In some case the antivirus was temporarily switched off and in other cases it was not (It might be wise to be on the safe side and switch if off for the duration of resolving the issue) and you will definitely need to be on the internet during the download processes. I recommend you download all the required files prior to the installations and that way while your antivirus is down you will not be connected to the internet which will on one hand provide a level of security and protection. Also you must plug your computer to an uninterrupted power supply for the duration of this exercise.
Finally to others facing this issue don't give up and resort to formatting your system. This issue will be resolved, just keep at it and don't give up. Good luck.

Sunday, March 22, 2015

How To Upgrade (Flash) Linksys' WRT54G/GL/GS Firmware to Tomato Firmware For IP Address and Bandwidth Monitoring

As a System Administrator one is usually faced with the challenge of providing the best possible technology solution within the confinement of an economic budget. Hence most small scale companies will definitely prefer the zero cost of a firmware upgrade of a Linksys' WRT54G/GL/GS  router as opposed to the cost of purchasing a Server to perform the same function.

Below are the Stages involved:
  • Download Tomato_1_28 version, it can be gotten from various sites, simply type "download Download Tomato_1_28" into google search bar and a list of possible sites will popup an example of such sites is the URL : 
  • Download Toastman Firmware 1.28.7636 VLAN for Tomato wrt54gl. Most links for this download will take you to this URL below:

  • Download B1 Free Archiver (B1FreeArchiver.exe) From the URL below:

Having downloaded all these you are ready to proceed. You will need to extract the Tomato_1_28 folder from its .Zip file and for that you can use WinRAR or WinZip archiver software both easily available. You will also need to extract the Toastman Firmware 1.28.7636 VLAN from its .B1 file and thats what the B1 Free Archiver is for. Simply install it and use it to extract the Toastman Firmware 1.28.7636 VLAN folder. Now we go to work.

  • Using a network cable connect your PC to the LAN of your Linksys' WRT54G/GL/GS router
  • Open a browser and type (login: admin, passwd: admin)
  • Go to Administration > Firmware upgrade.
  • Click the "Browse" button and navigate to the Tomato folder and select the WRT54G_WRT54GL.bin file and click OK and then click the Upgrade button and the bottom of this page. Please note that during the upgrade there must be no interruption what so  ever , you are not to touch the computer or the router during this process and for places where power shortage is likely, you are advised to use an Uninterrupted Power Supply (UPS) device.
  • navigate to the Tomato folder and select the WRT54G_WRT54GL.bin file
  • Click the upgrade button and wait. Usually it takes between 10-25 minutes for the firmware upgrade to be complete.

  • After the firmware flashing has run its cause, you will realize that you can now monitor Bandwidth usage but IP Address monitoring is not yet available. For that you will have to upgrade the firmware from this basic Tomato firmware to Toastman firmware.
  • Go to Administration > Firmware upgrade.
  • Click the Browse button and navigate to Toastman Firmware 1.28.7636 VLAN and select tomato-NDUSB-1.28.7636Toastman-IPT-ND-VLAN-VPN.trx file.
  • Click the Upgrade button and wait for the flashing to run its cause without interruption and afterwards you will be able to monitor IP Address on the Linksys router.
  • Enjoy!!!
  • NOTE: If you want to clear all the data and settings on the Links router before the upgrade you will have to perform the 30-30-30 reset. i.e Hold down the reset button and count for 30 seconds and then unplug the power cable while still holding down the reset button, after another 30 seconds count plug back the power cable while still holding down the reset button and finally after a final 30 seconds remove your hand from the reset button and all the data will be cleared. If you do this before the first firmware flashing all the data will be cleared and after the flashing you will be required to create a new password and reconfigure all the settings again i.e wireless name and passwords e.t.c. but this is not necessary if you don't wish to clear all the current data, you can very well successfully flash the device without resetting it.

How To Upgrade (Flash) Linksys' WRT54G/GL/GS Firmware to Tomato Firmware For IP Address and Bandwidth Monitoring

As a System Administrator one is usually faced with the challenge of providing the best possible technology solution within the confinem...