PySpark - Classification, Regression and Clustering Job

This week, I was able to run Logistic Regression, Linear Regression and k-means using data streamed into PySpark using HPCCFuseJ. The code can be found here.

Note:
- PySpark + HPCCFuseJ currently treats HPCC file as local file.
- The implementation has been used only to simple flat thor files. It has not been tested on more complicated datasets such as datasets with child datasets or records.
- The performance of the HPCCFuseJ has not been assessed yet. A good idea would be to test the performance of HPCCFuseJ and compare it to just scp'ng the file to the local machine.

Comments

Post a Comment

Popular posts from this blog

PySpark Error - EOFError

I send almost the whole afternoon trying to debug an EOFError while implementing KMeansStreaming which uses a streaming THOR file as input. ( code ). I found the reason why I was getting the error. There were two reasons: The first clue was “java.lang.IllegalArgumentException: requirement failed” – which means the training and the testing data are not of the same dimension. The second clue was actually the EOFError – which is actually an out of memory error. The workaround was to increase the memory allocated using the option –executor-memory. 
Read more

How to set up a Spark multi-node cluster on AWS-EC2

Context Our, Timothy Humphrey and I, aim was to benchmark the performance of Apache Spark against HPCC.  Tim had already a set up HPCC on AWS and had all the worker nodes on the same placement group so that the network latency is low. The easiest way to start a Spark multi-node cluster in AWS is to use AWS EMR. There are various resources like this , which explain  how to Spark multi-node cluster.  But, EMR service does not support placement group. Since, it only supports a small number of EC2 instance types, as compared to the EMR service. Hence, we decided to set up Spark on EC2 rather than using AWS EMR. The rest of the article will document the process. Disclaimer Most of the content of this document is from here  with some changes for example, the tutorial talks about install Spark on Ubuntu system and for our setup we wanted to use Centos (for legacy reasons). Steps 1. Spin N nodes on EC2. For this article, we assume N=5. Please make sure that the port...
Read more

Performance comparison between FUSE plugin based download and Browser based download - 2.0

Image
Updated version of this To measure performance between HPCCFuseJ based download and browser-based download, we measure the time taken to copy mounted folder (using HPCCFuseJ) to a local folder and compare it with the download time using the browser (for details refer to this ). The graph below shows the ratio:  In the figure, it can be seen that the time required to download using HPCCFuseJ increases as the file size increase. The exponential trend of the graph is because of multiple fetch nature of the HPCCFuseJ. HPCCFuseJ works in two phases namely: data fetching and data processing. Data fetching phase uses web service calls to fetch data. Data processing uses the data fetched (during data fetching phase) and converts it to JSON format. This data (in JSON format) is then consumed by the application. The figure below shows the ratio of the time required by total time required (by both phases) to the time required by the data fetch phase. In the figure,...
Read more