PySpark installation

PySpark is very easy to install by just following the steps listed in this link. The only catch is compilation step takes close to 30 mins. 

Comments

Post a Comment

Popular posts from this blog

How to set up a Spark multi-node cluster on AWS-EC2

Context Our, Timothy Humphrey and I, aim was to benchmark the performance of Apache Spark against HPCC.  Tim had already a set up HPCC on AWS and had all the worker nodes on the same placement group so that the network latency is low. The easiest way to start a Spark multi-node cluster in AWS is to use AWS EMR. There are various resources like this , which explain  how to Spark multi-node cluster.  But, EMR service does not support placement group. Since, it only supports a small number of EC2 instance types, as compared to the EMR service. Hence, we decided to set up Spark on EC2 rather than using AWS EMR. The rest of the article will document the process. Disclaimer Most of the content of this document is from here  with some changes for example, the tutorial talks about install Spark on Ubuntu system and for our setup we wanted to use Centos (for legacy reasons). Steps 1. Spin N nodes on EC2. For this article, we assume N=5. Please make sure that the port...
Read more

Objectives

The objective of my project is to build a connector between HPCC world and SPARK world. Motivation HPCC is the heart of LN business and almost all the important data is stored in the HPCC systems. Hence, if an analyst/statistician wants to build a model of (some or all) data using SPARK, she needs to download data to either her local system or move it to a cluster. This can be time-consuming and she needs to be very careful of being compliant with strict data rules. A possible solution to this problem can be having a bridge between these two (SPARK and HPCC) worlds. Expected Output  We expect our output to be in form of a connector when installed can enable ECL programmers to use SPARK algorithms on data stored in HPCC as well as PySpark programmers to use HPCC data. Github Repo  
Read more