Consulting to improve Apache Spark program performance

$100-400 USD

Closed

Posted

over 8 years ago

$100-400 USD

Paid on delivery

I have a Spark application (coded with Scala) running on a cluster of 5 nodes. Each node has 125GB memory and 32 cores. There is a function/method containing 8000 loops totally. Inside the loop it calls MLlib k-Means, so the k-Means runs about 8000 thousand times. The running time of the big loop is around 5.5 hours currently on the Spark clusters. I want to reduce the running time significantly. May need to tune JVM parameters, Spark configuration parameters, or restructure the Scala application code, etc.

Scala

Spark

Project ID: 9103352

About the project

3 proposals

Remote project

Active 8 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

3 freelancers are bidding on average $348 USD for this job

@ahmsak

Hello Sir... I am a computer science tutor, I teach (among others) Scala, Haskell, Prolog and Algorithms. I have done many projects like this, and I'm the top Scala developer, you can check here please: https://www.freelancer.com/freelancers/skills/Scala/ Please contact me for more details when possible. I look forward to work for you Sir. Best Regards.

$300 USD in 3 days

5.0

(3 reviews)

3.5

@HappyRainy

First of all, I Need to know what the current settings you are using for running the applivation now. Kmeans could be very expensive if you have big number of clusters and too many of the dimensions.

$444 USD in 4 days