Managing Java versions and Eclipse

Some little bits on managing Java versions and Eclipse:

Managing JRE installations in Eclipse
http://www.codejava.net/ides/eclipse/managing-jre-installations-in-eclipse

How to switch JDK version on Mac OS X
https://www.jayway.com/2014/01/15/how-to-switch-jdk-version-on-mac-os-x-maverick/
Edit your ~/.bash_profile and add the following:
function setjdk() {
  if [ $# -ne 0 ]; then
   removeFromPath '/System/Library/Frameworks/JavaVM.framework/Home/bin'
   if [ -n "${JAVA_HOME+x}" ]; then
    removeFromPath $JAVA_HOME
   fi
   export JAVA_HOME=`/usr/libexec/java_home -v $@`
   export PATH=$JAVA_HOME/bin:$PATH
  fi
 }
 function removeFromPath() {
  export PATH=$(echo $PATH | sed -E -e "s;:$1;;" -e "s;$1:?;;")
 }
setjdk 1.7

Installing Java 8 on Ubuntu

Quick, easy point of reference for installing Java 8 on Ubuntu:

http://stackoverflow.com/questions/25549492/install-jdk8-in-ubuntu-14-04

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

http://askubuntu.com/questions/315646/update-java-alternatives-vs-update-alternatives-config-java
sudo update-alternatives --config java

Debugging spark job incorrect JVM jar file loaded

I recently had an issue with incorrect version of a JVM jar file being loaded in a Spark job on Hadoop.

This Scala code snippet helped debug the issue and determine the loaded jar file path:

val jarPath = classOf[MyObject].getProtectionDomain().getCodeSource().getLocation().getPath()

In my case this pointed to an old version of guava:
/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/guava-11.0.jar

Then by setting spark.driver.extraClassPath and spark.executor.extraClassPath arguments for spark-submit, the correct version of the jar file was loaded successfully:

spark-submit --class com.MyClass <other_spark_args> --conf "spark.driver.extraClassPath=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/guava-15.0.jar" --conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/guava-15.0.jar" /home/mypath/myjarfile.jar <my_job_params>

For more info with extraClassPath, see: http://spark.apache.org/docs/latest/configuration.html

Convert binary word2vec model to text vectors

If you have a binary model generated from google's awesome and super fast word2vec word embeddings tool, you can easily use python with gensim to convert this to a text representation of the word vectors.

Input: binary word embedding model from google's word2vec tool

Output: text vectors for word embeddings

Python conversion code:
from gensim.models import word2vec
model = word2vec.Word2Vec.load_word2vec_format('path/to/mymodel.bin', binary=True)
model.save_word2vec_format('path/to/mymodel.txt', binary=False)

I recommend using Anaconda from Continuum Analytics for a bundled python distribution.  To install gensim in Anaconda just type: conda install gensim :)


Original ref: https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/13828/how-to-convert-bin-file-of-word2vec-model-into-txt-r/91564

Analyse consecutive timeseries pairs in Spark

With time series data it is useful to construct pairs of elements for analysis.

Here is a way to construct a Spark RDD from a time series that has the pairs together in the final RDD:


val arr = Array((1, "A"), (8, "D"), (7, "C"), (3, "B"), (9, "E"))
val rdd = sc.parallelize(arr)
val sorted = rdd.sortByKey(true)
val zipped = sorted.zipWithIndex.map(x => (x._2, x._1))
val pairs = zipped.join(zipped.map(x => (x._1 - 1, x._2))).sortBy(_._1)

Which produces the consecutive elements as pairs in the RDD for further processing:
(0,((1,A),(3,B)))
(1,((3,B),(7,C)))
(2,((7,C),(8,D)))
(3,((8,D),(9,E)))

Ref: https://www.mail-archive.com/user@spark.apache.org/msg39353.html

Data Science Blogs around the Interwebs

So many great data science/machine learning blogs, so little time!

Company Blogs

http://googleresearch.blogspot.com.au/

https://research.facebook.com/blog/ai/
https://research.facebook.com/blog/datascience/

https://blog.twitter.com/data

http://data.quora.com/

https://engineering.linkedin.com/big-data
http://blog.linkedin.com/topic/economic-graph/

http://www.ebaytechblog.com/category/machine-learning/

http://blogs.aws.amazon.com/bigdata/

http://nerds.airbnb.com/data/

https://codeascraft.com/category/data/  (Etsy)

http://blog.kaggle.com/


Personal Blogs

http://blog.echen.me/ (Edward Chen)

http://fastml.com/ (Zygmunt Zając)

http://flowingdata.com/ (Nathan Yau)

http://simplystatistics.org/ (Jeff Leek, Roger Peng, and Rafa Irizarry)

http://hunch.net/ (John Langford et al.)

http://www.walkingrandomly.com/ (Mike Croucher)


Aggregators and other links

http://www.r-bloggers.com/

https://www.reddit.com/r/machinelearning

http://www.datatau.com/

http://stats.stackexchange.com/

Ubuntu 14 Classic Desktop set up

Whenever I set up a VM with Ubuntu I want to fall back to a lighter desktop with fewer effects.  Here's how to install the classic Gnome desktop and make the new Unity desktop optional on login.

Ref: http://www.howtogeek.com/189912/how-to-install-the-gnome-classic-desktop-in-ubuntu-14.04/

1) sudo apt-get update; sudo apt-get install gnome-session-fallback

2) Logout

3) Login with Metacity icon option

Happy days.