Making git on Windows behave with CRLF line endings

Git on Windows by default is a bit too clever for itself with line endings, typically having the config autocrlf=true

When I checkout a Linux/OSX repo that contains shell scripts that are used in a built Docker image - please leave line endings as LF as per the repo - don't convert to CRLF.

To achieve this for the entire repo, git clone like so:
git clone --config core.autocrlf=input <repo>

More info: https://git-scm.com/docs/git-config#git-config-coreautocrlf

Serialising a RandomForestClassificationModel from PySpark to a SequenceFile on hdfs

Prior to Spark 2.0 the org.apache.spark.ml.classification.RandomForestClassificationModel doesn't have a save() method in it's Scala API as it doesn't implement the MLWritable interface.

If it did, then from PySpark we could easily call this like so:
lrModel._java_obj.save(model_path)
loaded_lrModel_JVM = sqlContext._jvm.org.apache.spark.ml.classification.LogisticRegressionModel.load(model_path)
loaded_lrModel = LogisticRegressionModel(loaded_lrModel_JVM)

(Note that this issue doesn't apply to the older deprecated org.apache.spark.mllib.tree.model.RandomForestModel from Spark MLlib which does have a save() method in v1.6)

This is a problem as my current client is constrained to using Spark v1.6

rdd.saveAsObjectFile() is an alternative way to serialise/deserialise a model using the Hadoop API to a SequenceFile.


Here is the relatively simple Scala approach:
// Save
sc.parallelize(Seq(model), 1).saveAsObjectFile("hdfs:///some/path/rfModel")

// Load
val rfModel = sc.objectFile[RandomForestClassificationModel]("hdfs:///some/path/rfModel").first()


Due to serialisation issues with Py4J the PySpark approach is more complex:
# Save
gateway = sc._gateway
java_list = gateway.jvm.java.util.ArrayList()
java_list.add(rfModel._java_obj)
modelRdd = sc._jsc.parallelize(java_list)
modelRdd.saveAsObjectFile("hdfs:///some/path/rfModel")

# Load
rfObjectFileLoaded = sc._jsc.objectFile("hdfs:///some/path/rfModel")
rfModelLoaded_JavaObject = rfObjectFileLoaded.first()
rfModelLoaded = RandomForestClassificationModel(rfModelLoaded_JavaObject)
predictions = rfModelLoaded.transform(test_input_df)



Reference source of RandomForestClassifier v1.6 vs. v2.2:
https://github.com/apache/spark/blob/v1.6.2/mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala
https://github.com/apache/spark/blob/v2.2.0/mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala

Reference for MLWritable:
https://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/ml/util/MLWritable.html
https://spark.apache.org/docs/2.0.0/api/java/org/apache/spark/ml/util/MLWritable.html

Calling Scala API methods from PySpark when using the spark.ml library

While implementing Logistic Regression on an older Spark 1.6 cluster I was surprised by how many Python API methods were missing so that the task of saving and loading a serialised model was unavailable.

However using the Py4j calls we can reach directly into the Spark Scala API.

Say we have a `pyspark.ml.classification.LogisticRegressionModel` object, we can call save like so:

lrModel._java_obj.save(model_path)

And load, which is different due being a static method:

loaded_lrModel_JVM = sqlContext._jvm.org.apache.spark.ml.classification.LogisticRegressionModel.load(model_path)

loaded_lrModel = LogisticRegressionModel(loaded_lrModel_JVM)

This helps future proof SparkML development since `spark.mllib` is effectively deprecated, and on any Spark 2.x upgrade there should be minimal breaking changes to the API.

Find maximum row per group in Spark DataFrame

This is a great Spark resource on getting the max row, thanks zero323!
https://stackoverflow.com/questions/35218882/find-maximum-row-per-group-in-spark-dataframe

Using join (it will result in more than one row in group in case of ties):

import pyspark.sql.functions as F
from pyspark.sql.functions import count, col 

cnts = df.groupBy("id_sa", "id_sb").agg(count("*").alias("cnt")).alias("cnts")
maxs = cnts.groupBy("id_sa").agg(F.max("cnt").alias("mx")).alias("maxs")

cnts.join(maxs, 
  (col("cnt") == col("mx")) & (col("cnts.id_sa") == col("maxs.id_sa"))
).select(col("cnts.id_sa"), col("cnts.id_sb"))

Using window functions (will drop ties):

from pyspark.sql.functions import row_number
from pyspark.sql.window import Window

w = Window().partitionBy("id_sa").orderBy(col("cnt").desc())

(cnts
  .withColumn("rn", row_number().over(w))
  .where(col("rn") == 1)
  .select("id_sa", "id_sb"))

Using struct ordering:

from pyspark.sql.functions import struct

(cnts
  .groupBy("id_sa")
  .agg(F.max(struct(col("cnt"), col("id_sb"))).alias("max"))
  .select(col("id_sa"), col("max.id_sb")))

Adding a custom Python library path in a Jupyter Notebook

This code adds a Python library path relative to your working folder to enable you to load custom library functions from a Jupyter Notebook:

import sys, os
extra_path = os.path.join(os.getcwd(), "lib")
if extra_path not in sys.path:
    sys.path.append(extra_path)
    print('Added extra_path:', extra_path)

Then import like so:
import <my_file_name_in_lib_folder> as funcs

If in a Jupyter Notebook with Python 3.4+ the following will automatically reload the library:
import importlib
importlib.reload(funcs)

Preparing a hashed password with Jupyter Notebook

http://jupyter-notebook.readthedocs.io/en/latest/public_server.html#preparing-a-hashed-password

You can prepare a hashed password manually, using the function notebook.auth.security.passwd():

In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed' 

You can then add the hashed password to your jupyter_notebook_config.py. The default location for this file jupyter_notebook_config.py is in your Jupyter folder in your home directory, ~/.jupyter, e.g.:

c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

Resolving conflicts on a git branch

You'll need to update your branch with new commits from master, resolve those conflicts and push the updated/resolved branch to GitHub.

Resolve conflicts

I found these instructions much better than the instructions from Bitbucket involving a detached head! Thanks jaw6!

git checkout master
git pull
git checkout <branch>
git merge master
[ ... resolve any conflicts ... ]
git add [files that were conflicted]
git commit
git push

Direct Reference: https://github.com/githubteacher/github-for-developers-sept-2015/issues/648


Merge master

git checkout somebranch # gets you on branch somebranch
git fetch origin        # gets you up to date with origin
git merge origin/master

Reference: https://stackoverflow.com/questions/20101994/git-pull-from-master-into-the-development-branch


If it all goes wrong somehow, reset with the following (but ensure you're previously checked in as you can lose work with this command!!)

git reset –hard
git reset --hard <SOME-COMMIT>

Reference: https://stackoverflow.com/questions/9529078/how-do-i-use-git-reset-hard-head-to-revert-to-a-previous-commit


Bonus: reverting local changes

Revert changes made to your working copy:
git checkout .

Revert changes made to the index (i.e., that you have added), do this. Warning this will reset all of your unpushed commits to master!:
git reset

Revert a change that you have committed:
git revert <commit 1> <commit 2>

Remove untracked files (e.g., new files, generated files):
git clean -f

Remove untracked files directories (e.g., new or automatically generated directories):
git clean -fd

Reference: https://stackoverflow.com/questions/1146973/how-do-i-revert-all-local-changes-in-git-managed-project-to-previous-state

Cleaning up Docker disk space

Docker seems to eat disk space and leave a lot of left overs around. Here are some resources to fix that.

Docker image prune (the new standard fix)

docker image prune

https://docs.docker.com/config/pruning/
https://docs.docker.com/engine/reference/commandline/image_prune/
https://gist.github.com/anildigital/862675ec1b7bccabc311

General Info
https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes

https://stackoverflow.com/questions/17665283/how-does-one-remove-an-image-in-docker


Is the /var/lib/docker/ folder large?
sudo du -sh /var/lib/docker/

If it's large, may need to zap it (careful with this one!):
Ref: https://stackoverflow.com/questions/45798076/how-to-clean-up-docker

$ sudo su
# service docker stop
# cd /var/lib/docker
# rm -rf *
# service docker start

For more info about this folder see: https://stackoverflow.com/questions/19234831/where-are-docker-images-stored-on-the-host-machine

And also......try removing dangling images

https://forums.docker.com/t/how-to-remove-none-images-after-building/7050/3

List dangling:
docker images -f "dangling=true" -q

Delete dangling:
docker rmi $(docker images -f "dangling=true" -q)

Force delete dangling:
docker rmi -f $(docker images -f "dangling=true" -q)

Rsync Broken Pipe Error with SSH

Try adding these settings:

KeepAlive yes
ServerAliveInterval 20
ServerAliveCountMax 6

to your /etc/ssh/ssh_config or ~/.ssh/config file.

To solve Broken Pipe errors.

Reference: https://unix.stackexchange.com/questions/68775/rsync-timed-out

Installing Docker on an Ubuntu AWS EC2 machine

sudo apt-get update

# Install packages to allow apt to use a repository over HTTPS:
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

# For "sudo: unable to resolve host ip-x-x-x-x" errors, follow the following link to set "DNS Hostnames" to True:
# http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html#vpc-dns-updating
sudo true

# Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Verify that the key fingerprint is 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88.
sudo apt-key fingerprint 0EBFCD88

# Use the following command to set up the stable repository.
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

# Install the latest version of Docker CE
sudo apt-get update
sudo apt-get install docker-ce
sudo docker run hello-world

# Ref: https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-using-the-repository

Set up RVM, Ruby, Gem & Jykll to serve a username.github.io blog

Here are the steps I took to set up RVM, Ruby, Gem & Jykll to serve a username.github.io blog on OSX:

# Install RVM (Ruby version manager) 
# Also read: http://rvm.io/rvm/install and check the script before running
curl -sSL https://get.rvm.io | bash -s stable --ruby

# Install Ruby (replace x.y.z with latest stable version of ruby)
rvm install x.y.z
rvm docs generate-ri
rvm use x.y.z --default

# Check other requirements are available for jekyll and install if missing
gem --version
gcc --version
make --version

# Install jekyll
gem install jekyll

# Install bundler and get all dependencies
gem install bundler
bundle install

# Clone an existing jekyll template, this one from BlackrockDigital is nice
git clone https://github.com/BlackrockDigital/startbootstrap-clean-blog-jekyll.git
cd startbootstrap-clean-blog-jekyll
bundle exec jekyll serve

# Periodically update Jekyll to match the GitHub Pages server
bundle update github-pages

Reference:
https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/
http://jekyllrb.com/docs/installation/
https://jekyllrb.com/docs/posts/
https://github.com/planetjekyll/awesome-jekyll-editors

Running PaCAL on OSX with conda

PaCAL is a great library for performing arithmetic on probabilistic random variables just like you do with ordinary program variables. Here is a more detailed paper on PaCAL.

There are a couple of gotchas setting it up on OSX with conda:

• Requires Python 2.7.x (3.x not supported as of March 2017)
• Required dependencies: numpy matplotlib sympy scipy
• Workaround for "RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework".

Here are the steps to set up PaCAL under a conda environment on OSX:

$ conda create -n pacal python=2.7
$ source activate pacal

(pacal) $ pip install numpy matplotlib sympy scipy
(pacal) $ pip install pacal

(pacal) $ python
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
>>>
>>> # This TkAgg bit is to prevent the OSX 'Python is not installed as a framework' error.
>>> import matplotlib as mpl
>>> mpl.use('TkAgg')
>>>
>>> from pacal import *
Compiled interpolation routine not available
Compiled sparse grid routine not available
>>> dL = UniformDistr(1,3)
>>> L0 = UniformDistr(9,11)
>>> dT = NormalDistr(1,1)
>>> K = dL / (L0 * dT)
>>> K.plot()
>>> show()

Conda create environment commands

Conda create environment commands - super easy but nice to have as a reference:

# Python 2.7 $ conda create -n myenvname python=2.7 # Python 3.4 $ conda create -n myenvname python=3.4 # Python 3.x latest $ conda create -n myenvname python=3

$ source activate myenvname

More info: https://conda.io/docs/using/envs.html

If you then have a requirements.txt file, you can install the listed packages with:
conda install --yes --file requirements.txt

If you want to reproduce an existing conda environment on another machine, you can get a snapshot with:
conda env export > freeze.yml

which can then be restored via:
conda env create -f freeze.yml

Install xgboost on OSX with full OpenMP support in an anaconda virtualenv

The default gcc with OSX doesn't support OpenMP which enables xgboost to utilise multiple cores when training.

These steps show how to install gcc-6 with OpenMP support and build xgboost to support multiple cores and contain the python setup in an Anaconda virtualenv.

Install/update brew to support installing gcc-6:
See http://brew.sh/

Install gcc-6 with OpenMP support:
brew install gcc --without-multilib
Run brew doctor to ensure gcc-6 and g++-6 are linked correctly

Get latest xgboost from github:
git clone --recursive https://github.com/dmlc/xgboost

Build xgboost with gcc-6:
See https://xgboost.readthedocs.io/en/latest/build.html
cd xgboost; cp make/config.mk ./config.mk;
Ensure that the newly installed gcc-6 compliers are correct in config.mk (e.g. export CC = gcc-6; export CXX = g++-6)
make -j4
(This can take around 30 mins to build)

Create anaconda xgboost virtualenv:
conda create -n xgboost python=3.5
source activate xgboost

Install prerequisite numpy package in xgboost env:
conda install numpy

Set up xgboost for python xgboost virtualenv without sudo:
Ensure you are in the xgboost virtualenv (e.g. which python -> /Users/<user>/anaconda/envs/xgboost/bin/python)
cd <git_clone_location>/xgboost/python-package
python setup.py install
(This can take around 10 mins)

Test xgboost:
open a python session
Try import xgboost as xgb

Finally, set up Jupyter notebook:
If you want to access xgboost via Kernel > Change Kernel in Jupyter:
conda install jupyter

Helpful reference:
https://www.ibm.com/developerworks/community/blogs/jfp/entry/Installing_XGBoost_on_Mac_OSX?lang=en