Adding a custom Python library path in a Jupyter Notebook

This code adds a Python library path relative to your working folder to enable you to load custom library functions from a Jupyter Notebook:

import sys, os
extra_path = os.path.join(os.getcwd(), "lib")
if extra_path not in sys.path:
    sys.path.append(extra_path)
    print('Added extra_path:', extra_path)

Then import like so:
import <my_file_name_in_lib_folder> as funcs

If in a Jupyter Notebook with Python 3.4+ the following will automatically reload the library:
import importlib
importlib.reload(funcs)

Preparing a hashed password with Jupyter Notebook

http://jupyter-notebook.readthedocs.io/en/latest/public_server.html#preparing-a-hashed-password

You can prepare a hashed password manually, using the function notebook.auth.security.passwd():

In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed' 

You can then add the hashed password to your jupyter_notebook_config.py. The default location for this file jupyter_notebook_config.py is in your Jupyter folder in your home directory, ~/.jupyter, e.g.:

c.NotebookApp.password = u'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

Running PaCAL on OSX with conda

PaCAL is a great library for performing arithmetic on probabilistic random variables just like you do with ordinary program variables. Here is a more detailed paper on PaCAL.

There are a couple of gotchas setting it up on OSX with conda:

• Requires Python 2.7.x (3.x not supported as of March 2017)
• Required dependencies: numpy matplotlib sympy scipy
• Workaround for "RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework".

Here are the steps to set up PaCAL under a conda environment on OSX:

$ conda create -n pacal python=2.7
$ source activate pacal

(pacal) $ pip install numpy matplotlib sympy scipy
(pacal) $ pip install pacal

(pacal) $ python
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
>>>
>>> # This TkAgg bit is to prevent the OSX 'Python is not installed as a framework' error.
>>> import matplotlib as mpl
>>> mpl.use('TkAgg')
>>>
>>> from pacal import *
Compiled interpolation routine not available
Compiled sparse grid routine not available
>>> dL = UniformDistr(1,3)
>>> L0 = UniformDistr(9,11)
>>> dT = NormalDistr(1,1)
>>> K = dL / (L0 * dT)
>>> K.plot()
>>> show()

Conda create environment commands

Conda create environment commands - super easy but nice to have as a reference:

# Python 2.7 $ conda create -n myenvname python=2.7 # Python 3.4 $ conda create -n myenvname python=3.4 # Python 3.x latest $ conda create -n myenvname python=3

$ source activate myenvname

More info: https://conda.io/docs/using/envs.html

If you then have a requirements.txt file, you can install the listed packages with:
conda install --yes --file requirements.txt

If you want to reproduce an existing conda environment on another machine, you can get a snapshot with:
conda env export > freeze.yml

which can then be restored via:
conda env create -f freeze.yml

Install xgboost on OSX with full OpenMP support in an anaconda virtualenv

The default gcc with OSX doesn't support OpenMP which enables xgboost to utilise multiple cores when training.

These steps show how to install gcc-6 with OpenMP support and build xgboost to support multiple cores and contain the python setup in an Anaconda virtualenv.

Install/update brew to support installing gcc-6:
See http://brew.sh/

Install gcc-6 with OpenMP support:
brew install gcc --without-multilib
Run brew doctor to ensure gcc-6 and g++-6 are linked correctly

Get latest xgboost from github:
git clone --recursive https://github.com/dmlc/xgboost

Build xgboost with gcc-6:
See https://xgboost.readthedocs.io/en/latest/build.html
cd xgboost; cp make/config.mk ./config.mk;
Ensure that the newly installed gcc-6 compliers are correct in config.mk (e.g. export CC = gcc-6; export CXX = g++-6)
make -j4
(This can take around 30 mins to build)

Create anaconda xgboost virtualenv:
conda create -n xgboost python=3.5
source activate xgboost

Install prerequisite numpy package in xgboost env:
conda install numpy

Set up xgboost for python xgboost virtualenv without sudo:
Ensure you are in the xgboost virtualenv (e.g. which python -> /Users/<user>/anaconda/envs/xgboost/bin/python)
cd <git_clone_location>/xgboost/python-package
python setup.py install
(This can take around 10 mins)

Test xgboost:
open a python session
Try import xgboost as xgb

Finally, set up Jupyter notebook:
If you want to access xgboost via Kernel > Change Kernel in Jupyter:
conda install jupyter

Helpful reference:
https://www.ibm.com/developerworks/community/blogs/jfp/entry/Installing_XGBoost_on_Mac_OSX?lang=en

How to list word occurences using CountVectorizer from Scikit Learn

A simple and efficient way to get document frequency counts of words from a corpus is to use CountVectorizer from Scikit Learn

Getting back to the word from the index is not immediately obvious, here's how to do it:

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer

docs = <load your docs as an iterable>

count_vect = CountVectorizer()
doc_counts = count_vect.fit_transform(docs)  # this is of type scipy.sparse.csr.csr_matrix which is why we need to use
.ravel() below.

word_counts = zip(count_vect.get_feature_names(), np.asarray(doc_counts.sum(axis=0)).ravel())
word_counts = sorted(word_counts, key=lambda idx: -1 * idx[1] )
 
# Display top 100 words by frequency
word_counts[:100]

Convert binary word2vec model to text vectors

If you have a binary model generated from google's awesome and super fast word2vec word embeddings tool, you can easily use python with gensim to convert this to a text representation of the word vectors.

Input: binary word embedding model from google's word2vec tool

Output: text vectors for word embeddings

Python conversion code:
from gensim.models import word2vec
model = word2vec.Word2Vec.load_word2vec_format('path/to/mymodel.bin', binary=True)
model.save_word2vec_format('path/to/mymodel.txt', binary=False)

I recommend using Anaconda from Continuum Analytics for a bundled python distribution.  To install gensim in Anaconda just type: conda install gensim :)


Original ref: https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/13828/how-to-convert-bin-file-of-word2vec-model-into-txt-r/91564

Python 2.7 Setup Steps

UPDATE 2013:
Now follow these instructions: https://python-guide.readthedocs.org/en/latest/starting/install/win.html
Use distribute, not setuptools as described here
/UPDATE

I've set up Python 2.7 on several wind0ze machines recently, these are the "best practice" steps I now use:

• Download and install Python 2.7 to C:\Python27
• Download and install setuptools to C:\Python27\Lib\site-packages\
  (setuptools includes easy_install, so we can install pip! Read instructions for 32 vs 64bit versions)
• Add env variable PYTHON_HOME=C:\Python27
• Append env variable PATH with ;%PYTHON_HOME%;%PYTHON_HOME%\Scripts;
• Run C:\Python27\Scripts>easy_install-2.7 pip to install pip
• Install virtualenv: easy_install-2.7 virtualenv (now all pip install commands I run from a new virtualenv)

That takes care of the base environment.

Next is to install packages as required, remembering that matplotlib is best installed using an installer after executing "pip install numpy".

References:

pip Installation instructions (recommends installing pip using virtualenv)

Public service announcement (tool transition infographic)

Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip (covers the new cool tools)

How to install pip on Windows (installing pip globally using setuptools)
Installing matplotlib

python to solve linear system

Solving good ol' Ax=b:

>>> import numpy as np
>>> A = np.matrix('[3 -1 1; -1 1 -1; 1 -1 3]')
>>> A
matrix([[ 3, -1,  1],
        [-1,  1, -1],
        [ 1, -1,  3]])
>>> b = np.matrix('[1;1;1]')
>>> from scipy import linalg
>>> x = linalg.solve(A,b)
>>> x
array([[ 1.],
       [ 3.],
       [ 1.]])

See docs:
numpy.linalg.solve
routines.linalg
numpy solve examples (prob better than above!)

python to visualise vectors

I've been working on a simple example SVM problem and wanted to be able to visualise the vectors in 3D. Python with Matplotlib works nicely.

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt


fig = plt.figure()
ax = Axes3D(fig)


ax.plot3D([0,-1], [0,1], [0,1], zdir='z', label='x1 +1')
ax.plot3D([0,0], [0,1], [0,0], zdir='z', label='x2 -1')
ax.plot3D([0,1], [0,1], [0,1], zdir='z', label='x3 +1')
ax.plot3D([0,0], [0,0], [0,0], zdir='z', label='x4 +1')


ax.scatter([0,-1], [0,1], [0,1], zdir='z', label='x1')
ax.scatter([0,0], [0,1], [0,0], zdir='z', label='x2')
ax.scatter([0,1], [0,1], [0,1], zdir='z', label='x3')
ax.scatter([0,0], [0,0], [0,0], zdir='z', label='x4')


ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')


ax.legend()
plt.show()

To add an equation to the graph:

import numpy as np
x = np.linspace(-1, 1, 50)
y = 2*x**2
ax.plot3D(x, y, 0);

LibSVM v3.1 python example

LibSVM v3.1 python example: (many older examples break in v3.1)


from svmutil import *
svm_model.predict = lambda self, x: svm_predict([0], [x], self)[0][0]
prob = svm_problem([1,-1], [[1,0,1], [-1,0,-1]])
param = svm_parameter()
param.kernel_type = LINEAR
param.C = 10
m=svm_train(prob, param)
m.predict([1,1,1])

From: http://stackoverflow.com/questions/4214868/an-example-using-libsvm-in-python/4215056

python SimpleHTTPServer solves XMLHttpRequest errors

Loading an html file in a browser from the filesystem (i.e. file:///) that uses XMLHttpRequest will throw a javascript error since for security purposes file:/// requests have a null origin.

Example error:
XMLHttpRequest cannot load file:///somepath. Origin null is not allowed by Access-Control-Allow-Origin.

The solution is to serve the html file using http from a local webserver.  A very easy and simple webserver to run is offered with python by typing this in the folder you want to serve files from:
python -m SimpleHTTPServer 1234

Then navigate to http://127.0.0.1:1234/somepath to view the file.