Installing Cloudera Hadoop 0.20 and Mahout on Ubuntu 11

Notes for installing Cloudera Hadoop 0.20 and Mahout on Ubuntu 11.10
April 2012

UPDATE SEPT 2014 - ok this is very old now.  A better way is to just use the Cloudera CDH installer!
# wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin
# chmod u+x cloudera-manager-installer.bin
# sudo ./cloudera-manager-installer.bin
Then follow the local installer steps before navigating to the web installer when prompted.

Original April 2012 Instructions:

0. Ubuntu
Fresh install of Ubuntu ubuntu-11.10-desktop-amd64.iso on VirualBox with all updates and Guest Addons
Also mount a shared folder created in VirtualBox:
$ sudo mount -t vboxsf _Shared ~/_Shared
or add this line (without sudo) to /etc/rc.local

1. Java and ssh prerequisites:
http://cloudblog.8kmiles.com/2011/12/02/hadoop-prerequisite-for-hadoop-setup-in-ubuntu/
http://softwareinabottle.wordpress.com/2011/11/17/install-sun-jdk-6-on-ubuntu-11-10/

2. Update Java env by adding the following to: /etc/bash.bashrc
http://blog.sanaulla.info/2009/04/02/installing-jdk-setting-java_home-in-ubuntu/
JAVA_HOME=/usr/lib/jvm/java-6-sun
export JAVA_HOME
PATH=$PATH:$JAVA_HOME/bin:
export PATH
CLASSPATH=$JAVA_HOME/lib/:.
export CLASSPATH

3. CDH3 install hadoop core:
http://cloudblog.8kmiles.com/2011/12/06/hadoop-cdh3-setup-in-ubuntu/
3.1 create cloudera.list:
$ sudo pico /etc/apt/sources.list.d/cloudera.list
with content:
deb http://archive.cloudera.com/debian maverick-cdh3 contrib
deb-src http://archive.cloudera.com/debian maverick-cdh3 contrib
$ curl -s http://archive.cloudera.com/debian/archive.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install hadoop-0.20
$ hadoop version

4. CDH3 install daemons for Pseudo Distributed Mode:
http://cloudblog.8kmiles.com/2011/12/07/hadoop-cdh3-pseudo-distributed-mode-setup/
to set up Standalone mode see: http://cloudblog.8kmiles.com/2011/12/07/hadoop-cdh3-standalone-mode-setup/

$ sudo apt-get install hadoop-0.20-namenode
$ sudo apt-get install hadoop-0.20-secondarynamenode
$ sudo apt-get install hadoop-0.20-jobtracker
$ sudo apt-get install hadoop-0.20-datanode
$ sudo apt-get install hadoop-0.20-tasktracker

$ sudo apt-get install hadoop-0.20-conf-pseudo
$ sudo alternatives --display hadoop-0.20-conf

5. Test the daemons:
for service in /etc/init.d/hadoop-0.20-*; do sudo $service start; done
sudo jps
hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar pi 2 100000

Navigate to: http://localhost:50070/ for Namenode details
Navigate to: http://localhost:50030/ for Job details

for service in /etc/init.d/hadoop-0.20-*; do sudo $service stop; done

6. Install mahout:
http://cloudblog.8kmiles.com/2012/02/16/cdh3-mahout-setup/
$ sudo apt-get install mahout


7. Install Hue interface:
https://ccp.cloudera.com/display/CDHDOC/Hue+Installation#HueInstallation-Installing%2CConfiguring%2CandStartingHueonOneMachine
$ sudo apt-get install hue

Open the /etc/hue/hue.ini configuration file and add:
[desktop]
secret_key=jFE93j;2[280-eiw.tyXrN2s3['d:/.q[eI^^y#e=+Iei*@Mn<qW5o

$ for service in /etc/init.d/hadoop-0.20-*; do sudo $service start; done
sudo /etc/init.d/hue start
Navigate to: http://localhost:8088/ for Hue

I hope these notes and links help someone out.  Please leave a comment if you have have any problems or additions.

Adding external libs to an Eclipse Galileo project

In eclipse Galileo :
Open the project's properties
Select Java Build Path
Select Libraries tab
From there you can Add External Jars
http://stackoverflow.com/questions/179024/adding-a-jar-to-an-eclipse-java-library

Also, when importing java source files into the project, the root import folder chosen must match the package structure.

Get object dependencies in SQL Server

Get object dependencies of a sproc via script instead of Management Studio:

DECLARE @my_object nvarchar(1024) = N'dbo.my_sproc'
CREATE TABLE #tempTable (name nvarchar(1024), [type] nvarchar(1024), updated nvarchar(1024), selected nvarchar(1024), [column] nvarchar(1024))
INSERT INTO #tempTable EXEC sp_depends @objname = @my_object
SELECT name FROM #tempTable group by name
DROP TABLE #tempTable
GO

Also useful:

SELECT * FROM sys.Tables

http://blog.sqlauthority.com/2009/09/23/sql-server-insert-values-of-stored-procedure-in-table-use-table-valued-function/

Increasing heap for Weka

Any OS
From command/terminal:
java -Xmx1024m -jar weka.jar

OSX gui
From terminal:
cd /Applications/weka-3-5-7.app/Contents
open Info.plist
Under  "Java" you will find an entry called "VMOptions" with the value "-Xmx256M".

Windows gui
Edit the line maxheap=1536m from C:\Program Files\Weka-3-X\RunWeka.ini

SQL to concatenate many rows into a single string

I used this SQL script to convert multiple returned rows into a single text string in SQL Server 2008.
SELECT REPLACE(
(SELECT CASE TBL.DayNumber
WHEN 1 THEN 'Sun'
WHEN 2 THEN 'Mon'
WHEN 3 THEN 'Tue'
WHEN 4 THEN 'Wed'
WHEN 5 THEN 'Thu'
WHEN 6 THEN 'Fri'
WHEN 7 THEN 'Sat'
ELSE NULL
END AS 'data()' 
FROM [dbo].[MyTableName] TBL
FOR XML PATH (''))
, ' ', ',') AS CommaDelimDayNames

This will convert:
Id          DayNumber
----------- --------------------
77          4
77          5
77          6

into:
Id          CommaDelimDayNames
----------- --------------------
77          Wed,Thu,Fri


As usual I had inspiration from the wonderful stackoverflow, in particular Diwakar's answer:
http://stackoverflow.com/questions/194852/concatenate-many-rows-into-a-single-text-string/5580166#5580166
This has the answer:

REPLACE((select FName AS 'data()'  from NameList  for xml path('')), ' ', ', ')

My hg/mercurial settings

My .hgignore for Visual Studio projects:


# use glob syntax
syntax: glob


# Ignore Visual Studio files
*.obj
*.exe
*.pdb
*.user
*.vspscc
*.bak
*.cache
*.ilk
*.log
*.sbr
*.scc
[Bb]in
[Dd]ebug*/
obj/
[Rr]elease*/
_ReSharper*/
[Tt]est[Rr]esult*
[Bb]uild[Ll]og.*
*.[Pp]ublish.xml


#*.suo # depending on public/private repo

SQL Server PIVOT and UNPIVOT to transpose a table

I had to rotate/transpose a SQL table variable for reporting purposes and eventually got SQL Server 2008 PIVOT and UNPIVOT operators to achieve this (rather that using a longer UNION ALL approach.)

Source table definition:
DECLARE @ResultTable TABLE
(
Ordinal INT,
Calc1 FLOAT,
Calc2 FLOAT,
Calc3 FLOAT
)

SELECT * FROM @ResultTable:
1 | 1000000 | 0.438556290234799 | 3.53674427608709
2 | 0 | 0.877112580469597 | 7.07348855217417
3 | 1000000 | 0.877112580469597 | 7.07348855217417
4 | 0 | 1.3156688707044 | 10.6102328282613
5 | 1000000 | 1.3156688707044 | 10.6102328282613
6 | 0 | 0 | 0

SQL to rotate/transpose rows to columns:
;WITH CTE AS 
  (
    SELECT Ordinal, Col1 as SeriesName, Col as Value FROM (SELECT * FROM @ResultTable) P
      UNPIVOT (Col FOR Col1 IN (
        Calc1, Calc2, Calc3 -- Series to return as rows
      )) UN
  )
  SELECT * FROM CTE
    PIVOT 
    (MAX(value) FOR Ordinal IN (
      [1],[2],[3],[4],[5],[6]  -- Ordinals to return as columns
    )) PVT

Transposed table results:
Calc1 | 1000000 | 0 | 1000000 | 0 | 1000000 | 0
Calc3 | 3.53674427608709 | 7.07348855217417 | 7.07348855217417 | 10.6102328282613 | 10.6102328282613 | 0
Calc2 | 0.438556290234799 | 0.877112580469597 | 0.877112580469597 | 1.3156688707044 | 1.3156688707044 | 0

Note that the limitation with this solution is the number rows in source table/columns in transposed table is fixed and requires SQL Server 2005+

Also thanks to Rishabh K from http://social.msdn.microsoft.com/Forums/en-AU/transactsql/thread/680023ea-bd2e-4569-a016-62810fee2001