Efficient uploading of Jar files to your Hadoop cluster

Copying fat Jar files up to your Hadoop cluster to execute jobs on production-sized data sets in order to find bottlenecks can be painful when you want a quick turn around whilst debugging.

Sometimes local mode just doesn't cut it.

A good solution is to use rsync which supports incremental checking to only transfer file differences.

Command:
rsync -avz /your_source_directory/somejob-0.0.1.jar login@servername:/target_directory/somejob-0.0.1.jar

Options:
-a archive mode
-v verbose mode
-z compress file data during the transfer

Solution to Maven solr-core artifact causes Eclipse error: "Missing artifact jdk.tools:jdk.tools:jar:1.6"

In Eclipse with a Maven project, if referencing artifact solr-core (v4.x) Eclipse can return a Maven Dependency Problem "Missing artifact jdk.tools:jdk.tools:jar:1.6".

This also causes a build path problem: The container 'Maven Dependencies' references non existing library 'C:\Users\<userdir>\.m2\repository\jdk\tools\jdk.tools\1.6\jdk.tools-1.6.jar'

The tools jar file is supplied by the JDK so we can exclude it in the pom.xml by adding an <exclusions> section like so:

<dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr-core</artifactId>
    <version>4.5.1</version>
    <exclusions>
        <exclusion>
            <artifactId>jdk.tools</artifactId>
            <groupId>jdk.tools</groupId>
        </exclusion>
    </exclusions>
</dependency>

Does anyone know if there is a better way to resolve this issue?

Export SQL Server table to csv file with headers using bcp

Ref: http://pastebin.com/x8Kk4Dn9
Ref: http://stackoverflow.com/questions/1355876/export-table-to-file-with-column-headers-column-names-using-the-bcp-utility-an/9754485#9754485


I use a method that outputs one file for the column headers read from INFORMATION_SCHEMA.COLUMNS and then appends a second file with the table data, both of which are generated using BCP.

Here is the batch file that creates TableData.csv, just replace the environment variables at the top.

Note that if you need to supply credentials, replace the -T option with -U my_username -P my_password

set BCP_EXPORT_SERVER=put_my_server_name_here
set BCP_EXPORT_DB=put_my_db_name_here
set BCP_EXPORT_TABLE=put_my_table_name_here

BCP "DECLARE @colnames VARCHAR(max);SELECT @colnames = COALESCE(@colnames + ',', '') + column_name from %BCP_EXPORT_DB%.INFORMATION_SCHEMA.COLUMNS where TABLE_NAME='%BCP_EXPORT_TABLE%'; select @colnames;" queryout HeadersOnly.csv -c -T -S%BCP_EXPORT_SERVER%

BCP %BCP_EXPORT_DB%.dbo.%BCP_EXPORT_TABLE% out TableDataWithoutHeaders.csv -c -t, -T -S%BCP_EXPORT_SERVER%

set BCP_EXPORT_SERVER=
set BCP_EXPORT_DB=
set BCP_EXPORT_TABLE=

copy /b HeadersOnly.csv+TableDataWithoutHeaders.csv TableData.csv

del HeadersOnly.csv
del TableDataWithoutHeaders.csv

This method has the advantage of always having the column names in sync with the table by using INFORMATION_SCHEMA.COLUMNS. The downside is it's a bit messy and creates temporary files. Microsoft should really fix the bcp utility to support this.

It uses the row concatenation trick from Concatenate many rows into a single text string? combined with ideas from http://social.msdn.microsoft.com/forums/en-US/sqlgetstarted/thread/812b8eec-5b77-42a2-bd23-965558ece5b9/

2 Ways to Count Rows in HBase

There are 2 ways to count the number of rows in an HBase table:

Run the following from the linux command line:
$ hbase org.apache.hadoop.hbase.mapreduce.RowCounter MyTableName

Run the following from the hbase shell (which is accessible from the Hue shell):
> count 'MyTableName'

C# LINQ Group by key and get max value of group

This is a handy C# LINQ snippet to group by a key in a list of objects and return a new list with only max values based on another property:

SomeList.GroupBy(x => x.MyKey).Select(g => g.OrderByDescending(x => x.DateCreated).First());

This was a stackoverflow answer - thanks driis!

Decode a mongo BinData field into a Guid with C#

This C# snippet will decode a mongo BinData field into a Guid

got this from Mongo:
{
  "_id" : ObjectId("50b40a5db14ea8902cd5d5ed"),
   "MyGuid : new BinData(3, "MC/0P9nkEeGgjYQrK2V3pQ=="),
   ...
}

but want to know the value of MyGid?  Try this little snippet:

using System;
namespace MongoGuidDecoder
{
    class Program
    {
        static void Main(string[] args)
        {
            var input = args[0];
            var bytes = Convert.FromBase64String(input.Trim());
            var guid = new Guid(bytes);
            Console.WriteLine(input);
            Console.WriteLine(guid);
        }
    }
}

Now run to get the Guid representation:
>MongoGuidDecoder MC/0P9nkEeGgjYQrK2V3pQ==
MC/0P9nkEeGgjYQrK2V3pQ==
3ff42f30-e4d9-e111-a08d-842b2b6577a5

My CentOS Gottchas

Lessons for CentOS 6.3 in Virtualbox

Configure Bridged Adapter to see the internet:

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
edit to have NM_CONTROLLED="no" and ONBOOT="yes"
Ref:  http://serverfault.com/questions/349993/virtualbox-centos-no-eth0

Then run:
# service network restart