Setting up an FTP server on CentOS 6.x

Need FTP access on a CentOS machine? Easy.

sudo yum install vsftpd
sudo service vsftpd start
sudo chkconfig vsftpd on
chkconfig --list

sudo vi /etc/vsftpd/vsftpd.conf # edit default conf file to have these lines:
    anonymous_enable=NO
    chroot_local_user=YES
    chroot_list_enable=NO
    chroot_list_file=/etc/vsftpd/chroot_list
    pasv_min_port=3000
    pasv_max_port=3050
   
sudo touch /etc/vsftpd/chroot_list
sudo /sbin/service vsftpd restart

Done.

Note that these steps have been grabbed from this Rackspace article that includes much more detail:
http://www.rackspace.com/knowledge_center/article/rackspace-cloud-essentials-centos-installing-vsftpd

New Eclipse workspace settings

Each new Eclipse workspace has the default settings, arg! I'm going to collect my customisations here.


Window > Preferences...

General > Editors > Text Editors > Spelling | Turn off

General > Editors > Text Editors | Show line numbers

General > Keys | Next Editor bound to Ctrl-Tab; Prev Editor bound to Ctrl-Shift-Tab


Need to check out:
https://code.google.com/a/eclipselabs.org/p/workspacemechanic/
http://mcuoneclipse.com/2012/04/04/copy-my-workspace-settings/

Setting up Hive with HBase external storage on CDH

If you have existing HBase tables it can be very handy to create Hive external tables wrapping these so that you can run HiveQL queries.

The following HiveQL will create the metastore table schema on top of MyTableName:

CREATE EXTERNAL TABLE MyTableName(key string, Column1 string, Column2 string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:Column1,cf:Column2")
TBLPROPERTIES ('hbase.table.name' = 'MyTableName');


The next step is to add the hbase auxlib jars to hive-site.xml to ensure certain HiveQL queries will run (e.g. select count(1) from MyTableName):
<property>
<name>hive.aux.jars.path</name>
<value>file:///opt/cloudera/parcels/CDH/lib/hive/lib/zookeeper.jar,file:///opt/cloudera/parcels/CDH/lib/hive/lib/hbase.jar,file:///opt/cloudera/parcels/CDH/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///opt/cloudera/parcels/CDH/lib/hive/lib/guava-11.0.2.jar</value>
</property>

This property must be added to the hive1 service > Config > Service-Wide > Advanced > Hive Service Configuration Safety Valve for hive-site.xml section to be able to execute certain HiveQL commands from the host command line.

To execute HiveQL from the Hue HiveUI add it to the hue1 service > Config > Beeswax Server (Default) > Advanced > Hive Configuration Safety Valve section.

Refs:
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration
http://www.confusedcoders.com/bigdata/hive/hbase-hive-integration-querying-hbase-via-hive
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_18_10.html
https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/E8GfiwMOIPw

Free disk space email alert on Windows Server

Schedule the following PowerShell script to run periodically to receive email alerts on low disk space:


# Schedule with: %windir%\SysWOW64\WindowsPowerShell\v1.0\powershell.exe -WindowStyle "Hidden" -File C:\path_to_scripts\freedisk_space_alert.ps1

$minGbThreshold = 10;

$computers = "localhost"
$smtpAddress = "localhost";
$toAddress = "someone@test.com";
$fromAddress = "someone@test.com";

foreach($computer in $computers)
{
    $disks = Get-WmiObject -ComputerName $computer -Class Win32_LogicalDisk -Filter "DriveType = 3";
    $computer = $computer.toupper();
    $deviceID = $disk.DeviceID;
    foreach($disk in $disks)
    {
        $freeSpaceGB = [Math]::Round([float]$disk.FreeSpace / 1073741824, 2);
        if($freeSpaceGB -lt $minGbThreshold)
        {
            $smtp = New-Object Net.Mail.SmtpClient($smtpAddress)
            $msg = New-Object Net.Mail.MailMessage
            $msg.To.Add($toAddress)
            $msg.From = $fromAddress
            $msg.Subject = "Diskspace below threshold: " + $computer + "\" + $disk.DeviceId
            $msg.Body = $computer + "\" + $disk.DeviceId + " " + $freeSpaceGB + "GB Remaining";
           
            $smtp.UseDefaultCredentials = $false;       
            $cred = New-Object System.Net.NetworkCredential("\smtpuser", "<enter password>");  # Ensure smtpuser is authorised to send emails
            $smtp.Credentials = $cred;

            $smtp.Send($msg)
        }
    }
}


This script is slightly modified from the one found here to include credentials:
http://gavindraper.com/2012/09/22/automatic-low-hard-disk-alerts-for-windows-server/
Thanks Gavin!

Also you may need to enable script execution in PowerShell by running the following:
%windir%\SysWOW64\WindowsPowerShell\v1.0\powershell.exe set-executionpolicy remotesigned
Which will still restrict internet downloaded scripts and is better than setting this value to unrestricted
See: http://superuser.com/questions/106360/how-to-enable-execution-of-powershell-scripts

Efficient uploading of Jar files to your Hadoop cluster

Copying fat Jar files up to your Hadoop cluster to execute jobs on production-sized data sets in order to find bottlenecks can be painful when you want a quick turn around whilst debugging.

Sometimes local mode just doesn't cut it.

A good solution is to use rsync which supports incremental checking to only transfer file differences.

Command:
rsync -avz /your_source_directory/somejob-0.0.1.jar login@servername:/target_directory/somejob-0.0.1.jar

Options:
-a archive mode
-v verbose mode
-z compress file data during the transfer

Solution to Maven solr-core artifact causes Eclipse error: "Missing artifact jdk.tools:jdk.tools:jar:1.6"

In Eclipse with a Maven project, if referencing artifact solr-core (v4.x) Eclipse can return a Maven Dependency Problem "Missing artifact jdk.tools:jdk.tools:jar:1.6".

This also causes a build path problem: The container 'Maven Dependencies' references non existing library 'C:\Users\<userdir>\.m2\repository\jdk\tools\jdk.tools\1.6\jdk.tools-1.6.jar'

The tools jar file is supplied by the JDK so we can exclude it in the pom.xml by adding an <exclusions> section like so:

<dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr-core</artifactId>
    <version>4.5.1</version>
    <exclusions>
        <exclusion>
            <artifactId>jdk.tools</artifactId>
            <groupId>jdk.tools</groupId>
        </exclusion>
    </exclusions>
</dependency>

Does anyone know if there is a better way to resolve this issue?

Export SQL Server table to csv file with headers using bcp

Ref: http://pastebin.com/x8Kk4Dn9
Ref: http://stackoverflow.com/questions/1355876/export-table-to-file-with-column-headers-column-names-using-the-bcp-utility-an/9754485#9754485


I use a method that outputs one file for the column headers read from INFORMATION_SCHEMA.COLUMNS and then appends a second file with the table data, both of which are generated using BCP.

Here is the batch file that creates TableData.csv, just replace the environment variables at the top.

Note that if you need to supply credentials, replace the -T option with -U my_username -P my_password

set BCP_EXPORT_SERVER=put_my_server_name_here
set BCP_EXPORT_DB=put_my_db_name_here
set BCP_EXPORT_TABLE=put_my_table_name_here

BCP "DECLARE @colnames VARCHAR(max);SELECT @colnames = COALESCE(@colnames + ',', '') + column_name from %BCP_EXPORT_DB%.INFORMATION_SCHEMA.COLUMNS where TABLE_NAME='%BCP_EXPORT_TABLE%'; select @colnames;" queryout HeadersOnly.csv -c -T -S%BCP_EXPORT_SERVER%

BCP %BCP_EXPORT_DB%.dbo.%BCP_EXPORT_TABLE% out TableDataWithoutHeaders.csv -c -t, -T -S%BCP_EXPORT_SERVER%

set BCP_EXPORT_SERVER=
set BCP_EXPORT_DB=
set BCP_EXPORT_TABLE=

copy /b HeadersOnly.csv+TableDataWithoutHeaders.csv TableData.csv

del HeadersOnly.csv
del TableDataWithoutHeaders.csv

This method has the advantage of always having the column names in sync with the table by using INFORMATION_SCHEMA.COLUMNS. The downside is it's a bit messy and creates temporary files. Microsoft should really fix the bcp utility to support this.

It uses the row concatenation trick from Concatenate many rows into a single text string? combined with ideas from http://social.msdn.microsoft.com/forums/en-US/sqlgetstarted/thread/812b8eec-5b77-42a2-bd23-965558ece5b9/