Getting Active Directory working with Hadoop, Hive, Sqoop, etc.

Well... I have found a little free tool for integrating linux operating systems with Windows Active Directory fairly seamlessly. It is called LikeWise Open (http://www.likewise.com/community/index.php/download/). Just using a couple of simple command-line statements, I was able to
a)Add the machine to the active directory
b)Allow users to log in, changing the default shell to /bin/bash

The only thing I needed to do was manually modify the /etc/sudoers file to add our ADs group called "hadoop users" be able to run all commands

DOMAIN\\hadoop^users ALL=(ALL) NOPASSWD: ALL

Done! Now Hive and Sqoop can be executed using 'sudo hive -e "SELECT * FROM table1;"', or "sudo sqoop-import ......."

This is by far the easiest way to get your windows users to get access to the the cluster without having to manage every account individually.

Sqoop, SQLServer, DateTime, and Fun!

Ok, so writing a simple query like the following:

SELECT a.* FROM table a WHERE a.timestamp >= '20101125'

Should be pretty straightforward for importing data into Hive through Sqoop, but alas there are some hurdles.

I'll just get straight to the solution, I know people hate to read, so here is the final command line example:

sqoop import --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --connect 'jdbc:sqlserver://localhost;user=xxxxxx;password=xxxxxxx;database=MyDB' --query "select a.* from tableX a where (a.timestamp >= '2010-11-25 00:00:00') AND \$CONDITIONS" --target-dir /data/feedlog --split-by 'MyIDColumn' --fields-terminated-by '\t'

Basically, enclose the entire --query in double-quotes, and then escape the $ on the $CONDITIONS value with a \ as well so that is escapes the initial bash evaluation.

Happy hunting!