Hue 1.2.0 upgrade and Beeswax

Hey, its been awhile since I've typed anything out on this blog.

This will be short and sweet. Basically, my configuration is this:

node0 contains all user-based applications:
-pig
-hive
-hue
-sqoop

upgrading everything BUT hue was as easy as can be. I already had the proper nodes added to my hdfs-site.xml, core-site.xml, and mapred-site.xml files, including port numbers, etc. but I would still see the dreaded "localhost:8003" error on the web page whenever I loaded it up.

The secret ended up being in the /var/log/hue/beeswax*.out file, indicating a missing com.mysql.jdbc jar file in the classpath, so I simply copied "mysql-connector-java-5.1.11-bin.jar" to "/usr/share/hue/apps/beeswax/hive/lib/" and restarted the hue service. Beeswax now runs without issue. That was the only thing holding me back for 6 hours :( I feel so dumb, but hopefully this will help others in their quest to get beeswax working properly.

Oh, and the "hive.aux.jars.path" property in hive-site.xml doesn't do a damn thing to help. Don't even bother with that property, it doesn't work. This is as of 1.2.0.

Modifying hadoop configuration files on a Windows Machine == TROUBLE!!!

Ok, so I have all of our hadoop configuration files in subversion, and I have a cron job that runs every minute to sync up the puppetmaster's copy to what is in svn, and then touch the site.pp file so that all of the puppet agents eventually get the changes that have been committed to our revision control system. It works really well, and I like the fact that I can work on configuration files on my own machine without having to PuTTY into another machine and having to work with nano or vi (which are great programs, just not as convenient as notepad or EditPlus).

That said, one must be careful to not lose the "LINUX"ness from the files if you decide to modify these guys on a windows box, using tortiseSVN and a windows text editor.

You will see an error like the following when you attempt to start your services:

[root@HANODE2 ~]# service hadoop-0.20-datanode restart
Stopping Hadoop datanode daemon (hadoop-datanode): /etc/hadoop-0.20/conf/hadoop-: command not found
: command not foundnf/hadoop-env.sh: line 7:
......
: command not foundnf/hadoop-env.sh: line 52:
no datanode to stop
[ OK ]
Starting Hadoop datanode daemon (hadoop-datanode): /etc/hadoop-0.20/conf/hadoop-: command not found
: command not foundnf/hadoop-env.sh: line 7:
......
: command not foundnf/hadoop-env.sh: line 10:
: command not foundnf/hadoop-env.sh: line 52:
/hadoop-hadoop-datanode-HANODE2.outlog/hadoop
: command not foundnf/hadoop-env.sh: line 2:
......
: command not foundnf/hadoop-env.sh: line 49:
: command not foundnf/hadoop-env.sh: line 52:
Exception in thread "main" java.lang.NoClassDefFoundError:
Caused by: java.lang.ClassNotFoundException:
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
. Program will exit.in class:
[ OK ]

The one-time fix is easy, just make the following call on your hadoop configuration files:

dos2unix /etc/hadoop/conf.MYCONFIGURATION/*

Though this does not scale well with the number of machines that your configuration will be deployed to in your cluster, so we have two options here:

1) Use a text editor that is linux file friendly
2) Add a step to the download on the puppetmaster that makes these files "linux"ee again before the agents get a chance to grab the latest changes.

I've chosen to simply add the following lines to my cron script so that on download of the latest from SVN, the files are forced into linux format for all to love:

dos2unix -q /etc/puppet/modules/hadoop/files/*
dos2unix -q /etc/puppet/modules/hadoop/files/conf.my_cluster/*
dos2unix -q /etc/puppet/modules/hadoop/manifests/*
dos2unix -q /etc/puppet/modules/hadoop/manifests/classes/*

A Brief Overview of Our Setup

Ok, so setting up a Hadoop cluster isn't super straight-forward, but there are lots of folks out there attempting to bring it all together for us. If I miss a reference here or there, please email me and let me know so that I can source the entries that I place in this blog.

Our setup, for the time being is rather simple. We have 4 nodes in total, one is our NameNode/JobTracker (herein refered to as Node1), and the other three are DataNodes (refered to as Node2-4). This will grow over the next couple of months into at least a 10 node configuration, so having a proper Configuration Management system installed will be crucial to the survival of the opscon (me).

We also have a server that is our "Staging" instance (referred to as Node0, but perhaps another name would be more fitting). This server has Active Directory integration and will be where all users will go to load data, run queries, etc. For the time being this server will also be the host of the MySQL instance for Hive and Puppet, as well as the host for the Puppet Master daemon.

We are implementing Cloudera's distribution of Hadoop, including Hive, Pig, Sqoop, and a variety of other utilities.

My next entry will be around Active Directory installation on our Staging server. It will be fairly brief, and will detail how to ensure that Pig can be executed by a user with near-zero permissions.