Hey, its been awhile since I've typed anything out on this blog.
This will be short and sweet. Basically, my configuration is this:
node0 contains all user-based applications:
-pig
-hive
-hue
-sqoop
upgrading everything BUT hue was as easy as can be. I already had the proper nodes added to my hdfs-site.xml, core-site.xml, and mapred-site.xml files, including port numbers, etc. but I would still see the dreaded "localhost:8003" error on the web page whenever I loaded it up.
The secret ended up being in the /var/log/hue/beeswax*.out file, indicating a missing com.mysql.jdbc jar file in the classpath, so I simply copied "mysql-connector-java-5.1.11-bin.jar" to "/usr/share/hue/apps/beeswax/hive/lib/" and restarted the hue service. Beeswax now runs without issue. That was the only thing holding me back for 6 hours :( I feel so dumb, but hopefully this will help others in their quest to get beeswax working properly.
Oh, and the "hive.aux.jars.path" property in hive-site.xml doesn't do a damn thing to help. Don't even bother with that property, it doesn't work. This is as of 1.2.0.
Showing posts with label MySQL. Show all posts
Showing posts with label MySQL. Show all posts
2011-03-17
2010-11-30
Getting Puppet up and Running
First off if after reading all of the articles, documentation, and googling that you can possibly muster in order to solve your problems, go onto IRC at
The folks there will be more than happy to help you, LIVE!
Many thanks go out to "ZipKid", "Volcane", and "whack" for helping me out!
Ok, so I had some real trouble even getting Puppet up and running. Again, I'm running CentOS 5.5, and I decided to have puppet run on it's own special VM so that I have some segregation of responsibilities in our infrastructure.
That said, I decided to try and install the latest and greatest version, Puppet; 2.6.3.
In order to accomplish the install using yum, please go to the following VERY helpful site and read the instructions thoroughly before starting on your journey:
Ok, so when you run into certificate errors, you need to keep a couple of things in mind:
1) Run everything with --no-daemonize --debug (puppetmasterd, puppetd)
2) If you get the error, start ALL OVER AGAIN, by going onto the master server and performing the following steps
2 a) puppet cert --list --all
2 b) puppet cert --remove <certmachine.domain.com>
3) deleting EVERYTHING inside /var/lib/puppet/ssl/, and /etc/puppet/ssl/ on ALL machines in question (master and agent(s))
Now re-run the cert request line as shown
You'll see the request come in on the server output. Open another console and run the following on the master:
Find the line that shows the FQDN of the agent requesting the certificate and run
That should do it, you can run the following to see if they are talking properly from the agent:
You shouldn't see any errors, if you do, go through your .pp files and clean them up as needed.
Freenode.net #puppet
The folks there will be more than happy to help you, LIVE!
Many thanks go out to "ZipKid", "Volcane", and "whack" for helping me out!
Ok, so I had some real trouble even getting Puppet up and running. Again, I'm running CentOS 5.5, and I decided to have puppet run on it's own special VM so that I have some segregation of responsibilities in our infrastructure.
That said, I decided to try and install the latest and greatest version, Puppet; 2.6.3.
In order to accomplish the install using yum, please go to the following VERY helpful site and read the instructions thoroughly before starting on your journey:
http://www.craigdunn.org/2010/08/part-1-installing-puppet-2-6-1-on-centos-with-yumrpm/
Ok, so when you run into certificate errors, you need to keep a couple of things in mind:
1) Run everything with --no-daemonize --debug (puppetmasterd, puppetd)
2) If you get the error, start ALL OVER AGAIN, by going onto the master server and performing the following steps
2 a) puppet cert --list --all
2 b) puppet cert --remove <certmachine.domain.com>
3) deleting EVERYTHING inside /var/lib/puppet/ssl/, and /etc/puppet/ssl/ on ALL machines in question (master and agent(s))
Now re-run the cert request line as shown
puppet agent --no-daemonize --debug --waitforcert 60 --test
You'll see the request come in on the server output. Open another console and run the following on the master:
puppet cert --list
Find the line that shows the FQDN of the agent requesting the certificate and run
puppet cert --sign <machine.domain.com>
That should do it, you can run the following to see if they are talking properly from the agent:
puppet agent --noop --test --server=<servername.domain.com>
You shouldn't see any errors, if you do, go through your .pp files and clean them up as needed.
2010-11-23
A Brief Overview of Our Setup
Ok, so setting up a Hadoop cluster isn't super straight-forward, but there are lots of folks out there attempting to bring it all together for us. If I miss a reference here or there, please email me and let me know so that I can source the entries that I place in this blog.
Our setup, for the time being is rather simple. We have 4 nodes in total, one is our NameNode/JobTracker (herein refered to as Node1), and the other three are DataNodes (refered to as Node2-4). This will grow over the next couple of months into at least a 10 node configuration, so having a proper Configuration Management system installed will be crucial to the survival of the opscon (me).
We also have a server that is our "Staging" instance (referred to as Node0, but perhaps another name would be more fitting). This server has Active Directory integration and will be where all users will go to load data, run queries, etc. For the time being this server will also be the host of the MySQL instance for Hive and Puppet, as well as the host for the Puppet Master daemon.
We are implementing Cloudera's distribution of Hadoop, including Hive, Pig, Sqoop, and a variety of other utilities.
My next entry will be around Active Directory installation on our Staging server. It will be fairly brief, and will detail how to ensure that Pig can be executed by a user with near-zero permissions.
Our setup, for the time being is rather simple. We have 4 nodes in total, one is our NameNode/JobTracker (herein refered to as Node1), and the other three are DataNodes (refered to as Node2-4). This will grow over the next couple of months into at least a 10 node configuration, so having a proper Configuration Management system installed will be crucial to the survival of the opscon (me).
We also have a server that is our "Staging" instance (referred to as Node0, but perhaps another name would be more fitting). This server has Active Directory integration and will be where all users will go to load data, run queries, etc. For the time being this server will also be the host of the MySQL instance for Hive and Puppet, as well as the host for the Puppet Master daemon.
We are implementing Cloudera's distribution of Hadoop, including Hive, Pig, Sqoop, and a variety of other utilities.
My next entry will be around Active Directory installation on our Staging server. It will be fairly brief, and will detail how to ensure that Pig can be executed by a user with near-zero permissions.
Subscribe to:
Posts (Atom)