Scratch Co.

Before installing Hudson I had a very basic idea about its purpose and capabilities.  Eventually this understanding came pummeling down when I went to unit test a Ruby project I have been working on.  Luckily, this post was a google search away.  It assumes you already have ruby, rubygems, and rails installed on your build machine.

*(I ran into some problems loading rails and mysql onto the hudson server.  This post and this post really helped me with the errors I received.)

ci_reporter is a plugin that allows me to track the unit tests within my rails project.  There are two methods of installation: as a gem or a plugin.  A gem is available system wide while a plugin is added directly to one projects codebase.  I chose to install it as a plugin so all required code is shipped along with the project. 

Throughout the day today I have been struggling to set up a successful configuration between a server and client instance of git repositories.  To track down the perfect, error free solution, I have decided to write a blog post recording every little detail.  If my configuration fails, I will alter my steps and try again until I no longer run into errors.  This should be fun!

Server

First I must set up the remote (server) repository on my scm instance.  In order to the repository to be accessible from over http, I must add a location tag to the list of sites-enables.

vim /etc/apache2/sites-enabled/000-default

and add:

<Location /git/webservice.git>
DAV on
AuthType Basic
AuthName "Git"
AuthUserFile /etc/apache2/svn.htpasswd
Require valid-user
</Location>

Once this is in place, I must create the directory provided in the Location tag.

mkdir webservice.git
cd webservice.git

Now that I am inside the directory of my repository, I will run a few commands that sets up the general structure of a git repository.

git --bare init --shared
git --bare update-server-info

After these commands are successfully executed, I will give ownership of the directories to apache.

chown -R www-data:www-data

And just to be safe I usually restart apache so everything gets registered correctly.

/etc/init.d/apache2 restart

Client

On the client side we can create a repository out of any already existing directory.  I will be selecting the directory that holds the rails code for my api.

cd /path/to/directory
git init
git add .
git commit


Before we can try to communicate with the remote repository we must store some information for authentication.

vim ~/.netrc


and add:

machine <servername>
login <username>
password <password>


then change the permissions for the file

chmod 600 ~/.netrc

Finally we must link the remote repository to our local repository with the following commands:

git remote add origin http://svn.scratchco.dnsalias.com/webservice.git/

This command places some code into /local/repo/.git/config that will allow us to more conveniently push our repostory to the remote server using:

git push origin -all


webservice

• Trac
• Git
• Hudson
• Index

sdk

• Trac
• SVN
• Hudson

application

• Trac
• SVN
• Hudson

hudson

• home

mysql

• phpmyadmin
• url - mysql.scratchco.dnsalias.com:3306

api

• url - http://api.scratchco.dnsalias.com/

I followed the same steps as before to get the virtual server loaded and configured.  Since you have already seen these steps three time previous, I have chosen to leave out the details and spare you the time of scrolling past them.  The real importance of this post is to discuss a new module for apache that supports Rails communication.  The name of the module is named passenger (nickname mod_rails) and was developed by the great group over at Phusion.   Before this module was created, the ways in which rail projects could be deployed were limited.  Most of them required you to jump through hoops just to get it installed.  “mod_rails” is a nice relief for the average developer who is already familiar with apache.  The installation consists of three steps that take a matter of minutes.

Even though the module is in the Beta stages of development, I have decided to implement it into my production server in hopes of becoming familiar with it before its official release.  Plus, I dont need to worry about stability at the moment since I am no where near launch.  This post will document the steps taken to get passenger running on a clean debian based OS.

These steps derive from this reference

Dependencies
apt-get install make gcc build-essential

Ruby
aptitude install ruby libzlib-ruby rdoc irb ruby1.8-dev

Gems
wget http://rubyforge.org/frs/download.php/35283/rubygems-1.1.1.tgz
tar zxvf rubygems-1.1.1.tgz
cd rubygems-1.1.1
ruby setup.rb all
ln /usr/bin/gem1.8 gem


Rails
gem install rails --include-dependencies

Apache
aptitude install apache2 apache2-doc apache2-dev libopenssl-ruby
mkdir /var/rails
chown -R www-data:www-data /var/rails


Database

aptitude install libmysql-ruby


Passenger (mod_rails)
gem install passenger
passenger-install-apache2-module

Configure Apache

vim /etc/apache2/httpd.conf

and add

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-1.0.1/ext/apache2/mod_passenger.so
RailsSpawnServer /usr/lib/ruby/gems/1.8/gems/passenger-1.0.1/bin/passenger-spawn-server
RailsRuby /usr/bin/ruby1.8

NameVirtualHost *:80
<virtualhost>
ServerName api.scratchco.dnsalias.com
#DocumentRoot /var/rails/public
DocumentRoot /var/www
RailsBaseURI /rails
</virtualhost>

Result 
api.scratchco.(dnsalias).com/rails/

I have decided to use Git Version Control System in place of Subversion for the Ruby portion of my project. This decision is influenced by the strong commitment the ruby community has shown towards it. In order to do this I am required to do four things:

1. Install Git
2. Update Apache virtual servers to allow for updates over http
3. Install plugin for eclipse to handle Git repositories in addition to Subversion
4. Install and configure git plugin for Trac Instance
5. Install and configure git plugin for Hudson

Tonight I managed to knock out the third requirement on the list. The link provided was very helpful through the process. Tomorrow I will begin work on the first three.

Ruboss is a new framework, developed by dima berastau and peter armstrong, which aims to automate a large portion of communication between Flex and RESTful Rails controllers. This fits perfectly into my design for a web service, SDK, and application. Ruboss will help while developing the SDK portion of the project. The great thing about ruboss is it’s ability to communicate to many different RESTful backends (simpleDB, rails, merb) while using many different languages (xml, json, amf, etc.) to store and transport information.

I have walked through the example project without any complications. This week I will be digging through the code in hopes of discovering the current capabilities as well as future possibilities. This framework is said to work with both PureMVC and Cairngorm, each of which are “micro-architectures” within the flex community. Not only does it allow for flexibility, but it also provides a quicker and easier path for my development. The only downside of the project is its immature state and loyalty to the REST methodology. From what I understand at this point, I must follow strict guidelines in order to make my controllers RESTful before I can use the Ruboss framework.

This is a very interesting project. I will keep the blog updated with my findings and hopefully a solution to my SDK development. Talk to you soon.

In the past two installations, the command for creating a xen image was a bit exaggerated to emphasize the options available. Now that we have the general config file (/etc/xen-tools/xen-tools.conf) how we want it, all we need is the following code to get a debian 4.0 image up and running:

xen-create-image --hostname=hudson.scratchco.com --size=10Gb --ide \
--ip=10.0.0.3 --force --dir=/vserver --debootstrap

after the image is created, we need to create a symbolic link so the image starts up with the host server

ln -s /etc/xen/hudson.scratchco.com.cfg /etc/xen/auto

Next, ssh into the new image and install the standard dependencies along with apache. Once we have apache installed we need to create Virtual Hosts on the host server to point to the newly created image.

On the host server

vim /etc/apache2/sites-enabled/hudson.scratchco.com

and place the following code within it

<VirtualHost *>
# Basic setup
ServerAdmin jacob@scratchco.com
ServerAlias hudson.scratchco.dnsalias.com
Servername hudson.scratchco.dnsalias.com
ProxyRequests off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://10.0.0.3:8180/hudson
ProxyPassReverse / http://10.0.0.3:8180/hudson
</VirtualHost>

Install Java and Tomcat

first install Java

apt-get install sun-java5-jdk

then install tomcat

apt-get install tomcat5.5
apt-get install tomcat5.5-admin
apt-get install tomcat5.5-webapps

Install and Configure Hudson

Go to tomcat webapps directory and get hudson

cd /var/lib/tomcat5.5/webapps/
wget https://hudson.dev.java.net/files/documents/2402/85741/hudson.war


We have to tweak tomcat by going to its configuration file

vim /etc/default/tomcat5.5


and making it look like this

# Use the Java security manager? (yes/no, default: yes)
# WARNING: Do not disable the security manager unless you understand
# the consequences!
# NOTE: java-gcj-compat-dev currently doesn't support a security
# manager.
TOMCAT5_SECURITY=no


Since hudson stores its records in files rather than a database, we must set HUDSON_HOME to the directory where we would like to store the files.

mkdir /path/to/hudson_home
chown -R tomcatUser:group /path/to/hudson_home <-- very important
export HUDSON_HOME=/path/to/hudson_home
catalina.sh start


And that should wrap it up. Now when I visit hudson.scratchco.dnsalias.com I will see the hudson control panel. My next task is to become familiar with hudson and its best practices.

*I still have not figured out why the ip addresses need to be changed (/etc/hosts), but I will give it a higher priority on my to-do list.

So, to this point I have a xen master server with a xen slave instance. The slave instance is actually a virtual server with the purpose of hosting subversion and trac. The next component every network needs is a db server. This should be a painless process since I had documented the steps while creating the first instance. I will do the same for this instance just to see what steps overlap and what steps are independent to the build. This will also help me determine the common commands for creating/starting a new xen slave instance; which I will later document as well.

*The LAMP (Linux, Apache, MySQL, PHP) installation instructions were gathered from HowToForge

Create Xen Server


xen-create-image --hostname=mysql.scratchco.com --size=10Gb --swap=512Mb --ide \
--ip=10.0.0.2 --netmask=255.0.0.0 --gateway=10.0.0.254 --force \
--dir=/vserver --memory=512Mb --arch=i386 --kernel=/boot/vmlinuz-2.6.18-xen \
--debootstrap --dist=etch --mirror=http://ftp.us.debian.org/debian/ --passwd


Start Server

xm create /etc/xen/mysql.scratchco.com.cfg


Configure to Start at Boot

ln -s /etc/xen/mysql.scratchco.com.cfg /etc/xen/auto


SSH into Server

ssh 10.0.0.2


Edits Hosts

vim /etc/hosts

and change

192.168.0.101 svn.scratchco.com

to

10.0.0.1 svn.scratchco.com

* I am not sure why this needs to be changed, I will research this issue and update the post when I have a better solution

Install Dependencies

apt-get update
apt-get upgrade
apt-get install vim-full
apt-get install build-essential
apt-get install locales
dpkg-reconfigure locales


select en_US.UTF-8

/usr/bin/tzconfig or /usr/sbin/tzconfig

Select time zone

Install MySQL
Install packages

apt-get install mysql-server mysql-client libmysqlclient15-dev

Edit Configuration File

vim /etc/mysql/my.cnf

comment out

...
#bind-address = 127.0.0.1
...


Setup User
mysqladmin -u root password yourrootsqlpassword
mysqladmin -h mysql.scratchco.com -u root password yourrootsqlpassword

Give Permissions
GRANT ALL PRIVILEGES ON *.* TO root@'localhost' IDENTIFIED BY 'root-password'

Install Apache/PHP5
Install Apache

apt-get install apache2 apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert


Install PHP
apt-get install libapache2-mod-php5 php5 php5-common php5-curl php5-dev php5-gd php5-idn php-pear php5-imagick php5-imap php5-json php5-mcrypt php5-memcache php5-mhash php5-ming php5-mysql php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl phpmyadmin


Install Apache Modules

a2enmod ssl
a2enmod rewrite
a2enmod suexec
a2enmod include

Configure Master Server
Open iptables

vim /etc/network/if-up.d/iptables

and add this line to forward all requests on port 3306 to our new server

iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 3306 -j DNAT --to 10.0.0.2:3306


Create VirtualHost

create new file

vim /etc/apache2/sites-enabled/mysql.scratchco.com

and add

<VirtualHost *>
# Basic setup
ServerAdmin jacob@scratchco.com
ServerAlias mysql.scratchco.dnsalias.com
Servername mysql.scratchco.dnsalias.com
ProxyRequests off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://10.0.0.2/
ProxyPassReverse / http://10.0.0.2/
</VirtualHost>

Configure DB Server

Now we have to configure the db server to default to phpmyadmin when accessed from the web. First we need to delete the web directory and create a symbolic link to the phpmyadmin directory which was installed with apt-get.


rm -rf /var/www/
ln -s /usr/share/phpmyadmin/ /var/www


Second, we must edit the default virtualhosts to make /var/www the root content for htttp://mysql.scratchco.(dnsalias).com/

vim /etc/apache2/sites-available

and comment out

...
# RedirectMatch ^/$ /apache2-default/
...


Plugins:

• Trac Admin (required)
• IniAdminPlugin
• AccountManagerPlugin
• SvnAuthzAdminPlugin
• TracSvnHooksPlugin
• HttpAuthPlugin
• EclipseTracPlugin
• Hudson

Today I embarked on a journey to improve the capabilities of my xen server. In the original xen configuration, I chose to use iptables in order to pass requests onto the virtual instances. This has one major drawback that prohibited decoupling of the individual components (svn, trac, mysql, etc.). Since each of these components have a web interface running on port 80, I had to find a solution to separate the components but still provide web access to each instance.

My first option was to change the ports for each of the components (svn on 81, trac on 82, mysql on 83). In the past, I have experienced numerous complications when trying to switch the port of open source projects. I find there is atleast one piece of the application that breaks because of the change. My second option, and the one I chose to pursue, was a solution using an apache module which allows requests to be redirected to another domain/ip that is on the local network.

Attempt #1

After I had a general idea of the direction I was heading, I began digging into the apache documentation to search for a module that fit my requirements. A couple hours of googling later, I came up with a solution that took advantage of the mod_rewrite module within apache. I intended to use the module to receive a url and then redispatch the request on the local network. This technique would also require a dns server to keep track of the local ip addresses and domains for the individual server instances. I later found out that my understanding of the rewrite module was a bit skewed. Its purpose and usage was to take a url and manipulate it before sending it back to the user as a redirect. This meant that the request was never fired again in my local network. Onto plan B…
Attempt #2

After another hour or so of searching I came across this blog which showed how to use the Apache mod_proxy module to redirect the request locally. Before I could redirect the user to a specific server I had to find out what server they would like to go to. This task is rather simple with the help of Apache’s VirtualHosts. Once I had the virtual hosts setup, I added a ProxyPass and ProxyPassReserve to its configuration. As you’ll see later in the code, the Proxy pass redirects all requests to the appropriate server instance without the user even knowing. Now that you have the general gist of how the system works, I will walk you through the steps required for installation and configuration

Installation

Install Apache


apt-get install apache2 apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert


Install and Load Modules


apt-get install libapache2-mod-proxy-html
a2enmod include
a2enmod rewrite
a2enmod headers
a2enmod deflate
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_html

Restart Apache

/etc/init.d/apache2 force-reload

Create VirtualHost File

vim /etc/apache2/sites-enabled/svn.scratchco.com

And Add

<VirtualHost *>
# Basic setup
ServerAdmin jacob@scratchco.com
ServerAlias svn.scratchco.dnsalias.com
Servername svn.scratchco.dnsalias.com
ProxyRequests off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://10.0.0.1/svn/
ProxyPassReverse / http://10.0.0.1/svn/
</VirtualHost>

I also created a virtual host for trac, since I wanted it to have its own subdomain.

Create VirtualHost File

vim /etc/apache2/sites-enabled/trac.scratchco.com

And Add

<VirtualHost *>
# Basic setup
ServerAdmin jacob@scratchco.com
ServerAlias trac.scratchco.dnsalias.com
Servername trac.scratchco.dnsalias.com
ProxyRequests off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://10.0.0.1/
ProxyPassReverse / http://10.0.0.1/
</VirtualHost>

Restart Apache

/etc/init.d/apache2 force-reload

Results

http://svn.scratchco.com/scratchco

http://trac.scratchco.com/scratchco

Conclusion

I am glad I was able to complete this portion of the project. Now I have a system that will ease development. The solution I presented today is only temporary. Eventually I would like to install a dns server that will control the internal ip and domains. At this moment I am using a free DDNS which makes my url *.scratchco.dnsalias.com. A dns server will allow me to change the ServerName and ServerAlias fields of the Virtual host to svn.scratchco.com and trac.scratchco.co, respectively. In order to do this I will also need to make my local dns server the first nameserver I check (/etc/resolv.conf) when searching for records. This will mainly result in less configuration once I get a static IP address.