Roll your own Subversion server, Warehouse Setup

Warehouse is a Ruby on Rails application developed by entp which provides a front end to your SVN repositories. It is quite useful because it provides a much better system to manage svnserve user management and access. It also lets you show your code to other users by providing them a direct link to the code which they can view in their browser instead of having to checkout a copy via SVN. Let’s get it installed.

Create Warehouse DB

The first step is to create the MySQL databases which will house our Warehouse install. Login to mysql and execute the following commands:

mysql> CREATE DATABASE warehouse;
Query OK, 1 row affected (0.04 sec)

mysql> CREATE DATABASE warehouse_test;
Query OK, 1 row affected (0.00 sec)

Create Warehouse User

Next we will create a MySQL user specifically for Warehouse. By limiting the user’s access we can keep our MySQL database more secure as a compromise of this username and password will only give access to the Warehouse DB. Be sure to replace some_pass with a password of your choice:

mysql> CREATE USER 'warehouse'@'localhost' IDENTIFIED BY 'some_pass';
Query OK, 0 rows affected (0.01 sec)

mysql> GRANT ALL PRIVILEGES ON warehouse.* TO 'warehouse'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON warehouse_test.* TO 'warehouse'@'localhost';
Query OK, 0 rows affected (0.00 sec)

Create the web root

Now let’s create a web root in our home folder where Warehouse will live:

mkdir /home/demo/public_html

Install Ruby Subversion bindings

Before Warehouse can run, we need to install the Ruby Subversion bindings which, as the name suggests, lets Ruby interact with your SVN repo. I’ve never been able to get them installed by following their guide but luckily we can just use yum to install the necessary components:

sudo yum install subversion-ruby

Now fire up IRB and let’s make sure it works:

irb
...
irb(main):001:0> require 'svn/core'
=> true

Download Warehouse

Our system is finally ready for Warehouse. To download it, we simply clone Warehouse via git by doing the following:

cd /home/demo/public_html
git clone git://github.com/entp/warehouse.git

Bootstrap Warehouse

Now tell Warehouse to initialize itself by running the following command and follow the instructions presented on screen. You might run into a couple of “errors” I’ve detailed below but don’t worry about them for now, they won’t stop Warehouse from running.

cd ~/public_html/warehouse
rake warehouse:bootstrap
...

Now it's time for Step 3: Load the database schema.  All of your data will be overwritten. Are you sure you wish to continue? [y/n]
y

mkdir -p /home/demo/public_html/warehouse/log
!! No Ultraviolet gem found, defaulting to javascript syntax highlighting.  Do not be afraid.
!! Error loading plugins: Mysql::Error: Table 'warehouse.plugins' doesn't exist: SELECT * FROM `plugins`   WHERE (name IN ('photo_gallery'))
!! Make sure the database was created successfully and migrated.
...

Warehouse v1.1.6 is ready to roll.
Okay, thanks for bootstrapping!  I know I felt some chemistry here, did you?
Now, start the application with 'script/server' and visit http://mydomain.com/ to start the installation process.

Sweet, we are ready to roll.

Configuring Thin

Create configuration file

cd ~/public_html/warehouse
sudo thin install
...
sudo thin config -C /etc/thin/warehouse.yml -c /home/demo/public_html/warehouse/  --servers 3 -e production

Start Thin

sudo /etc/init.d/thin start
...
[start] /etc/thin/warehouse.yml ...
Starting server on 0.0.0.0:3000 ...
!! Ruby 1.8.5 is not secure please install cgi_multipart_eof_fix:
   gem install cgi_multipart_eof_fix
Starting server on 0.0.0.0:3001 ...
!! Ruby 1.8.5 is not secure please install cgi_multipart_eof_fix:
   gem install cgi_multipart_eof_fix

Oops, install cgimultiparteof_fix

sudo gem install cgi_multipart_eof_fix

Start Thin again

sudo /etc/init.d/thin stop
...
sudo /etc/init.d/thin start
...
[start] /etc/thin/warehouse.yml ...
Starting server on 0.0.0.0:3000 ...
** Ruby version is not up-to-date; loading cgi_multipart_eof_fix
Starting server on 0.0.0.0:3001 ...
** Ruby version is not up-to-date; loading cgi_multipart_eof_fix

Optimize Thin

Let’s check our memory footprint:

Thin with three servers

Yikes! Our memory usage is close to our 256MB limit and we are starting to page in/out of the swap space. Looks like three Thin servers is too much for our 256 MB slice. Let’s fix this. Stop the Thin cluster:

sudo /etc/init.d/thin stop

Edit the Thin configuration file to deploy only two servers:

sudo nano /etc/thin/warehouse.yml

Change the servers option From:

servers: 3

To:

servers: 2

Restart the Thin cluster:

sudo /etc/init.d/thin start

Note: Be sure to stop thin BEFORE changing the amount of servers or you will end up with stray servers that don’t get shut down.

Check memory usage again:

Thin with two servers

Perfect, memory usage is now under 200 MB. Now, while it seems that the swap is still in use, it is actually the remains of what got paged there from our overload. It is not currently being paged in and out of. This is OK!

Configure Nginx

Proxy requests to Thin

Open up the virual hosts configuration file:

sudo nano /etc/nginx/conf.d/virtual.conf

and change the file contents to:

upstream domain1 {
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
    }

server {
    listen   80;
    server_name code.example.com;

    access_log /home/demo/public_html/warehouse/log/access.log;
    error_log /home/demo/public_html/warehouse/log/error.log;

    root   /home/demo/public_html/warehouse/public/;
    index  index.html;

    location / {
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect false;

        if (-f $request_filename/index.html) {
            rewrite (.*) $1/index.html break;
        }

        if (-f $request_filename.html) {
            rewrite (.*) $1.html break;
        }

        if (!-f $request_filename) {
            proxy_pass http://domain1;
            break;
        }
    }
}

This setup proxies all non-static requests (E.g. files that don’t exist on the filesystem in the public folder) to our Thin cluster.

Restart Nginx to apply the changes. I’ve heard that there’s some issues with the restart command so it doesn’t hurt to just start and stop the server:

sudo /etc/init.d/nginx stop
...
sudo /etc/init.d/nginx start

Now cross your fingers and navigate to your server to view the Warehouse app:

http://code.example.com/

Nice!

Believe it or not, we now have our Subversion system up and running. We have svnserve listening in the background to check our code in/out while a web-facing Warehouse allows us to browse our code in our favorite Browser. Warehouse is running off a MySQL database backend and powered by a small Thin cluster in a proxy setup with Nginx. All of this on a small 256 MB slice which costs about $10 a month.

Alright, well let’s not get over excited. There’s still plenty of work to do. We can’t actually do much with Warehouse yet! In the next article, we will configure Warehouse to actually interface with our SVN repositories for web access and even user management and access control.

Tags: , , ,


  • ¬†Your imformation is so good. It efforted more for my job. Thank for your help! Hope you post other imformation soon. See you again!

  • TJ

    Thanks for the quick reply...the problem was on my side...the page that was reporting the error was a different page than you were editing above...looks like a random number 4 got inserted into the page...I am able to restart the nginx service now and get a 502 Bad Gateway error when i browse to the site...but that's alot better than the service not starting !!! Looks like i just needed to step away from this for a while !!! Appreciate the reply!

  • TJ

    I have downloaded and installed Warehouse and am now at the step: Configure Nginx / Proxy requests to Thin....I am stuck here...I have taken the file "/etc/nginx/conf.d/virtual.conf" and updated it with the content that you provided (changing "demo" to "tboykin") but when i try to restart nginx, i get an error:

    Starting nginx: 2010/01/22 11:10:53 [emerg] 4436#0: unknown directive "4#######################################################################" in /etc/nginx/nginx.conf:18

blog comments powered by Disqus