Creating a virtualized fully-distributed Hadoop cluster using Linux Containers

TL;DR Why and how I created a working 9-node Hadoop Cluster on my  laptop

In this post I’ll cover why I wanted to have a decent multi-node Hadoop cluster on my laptop, why I chose not to use virtualbox/VMware player, what is LXC (Linux Containers) and how did I set it up. The last part is a bit specific to my desktop O/S (Ubuntu 13.10).

Why install a fully-distributed Hadoop cluster on my laptop?

Hadoop has a “laptop mode” called pseudo-distributed mode. In that mode, you run a single copy of each service (for example, a single HDFS namenode and a single HDFS datanode), all listening under localhost.

This feature is rather useful for basic development and testing – if all you want is write some code that uses the services and check if it runs correctly (or at all). However, if your focus is the platform itself – an admin perspective, than that mode is quite limited. Trying out many of Hadoop’s feature requires a real cluster – for example replication, failure testing, rack awareness, HA (of every service types) etc.

Why VirtualBox or VMware Player are not a good fit?

When you want to run several / many virtual machines, the overhead of these desktop virtualization sotware is big. Specifically, as they run a full-blown guest O/S inside them, they:

  1. Consume plenty of RAM per guest – always a premium on laptops.
  2. Consume plenty of disk space – which is limited, especially on SSD.
  3. Take quite a while to start / stop

To run many guests on a laptop, a more lightweight infrastructure is needed.

What is LXC (Linux Containers)

LXC is a lightweight virtualization solution for Linux, sometimes called “chroot on steroids”. It leverages a set of Linux kernel features (control groups, kernel namespaces etc) to provide isolation between groups of processes running on the host. In other words, in most cases the containers (virtual machines) are using the host kernel and operating system, but are isolated as if they are running on their own environent (including their own IP addresses etc).

In practical terms, I found that the LXC containers start in a sub-second. They have no visible memory overhead  – for example, if a container runs the namenode, only the namenode consumes RAM, based on its java settings. They also consume very little disk space, and their file system is just stored directly in the host file system (under /var/lib/lxc/container_name/rootfs), which is very useful. You don’t need to statically decide how much RAM or virtual cores to allocate each (though you could) – they consume just what they need. Actually, while the guests are isolated from each other, you can see all their processes together in the host using the ps. command as root..

Some other notes – LXC is already used by Apache Mesos and Dockers, amongst others. A “1.0″ release is targeted next month (February). If you are using RHEL/CentOS, LXC is provided as a “technical preview” on RHEL 6 (likely best to use a current update).

If you want a detailed introduction, Stéphane Graber which is one of the LXC maintainer started an excellent ten-post series on LXC a couple of weeks ago.

So, how to set it up?

The following are some of my notes. I only tried it on my laptop, with Ubuntu 13.10 64-bit as a guest, so if you have a different O/S flavor, expect some changes.

All commands below are as root.
start by installing LXC:

apt-get install lxc

The installation also creates a default virtual network bridge device for all containers (lxcbr0). Next, create your first Ubuntu container called test1 – it will take a few minutes as some O/S packages will be downloaded (but they will be cached, so the next containers will be created in a few seconds):

lxc-create -t ubuntu -n test1

Start the container with:

lxc-start -d -n test1

List all your containers (and their IP addreses) with:

lxc-ls --fancy

and open a shell on it with:

lxc-attach -n test1

stop the container running on your host:

lxc-stop -n test1

You can also clone a container (lxc-clone) or freeze/unfreeze a running one (lxc-freeze / lxc-unfreeze). LXC also has a web GUI with some nice visualizations (though some of it was broken for me), install it with:

wget http://lxc-webpanel.github.io/tools/install.sh -O - | bash

and access it with http://localhost:5000 (default user/password is admin/admin):

lxc-web-panel

Now, some specific details for an Hadoop container

1. Install a JVM inside the container. The default LXC Ubuntu container has a minimal set of packages . At first, I installed openjdk in it:

apt-get install openjdk-7-jdk

However, I decided to switched to Oracle JDK as that is what’s typically being used. That required some extra steps (the first step adds the “add-apt-repository” command, the others set a java7 repository to ease installation and updates):

apt-get install software-properties-common
add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java7-installer

2. Setting the network

By default, LXC uses DHCP to provide dynamic IP addresses to containers. As Hadoop is very sensitive to IP addresses (it occasionally uses them instead of host names), I decided that I want static IPs for my Hadoop containers.

In Ubuntu, the lxc bridge config file is /etc/default/lxc-net – I uncommented the following line:

LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf

and created /etc/lxc/dnsmasq.conf with the following contents:

dhcp-host=hadoop11,10.0.1.111
dhcp-host=hadoop12,10.0.1.112
dhcp-host=hadoop13,10.0.1.113
dhcp-host=hadoop14,10.0.1.114
dhcp-host=hadoop15,10.0.1.115
dhcp-host=hadoop16,10.0.1.116
dhcp-host=hadoop17,10.0.1.117
dhcp-host=hadoop18,10.0.1.118
dhcp-host=hadoop19,10.0.1.119
dhcp-host=hadoop21,10.0.1.121
dhcp-host=hadoop22,10.0.1.122
dhcp-host=hadoop23,10.0.1.123
dhcp-host=hadoop24,10.0.1.124
dhcp-host=hadoop25,10.0.1.125
dhcp-host=hadoop26,10.0.1.126
dhcp-host=hadoop27,10.0.1.127
dhcp-host=hadoop28,10.0.1.128
dhcp-host=hadoop29,10.0.1.129

That should cover more containers than I’ll ever need… My naming convention is that the host name is hadoop[rack number][node number] and the IP is 10.0.1.1[rack number][hostname]. For example, host hadoop13 is located at rack 1 node 3 with IP address of 10.0.113. This way it is easy to figure out what’s going on even if the logs only have IP.

I’ve also added all of the potential containers to my /etc/hosts and copied them to hadoop11, which I use as a template to clone the rest of the nodes:

10.0.1.111	hadoop11
10.0.1.112      hadoop12
10.0.1.113      hadoop13
10.0.1.114      hadoop14
10.0.1.115      hadoop15
10.0.1.116      hadoop16
10.0.1.117      hadoop17
10.0.1.118      hadoop18
10.0.1.119      hadoop19
10.0.1.121      hadoop21
10.0.1.122      hadoop22
10.0.1.123      hadoop23
10.0.1.124      hadoop24
10.0.1.125      hadoop25
10.0.1.126      hadoop26
10.0.1.127      hadoop27
10.0.1.128      hadoop28
10.0.1.129      hadoop29

I ran into one  piece of ugliness regarding DHCP. I use hadoop11 container as a template, so I as I develop my environment I have destroyed the rest of the containers and cloned them again a few times. At that point, they got a new dynamic IP – it seems the DHCP server remembered that the static IP that I wanted was already allocated. Eventually, I added this to the end of  my “destroy all nodes” script to solve that:

service lxc-net stop
rm /var/lib/misc/dnsmasq.lxcbr0.leases 
service lxc-net start

If you run into networking issues – I recommend checking the Ubuntu-specific docs, as lots of what you find on the internet is scary network hacks that are likely not needed in Ubuntu 13.10 (and didn’t work for me).

Another setup thing – if you run on BTRFS (a relatively new copy-on-write filesystem), the lxc-clone should detect it and leverage it. It doesn’t seem to work for me, but instead copying files directly into /var/lib/lxc/container_name/rootfs using cp –reflink works as expected (you can verify disk space after copying by waiting a few seconds and then issuing btrfs file df /var/lib/lxc/ )

3. Setting Hadoop

This is pretty standard… You should just set up ssh keys, maybe create some O/S users for Hadoop, download Hadoop bits from Apache or a vendor and get going… Personally, I used Apache Hadoop (1.2.1 and 2.2.0) – I have set things up on a single node and then clone it to create a cluster.

In addition, to ease my tests, I’ve been scripting the startup, shutdown and full reset of the environment based on a config file that I created. For example, here is my current config file for Apache Hadoop 1.2.1 (the scripting works nicely but is still evolving – ping me if you are curious):

hadoop11 namenode zookeeper
hadoop12 datanode tasktracker
hadoop13 datanode tasktracker
hadoop14 datanode tasktracker
hadoop21 jobtracker zookeeper
hadoop22 datanode tasktracker
hadoop23 datanode tasktracker
hadoop24 datanode tasktracker
hadoop29 secondarynamenode hiveserver2 zookeeper
Obviously, you can set a smaller cluster, based on your cores / RAM / disk / workload etc.
Summary
If you’re working with Hadoop, having a full blown cluster on your laptop is just awesome… Stay tuned for some examples in the coming weeks :)

23 thoughts on “Creating a virtualized fully-distributed Hadoop cluster using Linux Containers

  1. Pingback: Exploring The Hadoop Network Topology | Big Data, Small Font

  2. Pingback: Exploring HDFS Block Placement Policy | Big Data, Small Font

  3. Pingback: Exploring the HDFS Default Value Behaviour | Big Data, Small Font

  4. Total noob here, but wouldn’t the entry ‘LXC_DOMAIN=”lxc”‘ in lxc.net allow you to use names like this: hadoop11.lxc and you don’t have to worry about making static ip addresses? I am looking to setting it up like that…but not sure how to do with dnsmasq.conf file.

  5. Thanks for your great posts. These are so helpful for me to study pattern of replication.
    Anyway i got a problem, which is a discordance between lxc-net i set and address derived from fsck(-racks option). I made 3 racks, and each rack has 2 nodes. My naming pattern was same as yours.

    - rack01
    10.0.3.111 : namenode
    10.0.3.112 : datanode, secondary namenode
    - rack02
    10.0.3.121 : datanode
    10.0.3.122 : datanode
    - rack03
    10.0.3.131 : datanode
    10.0.3.132 : datanode

    But i set these address on GUI LXC Web Panel not using CUI.
    Of course, i wrote correctly /etc/hosts and Hadoop’s conf/masters, conf/slaves.
    LXC/Hadoop works very well. I didn’t get any error message related to IP address.

    But when i wrote ‘bin/hadoop fsck /test -files -blocks -racks’, i got unexpected IP addresses.
    [/d_rack01/10.0.3.133:50010, /d_rack01/10.0.3.114:50010, /d_rack01/10.0.3.124:50010]
    (d_rack01 is default rack value. i used your topology.script.sh.) So, i checked topology.log.

    topology.log
    Fri Feb 14 10:27:03 UTC 2014 input: 10.0.3.114
    Fri Feb 14 10:27:17 UTC 2014 input: 10.0.3.124
    Fri Feb 14 10:27:18 UTC 2014 input: 10.0.3.133
    Fri Feb 14 10:27:26 UTC 2014 input: 10.0.3.108
    Fri Feb 14 10:27:27 UTC 2014 input: 10.0.3.110

    It should’ve been 10.0.3.111, 10.0.3.121 …. 10.0.3.131, 10.0.3.132.
    It’s really make me become a yellow-faced elephant. Could you tell me details that you set? I want to know how you did ’2. Setting the network’. I added ‘LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf’ to ‘/etc/default/lxc’ file because I don’t have ‘/etc/default/lxc-net’ file. And I made ‘/etc/lxc/dnsmasq.conf’. But It still doesn’t work correctly.

    Ubuntu 12.04 LTS / Hadoop 1.2.1 / LXC 0.7.5

    • haneul@haneul:~$ cat /var/lib/misc/dnsmasq.leases
      1392609217 00:16:3e:44:dc:2a 10.0.3.108 hadoop32 *
      1392609215 00:16:3e:b6:de:38 10.0.3.114 hadoop31 *
      1392609212 00:16:3e:6c:b4:6e 10.0.3.133 hadoop22 *
      1392609206 00:16:3e:c7:3b:a7 10.0.3.124 hadoop21 *
      1392609198 00:16:3e:5b:0a:ab 10.0.3.110 hadoop12 *
      1392609191 00:16:3e:8d:f7:ec 10.0.3.141 hadoop11 *

      It is the problem that is caused by allocating static ip to host.
      I think that allocating IP using LXC Web Panel is not applied correctly.
      So, how can i allocate ip addresses correctly. What’s wrong with my way :(

      • Hi,
        I just saw your messages.
        It seems that your static IP definitions don’t work, that’s why you see that DNS has leased dynamic IPs to your servers.
        I didn’t try to assign static IP address using the GUI, so I can’t comment on it.
        I suggest you try to follow the instructions in my post:
        1. Create a new file called /etc/lxc/dnsmasq.conf and put inside your definition, following my example from the post:
        Syntax is dhcp-host=hostname,static_IP_address
        Like:
        dhcp-host=hadoop11,10.0.1.111
        2. Point LXC_DHCP_CONFILE to that file.
        3. Stop networking, delete the lease file (dnsmasq.leases), start the networking , start a container and see if it working… Repeat until you fixed it…

        Also, I see that you are running 12.04. I haven’t tried it, but it may be possible that the version included in it is less functional… Check the 12.04-specific documentation here:
        https://help.ubuntu.com/12.04/serverguide/lxc.html

  6. Pingback: Exploring The Hadoop Network Topology | Big Data Analytics News

  7. Hi Ofir ;-) nice to see you in fantastic technical shape again! We miss you definitely.
    LXC looks like Solaris legacy.

    • Thanks a lot Marek! I hope HAWK is starting to get some traction in Europe.
      LXC (and BTRFS) are all about Linux catching up… But regardless, it is very useful and it actually works :)

  8. I got stuck on static ip assignment to the containers using LXC_DHCP_CONFILE flag. I am using Ubuntu 13.10 and installed lxc via apt-get.

    Fist of all there in my configuration I did not have the lxc bridge config file “/etc/default/lxc-net” as mentioned in the original article. Rather, I had “/etc/default/lxc” instead. Setting the LXC_DHCP_CONFILE=”/etc/lxc/dnsmasq.conf” in this file did not help either. I traced the problem down to the /etc/init/lxc-net.conf file where the dnsmasq command is actially issued. On inspecting the /etc/init/lxc-net.conf file, I found no references to the “LXC_DHCP_CONFILE” variable.

    I had to update the following command
    dnsmasq -u lxc-dnsmasq –strict-order –bind-interfaces –pid-file=${varrun}/dnsmasq.pid –conf-file –listen-address ${LXC_ADDR} –dhcp-range

    to include “LXC_DHCP_CONFILE” as follows

    dnsmasq -u lxc-dnsmasq –strict-order –bind-interfaces –pid-file=${varrun}/dnsmasq.pid –conf-file=${LXC_DHCP_CONFILE} –listen-address ${LXC_ADDR} –dhcp-range

    I than restarted the service sudo nano /etc/init/lxc-net.conf and sure enough the container picked up the ip address I had specified in /etc/lxc/dnsmasq.conf file.

  9. Interesting, you are directly using LXC instead of docker. Would you agree that using docker would simplify the steps creating those 29 containers?

    • I would likely try to move to docker sooner or later… At the time I thought about it, it seems to me docker (was version 0.6) was still not mature enough. Anyway, I thought trying lxc first would be useful (gain experience with the infrastructure).
      In my case, I found LXC to be very simple to work with – I installed one container once, then duplicated it with a script to as many container as I want (actually not 29, typically just nine). So, I don’t think Docker will simplify much, if any, but still, can’t really know until I giive it a serious try one day…

      • Thanks for the reply, Ofir. I’m interested to know how you set up the single Hadoop node and then clone it to create a cluster, because both LXC and Hadoop are new to me. Care for a follow up blog? Thx.

        • Thanks for the followup and the link (below) for your Dockers blog.
          I think I’ll have time to follow up in another post only in a month (busy days…), likely for Ubuntu 14.04.
          Now, if you just want to get started and learn the basics, you should definitely download a pre-built single-instance virtual machine and run the tutorials. Try this excellent one – http://hortonworks.com/products/hortonworks-sandbox/
          If you then want to get your hands dirty, then my multi-node version makes sense…. Apache Hadoop is basically just a bunch of Java applications. So, you just need an empty container, install Java, likely create one or a few O/S users, download and unzip the hadoop project (example: hadoop-2.3.0.tar.gz ), likely tweak some config files and you are ready to clone (a basic system)… Dark magic is only inside the config files… I think I covered it all, except the dak magic :)
          Ofir

  10. can someone clarify my below doubts about lxc container.

    1) Is it possible to install jdk at host level and use the same jdk in all containers?

    2) Is it possible to customize the lxc ubuntu template to contain by default java or any common software which is required for all containers (like chef, git..etc)?

    • Hey Raj,
      Don’t think so, its chroot environment..

      But sure you can have that freedom based on what apt sources you are using since you always use custom course into the template in-fact, write you own handy template instead use the default one, this is plenty straight forward and plane shell script..

  11. Pingback: Reduce resource consumption and clone in seconds your oracle virtual environment on your laptop using linux containers and btrfs | bdt's oracle blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s