From Hackerspace Brussels
Jump to: navigation, search


[edit] Project Nexus

[edit] Goal

"Configure a dedicated server so that it can host several virtual machines connected on the same virtual network, routed on the IPv6 Internet."


  • IPs are assigned by the Nexus with DHCPv4 and DHCPv6
  • Adding a new machine should be easy

[edit] Why

The hackerspace needs a new server infrastructure, and we decided to host it on a dedicated server.

[edit] Steps

[edit] Notes

  • In this document, we will use the demonstration IPv6 network 2001:db8:1:2::/64.
  • Our dom0 has the public IPv6 2001:db8:1:2::1, and our domUs will use the network 2001:db8:1:2:100::/72.
  • Similarly, the dom0 has the public IPv4 address, and our domUs will use the private network range
  • The dom0 will have a LVM virtual group nexus with enough space on it to create new logical volumes on it.

These volumes will be used as the domUs' hard drives.

[edit] Installing Linux

This won't be described here. However, the following steps were executed on Ubuntu quantal (12.10).

Note that as it was automatically installed by OVH on a Kimsufi server, the custom OVH kernel does not support Xen. It had to be changed:

aptitude install linux-server

[edit] Installing Xen

Installing Xen on Debian or Ubuntu is easy with the package manager:

aptitude install xen-hypervisor xen-tools xen-utils

However, the system keeps booting on the first entry, which is not the one we want. However, it can be changed:

mv /etc/grub.d/{06,23}_OVHkernel
mv /etc/grub.d/{10,25}_linux

Reboot to use the correct entry:


Make sure Xen is functional:

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 15852     4     r-----      7.6

[edit] Prepare the network

We are going to make a virtual network bridged between all the virtual machines (domUs), but not bridged to the external network, so that the hosting machine (dom0) will route and firewall the traffic.

Nexus Network.svg

[edit] Internal bridge

To create the xenbr0 bridge, we can't use eth0, the unique network interface of the server. We must create a virtual network interface. That's where the kernel module dummy can help us.

modprobe dummy
echo dummy >> /etc/modules

This will have created a virtual network interface named dummy0. We will use it as a basis for a bridge for all our domUs.

cat <<EOF >> /etc/network/interfaces
# Xen Backend
auto xenbr0
iface xenbr0 inet static
    bridge_ports dummy0
iface xenbr0 inet6 static
    address 2001:db8:1:2:100::1
    netmask 72
ifup xenbr0

[edit] IP forwarding

Our dom0 will act as a router in our configuration. Therefore, we have to configure it for packet forwarding on both IPv4 and IPv6.

cat <<EOF >> /etc/sysctl.conf
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.proxy_ndp=1

We also need to tell iptables to perform network address translation (NAT) for our private network:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

More iptables configuration won't be discussed in this document. There is plenty documentation everywhere on the Internet. However, here is a nice configuration to restore the rules at boot time:

cat <<EOF > /etc/network/if-pre-up.d/iptablesload
iptables-restore < /etc/iptables.rules
exit 0
cat <<EOF > /etc/network/if-post-down.d/iptablessave
iptables-save -c > /etc/iptables.rules
if [ -f /etc/iptables.downrules ]; then
   iptables-restore < /etc/iptables.downrules
exit 0
chmod +x /etc/network/if-pre-up.d/iptablesload /etc/network/if-post-down.d/iptablessave

[edit] DHCP and router advertisement

As we (obviously) want automatic network configuration on our domUs, we need a DHCP server. Additionnally, as IPv6 DHCP doesn't include router information, we need a router advertisement daemon like radvd.

  • Install DHCP and radvd daemons:
aptitude install isc-dhcp-server radvd
  • Configure DHCP for IPv4:
cat <<EOF > /etc/dhcp/dhcpd.conf 
ddns-update-style none;
option domain-name "";
option domain-name-servers,;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;
subnet netmask {
  option routers;
EOF is reserved for host configuration, based on MAC address.

  • Configure DHCP for IPv6:
cat <<EOF > /etc/dhcp/dhcpd6.conf 
ddns-update-style none;
option domain-name "";
option domain-name-servers,;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;
subnet6 2001:db8:1:2:100::/72 {
  range6 2001:db8:1:2:180::/73;

2001:db8:1:2:180::/73 is reserved for host configuration, based on MAC address.

  • Tell DHCP to listen only on xenbr0:
patch -p0 <<EOF
--- /etc/default/isc-dhcp-server	2013-01-18 01:47:25.000000000 +0100
+++ /etc/default/isc-dhcp-server	2013-01-18 01:54:19.000000000 +0100
@@ -18,4 +18,4 @@
 # On what interfaces should the DHCP server (dhcpd) serve DHCP requests?
 #	Separate multiple interfaces with spaces, e.g. "eth0 eth1".
  • Configure radvd:
cat <<EOF > /etc/radvd.conf
interface xenbr0
    AdvSendAdvert on;
    prefix 2001:db8:1:2:100::/72
        AdvOnLink on;
        AdvAutonomous on;

This will send router advertisements on interface xenbr0, telling that the network prefix is 2001:db8:1:2:100::/72 and that the router is 2001:db8:1:2:100::1 (as this is the IP of xenbr0).

  • At last, restart the services to use this configuration:
service radvd restart
service isc-dhcp-server restart
service isc-dhcp-server6 restart

[edit] Neighbor solicitations

IPv6 routing is done by answering to neighbor solicitations. Our router will have to answer our ISP's requests for known IPs.

The problem is that, in Linux, the list of IPs for which the router will answer is statically configured.

The project npd6 (Neighbor Proxy Daemon IPv6) will answer to *all* the requests for a given IPv6 range.

  • Add the npd6 launchpad:
add-apt-repository ppa:sgroarke/npd6
  • Install it
aptitude update
aptitude install npd6
  • Configure it:
cat <<EOF > /etc/npd6.conf
listtype = none
listlogging = off
collectTargets = 100
linkOption = false
ignoreLocal = true
routerNA = true
maxHops = 255
pollErrorLimit = 20

Here, we will answer 'yes' for *all* IPv6 addresses in the range 2001:db8:1:2::/64

  • Start it:
service npd6 start
  • Add it to the system startup
update-rc.d npd6 defaults

[edit] Create a domU

[edit] Create a LVM virtual disk

If there is no LVM physical volume yet, you can create it.

  • Create the physical volume:
pvcreate /dev/sda4
  • Create the virtual group named nexus:
vgcreate nexus /dev/sda4

Now you can create a new logical volume:

lvcreate -L 4G -n probe0 nexus

[edit] Get the boot files

Ubuntu provides light netboot images for installation. We are going to use them to install our dom0.

Change the mirror and the version if needed ;)

mkdir -p /var/lib/xen/images/ubuntu-netboot
cd /var/lib/xen/images/ubuntu-netboot

[edit] Create the domU

Now we can create a configuration file for the domU probe0:

cat <<EOF >/etc/xen/probe0.cfg
name = "probe0"
memory = 256
disk = ['phy:/dev/nexus/probe0,xvda,w']
vif = ['mac=0a:00:00:00:00:01']
kernel = "/var/lib/xen/images/ubuntu-netboot/vmlinuz"
ramdisk = "/var/lib/xen/images/ubuntu-netboot/initrd.gz"
extra = "debian-installer/exit/always_halt=true -- console=hvc0"

Boot it to install the system:

xm create /etc/xen/probe0.cfg -c

Once the installation complete, change the configuration file:

patch -p 0 <<EOF
--- /etc/xen/probe0.cfg
+++ /etc/xen/probe0.cfg
@@ -3,5 +3,6 @@
 disk = ['phy:/dev/nexus/probe0,xvda,w']
 vif = ['mac=0a:00:00:00:00:01']
-kernel = "/var/lib/xen/images/ubuntu-netboot/vmlinuz"
-ramdisk = "/var/lib/xen/images/ubuntu-netboot/initrd.gz"
-extra = "debian-installer/exit/always_halt=true -- console=hvc0"
+#kernel = "/var/lib/xen/images/ubuntu-netboot/vmlinuz"
+#ramdisk = "/var/lib/xen/images/ubuntu-netboot/initrd.gz"
+#extra = "debian-installer/exit/always_halt=true -- console=hvc0"
+bootloader = "pygrub"

And boot this domU:

xm create /etc/xen/vmtest.cfg -c

The last thing to do is to tell the domU to ask for an IPv6 address:

echo iface eth0 inet6 dhcp >> /etc/network/interfaces