rui lopes notebook

Using Packer, Vagrant and Boxstarter to create Windows environments

Sun, 24 May 2015 00:00:00 +0000

Testing your application in different environments can be hard to do manually, as its time consuming, error prone, and not easily reproducible. This article will show you a way on how to automatically create Windows environments, where you can test your application.

Tools

You will use the following tools:

Packer and packer-windows to create an up-to-date Windows base image from an ISO file.
Vagrant to launch and configure a base image tailored to a specific scenario.
Boxstarter to easily install software from Chocolatey packages.
VirtualBox to run virtual machines.

Install

You will need to run several commands. These must be run on a bash shell, which I’ll assume that you have installed following the instructions from Sane shell environment on Windows.

Make sure you have previously installed Packer, Vagrant and VirtualBox and have them available on your PATH, e.g. with:

export PATH="$PATH:/c/Development/packer"
export PATH="$PATH:/c/Development/Vagrant/bin"
export PATH="$PATH:/c/Program Files/Oracle/VirtualBox"

Windows base image

Packer is a tool to automatically create a machine image from a template file.

packer-windows is a repository with pre-configured template files for installing different Windows versions.

We will use them to drive the installation of Windows 2012 R2 into a virtual machine image. This image will have the bare minimum needed to be later used by Vagrant.

This base image will be automatically configured with:

SSH server.
WinRM.
vagrant user account with a vagrant password.
Latest Windows updates.
VirtualBox guest additions.
60 GB disk.

Start by getting the Windows template files:

git clone https://github.com/joefitzgerald/packer-windows.git
cd packer-windows

NB These templates are for the trial versions of Windows, namely Windows 2012 R2 Standard, but you can also install from your own Windows ISO file.

In order to makes things easier for us we will tweak a couple of files.

We want to see how the installation is going on, for that we need to change the template file to start the virtual machine in non-headless mode. For that, edit the windows_2012_r2.json template file:

vim windows_2012_r2.json
...
{
  "type": "virtualbox-iso",
  ...
  "headless": false,
...

By default, the template is configured to install the latest Windows updates, but the default timeout might now be enough for the updates to finish in time, so increase the ssh_timeout from the default 2h to 8h:

vim windows_2012_r2.json
...
{
  "type": "virtualbox-iso",
...
  "ssh_wait_timeout": "8h",
...

Also, sometimes, the last provisioning step fails, so you also need to change the line that has rm -rf to:

{
  "type": "shell",
  "inline": [
    "rm -rf /tmp/* || true"
  ]
}

Also consider removing the ./scripts/compact.bat (it takes ages to run) and the ./scripts/chef.bat (you might not need it) lines.

Finally, build the VirtualBox based image:

time packer build -only virtualbox-iso windows_2012_r2.json

Now be patient! It will download the Windows ISO file and install the latest Windows Updates. These steps take a lot of time to complete (about 2 hours in my environment). At the end you should have a .box file (about 1.8 GB), ready to be used by Vagrant.

Add the generated box file to vagrant:

vagrant box add --name windows_2012_r2 windows_2012_r2_virtualbox.box

NB You can distribute the box file to other system, or you can delete it (its no longer needed once added to vagrant)

We are now almost ready to launch a Windows virtual machine with Vagrant, for that we need to create a Vagrantfile:

mkdir testing && cd testing
cat<<"EOF">Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.require_version ">= 1.6.2"

Vagrant.configure("2") do |config|
    config.vm.define "testing"
    config.vm.box = "windows_2012_r2"

    config.vm.provider :virtualbox do |v, override|
        v.gui = true
        v.customize ["modifyvm", :id, "--memory", 2048]
        v.customize ["modifyvm", :id, "--cpus", 2]
    end
end
EOF

And launch it:

time vagrant up

NB The initial run will take about 4m to boot into Windows. Later runs take about 40s. But YMMV.

NB To troubleshoot you need pass the --debug flag to vagrant up.

Once Windows boots, try it a bit to check whether things are working fine, and then stop the virtual machine with:

vagrant halt

And destroy it:

vagrant destroy

You are now ready to customize a vagrant environment to fit your needs. For that we will modify the Vagrantfile to use a provision script.

Provisioning

Customizing a vagrant environment is a mater of creating one or more provision scripts. Lets start simple, with one embedded within the Vagrantfile, it will just change the keyboard layout and timezone:

cat<<"EOF">Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.require_version ">= 1.6.2"

# NB this is a PowerShell script that is run as Administrator.
$root_provision_script = <<'ROOT_PROVISION_SCRIPT'
# set keyboard layout.
# NB you can get the name from the list:
#      [System.Globalization.CultureInfo]::GetCultures('InstalledWin32Cultures') | out-gridview
Set-WinUserLanguageList pt-PT -Force

# set the date format, number format, etc.
Set-Culture pt-PT

# set the timezone.
# tzutil /l lists all available timezone ids
& $env:windir\system32\tzutil /s "GMT Standard Time"
ROOT_PROVISION_SCRIPT

Vagrant.configure("2") do |config|
    config.vm.define "testing"
    config.vm.box = "windows_2012_r2"

    config.vm.provider :virtualbox do |v, override|
        v.gui = true
        v.customize ["modifyvm", :id, "--memory", 2048]
        v.customize ["modifyvm", :id, "--cpus", 2]
    end

    config.vm.provision "shell", inline: $root_provision_script
end
EOF

And launch it:

vagrant up

If its running fine, stop the virtual machine with:

vagrant halt

Now its time to install applications. This is handled by Boxstarter. Lets modify the provision script to install Boxstarter and some applications:

vim Vagrantfile
...
$root_provision_script = <<'ROOT_PROVISION_SCRIPT'
...
# install Boxstarter.
# NB Do NOT install chocolatey before Boxstarter. If you do, strange things
#    will happen...
# TODO this should all be abstracted in a "boxstarter" provisioner.
$boxstarterSetupPath = "$env:TEMP\Boxstarter-setup"
$boxstarterSetupZipPath = $boxstarterSetupPath + ".zip"
Invoke-WebRequest http://boxstarter.org/downloads/Boxstarter.2.4.209.zip -OutFile $boxstarterSetupZipPath
 [System.Reflection.Assembly]::LoadWithPartialName("System.IO.Compression.FileSystem")
 [System.IO.Compression.ZipFile]::ExtractToDirectory($boxstarterSetupZipPath, $boxstarterSetupPath)
& $boxstarterSetupPath\setup.bat -Force

echo NBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNB
echo NB
echo 'NB Boxstarter might need to reboot the machine, in that case, vagrant will'
echo 'NB fail, but that is expected. you need to monitor the install yourself to'
echo 'NB known when its done...'
echo NB
echo NBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNBNB

# NB if any of the choco packages need to access the desktop, you need to force
#    a reboot at start of the boxstarter script. when you do that Boxstarter
#    will run the script again, but in a auto-logon session. the easiest way to
#    do that, is to install google chrome.
$env:PSModulePath = "$([System.Environment]::GetEnvironmentVariable('PSModulePath', 'User'));$([System.Environment]::GetEnvironmentVariable('PSModulePath', 'Machine'))"
cp C:\vagrant\setup.ps1 $env:TEMP
Import-Module Boxstarter.Chocolatey
$credential = New-Object System.Management.Automation.PSCredential("vagrant", (ConvertTo-SecureString "vagrant" -AsPlainText -Force))
Install-BoxstarterPackage $env:TEMP\setup.ps1 -Credential $credential
ROOT_PROVISION_SCRIPT
...

Create the script to automatically install some software:

cat<<"EOF">setup.ps1
# NB this file has to be idempotent. it will be run several times if the computer needs to be restarted.
#    when that happens, Boxstarter schedules this script to run again with an auto-logon.
# NB always remember to pass -y to choco install!
# NB already installed packages will refuse to install again; so we are safe to run this entire script again.

# NB make sure this is the first software you install, because, as a side
#    effect, this will trigger a reboot, which in turn, will fix the vagrant bug
#    that prevents the machine from rebooting after setting the hostname.
choco install -y google-chrome-x64

# Enable Show Window Contents While Dragging
reg ADD "HKCU\Control Panel\Desktop" /v DragFullWindows /t REG_SZ /d 1 /f
taskkill /IM explorer.exe /F ; explorer.exe

# Pin an application to the task bar.
Install-ChocolateyPinnedTaskBarItem "$env:windir\system32\services.msc"

choco install -y notepad2

choco install -y nodejs.install

choco install -y visualstudiocode
EOF

And launch it:

vagrant up

You will notice that vagrant up seems to fail with something like:

The following WinRM command responded with a non-zero exit status. Vagrant assumes that this means the command failed!

powershell -ExecutionPolicy Bypass -OutputFormat Text -file c:\tmp\vagrant-shell.ps1

But that’s expected, you need to monitor the install yourself to known when its done. Doing that automatically would be nice, but I’ll leave that as an exercise for the reader (hint: You could create a new boxstarter Vagrant provisioner).

As a side note, you could also manually provision the machine and later export it as a new box file, e.g.:

vagrant package --base testing --output testing.box

Install from your own Windows ISO file

The templates contained in the windows-packer repository are configured to install the trial versions of windows, but you can also install from a specific ISO file. For that, you need to set the ISO path and its MD5 checksum, e.g.:

vim windows_2012_r2.json
...
  "iso_url": "d:/Images/windows_2012_r2.iso",
  "iso_checksum_type": "md5",
  "iso_checksum": "458ff91f8abc21b75cb544744bf92e6a",
...

The iso_checksum value can be computed with md5sum:

md5sum d:/Images/windows_2012_r2.iso
458ff91f8abc21b75cb544744bf92e6a *d:/Images/windows_2012_r2.iso

If you don’t have a volume license ISO, you also need to set the Product Key. For that open the answer_files/2012_r2/Autounattend.xml file, search for ProductKey and follow the instructions.

The End

And thats it! You should now be able to create Windows environments to fit your needs.

You should also look into configuration management tools like Ansible, Chef, and Puppet. These might useful to actually install and configure your own application.

Testing the Elasticsearch cluster behavior under network failure

Sun, 15 Mar 2015 00:00:00 +0000

Elasticsearch is a distributed system that depends heavily on the network, as such, you need to known how it behaves under different failures scenarios. This post shows a way of mounting these scenarios with Linux containers.

You can simulate a couple of failure scenarios:

Node network loss
Packet delay and/or loss

This setup should also be generally useful when you want to known how Elasticsearch behaves. For example:

Expand or shrink the cluster
Node maintenance
Backup/Restore
Shard re-allocation
Near disk full behavior

The setup will be created with the help of: VirtualBox, Vagrant, CoreOS and Docker.

Preparation

Make sure you install VirtualBox, Vagrant, and Git. And have them available on your $PATH.

Check the tools versions.

VirtualBox:

VBoxManage --version

4.3.24r98716

Vagrant:

vagrant version

Installed Version: 1.7.2

Git:

git --version

git version 1.9.5.msysgit.0

CoreOS and Docker

CoreOS will be running inside a VirtualBox Virtual Machine. It will be used as our Docker host. Docker will be used to create a Linux container for each Elasticsearch node.

Install and start CoreOS:

git clone https://github.com/coreos/coreos-vagrant test-elasticsearch
cd test-elasticsearch
vagrant up

Open a shell:

vagrant ssh

Check the Docker version:

docker version

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.3.3
Git commit (client): a8a31ef-dirty
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.3.3
Git commit (server): a8a31ef-dirty

Check the Docker info:

docker info

Containers: 0
Images: 0
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 3.19.0
Operating System: CoreOS 618.0.0
CPUs: 1
Total Memory: 998 MiB
Name: core-01
ID: QLVN:NP7B:SZDN:D7PM:DWNF:364W:O4XA:3JLG:MNJ4:2NFS:L5YU:GUOO

Setup

We test the different scenarios in a three node Elasticsearch cluster.

Each node is run in a Docker container that is based on the dockerfile/elasticsearch image.

Docker manages the network between the nodes. It does so by creating a pair of veth (virtual Ethernet) interfaces for each container. These form a point-to-point link between the container the host. The host side will be connected to a bridge, which allows the communication between nodes:

+-- docker0 bridge --+        +-- node0 container --+
|                    |        |                     |  
|      +-----> $node0_veth <----> eth1 interface    |
|      |             |        |                     |
|      |             |        +---------------------+
|      |             |
|      |             |        +-- node1 container --+
|      |             |        |                     |
|      +-----> $node1_veth <----> eth1 interface    |
|      |             |        |                     |
|      |             |        +---------------------+
|      |             |
|      |             |        +-- node2 container --+
|      |             |        |                     |
|      +-----> $node2_veth <----> eth1 interface    |
|                    |        |                     |
+--------------------+        +---------------------+

NB This setup allows for multicast, so you don’t have to do anything more for having a working Elasticsearch cluster.

NB For more details read the Docker Network Configuration documentation.

The interface name of each container is saved in a environment variable, e.g., node0_veth. This will later let us easily fiddle with a node network interface.

Each container will get its data from a volume that is stored on the host. This will let us easily see the logs and index data.

Each container will also have the normal Elasticsearch 9200 port forwarded to the host, e.g. node 1 will be available at http://localhost:9201.

Before you can start the nodes you need the docker-native-inspect tool. Build it with:

curl -o docker-native-inspect.tar.bz2 https://bitbucket.org/rgl/docker-native-inspect/get/default.tar.bz2
mkdir docker-native-inspect
sudo mkdir -p /opt/bin
tar xf docker-native-inspect.tar.bz2 --strip-components=1 -C docker-native-inspect
docker build -t docker-native-inspect docker-native-inspect
docker run --rm docker-native-inspect tar czf - docker-native-inspect | tar vxzf - -C /opt/bin
docker-native-inspect -h

And start the Elasticsearch nodes with:

for i in 0 1 2; do
  mkdir -p node$i/{data,work,logs,plugins}
  docker run -d \
    --name=node$i \
    --hostname=node$i \
    -p 920$i:9200 \
    -v "$PWD/node$i:/data" \
    dockerfile/elasticsearch \
    /elasticsearch/bin/elasticsearch \
    -Des.node.name=node$i \
    -Des.path.data=/data/data \
    -Des.path.work=/data/work \
    -Des.path.logs=/data/logs \
    -Des.path.plugins=/data/plugins \
    -Des.logger.level=DEBUG
  declare "node${i}_veth=`sudo docker-native-inspect -format '{{.network_state.veth_host}}' state node$i`"
done

Install some plugins:

for i in 0 1 2; do
  docker exec -it node$i /elasticsearch/bin/plugin --install mobz/elasticsearch-head       # /_plugin/head
  docker exec -it node$i /elasticsearch/bin/plugin --install royrusso/elasticsearch-HQ     # /_plugin/HQ
  #docker exec -it node$i /elasticsearch/bin/plugin --install lmenezes/elasticsearch-kopf  # /_plugin/kopf
done

NB Access them at http://localhost:9200/_plugin/head or HQ.

If you need to access the node, e.g. to inspect something, you can use:

docker exec -it node0 bash

Create a basic hello index:

curl -XPUT http://localhost:9200/hello -d '
index:
    number_of_shards: 3
    number_of_replicas: 1
'

Index some documents:

for i in 1 2 3 4 5 6 7 8 9 10; do
  curl -XPUT http://localhost:9200/hello/message/$i -d "{\"message\":\"hello $i\"}"
done

To see how things are going, open a different shell, and tail the logs:

tail -f node*/logs/elasticsearch.log

Also open the head and HQ pages:

http://localhost:9200/_plugin/head
http://localhost:9200/_plugin/HQ

You are now ready to test the several scenarios!

Test

Node network loss

To simulate a network loss we simply bring the node container veth interface down:

sudo ip link set dev $node1_veth down

If you now try to refresh the cluster state in the head plugin, you should see it does not really work. After a while, the Elasticsearch logs should starting to display a lot of activity.

For example, at the master node (in my case this was node2), things like:

[2015-03-14 14:50:55,265][DEBUG][action.admin.cluster.node.stats] [node2] failed to execute on node [gDtJuVYCQ5qHiU-aSKnj7g]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [node1][inet[/10.1.0.12:9300]][cluster:monitor/nodes/stats[n]] request_id [289] timed out after [15002ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[2015-03-14 14:51:25,263][DEBUG][action.admin.cluster.node.stats] [node2] failed to execute on node [gDtJuVYCQ5qHiU-aSKnj7g]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [node1][inet[/10.1.0.12:9300]][cluster:monitor/nodes/stats[n]] request_id [326] timed out after [15000ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[2015-03-14 14:51:55,265][DEBUG][action.admin.cluster.node.stats] [node2] failed to execute on node [gDtJuVYCQ5qHiU-aSKnj7g]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [node1][inet[/10.1.0.12:9300]][cluster:monitor/nodes/stats[n]] request_id [363] timed out after [15001ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[2015-03-14 14:51:58,692][DEBUG][discovery.zen.fd         ] [node2] [node  ] failed to ping [[node1][gDtJuVYCQ5qHiU-aSKnj7g][node1][inet[/10.1.0.12:9300]]], tried [3] times, each with  maximum [30s] timeout

[2015-03-14 14:51:58,700][DEBUG][cluster.service          ] [node2] processing [zen-disco-node_failed([node1][gDtJuVYCQ5qHiU-aSKnj7g][node1][inet[/10.1.0.12:9300]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout]: execute

[2015-03-14 14:51:58,726][DEBUG][cluster.service          ] [node2] cluster state updated, version [11], source [zen-disco-node_failed([node1][gDtJuVYCQ5qHiU-aSKnj7g][node1][inet[/10.1.0.12:9300]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout]

[2015-03-14 14:51:58,732][INFO ][cluster.service          ] [node2] removed {[node1][gDtJuVYCQ5qHiU-aSKnj7g][node1][inet[/10.1.0.12:9300]],}, reason: zen-disco-node_failed([node1][gDtJuVYCQ5qHiU-aSKnj7g][node1][inet[/10.1.0.12:9300]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout

[2015-03-14 14:51:58,732][DEBUG][cluster.service          ] [node2] publishing cluster state version 11

After which the cluster will scramble to recover the lost shards in the remaining nodes.

And on node1 (the one we cut the network cable) will have something like:

[2015-03-14 14:51:58,847][DEBUG][discovery.zen.fd         ] [node1] [master] failed to ping [[node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]]], tried [3] times, each with maximum [30s] timeout

[2015-03-14 14:51:58,851][DEBUG][discovery.zen.fd         ] [node1] [master] stopping fault detection against master [[node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]]], reason [master failure, failed to ping, tried [3] times, each with  maximum [30s] timeout]

[2015-03-14 14:51:58,854][INFO ][discovery.zen            ] [node1] master_left [[node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]]], reason [failed to ping, tried [3] times, each with  maximum [30s] timeout]

[2015-03-14 14:51:58,868][DEBUG][cluster.service          ] [node1] processing [zen-disco-master_failed ([node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]])]: execute

[2015-03-14 14:51:58,870][WARN ][discovery.zen            ] [node1] master left (reason = failed to ping, tried [3] times, each with  maximum [30s] timeout), current nodes: {[node0][tu2reCtdT2Shekx0gitB1g][node0][inet[/10.1.0.11:9300]],[node1][gDtJuVYCQ5qHiU-aSKnj7g][node1][inet[/10.1.0.12:9300]],}

[2015-03-14 14:51:58,898][DEBUG][cluster.service          ] [node1] cluster state updated, version [10], source [zen-disco-master_failed ([node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]])]

[2015-03-14 14:51:58,903][INFO ][cluster.service          ] [node1] removed {[node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]],}, reason: zen-disco-master_failed ([node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]])

[2015-03-14 14:51:58,918][DEBUG][cluster.service          ] [node1] set local cluster state to version 10

[2015-03-14 14:51:58,968][DEBUG][cluster.service          ] [node1] processing [zen-disco-master_failed ([node2][E3MoSOWDSDW_8AdCPznakg][node2][inet[/10.1.0.13:9300]])]: done applying updated cluster_state (version: 10)

After a while node1 will be marked as failed by node2 and will no longer be pinged. Which means, it has to manually re-join the cluster.

After a while node1 will elect itself as a master:

[2015-03-14 14:53:03,491][DEBUG][cluster.service          ] [node1] cluster state updated, version [11], source [zen-disco-join (elected_as_master)]

[2015-03-14 14:53:03,491][INFO ][cluster.service          ] [node1] new_master [node1][gDtJuVYCQ5qHiU-aSKnj7g][node1][inet[/10.1.0.12:9300]], reason: zen-disco-join (elected_as_master)

[2015-03-14 14:53:03,492][DEBUG][cluster.service          ] [node1] publishing cluster state version 11

[2015-03-14 14:53:33,504][WARN ][discovery.zen.publish    ] [node1] timed out waiting for all nodes to process published state [11] (timeout [30s], pending nodes: [[node0][tu2reCtdT2Shekx0gitB1g][node0][inet[/10.1.0.11:9300]]])

Which is somewhat odd. But this happens because the discovery.zen.minimum_master_nodes key was not changed from the default of 1. You should probably increase that value.

Bring the interface back up to the network:

sudo ip link set dev $node1_veth up

You’ll notice that the node will not re-join the other nodes in the cluster. You have to manually restart it:

docker stop node1
docker start node1
declare "node1_veth=`sudo docker-native-inspect -format '{{.network_state.veth_host}}' state node1`"

And it should now join the cluster.

Packet delay and/or loss

Linux lets you fiddle with the network stack in several ways. One of them lets you delay or discard packets.

You can influence the entire communication between all nodes by changing the bridge0 interface. You can also do it for a single node by changing the corresponding veth interface.

For example, to add 100ms+-10ms delay to all packets:

sudo tc qdisc add dev docker0 root netem delay 100ms 10ms

To change it to have 25% pseudo-correlation:

sudo tc qdisc change dev docker0 root netem delay 100ms 10ms 25%

To change it to have 1% of packet loss:

sudo tc qdisc change dev docker0 root netem loss 1%

To change it to combine delay and loss:

sudo tc qdisc change dev docker0 root netem delay 100ms 10ms 25% loss 1%

To see all rules:

sudo tc qdisc show dev docker0

To remove all rules:

sudo tc qdisc del dev docker0 root

To apply the above examples to a specific node, instead of the docker0 device use the corresponding veth device, e.g. $node2_veth.

For more information see:

Bootstrapping your GitHub Pages blog with Vagrant

Fri, 13 Mar 2015 00:00:00 +0000

This will show you a way of creating a Vagrant Virtual Machine with the tools needed for generating a blog based on the GitHub Pages stack. GitHub Pages uses the Jekyll static site generator. This VM will use the same version of the tools that are currently in use at GitHub.

The base Vagrant box is based on the one created in From Iso To Vagrant Box post (read it first) but with a Xubuntu desktop.

You need to modify the base box to install the xubuntu-desktop package. For that, modify the preseed pkgsel/include line to include its package name:

vi preseed.txt
...
d-i pkgsel/include string openssh-server, xubuntu-desktop
...

Build the image, and add it to Vagrant:

packer build ubuntu-14.10.json
vagrant box add xubuntu-14.10-amd64 ubuntu-14.10-amd64-virtualbox.box
rm ubuntu-14.10-amd64-virtualbox.box

Create the Vagrantfile that provisions the GitHub Pages tools:

cat<<"VAGRANTFILE">Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

$root_provision_script = <<'ROOT_PROVISION_SCRIPT'
#!/bin/bash
# abort this script on errors.
set -eux

# prevent apt-get et al from opening stdin.
# NB even with this, you'll still get some warnings that you can ignore:
#     dpkg-preconfigure: unable to re-open stdin: No such file or directory
export DEBIAN_FRONTEND=noninteractive

# install javascript support for Jekyll.
# See https://github.com/sstephenson/execjs
apt-get -y install software-properties-common
apt-add-repository -y ppa:chris-lea/node.js
apt-get update
apt-get -y install nodejs

# install git. we'll use it to later push the blog to github.
apt-get -y install git

# install ungit. a web-based gui for git.
npm install -g ungit

# install retext. a Markdown editor.
apt-get -y install retext

# install ruby. Jekyll is a ruby application.
apt-get -y install ruby-dev ruby

# install dependencies needed for the gems.
apt-get -y install zlib1g-dev

# install Jekyll -- the version that is currently hosted by GitHub.
gem install --verbose github-pages
ROOT_PROVISION_SCRIPT

Vagrant.configure(2) do |config|
  config.vm.box = "xubuntu-14.10-amd64"

  config.vm.hostname = "pages"

  config.vm.provider "virtualbox" do |vb|
    vb.gui = true
  end

  config.vm.provision "shell", inline: $root_provision_script
end
VAGRANTFILE

NB This blog has a similar Vagrantfile.

Enter the VM and generate a SSH key-pair:

vagrant ssh
ssh-keygen
cat ~/.ssh/id_rsa.pub

Add the public key (the content of the id_rsa.pub file) into your GitHub account at:

https://github.com/settings/ssh

Also create a new empty repository named after your username, e.g. example.github.io.

NB From here on I’ll use example as your username.

Setup git:

git config --global user.name "Example Doe"
git config --global user.email example@example.com
git config --global push.default simple
git config --list

Test that it works:

ssh -T git@github.com

Initialize the blog:

git clone git@github.com:example/example.github.io.git
cd example.github.io
jekyll new .

Start the server:

jekyll serve

And access the blog at:

http://localhost:4000

At another shell, open a file, edit and save it:

vi blog/about.md

After you save a file Jekyll should re-generate the blog. To see the result just refresh the browser page.

When you are ready, push the changes to the world:

git add --all
git commit -m 'Hello World!'
git push

After a couple of minutes the blog should be live at:

http://example.github.io

And that’s it!

From Iso To Vagrant Box

Thu, 12 Mar 2015 00:00:00 +0000

This is my recipe on how to automatically create an up-to-date Ubuntu Vagrant box file from an iso.

You need the following ingredients:

I use the minimal ISO because the regular ISO does not always contains the latest package versions, so you’ll not end up having to update your system just right after you install it. This is because the mini ISO installs the latest packages right of the internet; it is also much smaller than the regular ISO.

Packer is used to install Ubuntu into a Vagrant box file. Packer uses a template file to describe that process. The template uses a preseed file to drive the Debian installer (aka d-i) to automatically install Ubuntu in a VirtualBox Virtual Machine, which will be later exported into a Vagrant box file.

Vagrant is used to automatically create and setup (or provision) a development environment inside a Virtual Machine. Vagrant uses a Vagrantfile script to describe that process.

VirtualBox is used to actually run the Virtual Machine.

You can find the sources at rgl/ubuntu-vagrant, but read along to known more details.

The following sections assume that you already have Packer, Vagrant and VirtualBox installed and available on your PATH.

Packer Template

Create the template:

cat<<"EOF">ubuntu-14.10.json
{
  "variables": {
    "disk_size": 8192
  },
  "builders": [
    {
      "name": "amd64-virtualbox",
      "type": "virtualbox-iso",
      "guest_os_type": "Ubuntu_64",
      "guest_additions_mode": "attach",
      "headless": false,
      "http_directory": ".",
      "vboxmanage": [
        ["modifyvm", "{{.Name}}", "--memory", "2048"],
        ["modifyvm", "{{.Name}}", "--cpus", "2"],
        ["modifyvm", "{{.Name}}", "--vram", "32"]
      ],
      "disk_size": "{{user `disk_size`}}",
      "hard_drive_interface": "sata",
      "iso_url": "http://archive.ubuntu.com/ubuntu/dists/utopic/main/installer-amd64/current/images/netboot/mini.iso",
      "iso_checksum": "4f783f3917ed4c663c9a983f4ee046fc",
      "iso_checksum_type": "md5",
      "ssh_username": "vagrant",
      "ssh_password": "vagrant",
      "ssh_wait_timeout": "60m",
      "boot_wait": "5s",
      "boot_command": [
        "<esc><wait>",
        "/linux initrd=/initrd.gz",
        " nofb",
        " fb=false",
        " auto=true",
        " url=http://{{.HTTPIP}}:{{.HTTPPort}}/preseed.txt",
        " hostname=vagrant",
        " DEBCONF_DEBUG=5",
        " --",
        " nofb",
        "<enter>"
      ],
      "shutdown_command": "echo vagrant | sudo -S poweroff"
    }
  ],

  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo vagrant | sudo -S bash {{.Path}}",
      "scripts": ["setup.sh"]
    }
  ],

  "post-processors": [
    {
      "type": "vagrant",
      "output": "ubuntu-14.10-{{.BuildName}}.box"
    }
  ]
}
EOF

Notes

To known the available boot options open the boot/grub/grub.cfg file inside the mini.iso file.
While d-i is installing Ubuntu you can press alt+f2 to open a shell and inspect the environment. e.g. cat /proc/cmdline ; env ; ls -l /target
You can also press alt+f4 to see what is going on (its a console running tail -f /var/log/syslog). Press alt+f1 to return to the d-i.
The DEBCONF_DEBUG=5 was added into the kernel command line to aid us in debugging the preseed installation. At install time, when d-i asks a question, look at the last debconf variable that is mentioned on syslog (press alt+f4), its probably the one you need to set to answer the question.
At a console, to change the keyboard map, type e.g. loadkeys pt.
You only have the Nano text editor on the install console. To search text inside it press ctrl+w. To repeat the search press alt+w. To search backwards press alt+b.

Preseed

Create the preseed:

cat<<"EOF">preseed.txt
# save the log for debugging purposes.
d-i preseed/late_command string cp /var/log/syslog /target/home/vagrant/preseed.log

d-i debian-installer/locale string en_US.UTF-8
d-i localechooser/supported-locales multiselect en_US.UTF-8, pt_PT.UTF-8

d-i keyboard-configuration/layoutcode string pt
d-i console-setup/ask_detect boolean false

d-i mirror/country string manual
d-i mirror/http/hostname string nl.archive.ubuntu.com
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string

d-i clock-setup/utc boolean true
d-i time/zone string Europe/Lisbon

d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

d-i grub-installer/only_debian boolean true
d-i finish-install/reboot_in_progress note

d-i passwd/user-fullname string vagrant
d-i passwd/username string vagrant
d-i passwd/user-password password vagrant
d-i passwd/user-password-again password vagrant
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false

tasksel tasksel/first multiselect
d-i pkgsel/include string openssh-server
d-i pkgsel/upgrade select full-upgrade
d-i pkgsel/update-policy select none
EOF

Notes

Also see the example preseed at the Ubuntu Installation Guide.
A preseed/late_command was added to aid in debugging the preseed; it creates a preseed.log file with what happened at preseed.
keyboard-configuration/layoutcode accepts similar values as loadkeys(1), e.g. us, pt, etc.
time/zone accepts a filename relative to the /usr/share/zoneinfo/ directory.
tasksel/first is a comma separated list, .e.g. ubuntu-server, xubuntu-desktop.
You can later add support for other locales. e.g. I like to use ConEmu and it does not support UTF-8 at all, so I do:

sudo sh -c 'echo en_US.ISO-8859-15 ISO-8859-15 >> /var/lib/locales/supported.d/local'
sudo sudo dpkg-reconfigure locales
export LANG=en_US.ISO-8859-15 LANGUAGE=en_US
#sudo sh -c "echo 'LANG=$LANG' > /etc/default/locale" # optional. maybe just do this on your .bashrc

Provisioning script

Create the provisioning script:

cat<<"ENDSETUP">setup.sh
#!/bin/bash
# abort this script when a command fails or a unset variable is used.
set -eu
# echo all the executed commands.
set -x

# let our user use root permissions without sudo asking for a password (because
# d-i adds us into the sudo group, but we must be on the admin group instead).
# alternatively: echo 'vagrant ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/vagrant
groupadd -r admin
usermod -a -G admin vagrant
gpasswd -d vagrant sudo
sed -i -e 's,%admin ALL=(ALL) ALL,%admin ALL=(ALL) NOPASSWD:ALL,g' /etc/sudoers

# installing the vagrant public key.
# NB vagrant will replace it on the first run.
install -d -m 700 /home/vagrant/.ssh
pushd /home/vagrant/.ssh
wget --no-check-certificate https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub -O authorized_keys
chmod 600 ~/.ssh/authorized_keys
chown -R vagrant:vagrant .

# install the VirtualBox Guest Additions.
# this will be installed at /opt/VBoxGuestAdditions-VERSION.
# REMOVE_INSTALLATION_DIR=0 is to fix a bug in VBoxLinuxAdditions.run.
# See http://stackoverflow.com/a/25943638.
apt-get -y -q install gcc dkms
mkdir -p /mnt
mount /dev/sr1 /mnt
while [ ! -f /mnt/VBoxLinuxAdditions.run ]; do sleep 1; done
REMOVE_INSTALLATION_DIR=0 /mnt/VBoxLinuxAdditions.run --target /tmp/VBoxGuestAdditions
rm -rf /tmp/VBoxGuestAdditions
umount /mnt

# disable the DNS reverse lookup on the SSH server. this stops it from
# trying to resolve the client IP address into a DNS domain name, which
# is kinda slow and does not normally work when running inside VB.
echo UseDNS no >> /etc/ssh/sshd_config

# disable the graphical terminal. its kinda slow and useless on a VM.
sed -i -E 's,#(GRUB_TERMINAL\s*=).*,\1console,g' /etc/default/grub
update-grub

# use the up/down arrows to navigate the bash history.
# NB to get these codes, press ctrl+v then the key combination you want.
cat<<"EOF">>/etc/inputrc
"\e[A": history-search-backward
"\e[B": history-search-forward
set show-all-if-ambiguous on
set completion-ignore-case on
EOF

# clean packages.
apt-get -y autoremove
apt-get -y clean

# zero the free disk space -- for better compression of the box file.
#dd if=/dev/zero of=/EMPTY bs=1M ; rm -f /EMPTY
ENDSETUP

Build

Build it:

packer validate ubuntu-14.10.json
packer inspect ubuntu-14.10.json
#time packer build -var disk_size=81920 ubuntu-14.10.json
time packer build ubuntu-14.10.json

NB you can also modify the disk size with the disk_size user variable. e.g. for 80GB, build with: packer build -var disk_size=81920 ubuntu-14.10.json.

NB the box will compress to about 720MB (with 1.2G of used space).

NB in my machine, the build took about 15 minutes to create the final box file.

Add the box to vagrant:

vagrant box add ubuntu-14.10-amd64 ubuntu-14.10-amd64-virtualbox.box

Test

Create a test VM based on our created box:

mkdir -p test && cd test
vagrant init ubuntu-14.10-amd64
time vagrant up

NB in my machine, this takes about 45 seconds to boot.

Connect to the console, and kick the tires:

vagrant ssh
uname -a
id
df -h
less preseed.log
cat /proc/cmdline
dmesg
exit

Poweroff the VM, and destroy it:

vagrant halt
vagrant destroy
cd ..
rm -rf test

When you no longer need the box in vagrant, you can also remove it with:

vagrant box remove ubuntu-14.10-amd64

And that’s it!

You could now proceed on how to bootstrap a GitHub Pages blog inside a Xubuntu desktop.

From zero to a blank world map

Sun, 04 Jan 2015 17:54:00 +0000

When you need to show a web map you normally end up using something like:

var map = L.map('map').setView([38.736946, -9.142685], 3);
L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png').addTo(map);

But what does it take to show that map? What software stack lays behind the curtains? And how can you customize the map? Lets find out!

We start from a vanilla Linux Virtual Machine and end up showing a humble web map – a blank world – the hello world of maps.

Be warned that I barely describe the terms used in the GIS world. Be sure to read the referenced material for more information.

So, lets get cooking!

Ingredients

Recipe

Install Xubuntu

Install Xubuntu 14.10 (Utopic Unicorn) from torrent on a VirtualBox VM.

Make sure you give it some respectful resources. I gave mine:

60GB of disk
4GB of memory
2 CPUs
64MB of video memory

After install, bring it up to date. Open a Terminal Emulator and type:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get clean
sudo reboot

Install the VirtualBox Guest Additions. On the VM VirtualBox UI select Devices, Insert Guest Additions CD Image.... Xubuntu should automatically mount the CD. Open a terminal and type:

sudo apt-get install dkms
cd /media/$USER/VBOX*
sudo ./VBoxLinuxAdditions.run
sudo reboot

In order to copy the commands from this article into the VM you should enable the shared clipboard. On the VM VirtualBox UI select the Devices, Shared Clipboard, Bidirectional.

Mapbox Studio

Studio is used to quickly style maps.

Install it with:

sudo apt-add-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs

sudo apt-get install git
git clone https://github.com/mapbox/mapbox-studio.git
cd mapbox-studio
npm install

NB I’ve got revision 953ecc.

Start Studio:

npm start

On the Terminal, Right-Click the http://localhost:3000 text and select Open Link. Or open the Firefox Web Browser from the Xfce menu. Or in a second Terminal type:

exo-open http://localhost:3000

Follow the application steps to create a free Mapbox account. After that, you are ready to use Studio.

Coastline data

The Coastline is the line where the land meets the sea. We’ll use it to draw the world outline on our map.

The Coastline data is maintained by the OpenStreetMap contributors. Its periodically extracted from OpenStreetMap and made available at the OpenStreetMap Data site as a Shapefile – created by the OSMCoastline application.

This data is available in two different projections (aka SRS):

EPSG:4326 (aka WGS84). Used in GPS and raw OpenStreetMap data found on Planet.osm.
EPSG:3857 (aka Web Mercator or Spherical Mercator aka 900913). Used in web maps (e.g. Google/Bing/Leaftlet/Mapbox.js).

NB You should prefer EPSG:3857 because its the one normally used in web maps – no re-projection will be needed.

NB Be sure to known your coordinates projection – otherwise confusion will surely arise!

Download the Coastline water polygons Shapefile (~390MB) – EPSG:3857 projection:

sudo apt-get install curl
curl -O http://data.openstreetmapdata.com/water-polygons-split-3857.zip
unzip water-polygons-split-3857.zip

And index it:

~/mapbox-studio/node_modules/mapnik/lib/binding/node-v11-linux-*/shapeindex water-polygons-split-3857/water_polygons.shp

Source

To show any data in Studio we need to have a Source.

NB A Studio Source handles both of the projections mentioned before. But if you use EPSG:3857 no re-projection will be needed.

To create one, go to Studio, and:

Click the Projects icon (bottom-left corner).
Click the New Project icon.
Click the Blank source icon (right column).

Create a new layer for the data:

Click the New Layer icon (top-right).
Click the Browse button and select the water-polygons-split-3857/water_polygons.shp file.
Click the rename link and rename the layer to coastalarea. You will later use it in a stylesheet selector as #coastalarea.
Click the Done button.

Change the project settings:

Click the Settings button (left toolbar).
Change the Maxzoom value to 22.
Click the Save as button (left toolbar).
Save the Source project as blank-world.

You should now have on your disk a blank-world.tm2source directory. It contains all the Source project files.

You are now ready to create a Style project.

Style

Maps are styled with CartoCSS – a language similar to CSS.

Lets use it to style the map.

Go to Studio and create a new Style project:

Click the Projects icon (bottom-left corner).
Click the New Project icon.
Click the Basic style icon (left column).

You should now see a basic map; but we want to use our own Source, so:

Click the Layers icon.
Click the Change source icon.
Toggle the Sources to Local.
Select the blank-world.tm2source Source that we’ve created before.
Click the Save as button (left toolbar). And save the project as blank-world.

On your disk, you should now have a blank-world.tm2 directory. It contains all the Style project files.

You should also notice a Style pane on the right hand side of the window. It contains the stylesheet that is used to style the map. Replace the text with:

@land: #f8f4f0;
@water: #a0c8f0;

Map {
  background-color: @land;
}

#coastalarea {
  line-color: @water;
  polygon-fill: @water;
}

Save the Project (press Ctrl+S).

After a bit, you should see a map!

Tile server

To display a map on a web page we need:

A server that returns the map as Tile images.
A HTML widget that displays them.

We’ll use the tessera server. Install it with:

mkdir blank-world && cd blank-world

cat<<"EOF">package.json
{
  "name": "blank-world",
  "version": "1.0.0",
  "private": true,
  "dependencies": {}
}
EOF

npm install tessera --save
npm install tilelive-tmsource --save
npm install tilelive-tmstyle --save

See the resulting dependencies and versions:

cat package.json

I got:

"tessera": "^0.5.1",
"tilelive": "^5.4.1",
"tilelive-tmsource": "^0.1.2",
"tilelive-tmstyle": "^0.3.0",
"tilelive-vector": "^2.3.1"

Start tessera:

./node_modules/.bin/tessera tmstyle://$HOME/blank-world.tm2

It should now be running at http://localhost:8080.

In another Terminal, get the TileJSON document that describes the map:

sudo apt-get install httpie
http get http://localhost:8080/index.json

You should see something like:

{
    "attribution": "...", 
    "bounds": [
        -180, 
        -85.0511, 
        180, 
        85.0511
    ], 
    "center": [ ... ], 
    "format": "png8:m=h", 
    "maxzoom": 22, 
    "minzoom": 0, 
    "name": "Untitled", 
    "scale": "1", 
    "source": "tmsource:///home/rgl/blank-world.tm2source", 
    "tilejson": "2.0.0", 
    "tiles": [
        "http://localhost:8080/{z}/{x}/{y}.png"
    ]
}

You can also get a Tile image, e.g. the south of Portugal:

curl -O http://localhost:8080/6/30/24.png
exo-open 24.png

Tiles are normally PNG images with 256x256 pixels.

You are almost done! Off to showing the web map…

Web map

To show the map we’ll use the Mapbox.js library.

Create the web page:

cat<<"EOF">map.html
<!DOCTYPE html>
<html>
<head>
<meta charset='utf-8' />
<title>blank world</title>
<meta name='viewport' content='initial-scale=1,maximum-scale=1,user-scalable=no' />
<script src='https://api.tiles.mapbox.com/mapbox.js/v2.1.4/mapbox.js'></script>
<link href='https://api.tiles.mapbox.com/mapbox.js/v2.1.4/mapbox.css' rel='stylesheet' />
<style>
  body { margin:0; padding:0; }
  #map { position:absolute; top:0; bottom:0; width:100%; }
</style>
</head>
<body>
<div id='map'></div>
<script>
var map = L.mapbox.map('map', 'http://localhost:8080/index.json')
    .setView([38.736946, -9.142685], 3);
</script>
</body>
</html>
EOF

Open it:

exo-open map.html

You should now see the blank world map!

And there you have it! Let me known how it worked for you.

In a future article I’ll show you how to import the water areas from OpenStreetMap.

More information

This article mentioned a lot of pieces that you might not be familiar with. To get more insight you should read the following references:

Map Styles:

Map projections (aka SRS):

EPSG:4326 (aka WGS84). Used in GPS and raw OpenStreetMap data found on Planet.osm.
EPSG:3857 (aka Web Mercator or Spherical Mercator). Used in web maps (e.g. Google/Bing/Leaftlet/Mapbox.js).
FAQ
The Google Maps / Bing Maps Spherical Mercator Projection confusion.

OpenStreetMap:

As an alternative software stack, you should look at:

And finally, feel free to contact me!

– RGL

Taking web application screenshots with CasperJS

Sun, 20 Apr 2014 15:51:00 +0000

When you need to take screenshots of a web application, the normal route is to navigate to a page, fill in the forms with sample data, use the Print screen (or Alt+Print screen) key to take the screenshot, then crop and export it with a normal application like GIMP… which is quite time consuming… and god forbid if you need to do it again, with a slightly different data or page design…

A better route would be to automate the whole process. Let me show you how to do it with CasperJS!

Lets start!

NB if you need some help installing CasperJS, see the end of this article.

I’ll be using my home page as an example.

Create a simple casper test script that captures the entire page inside the file example.js:

// set the viewport size to include all our page content.
casper.options.viewportSize = {width: 700, height: 600};

casper.test.begin("test", function(test) {
  // step 1: open the page.
  casper.start("http://ruilopes.com", function() {
    // do an example test.
    test.assertTitle("rui lopes home");
  });

  // step 2: take some screenshots.
  casper.then(function() {
    // capture the entire page.
    casper.capture("page.png");

    // capture the nav element.
    casper.captureSelector("selector.png", "nav");
  });

  // actually run the steps we defined before.
  casper.run(function() {
    test.done();
  });
});

NB As you can see its quite self-explanatory; but you should really read The Fine Manual.

Run casper:

casperjs test example.js

Should output something like:

Test file: example.js
# test
PASS Page title is: "rui lopes home"
PASS 1 test executed in 0.087s, 1 passed, 0 failed, 0 dubious, 0 skipped.

And look at the generated screenshots.

the entire page

the nav element

Mouse Cursor

CasperJS uses PhantomJS uses QtWebKit in headless mode. But, unfortunately, this combo does not seem to support the mouse cursor. So I had to roll my own solution by creating casperjs-screenshooter. It simulates the mouse cursor by adding an img element with id screenshooterCursor to the page DOM and the hover class into a element to simulate the hover effect.

Lets see how we can use it:

// step 3: take some screenshots with the mouse over a element.
screenshooter.thenMoveCursor("pointer", "a[href='http://blog.ruilopes.com/']", function() {
    // capture the area taken by the h1 and mouse cursor elements.
    screenshooter.capture("multiple-selector-with-mouse-cursor.png", "h1, #screenshooterCursor");
});

NB by default, you can use the following cursor names: default, pointer, text, help or none. You can add more yourself by using cur2png or png2json (you can download the binaries too).

NB the default cursors come from the excellent Ubuntu Human theme.

NB with screenshooter.capture (instead of casper.capture) you can capture the area taken by several elements, each selected by a different selector separated by a comma.

And the resulting screenshot image:

And thats it… see the full source and let me known what you think!

– RGL

Install on Ubuntu

NB I’m using Ubuntu 14.04; but it should be the same on other versions.

Run the following commands:

sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs
sudo npm install -g casperjs

Install on Windows

On Windows (I’m using 8.1), we need to jump through some hoops… download the installer (e.g. node-v0.10.26-x86.msi) from:

http://nodejs.org/download/

And install it (I’ve installed mine at C:\Development\nodejs).

Next, open a Command Prompt, and make sure you can run node. If you didn’t let the installer modify your PATH, you need to add the directories manually with:

set PATH=%PATH%;C:\Development\nodejs;%APPDATA%\npm

NB %APPDATA% should expand to something like C:\Users\rgl\AppData\Roaming.

Check the installed node version:

node --version

Should output something like:

v0.10.26

Check the installed npm version:

npm --version

Should output something like:

1.4.3

Install CasperJS with:

npm install -g casperjs

And check its version:

casperjs --version

Should output something like:

'python' is not recognized as an internal or external command,
operable program or batch file.

Errr what? needs python? Oh well… as I found out its possible to run without it, lets add the needed directories into the PATH:

set PATH=%APPDATA%\npm\node_modules\casperjs\bin;%APPDATA%\npm\node_modules\casperjs\node_modules\phantomjs\lib\phantom;%PATH%

NB You need to add the caspjerjs\bin directories before the ones we’ve set before.

NB If this sounds too hackish, install python and add it into your PATH.

And try again:

casperjs --version

Should finally output something like:

1.1.0-beta3

Fix magnet link openning in Xfce / Arch Linux

Sun, 25 Aug 2013 18:18:00 +0000

The current Xfce (4.10) that ships with Arch Linux has an annoying bug of not correctly opening Magnet URIs (e.g. when you click a magnet link from within Chromium). So I had to have that fixed, but doing it took longer that I expected… so I’m creating this recipe for helping you solving the problem more easily.

Chromium handles unknown URI schemes by delegating them to the xdg-open command. Which in turn, detects it’s running under the Xfce desktop and delegates the control the exo-open command. Which fails to open the magnet URIs.

In these instructions you’ll patch the exo package and re-install it with a little patch that fixes that annoying problem.

So let’s get to it! Open a terminal and issue the following commands.

Configure xdg-open to open magnet URIs with the Transmission application:

xdg-mime default transmission-gtk.desktop x-scheme-handler/magnet

NB: this modifies the ~/.local/share/applications/mimeapps.list file.

Get the exo package (contains the exo-open command):

sudo pacman -S abs
sudo abs extra/exo
cd /tmp
cp -R /var/abs/extra/exo .
cd exo

Add the patch into a file:

cat<<"EOF">fix-exo_str_looks_like_an_uri-bug-10098.patch
diff --git a/exo/exo-string.c b/exo/exo-string.c
index 33f86f9..056b36a 100644
--- a/exo/exo-string.c
+++ b/exo/exo-string.c
@@ -429,7 +429,7 @@ exo_str_looks_like_an_uri (const gchar *str)
       for (++s; g_ascii_isalnum (*s) || *s == '+' || *s == '-' || *s == '.'; ++s);

       /* <scheme> must be followed by ":" */
-      return (*s == ':' && *(s+1) == '/');
+      return (*s == ':' && *(s+1) != '\0');
     }

   return FALSE;
EOF

Modify the PKGBUILD file to patch the source by adding the prepare function above the existing build function:

prepare() {
  cd "$srcdir/$pkgname-$pkgver"
  patch -p1 &lt; "$startdir/fix-exo_str_looks_like_an_uri-bug-10098.patch"
}

NB to properly finish the package you should also add the patch file name into the source array and its sha256 checksum into the sha256sums array… but I’ll not bother.

And finally build and install the package:

makepkg -s
sudo pacman -U *.pkg*

And that should be it! Try to click on a magnet file link now!

– RGL

from http to https with free certificates

Sun, 06 Mar 2011 14:51:00 +0000

Recently I’ve noticed that StartCom gives away free certificates through theirs StartSSL site. In this post I will explain how I managed to configure nginx to host two of my domains, on the same IP address, using HTTPS.

TL;DR summary

This is the recipe to create an RSA keypair, a Certificate Signing Request (CSR), install and configure nginx.

Create the RSA keypair:

cd /etc/ssl/private
openssl req -newkey rsa:2048 -nodes -keyout ruilopes.com.key -out ruilopes.com.csr && chmod 400 ruilopes.com.key

Set Common Name (eg, YOUR name) field to: ruilopes.com

Paste the contents of the ruilopes.com.csr file into the StartCom form, let’m generate the certificate, copy it into the ruilopes.com.crt file.

Append the StartCom intermediate CA certificate to our own:

curl -O https://www.startssl.com/certs/sub.class1.server.ca.pem
cat sub.class1.server.ca.pem >> ruilopes.com.crt

Install and configure nginx:

add-apt-repository ppa:nginx/stable
apt-get update
apt-get install nginx-full

mkdir /var/www/ruilopes.com

cat<<"EOF">/var/www/ruilopes.com/index.html
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <title>Hello World</title>
</head>
<body>
  <h1>Hello World!</h1>
</body>
</html>
EOF


cat<<"EOF">/etc/nginx/sites-available/ruilopes.com.conf
server {
  server_name ruilopes.com www.ruilopes.com;

  listen 80 default;
  listen 443 default ssl;

  ssl_certificate      /etc/ssl/private/ruilopes.com.crt;
  ssl_certificate_key  /etc/ssl/private/ruilopes.com.key;

  root /var/www/ruilopes.com;
  access_log /var/log/nginx/ruilopes.com.access.log;

  index index.html;
}
EOF

ln -s /etc/nginx/sites-available/ruilopes.com.conf /etc/nginx/sites-enabled/
rm /etc/nginx/sites-enabled/default

service nginx restart

Test with openssl:

openssl s_client -connect ruilopes.com:443 -servername ruilopes.com

Test by opening the following address:

https://ruilopes.com/

That’s it!

But really, keep reading! I’ll explain how this works and also add a second domain to the same IP address… this is a somewhat lengthy post… so bear with me! In the end, I hope you’ll understand what is going on.

From http to HTTPS

Roughly, we’ll follow this plan:

create a RSA keypair (two keys: a private and a public key).
create a Certificate Signing Request (CSR).
submit the CSR to StartCom; they will create and sign a certificate that binds our public key and domain name to their Certification Authority (CA).
configure nginx to use our private key and the certificate signed by StartCom.

I’ll use Ubuntu 10.04, so the nginx install and file-system paths are tied to this particular Linux flavor.

All the commands displayed here should be run inside a shell with root privileges.

Lets start by installing nginx:

apt-get install nginx

And a plain nginx configuration (no certificates involved) for my primary domain ruilopes.com:

cat<<"EOF">/etc/nginx/sites-available/ruilopes.com.conf
server {
  server_name ruilopes.com www.ruilopes.com;

  listen 80;

  root /var/www/ruilopes.com;
  access_log /var/log/nginx/ruilopes.com.access.log;

  index index.html;
}
EOF

Create a simple index page:

mkdir /var/www/ruilopes.com

cat<<"EOF">/var/www/ruilopes.com/index.html
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <title>Hello World</title>
</head>
<body>
  <h1>Hello World!</h1>
</body>
</html>
EOF

Enable the new site, remove the default site (which ships by default with Ubuntu), and notify nginx to reload its configuration:

ln -s /etc/nginx/sites-available/ruilopes.com.conf /etc/nginx/sites-enabled/
rm /etc/nginx/sites-enabled/default
service nginx reload

Try to access the site, to see if its working:

curl --silent --dump - http://ruilopes.com/

You should see something like:

HTTP/1.1 200 OK
Server: nginx/0.7.65
Date: Mon, 31 Jan 2011 21:54:22 GMT
Content-Type: text/html
Content-Length: 141
Last-Modified: Sun, 30 Jan 2011 20:26:43 GMT
Connection: keep-alive
Vary: Accept-Encoding
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <title>Hello World</title>
</head>
<body>
  <h1>Hello World!</h1>
</body>
</html>

Now that we have the baseline working, lets add some crypto to the mix.

Using openssl(1) genrsa(1) create an 2048-bit RSA keypair (without password protection; use -des3 for password protection):

cd /etc/ssl/private
openssl genrsa -out ruilopes.com.key 2048
chmod 400 ruilopes.com.key

NB You can later password protect a key with:

openssl rsa -des3 -in ruilopes.com.key -out ruilopes.com.key.new && mv ruilopes.com.key.new ruilopes.com.key

You can inspect the private and public key components with:

openssl rsa -noout -text -in ruilopes.com.key

It looks something like:

Private-Key: (2048 bit)
modulus:
    00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:
    ...
publicExponent: 65537 (0x10001)
privateExponent:
    ...
prime1:
    ...
prime2:
    ...
exponent1:
    ...
exponent2:
    ...
coefficient:
    ...

Or just inspect the public content:

openssl rsa -pubout -in ruilopes.com.key 2>/dev/null | openssl rsa -noout -text -pubin

Which should look something like:

Modulus (2048 bit):
    00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:
    ...
Exponent: 65537 (0x10001)

Using openssl req(1) create the Certificate Signing Request (CSR):

openssl req -new -key ruilopes.com.key -out ruilopes.com.csr

NB You can also generate the keypair and CSR with a single command:

openssl req -newkey rsa:2048 -nodes -keyout ruilopes.com.key -out ruilopes.com.csr

This step will ask you for information to include in the CSR, in our case, StartCom level 1 certificates only need the domain name (aka X.509 Common Name or CN; NB all the other information is ignored by StartCom), here’s how openssl req asks for this information:

Country Name (2 letter code) [AU]:.
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:.
Organization Name (eg, company) [Internet Widgits Pty Ltd]:.
Organizational Unit Name (eg, section) []:.
Common Name (eg, YOUR name) []:ruilopes.com
Email Address []:.

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

NB “.” is an openssl convention that means the field should be empty/omitted.

You can inspect the certificate signing request using:

openssl req -noout -text -in ruilopes.com.csr

It looks something like:

Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: CN=ruilopes.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:
                    ...
                Exponent: 65537 (0x10001)
        Attributes:
            a0:00
    Signature Algorithm: sha1WithRSAEncryption
        47:98:bd:ee:41:cb:29:de:29:54:f9:0d:d7:ff:6d:00:55:ec:
        ...

You are now ready to send the CSR to StartCom… just copy&paste its contents to the StartCom control panel and let them generate our certificate.

NB: They also let you add one sub-domain, I’ve choosen “www” (for www.ruilopes.com).

After the certificate is generated, copy&paste it into the ruilopes.com.crt file, and inspect the certificate with:

openssl x509 -noout -text -in ruilopes.com.crt

It will look something like:

Certificate:
...
        Issuer: C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Class 1 Primary Intermediate Server CA
        Validity
            Not Before: Jan 29 17:45:36 2011 GMT
            Not After : Jan 30 19:25:11 2012 GMT
        Subject: description=345115-eJ9Q0A1Uk83j6dMv, C=PT, O=Persona Not Validated, OU=StartCom Free Certificate Member, CN=www.ruilopes.com/emailAddress=me@example.com
...
            X509v3 Subject Alternative Name:
                DNS:www.ruilopes.com, DNS:ruilopes.com

You are now ready to configure nginx with the private keypair (inside ruilopes.com.key file) and the certificate (inside ruilopes.com.crt file). So lets do that:

cat<<"EOF">/etc/nginx/sites-available/ruilopes.com.conf
server {
  server_name ruilopes.com www.ruilopes.com;

  listen 80 default;
  listen 443 default ssl;

  ssl_certificate      /etc/ssl/private/ruilopes.com.crt;
  ssl_certificate_key  /etc/ssl/private/ruilopes.com.key;

  root /var/www/ruilopes.com;
  access_log /var/log/nginx/ruilopes.com.access.log;

  index index.html;
}
EOF

NB We’ve just added the listen 443, ssl_certificate and ssl_certificate_key lines.

Notify nginx to reload the configuration:

service nginx reload

And try to access our shinny HTTPS server! This time using openssl s_client(1):

openssl s_client -connect ruilopes.com:443 -servername ruilopes.com

The output should look something like:

CONNECTED(00000003)
depth=0 /description=345115-eJ9Q0A1Uk83j6dMv/C=PT/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=www.ruilopes.com/emailAddress=me@example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /description=345115-eJ9Q0A1Uk83j6dMv/C=PT/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=www.ruilopes.com/emailAddress=me@example.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /description=345115-eJ9Q0A1Uk83j6dMv/C=PT/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=www.ruilopes.com/emailAddress=me@example.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/description=345115-eJ9Q0A1Uk83j6dMv/C=PT/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=www.ruilopes.com/emailAddress=me@example.com
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----
... omitted ...
-----END CERTIFICATE-----
subject=/description=345115-eJ9Q0A1Uk83j6dMv/C=PT/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=www.ruilopes.com/emailAddress=me@example.com
issuer=/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
---

NB for now, ignore the errors that are displayed in this output.

Lets try with curl:

curl --silent --dump - https://ruilopes.com/

And you’ll notice that something is not right, as no text output is displayed… lets see whats is going on by removing the --silent switch, and try again. This time the output should be something like:

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

This happens because curl, by default, does not even try to validate intermediate CA certificates. Recall this piece of information from openssl s_client output:

---
Certificate chain
 0 s:/description=345115-eJ9Q0A1Uk83j6dMv/C=PT/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=www.ruilopes.com/emailAddress=me@example.com
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
---

This means that our HTTPS server is returning a single certificate, issued by the Certification Authority (CA) that has the Common Name (CN) StartCom Class 1 Primary Intermediate Server CA. Lets see if Ubuntu ships with a certificate that has that CN:

ls -F /etc/ssl/certs | grep -i startcom

It outputs something like:

StartCom_Certification_Authority.pem
StartCom_Ltd..pem

Lets see the CN:

openssl x509 -noout -text -in /etc/ssl/certs/StartCom_Certification_Authority.pem

It should look something like (we are only interested in the Issuer and Subject fields):

Issuer:  C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Certification Authority
Subject: C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Certification Authority

As you can see, the Subject CN is not the CN we are looking for… lets try the other certificate:

openssl x509 -noout -text -in /etc/ssl/certs/StartCom_Ltd..pem

It should look something like:

Issuer:  C=IL, ST=Israel, L=Eilat, O=StartCom Ltd., OU=CA Authority Dep., CN=Free SSL Certification Authority/emailAddress=admin@startcom.org
Subject: C=IL, ST=Israel, L=Eilat, O=StartCom Ltd., OU=CA Authority Dep., CN=Free SSL Certification Authority/emailAddress=admin@startcom.org

So neither certificate Subject has the CN we are looking for… so Ubuntu does not ship with the StartCom Class 1 Primary Intermediate Server CA certificate.

NB This is a simplification of the certificate resolution algorithm; eg. they are not matched just by their CN, but by their Distinguished Name (DN) and public key value/fingerprint (and other details I’m omitting here).

Lets download it from StartCom site, and see its contents:

curl -O https://www.startssl.com/certs/sub.class1.server.ca.pem
openssl x509 -noout -text -in sub.class1.server.ca.pem

It should look something like:

Issuer:  C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Certification Authority
Subject: C=IL, O=StartCom Ltd., OU=Secure Digital Certificate Signing, CN=StartCom Class 1 Primary Intermediate Server CA

So, the Subject we were looking for is here! Also notice that the Subject is different than the Issuer… this means this is an intermediate certificate; the ones that have the Issuer equal to Subject are called root certificates (aka self-signed certificates).

Also notice that the Issuer CN is the same has the one inside the /etc/ssl/certs/StartCom_Certification_Authority.pem file! So, Ubuntu ships with that root certificate. So, we are missing a link in the certificate chain we return in our HTTPS server… lets fix that by appending it there:

cat sub.class1.server.ca.pem >> ruilopes.com.crt

Notify nginx to reload the configuration:

service nginx reload

See how the certificate chain is now returned:

openssl s_client -connect ruilopes.com:443 -servername ruilopes.com

The output should look something like:

---
Certificate chain
 0 s:/description=345115-eJ9Q0A1Uk83j6dMv/C=PT/O=Persona Not Validated/OU=StartCom Free Certificate Member/CN=www.ruilopes.com/emailAddress=me@example.com
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
 1 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
---

So, now the chain is composed by two links:

Subject with CN=www.ruilopes.com; issued by Issuer with CN=StartCom Class 1 Primary Intermediate Server CA
Subject with CN=StartCom Class 1 Primary Intermediate Server CA; issued by Issuer with CN=StartCom Certification Authority

See how the first certificate Issuer is the same has the second certificate Subject? That’s how the first certificate is validated by the second, which in turn is validated by the root certificate installed on our local Ubuntu. You see, a chain is formed, starting on our certificate till the root certificate installed on our machine.

OK, looks good… lets see if it actually works by trying again with curl:

curl --silent --dump - https://ruilopes.com/

You should see something like:

HTTP/1.1 200 OK
Server: nginx/0.7.65
Date: Mon, 31 Jan 2011 21:54:22 GMT
Content-Type: text/html
Content-Length: 141
Last-Modified: Sun, 30 Jan 2011 20:26:43 GMT
Connection: keep-alive
Vary: Accept-Encoding
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <title>Hello World</title>
</head>
<body>
  <h1>Hello World!</h1>
</body>
</html>

So it worked!

Finally… lets try with a browser! Using Google Chrome you should see something like:

NB the window with the certificate information appears after you click the green padlock on the address bar.

So, that’s it! Our first domain is ready to use!

Now on to the other domain… well, now it much easier, just repeat everything, but with a different domain name and configuration file!

Create the configuration file:

cat<<"EOF">/etc/nginx/sites-available/statica.info.conf
server {
  server_name statica.info;

  listen 80 default;
  listen 443 default ssl;

  ssl_certificate      /etc/ssl/private/statica.info.crt;
  ssl_certificate_key  /etc/ssl/private/statica.info.key;

  root /var/www/statica.info;
  access_log /var/log/nginx/statica.info.access.log;

  index index.html;
}
EOF

ln -s /etc/nginx/sites-available/statica.info.conf /etc/nginx/sites-enabled/

Notify nginx to reload the configuration:

service nginx reload

And… it fails with:

Restarting nginx: [emerg]: a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/ruilopes.com.conf:4
configuration file /etc/nginx/nginx.conf test failed

This happens because we already had a default domain (ruilopes.com); so, remove the “default” word from the “listen” configuration lines:

cat<<"EOF">/etc/nginx/sites-available/statica.info.conf
server {
  server_name statica.info;

  listen 80;
  listen 443 ssl;

  ssl_certificate      /etc/ssl/private/statica.info.crt;
  ssl_certificate_key  /etc/ssl/private/statica.info.key;

  root /var/www/statica.info;
  access_log /var/log/nginx/statica.info.access.log;

  index index.html;
}
EOF

And try again:

service nginx reload

Now it fails with a different error:

Restarting nginx: [emerg]: "ssl" parameter can be specified for the default "listen" directive only in /etc/nginx/sites-enabled/statica.info.conf:5
configuration file /etc/nginx/nginx.conf test failed

It turns out we need a newer version of nginx… fortunately, there is a PPA at https://launchpad.net/nginx Lets, remove the old nginx, and add the new version to our system:

service nginx stop
add-apt-repository ppa:nginx/stable
apt-get update
apt-get remove nginx        # remove 0.7.65
apt-get install nginx-full  # add 0.8.54

And start it:

service nginx start

It should now work fine!

If you are still with me, you might be wondering how it possible to have two (or more) certificates on the same IP address? This is possible because nginx has support for Server Name Indication (SNI), which you can see with:

nginx -V

Which will output something like:

nginx version: nginx/0.8.54
TLS SNI support enabled

For this to work, the browser also has to support SNI. When it does, the browser will send the domain name (aka server name) that its trying to access in the initial TLS handshake.

This also means that older browsers (without SNI) will not be able to access our site…

And that’s it! I hope you now understand a bit more how certificates work!

– RGL

from zero to elasticsearch in a jiffy

Thu, 17 Feb 2011 16:39:00 +0000

In this article I will cut to the chase on how to start using elasticsearch, a distributed search engine that just works, You known, for Search!

Elasticsearch (ES) is based on Lucene, as so, you first need to download the Java Runtime Environment. I’ll assume you have installed it at C:\Dev\Java\jre6.

Next, download elasticsearch from:

http://www.elasticsearch.org/download/

And decompress it, eg., at C:\Dev\elasticsearch-0.14.4.

Open a Windows Command Prompt window, go into ES directory, and launch it:

cd c:\Dev\elasticsearch-0.14.4
set JAVA_HOME=C:\Dev\Java\jre6
bin\elasticsearch.bat -f

In a new Windows Command Prompt window, verify that ES is responding correctly. First check the cluster status by opening the following URL:

http://localhost:9200/_cluster/health

You should see something like:

{
  cluster_name: "elasticsearch"
  status: "green"
  timed_out: false
  number_of_nodes: 1
  number_of_data_nodes: 1
  active_primary_shards: 0
  active_shards: 0
  relocating_shards: 0
  initializing_shards: 0
  unassigned_shards: 0
}

And also check the cluster state at:

http://localhost:9200/_cluster/state

Which should return something like:

{
  cluster_name: "elasticsearch",
  master_node: "Sz86lynqSAKwb1X_NHsrIQ",
  blocks: {},
  nodes: {
    Sz86lynqSAKwb1X_NHsrIQ: {
      name: "Dredmund Druid",
      transport_address: "inet[/192.168.1.245:9300]",
      attributes: {}
    }
  },
  metadata: {
    templates: {},
    indices: {}
  },
  routing_table: {
    indices: {}
  },
  routing_nodes: {
    unassigned: [],
    nodes: {}
  },
  allocations: []
}

As ES seems to be working fine, we are now ready to index some documents; I’m going to use a tweet as an example.

Open a Bash shell window (I’m assuming you already have it installed as described at Sane shell environment on Windows).

You also need curl on your PATH, you can install it from:

http://curl.haxx.se/download/libcurl-7.19.3-win32-ssl-msvc.zip

Download a tweet:

curl -o 19529810560688128.json 'http://api.twitter.com/1/statuses/show/19529810560688128.json?include_entities=1'

And check its overall structure:

cat 19529810560688128.json

{
  "text": "why marking an operation as idempotent is important: http:\/\/www.zeroc.com\/faq\/whyIdempotent.html"
...

NB I’m just showing the attribute that is relevant to the search we are going to make. The actual tweet has much more attributes.

Lets index it:

curl -XPUT -d@19529810560688128.json http://localhost:9200/tweets/tweet/19529810560688128

ES should return something like:

{
  "_id": "19529810560688128",
  "_index": "tweets",
  "_type": "tweet",
  "ok": true
}

We are now ready for our first search! Lets search for tweets that have the “idempotent” word. For this we use the q parameter like:

curl 'http://localhost:9200/tweets/_search?pretty=true&q=idempotent'

Which returns something like:

{
  "_shards": {
    "failed": 0,
    "successful": 5,
    "total": 5
  },
  "hits": {
    "hits": [
      {
        "_id": "19529810560688128",
        "_index": "tweets",
        "_score": 0.023731936,
        "_source": {
          "text": "why marking an operation as idempotent is important: http:\/\/www.zeroc.com\/faq\/whyIdempotent.html"
          ...
        },
        "_type": "tweet"
      }
    ],
    "max_score": 0.023731936,
    "total": 1
  },
  "took": 1
}

The most important attribute in this result set is the hits.hits array, it contains all matched documents hits.

The previous search looked inside all tweet attributes. We can also search inside a particular attribute, for example, “text” attribute:

curl 'http://localhost:9200/tweets/_search?pretty=true&q=text:idempotent'

Which should return the same document as before, but this time, with a higher score.

ES lets us do much more advanced searches; we can use the simple Lucene Query Parser Syntax (as we did) and we can also use the JSON based search query DSL.

And that’s it! As you can see, the out-of-box experience is quite simple and strait forward! Of course, this was just the tip of the iceberg. elasticsearch has many tricks up its sleeve, my favorites are: simplicity, automatically distributed (just keep launching nodes, one or several, its the same; its really elastic), highly available (data is replicated; if a node fails, the data will come from another node) and thrift transport (at least one order of magnitude faster than HTTP).

You should read the documentation and join the community!

Also checkout the various client libraries, and the source code.

Oh, and in case you were wondering, the node names that appear on the logs are based on marvel characters, e.g.:

[2011-02-17 11:13:47,427][INFO ][node ] [D-Man] {elasticsearch/0.14.4}[6212]: stopping …

[2011-02-17 11:13:53,516][INFO ][node ] [Sublime, John] {elasticsearch/0.14.4}[5520]: initializing …

– RGL

Fix SyntaxHighlighter to display tumblr Markdown URLs

Sun, 12 Dec 2010 20:30:00 +0000

I write these tumblr blog posts using the Markdown syntax, which automatically detects and transforms URLs into proper HTML links. Though, things get messed up when I syntax highlight them with SyntaxHighlighter.

I see something like (as displayed by the browser):

http://example.com

instead of the expected:

http://example.com

So I dug into the code, and come up with this patch (apply into revision b7578b438a69):

--- a/scripts/shCore.js
+++ b/scripts/shCore.js
@@ -286,7 +286,6 @@
  highlight: function(globalParams, element)
  {
    var elements = this.findElements(globalParams, element),
-     propertyName = 'innerHTML', 
      highlighter = null,
      conf = sh.config
      ;
@@ -322,7 +321,7 @@
          continue;
      }
      
-     code = target[propertyName];
+     code = target.innerText || target.textContent;
      
      // remove CDATA from <SCRIPT/> tags if it's present
      if (conf.useScriptTags)

The text to highlight is now obtained from the innerText (works on IE / Chrome) or the textContent (works on Firefox / Chrome) property.

I’m left wondering why innerHTML was used in the first place… anyway, it now works as I expected!

– RGL