Saturday, January 30, 2010

JavaScript Pseudo Operator Hacks

Arrow operator, count down: -->

This hack is useful when iterating over an Array, albeit in reverse.

var foo = ["a", "b", "c", "d", "e"];
var i = foo.length;
while (i --> 0) {
    console.log(foo[i]);
}

Output: e d c b a

Spoiler: The arrow is a composite of -- and > operators. The i variable is post-decremented and then compared to 0 at each iteration.

Venus operator, convert to Number: +()

Exploit the duck typing nature of JavaScript to convert any literal string to a Number object. Enclose the value in the Venus operator's parentheses.

Example:

var foo = +("50"); // 50
var foo = +("050"); // 50
var foo = +("-50"); // -50
var foo = +("+50"); // 50
var foo = +("0x50"); // 80
var foo = +("0.50"); // 0.5
var foo = +("lol"); // NaN
var foo = +(50); // 50

Spoiler: The unary + and * operators in JavaScript convert to Number objects. The parentheses are optional and just add code bling. Conversion is done as base10 or base16. This is different from parseInt which also accepts implicit base8 for strings starting with "0". Hence parseInt("050") !== +("050") so be careful with octal values.

Note: This operator is similar to the Venus operator in Perl.

Truthiness operator, convert to Boolean: !!

This is another dynamic typing trick. Prepend !! to parse a value as Boolean true or false.

Examples:

var foo = !!"bar"; // true
var foo = !!"true"; // true
var foo = !!1; // true
var foo = !!0; // false
var foo = !!""; // false

But beware:

var foo = !!"false"; // true
var foo = !!"0"; // true

All String objects are converted to Boolean based on their length. Zero length strings are false while any other length is considered true, regardless of the string contents.

Spoiler: The two exclamation points are logical NOT operators. The right exclamation first converts the value to Boolean and inverts it. The left exclamation then inverts the Boolean again to produce the Boolean equivalent of the original value.

Scribe operator, convert to String: ''+

The Scribe is a simple hack to typecast any value to a String object. Useful when strict data types are required.

var foo = ''+ 123; // "123"
var foo = ''+ 0xDEADBEEF; // "3735928559"
var foo = ''+ new Date(); // "Sat Jan 30 2010 02:30:00 GMT+0100"

Spoiler: JavaScript evaluates the expression from left to right as an empty string and then adds the value by converting it to a String object by calling its toString method.

Match Maker operator, assign default arguments: ||

This is more of a code pattern than a pseudo operator but still a useful trick. It is often used in constructors when declaring the members of a class.

var MyFoo = function (prop) {
    this.bar = prop || "1337 h4x0r";
    this.toString = function () {return this.bar};
}
alert(new MyFoo()); // "1337 h4x0r"
alert(new MyFoo("js rocks")); // "js rocks"

Spoiler: The logical OR operator evaluates from left to right and short circuits. As soon as one operand evaluates to true it will return its value (the actual value, not the Boolean true). If the left hand operand evaluates as false then the value of the right hand operand will be assigned. Sounds complex but it's a really neat trick to make object declarations more readable.

Wednesday, January 27, 2010

Flash Hardware Acceleration for H.264

The HTML5 <video/> element is slowly but surely being supported by Firefox, Safari and Chrome. Youtube and Vimeo are experimenting with the open standard as well. But browser based video still relies, de facto, on the Adobe Flash Player. So here's how to make Flash suck less at high definition (720p/1080p) full screen video.

High definition video playback requires a lot of processing power but it's also an embarrassingly parallel task. If the machine has a dedicated graphics processing unit (GPU), either Nvidia or ATI, then Flash 10.1 beta 2 or later can use the GPU's massive power to decode H.264 high definition video. This reduces the load on the CPU and enables smooth full screen high definition video playback.

Upgrading Flash Player

  1. Uninstall the old Flash player through the Control Panel > Programs > Uninstall a program
  2. Download Flash Player 10.1 beta 2 or later from Adobe Labs.
  3. Close Firefox and Chrome
  4. Run the Flash Player setup

Upgrading Graphics Drivers

  1. Download the latest drivers for your GPU. ATI Catalyst 9.12 or later, or the Nvidia drivers 195 or later.
  2. For laptops with an ATI Mobility GPU use the Driver Modder utility to patch the Catalyst drivers.
  3. Uninstall your old drivers through Control Panel > Programs > Uninstall a program
  4. Install the new drivers
  5. Reboot

Enjoy!

Now compare the CPU usage in the Windows Task Manager. Even on a modest dual core laptop CPU usage should not go over 30-40%. Without hardware acceleration the frame rate would have been choppy with CPU usage completely maxing out one core. So enjoy watching silky smooth 1080p videos in the browser!

Thursday, January 7, 2010

Nginx Configuration and Popular Web Applications

The Pandion website runs on a Slicehost VPS with just 256 MB RAM. A few weeks ago I moved it from Apache2+mod_php to Nginx+fcgi_php+xcache. The result has been a drastic improvement in performance and stable memory usage.

Why Apache Cannot Scale

Apache handles each connection in a separate thread. But too many threads in a process slows down performance, so Apache spawns more processes. Each process however uses a lot of memory as it includes all enabled Apache modules including the PHP interpreter. This adds up to 2-5 MB per process and sometimes over a hundred processes may be running. At least once a month Apache would require so many processes that the server ran out of free memory and simply crashed. The primitive solution would be to add memory, but that costs money and only works to a point. The elegant solution lies in a more modern networking API.

Why Nginx Can Scale

Nginx uses epoll which can handle huge numbers of connections within a single process by using asynchronous I/O capabilities in the Linux 2.6 kernel. Slicehost VPS has two dual core Opteron CPUs so I run four Nginx worker processes, four fcgi_php workers and four MySQL workers. The server is now happily humming along at a stable 195-205 MB memory usage. The only fluctuation is from caching in MySQL and Xcache/fchi_php. CPU load is nigh constant at a ridiculous 0.00. Awesome!

DIY Configuration

One downside of Nginx is that there are no official, off-the-shelf configuration scripts for Drupal, phpMyAdmin, phpBB and other popular web applications. Luckily Nginx has a configuration scripting language that is much more sane than Apache's. So I created my own Nginx configurations for each of these tools.

Nginx Build Settings

The VPS runs Ubuntu 9.10 (Karmic Koala) which offers Nginx 0.7.62 through apt-get. Because Nginx is still in beta and development is extremely active I chose to build it by myself from the latest (as of this writing) 0.8.31 source code. I won't go over this process as there are many good tutorials available. But for reference, these were the build arguments:

./configure \
 --prefix=/usr \
 --sbin-path=/usr/sbin/nginx \
 --conf-path=/etc/nginx/nginx.conf \
 --pid-path=/var/run/nginx.pid \
 --lock-path=/var/lock/nginx.lock \
 --error-log-path=/var/log/nginx/error.log \
 --http-log-path=/var/log/nginx/access.log \
 --http-client-body-temp-path=/var/lib/nginx/body \
 --http-proxy-temp-path=/var/lib/nginx/proxy \
 --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
 --user=www-data \
 --group=www-data \
 --with-http_gzip_static_module \
 --with-http_stub_status_module \
 --without-select_module \
 --without-poll_module \
 --with-cpu-opt=opteron \
 --with-md5-asm \
 --with-sha1-asm \
 --with-zlib-asm=pentiumpro \
 --add-module=/home/sebastiaan/agentzh-headers-more-nginx-module-db9913e

Notice that the last line includes a custom module called headers-more. Nginx currently does not offer very good built-in control over HTTP output headers. I wanted to strip various headers from the HTTP response (Server, X-Powered-By, Date, Expires, Last-Modified) and use the Cache-Control header to fine-tune proxy server and web browser behaviour.

Drupal

Many Drupal sites already run on Nginx. In fact even Acquia, the Drupal consulting firm by Drupal founder Dries Buytaert, provides high performance Nginx-based Drupal hosting. Because there are many ways to use PHP with Nginx it's not easy to create a 100% compatible configuration script for Drupal. But this configuration works for me:

server {
 listen 80;
 # Strip "www." from the URL and redirect the old domain
 server_name www.pandion.im pandion.be www.pandion.be;
 rewrite ^/(.*) $scheme://pandion.im/$1 permanent;
}

server {
 listen 80;
 server_name pandion.im;
 root /home/sebastiaan/sites/pandion.im;
 index index.php index.xml index.html;

 # Serve static file
 try_files $uri @drupal;

 # Clean URLs for Drupal
 location @drupal {
  rewrite ^/(.*)$ /index.php?q=$1 last;
 }

 # Serve PHP scripts
 location ~ .php$ {
  # Prevent caching of admin panel
  if ($query_string ~ q=admin) {
   more_set_headers -s 200 "Cache-Control: no-cache, no-store, max-age=0";
  }
  # Forward to FastCGI daemon
  fastcgi_index index.php;
  include fastcgi_params;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
  fastcgi_pass 127.0.0.1:10005;
 }
}

phpMyAdmin

This is my Nginx port of the default Apache .htaccess file used by phpMyAdmin. It enables PHP, prevents caching and implements security measures to prevent access to unauthorised resources.

server {
 listen 80;
 server_name phpmyadmin.pandion.im;
 root /usr/share/phpmyadmin;
 index index.php;

 # Serve PHP scripts
 location ~ .php$ {
  # Prevent caching
  more_set_headers -s 200 "Cache-Control: no-cache, no-store, max-age=0";
  # Forward to FastCGI daemon
  fastcgi_index index.php;
  include fastcgi_params;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
  fastcgi_pass 127.0.0.1:10005;
 }

 # Authorize for setup
 location /setup {
  auth_basic "phpMyAdmin Setup";
  auth_basic_user_file /etc/phpmyadmin/htpasswd.setup;
 }

 # Disallow web access to directories that don't need it
 location /libraries {
  deny all;
 }
 location /setup/lib {
  deny all;
 }
}

phpBB

The archived forums of Pandion are locked and running the outdated phpBB 2. I did not test this with phpBB 3 so caveat emptor.

server {
 listen 80;
 server_name forums.pandion.im;
 root /home/sebastiaan/sites/forums.pandion.im;
 index index.php index.htm;

 # Serve PHP scripts
 location ~ .php$ {
  # Prevent caching of dynamic content
  more_set_headers -s 200 "Cache-Control: no-cache, no-store, max-age=0";
  # Forward to FastCGI daemon
  fastcgi_index index.php;
  include fastcgi_params;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
  fastcgi_pass 127.0.0.1:10005;
 }

 # Security
 location /cache {
  deny all;
 }
}

CoralCDN

The free (as in freedom and beer!) Coral Content Distribution Network is used to deliver the Pandion installer. CoralCDN automatically caches the file on any of hundreds of webservers distributed globally. Users automatically download from the Coral server nearest to them.

This configuration is ported from the official Apache rewrite rules. Requests for the file are forwarded to CoralCDN by appending ".nyud.net" to the hostname. Seeding requests from CoralCDN servers, identified by the user agent header, should be handled by Nginx. If CoralCDN cannot handle the request (eg. bandwidth limit reached) it will bounce the request back to us with "coral-no-serve" appended to the query string, and Nginx should serve the file directly to the client.

# CoralCDN caching of the download package
location = /pandion_setup.msi {
 if ($http_user_agent ~ ^CoralWebPrx) {
  break;
 }
 if ($query_string ~ (^|&)coral-no-serve$) {
  break;
 }
 rewrite ^/(.*) $scheme://$host.nyud.net/$1? redirect;
}

Saturday, October 24, 2009

Draw your own conclusions

Entrepreneurs in Kunshan, China, 24/10/2009
3G Industry Summit
3G Industry Summit

Entrepreneurs in Brussels, Belgium, 24/10/2009
Bizcamp
Bizcamp

Lest we forget...
Entrepreneurs in Brussels, Belgium, anno 1911
Solvay Conference
Solvay Conference
(Curie, Poincaré, Rutherford, Einstein, ...)

Friday, October 16, 2009

JavaScript Parser for RFC 3339 or ISO 8601 Timestamp Strings

A few days ago I wrote writing a generator for Atom 1.0 timestamps, which are defined as an RFC 3339 profile, which is defined as an ISO 8601 profile. Now I had to parse these strings in another application for chronological sorting. There are several implementations published already but for various reasons they are all lacking. Either the code base has huge dependencies, licensing is unclear, the implementation contains bugs, and so on. So why not invent a rounder wheel?

This is what I came up with:

/*
Internet Timestamp Parser
Copyright (c) 2009 Sebastiaan Deckers
License: GNU General Public License version 3 or later
*/
Date.prototype.setISO8601 = function (timestamp) {
var match = timestamp.match(
"^([-+]?)(\\d{4,})(?:-?(\\d{2})(?:-?(\\d{2})" +
"(?:[Tt ](\\d{2})(?::?(\\d{2})(?::?(\\d{2})(?:\\.(\\d{1,3})(?:\\d+)?)?)?)?" +
"(?:[Zz]|(?:([-+])(\\d{2})(?::?(\\d{2}))?)?)?)?)?)?$");
if (match) {
for (var ints = [2, 3, 4, 5, 6, 7, 8, 10, 11], i = ints.length - 1; i >= 0; --i)
match[ints[i]] = (typeof match[ints[i]] != "undefined"
&& match[ints[i]].length > 0) ? parseInt(match[ints[i]], 10) : 0;
if (match[1] == '-') // BC/AD
match[2] *= -1;
var ms = Date.UTC(
match[2], // Y
match[3] - 1, // M
match[4], // D
match[5], // h
match[6], // m
match[7], // s
match[8] // ms
);
if (typeof match[9] != "undefined" && match[9].length > 0) // offset
ms += (match[9] == '+' ? -1 : 1) *
(match[10]*3600*1000 + match[11]*60*1000); // oh om
if (match[2] >= 0 && match[2] <= 99) // 1-99 AD
ms -= 59958144000000;
this.setTime(ms);
return this;
}
else
return null;
}


Examples:

var foo = (new Date()).setISO8601("2009-10-16T20:11:36.456-07:00");
// Date: Sat Oct 17 05:11:36 UTC+0200 2009

var bar = (new Date()).setISO8601("-0448-05-27T05:52:08.97+11:30");
// Date: Mon May 26 20:51:38 UTC+0200 449 B.C.


Note:
This method can parse any RFC 3339 timestamp, but does not enforce strict validity. It can handle dates that are not valid RFC 3339 such as negative years or years with more than four digits. It also accepts lowercase "T" and "Z", as described in ISO 8601, and shows other minor leniencies. Do not use this method as a validator.

*Edit*
Fixed Firefox support.

*Edit*
Fixed offset minutes calculation.

Tuesday, October 13, 2009

Generating RFC 3339 or ISO 8601 Timestamps in JavaScript

I've been working on a generator for a custom Atom 1.0 feed. The Atom 1.0 specification (RFC 4287) requires Internet Timestamps (RFC 3339) which are based on ISO 8601 but these are not supported by the JavaScript Date object. Failing to find a drop-in JavaScript snippet which could generate the necessary timestamp format, I created the following function. It generates the Internet timestamp string from either the current time or a JavasScript Date object passed as an argument.

/*
Internet Timestamp Generator
Copyright (c) 2009 Sebastiaan Deckers
License: GNU General Public License version 3 or later
*/
var timestamp: function (date) {
var pad = function (amount, width) {
var padding = "";
while (padding.length < width - 1 && amount < Math.pow(10, width - padding.length - 1))
padding += "0";
return padding + amount.toString();
}
date = date ? date : new Date();
var offset = date.getTimezoneOffset();
return pad(date.getFullYear(), 4)
+ "-" + pad(date.getMonth() + 1, 2)
+ "-" + pad(date.getDate(), 2)
+ "T" + pad(date.getHours(), 2)
+ ":" + pad(date.getMinutes(), 2)
+ ":" + pad(date.getSeconds(), 2)
+ "." + pad(date.getMilliseconds(), 3)
+ (offset > 0 ? "-" : "+")
+ pad(Math.floor(Math.abs(offset) / 60), 2)
+ ":" + pad(Math.abs(offset) % 60, 2);
}


Examples:
timestamp();
// "2009-10-13T13:34:52.000+02:00"

timestamp(new Date(1983, 4, 9, 11, 5, 17));
// "1983-05-09T11:05:17.000+02:00"


Hope this helps anyone.

Wednesday, August 12, 2009

High Dynamic Range Photography

Yesterday I visited the Bulskampveld provincial domain. It is a forest area with a 19th century chateau that currently acts as nature museum and clinic for injured birds. I took a series of regular photos which are over in my Picasa gallery.

More interesting is playing with my camera's automatic exposure bracketing to create high dynamic range (HDR) photos. Cameras cannot capture accurate information from the brightest to the darkest areas in a single exposure. That is why either shadow areas are fully black or highlights are blown out as completely white. Either a short exposure can provide detail in the bright areas or a long exposure gives detail in the shadows. Now with HDR you can get overall detail from shadows to highlights.

Multiple exposures:


HDR image:


The process:
1. Mount the camera on a tripod or place it on an immobile surface.
2. Take multiple photos at the same ISO and aperture but vary the shutter speed.
3. Merge the photos using HDRI editing software.
4. Tone-map the HDR image into an 8-bit JPEG that can be viewed on screen or printed.

Automatically taking multiple exposures can be done using the camera's auto bracketing feature. Place the camera in aperture priority mode ("Av" on Canon models). Look in the settings menu for automatic exposure bracketing (AEB). This is a screenshot of a Canon 450D and it will be different on other models. Taking the exposures manually allows more control over the range and also lets you combine more than three shots.

Ensure that the camera remains in the same position between exposures. If there is movement of the subject you may also see a ghosting effect when the images are merged.

Store the photos in RAW format. This prevents information loss due to the camera's automatic post processing and JPEG compression.

Several software tools exist to merge and tone-map HDR images. Tone mapping is the process of converting the highly precise image to a lower precision image. This is necessary because monitors cannot handle more than 24 bit pixels: 8 bits each for red, green and blue. Our HDR image contains much more accurate pixel values. In this case 3 exposures of 3 times 14 bits per colour would give 126 bits per pixel. Tone mapping takes a subset of the contained information and renders a simple 24 bit RGB image. There are many different techniques ("algorithms") for determining this subset, each producing a different effect from neutral or realistic to highly stylised.

I tried Photoshop CS2 (File > Automate > Merge to HDR) which is fairly limited. The free and open source Qtpfsgui, despite its tongue twisting name, is much more powerful. It features many different tone-mapping algorithms and precise control over their settings. Keep in mind that this is a highly geeky app. It's a bit of a pain to get it working, the user interface is cryptic and intimidating, and crashes happen randomly.

Example exposures:


Result using Photoshop CS2:


Result using Qtpfsgui with the Mantiuk algorithm: