I'm not certain, but I think this project is relevant to yours. http://code.google.com/p/intrael. In short, it's a fast portable application server that uses a kinect to do blob tracking and serve detailed object info in the form of JSONP. It's not(just) multitouch, it's lower level than the Apple's Touch api. It's up to the JS dev to do things like labelling and following objects between frames etc I'd appreciate some review and suggestions especially on the JS part