How to set up your iPad with a Kinect motion sensor to work as a gesture based instrument

Find out how the Klezmer preset on Gestrument Pro transformed into a hand motion real-time playable instrument as seen in the video with Martin Fröst and Jesper Nordin. In this post Jesper shares how the setup was made.


Writer: Jesper Nordin
Jesper Nordin is a leading Swedish composer who has garnered considerable international acclaim in recent years. Jesper Nordin has also had huge success with his iOS app Gestrument that is based on his own composition technique.


Jamming with Martin Fröst

Jamming with one of the leading classical musicians in the world is a great test to see if a music technology can hold its ground!

When playing together with Martin Fröst the iPad with Gestrument Pro was connected to a laptop and used as a MIDI controller to Logic, where it played on the Dulcimer sound in the ESX24 sampler.

Martin Fröst plays on a Buffet/Crampon clarinet, as always.

Klezmer jam session with Martin Fröst and Jesper Nordin on Gestrument Pro

 

Connect your iPad to your Mac

To connect your iPad to your laptop you only need to plug in the USB charger. If you then go to the Audio/MIDI settings on you Mac you can enable IDAM (Inter-Device Audio and MIDI).

How to enable USB Audio and MIDI for iOS 

How to record digital audio from your iPhone to your Mac with iDAM

Use a Kinect motion sensor as MIDI signal

In this setup we used Gestrument’s internal beta software for Mac and PC that can translate the signal from a Kinect sensor into MIDI CC.

We will try to release this software to the public in one way or another as soon as possible, please email us at info@gestrument.com if you want to be notified when we do.

The musical preset played in real-time

Finally, the Gestrument PRO Klezmer preset (included in the Gestrument Pro 1.2 release) was used and performed together with Martin Fröst on a song he has played many times before. Martin’s brother Göran Fröst has made an amazing arrangement of the song that appears on this Sony Classical album.

In the video with the Klezmer preset it is used as a chord sequence, but only one part of the song is used – the fast section in the beginning, which you can find at 01.20 in the Spotify link. The rhythmic pattern of the string pizzicatos from Göran Frösts arrangement is used as a foundation to the rhythmic part.

 

Do you wanna try the Klezmer preset yourself? Download the free trial of Gestrument Pro on the App Store and get started!