The challenge was to create a simulation that trained medical practitioners how to guide a catheter into the right side of a patient's heart, then make a precise puncture. The target is the middle of the heart so that surgeons may gain access to the left side of the heart.
Ryan explained this common procedure prevents surgeons from threading a catheter through the heart, into the aorta to the left chamber. It's simpler to puncture through the middle. We wanted to understand how augmented reality could improve this procedure.
We set out to create a simulation that will provide a better look and practice, with a hologram of the heart. Dozens of xrays are usually conducted during the procedure, injecting dyes into the tissue for visual feedback. An advantage of the Hololens is that it can display a 2D close up and 3D view simultaneously. This was an early team UI decision. We also knew the Hololens would be great for monitoring vitals and managing patient information privately.
Ahmad Aljadaan and I set out to design the audio cues early on so that our voice actor, Bridget Swirski could record them. We quickly began to realize the importance of these cues in the user interface. The audio cues guide most of the experience, creating feedback for the user when they are on and off track in the procedure, and giving them directions as they guide the catheter and needle through the inside of the heart and arteries. We chose a female voice, as suggested in a lightning talk by the Hololens spatial audio design team. The human voice has excellent spatial recognition in the AR environment.
These audio cues helped refine our approach into the experience storyboard:
We knew the end user would need a granular method of controlling the catheter and needle. OSC seemed like a great choice since it's a low latency TCP protocol used for musical instrumentation. It works with iOS and Android already, and I found some packages for Unity for us to modify.
We crunched and coded to the very beginning of our presentation. The scope of our project was a little big for the weekend, and we did a lot. Sometimes the scope of your project outgrows the hackathon and it certainly did this time.