An Unusual Partnership and Bitmap Printing Has Resulted in a Solution for Pediatric Epilepsy Surgery

While TV shows and movies often portray the isolated genius as someone who can save the world by working alone, collaboration is necessary to accomplish the most extraordinary projects. Conferences are a great way to discover these collaborations. One notable example is actually shown by two of the keynote speakers at AMUG 2023, Nicholas Jacobson, a professor at the University of Colorado’s Anschutz Medical Campus, and Rob Ducey, a technical supervisor at animation specialist Laika Studios. At AMUG 2019, they met and fell in love with bitmap printing, a relatively new 3D modeling and printing technique. Now they’re working together to make innovative designs for pediatric cardiology, epilepsy, and cleft palates. We spoke with them to find out more.

3DN: Please introduce yourself.

Nick: Hi, I’m Nick Jacobson and I’ve been involved in 3D printing since 2004. My background is in architecture and I started 3D printing at an architecture school. I used binder jetting with refined cornstarch on a Zcorp printer. We were making architectural and landscape models at the height of the computational design boom. As students, we were just beginning to learn to program and code. We became fascinated by digital fabrication which allows us to translate these ideas into unusual forms.

My passion for form generation stayed with me throughout my architectural career. I worked for great architects like Zaha Hadid and Renzo Piano. After many years of practicing architecture, I was admitted to the Harvard Graduate School of Design as well as the Harvard Business School. Here, I was part a pioneering research program that looked into Voxel Printing using what was then Connex or Stratasys. Voxel technology was used to develop software that could model and 3D print. The initial focus was on design and structural engineering.

Currently, I am a researcher at the University of Colorado Anschutz Medical Campus. My lab focuses on 3D printing and computation for medical applications. Our Harvard software has been useful in the medical field. All it takes is to modify the input data. It may seem like a stretch but it’s very similar to working with structural analysis data. Currently, we are focused in 3 areas: Surgical guidance and education models, Prosthetics, and Implants… all revolving around soft tissue applications.

Rob Ducey (left), and Nick Jacobson (right).

Rob: Hello, I’m Rob Ducey and I’m currently the technical supervisor for what we call rapid prototype at LAIKA. Our group focuses on modeling, facial animation and fabrication of complex puppets, which are the stars in our films. The moniker “rapid prototype” is a legacy of the innovations we pioneered on our first film Coraline back in 2009 using 3D printing for facial animation. In 2016, this work was awarded a sci-tech Oscar. When Brian Mclean and Martin Munier arrived at the studio with the bold idea to use Objet polyjet printers in order to produce the many faces needed to create the facial performances for the characters in the film, I was immediately drawn to 3D printing. The task of building a pipeline to model and animate these faces, as well as print them, was my responsibility. Since that first production, we have innovated our processes to take full advantage of 3D printing with multiple materials. As a humble major in fine art, I get to play with the most advanced technology in computer-aided fabrication.

3DN: Your two disciplines are very different. How did you get together?

Nick: When LAIKA was the keynote speaker, we met at AMUG years ago. Their presentation was incredible and demonstrated how bitmap printing technology allowed them to create so much of their work. It was incredibly cathartic to see others using this esoteric technology. I was in awe of them and didn’t have the guts to chat with them eighth away. But, Brian and Rob Ducey later came to my talk on bitmap printing and voxel modeling for surgery. After the talk, they were the ones who came up to me first and talked about bitmap printing. This was a mutual love that we shared for bitmap printing. Next, we connected online and discussed our individual approaches. It was obvious immediately that we had simple solutions for many of the issues the other wanted to tackle. We had been trying to figure out how to work with thin hair structures for quite some time. The special effects community had already developed solutions that made it easy to use tractography, which we had been working on for a while. We still recall many of the ideas that the special effect world has been working on for years. They allow for advances in medical applications that no one else has seen. It was a simple connection that allowed for these conversations.

Rob: We are constantly on the lookout for technical innovations and new ideas to enhance our abilities and redefine stop motion animation. In our use of multi-material printing, we discovered a small community of like-minded tinkerers who had access to features still considered “in development”. We were able to visit universities such as Virginia Tech and MIT, and have discussions and visits with them. They were interested in what a group of filmmakers and artists working in a warehouse in the rain-soaked northwest was doing with these machines.  We were introduced to AMUG in 2015, and again in 2019, to deliver a keynote. AMUG is a gathering of people who are passionate about multi-material 3D printing. We always try to find others that can help us or share our excitement. I saw the description of Nick’s presentation and made sure we were there. We approached Nick excited to find someone in what was then a very small club of “voxel printers”. It was partly to share what we had learned but also to discover his secrets. It was like meeting someone from a lost tribe.

Laika has been using 3D printing for years to create its films (photo credit: Laika).

3DN: Can you tell us more information about your project? Also, could you tell us more about bitmap printing?

Nick: The following areas have been the focus of our projects: Advancement of bitmap printing techniques. Multimodal data Fusion for Medical Applications – Soft Tissue 3. Parametric design of prosthetics.

Bitmap printing allows you to print directly from a stack or bitmap images. It’s very much like how an inkjet printer works, except in this case the pages are layers. It’s very simple and leverages how a polyjet printer natively works, we are are essentially bypassing the internal slicer of the printer and sending our own slices. These slices are used to control the droplet’s location. However, this means that we need to define each droplet and that’s harder. A majority, if not all of the 3D modeling software is surface mesh-based. This model represents objects according to their boundaries, and ignores any internal volumetric data. Volumetric information is important to be able capture in many applications such as medical or structural. So, in order to bitmap print, we use a different paradigm for modeling called ‘voxel modeling’ which models volumetric data. This allows us to convert image-based data such as DICOM data into a 3D printed object without losing any data. Another benefit of bitmap printing and voxel modelling is the ability to create a gradient of material through droplet mixing rates by dithering.

1. Bitmap printing has a major drawback. You have to create the software necessary to create the files. There is no way to make a bitmap print by yourself. We have been combining our various workflows to create new workflows that help us better understand how to create bitmap prints. There are many things that need to be considered and those that still need to been worked out. It is still difficult to predict the material mixing of mechanical and colors properties and often we get unexpected results. Because of the number of calculations required to digitally preview and forecast how a print will look, this is due to the complexity of computers. Our primary focus is to quantify color-based mixing ratios that include translucency and other mechanical properties.
2. Multimodal data Fusion is an integral part of voxel based modeling. Voxel modeling employs compositing in place of booleaning which makes the combination of many complex data sources easy and computationally cheap. Voxel modeling also allows volumetrically changing color without having to modify the form. These factors enable us to combine multiple data sources (both bitmap and vector based), into one model. This includes MRI, fMRI. PET, MEG. Tractography and implanted device. Our work has concentrated on creating workflows that allow us control each data source individually and combine all the sources into one holistic model.
3. Parametric design is a very common modeling technique in both architecture and is the ‘bread and butter’ of stop motion animation. We have been working to create ‘rigged’ modeling software which allows for the facile manipulation of complex forms over time. This work was primarily focused on designing prosthetics for patients that can be modified over time. This work combines medical data with stop motion animation software to create a stop-motion series of models for human prosthetics.

This project could be used to treat epilepsy in children (photo credit: Nicholas Jacobson and Robert Ducey).

Rob: Our first meeting was an opportunity to discover a lot about how to design volumetrically to print. We noticed some inherent features of the medical imaging process that led to volumetric data. The studio was working on ways to take volumetric data from a visual effects workflow, and convert it to multi-material print. The key to that process was “bitmap printing” or, as it has now come to be known as “voxel printing”. We had been focusing on the outer appearance of printed objects and not the layers that affect it. Fraunhofer’s Cuttlefish product was crucial to this. This enabled us to print from slices. This is the same as bitmap printing. We devised a way to generate slices from volumetric data. Nick’s work was especially interesting in the various modalities of medical imaging he was working with in DICOM format. These were very similar to slices and are fascinating to view in printed form. There were obvious ways that this data could be integrated into VFX pipelines and generated some interesting prints. We began to discuss ways that we could feed this data back into the research areas Nick and his team at the University of Colorado were studying. We also discussed how to leverage techniques and methods that have been used for animation and visual effects to aid ongoing medical research. LAIKA has a unique overlap because we need the same precision, scales and level of detail in parts we design and create as you would expect for medical applications. We offer insight on how to visually make something compelling.

3DN: What are your ideas for applications?

Nick: This technology has many potential applications and we have just begun to scratch the surface. We are currently focusing on two areas: Pediatric Epilepsy Surgery, and Cleft Palate. Pediatric Epilepsy Surgery is the best use of bitmap printing in preoperative planning and intraoperative guidance. The reason for this is that epilepsy is mostly physiologically based. To plan treatment, surgeons need to review many 2D-based volumetric data source individually and sequentially in order to develop a mental understanding. We plan to combine the many, growing, datasets into a holistic 3D model for planning and guidance. A prospective trial will be launched before the end the year to assess the effectiveness of such a model for treatment planning as well as post-operative outcomes.

The presurgical technique known as Naso Alveolar Molding (also called Naso Alveolar Molding) is our focus in treating cleft lips and palates. This is a common procedure for infants with cleft palate and lip. It can be done almost immediately after birth. This treatment involves attaching an acrylic device to the palate that directs growth towards closing the cleft. This eliminates the need for surgery and makes subsequent surgeries easier. Families will need to make adjustments every week, which means they will have to visit the clinic once a month for three to nine months. Even if you live nearby, this can be difficult. Unfortunately, access to this treatment is not available to many patients and there are very few providers. We are working to develop a 3D printed solution that allows patients to receive a set via the mail. This would enable anyone who can take a dental image to be eligible for this treatment.

Rob: Nick and his team are driving the research about how to apply what we’re learning in the field of medicine. This has provided clarity for the studio about both how to technically design with voxels as well as how biology forms and functions volumetrically, from the perspective medical experts. The unique ability of additive processes to use these concepts in biology to design is a strength.

The procedure could aid in Naso Alveolar Molding (for cleft palate patients), which is a condition in which there are holes at birth in the palate (soft-hard) or in the lips (photo credits to Sunrise Orthodontics).

3DN: Last words for our readers?

BothAMUG 2023: Come learn more about our project and about us as a whole. The keynote speech will be delivered by us. “Collaborations Between an Animator, an Architect, and a Surgeon: The Keys to Impactful Innovation in Medicine” Tuesday, 21 March. If you haven’t already, you can register for the conference HERE.

How do you feel Rob and Nick are doing their work? Do you believe 3D printing conferences are a key part of encouraging collaboration in this industry? Please let us know by leaving a comment or using our contact form. LinkedIn, Facebook?, and Twitter pages! Don’t forget to sign up for our free weekly Get the NewsletterGet the most recent news about 3D printing straight to your inbox You can also view all of our videos at our YouTube channel.

*Cover Photo Credits: Nicholas Jacobson and Robert Ducey