Berkeley startup helps hospitals see medical mistakes with machine learning and object recognition

According to their research, the cost incurred by the misplacement of surgical tools — whether left inside patients or discarded — amounts to about $2 billion yearly in the U.S. alone. Armed with deep insight from their personal experiences in hospitals, the group decided to use artificial intelligence (AI), machine learning and computer vision to identify, count and track instruments in the operating room (OR) and anywhere on property.

As they developed a prototype system, the power of the idea won many supporters in technology and academic circles — as well as numerous prestigious awards and honors. They competed in Berkeley’s annual Big Ideas contest, earning a second-place award in May 2018 in the Hardware for Good category. They also were 2018 graduate finalists in the Collegiate Inventors Competition, sponsored by the National Inventors Hall of Fame, taking home the Innovation Prize from Arrow Electronics.

The cost of 20 minutes

The inspiration for their game-changing technology bloomed from the team’s extensive first-hand experience: All four founders are medical engineers and, collectively, have put in a lot of time observing hundreds of surgeries.

“I’ve seen quite a few cases as a medical student where the medical staff completed the surgery, closed the patient and were ready to clean up the OR for the next case — but something was missing,” said Robert Kim. “In one case, the nurses were counting instruments, and then they just couldn’t find one sponge. They looked for it everywhere — from the floor to the trash can. That led to a 15- to 20-minute delay before they found the sponge and were able to clean out the OR for the next case.”

Twenty minutes may not sound like a long time, but in a surgical environment, that brief window can have major impact on patients. It also adds up to significant operational costs, with hospitals seeing as much as $150 lost for every minute — that’s every 60 seconds — that an operating room sits unused.

The Skittles paradox

“While observing so many surgeries, we realized that the counting and logging process for instruments is extremely nagging,” Neira said. “It requires multiple people to cross-check — double-counting, essentially, to make sure that nothing is left behind.” A crucial step to be certain, but not necessarily the best value-added use of highly qualified surgical staff.

Yet the deeper issue is the risk. Upwards of 70 instruments are involved in an average surgery. Multiply that by many surgeries and many staff members, sorting and logging many tools, and from a risk-management perspective you’ve just cooked up a potentially sour recipe. Human skill and decision-making are second to none, but for routine tasks, sometimes human limitations can have a ripple effect. 

“The best way I can describe this is to imagine that I gave you a bag of Skittles and asked you to count the number of candies,” Neira said. “And then imagine I asked a second person to count the Skittles in that same bag.” Now, imagine this happening at a rapid pace, repeatedly, in the extreme pressure of a surgical setting. 
“In all likelihood, the two counts would be different. And so, in the operating room, this means that there has to be yet another count of surgical instruments by two more people.”

VIDI feels the cross-checking process creates distractions, eats up time and contributes greatly to a high incidence of surgical mistakes. 

Augmented peace of mind

With the challenge at hand, the team moved into research and development: an augmented reality platform to give surgeons peace of mind before stitching anyone up. They met with hospital staff, mined pain points across the campuses, observed how hospitals keep inventory and more. And meticulously logged the flow of all instruments — down to the individual scalpel — through each space, from where they are prepared, sterilized and packaged, to the OR and all the way back home.

It may sound tedious, but it’s precisely within this workflow that a lot of time and property is lost, commented Álvarez del Blanco. “More often than not, the instruments are simply thrown away,” he added frankly. 

VIDI believes image recognition is superior to alternative solutions on the market, including laser-engraving and radio-frequency-identification (RFID) technologies. For their use, the technology offers lower costs and greater durability and doesn’t require extensive retrofits of hospital systems. 

“What we envisioned was something along the lines of a supermarket scanner, where the user can place an instrument on a surface and it is recognized via a mounted camera,” Neira said. 

With the form in place, they turned to the bigger challenge, and greater opportunity, in the data. 

Training travails

Serving up a machine-learning library of surgical instrument data is VIDI’s core value add. The team envisioned multiple uses for the library, from tracking tools during individual surgeries to monitoring instrument utilization across clinic settings to improve efficiency and cut costs. 

There was just one small problem when they started out: That data did not exist. Anywhere. So they built their own extensive data set from scratch.

Training a machine-learning algorithm is notoriously difficult and time-consuming. To begin, they had to obtain physical examples of instruments to serve as data sets. A single tray full of surgical instruments can cost as much as $10,000. For a four-person startup, the costs to actually purchase could be a show-stopper. 

“Whenever we go out and talk to people … everyone is very enthusiastic,” Álvarez del Blanco noted. “Everyone in the field recognizes the problem and the fact that this is something new. We’ve had a great reception and people really want to help out and lend us their expensive instruments.”

Another obstacle lies in the surgical instruments themselves, with different tools often having a very similar appearance, making them hard to distinguish from each other.
“The real impact of what we are doing is the library of surgical instrument data that didn’t exist before,” Álvarez del Blanco added.  “So, to train those algorithms we need to collect videos, break down into frames and annotate then for specific instruments. This is something we are doing internally for now, but we’re looking at other approaches for scaling up the effort."

“Once we manipulate data on 70 types of instruments, and 20 different variants within a type, which, to our eyes, are very similar, there start to be a lot of issues.” Issues such as identifying which brand a scalpel is — or whether it’s a 1-34 or a 1-33. “This makes it very challenging.”

It’s a labor of love, taking as much as 60 minutes to complete all the steps required to enter just one instrument into the database.

Vision for the future

VIDI is considering running its innovation on Indiegogo, the global crowdfunding leader that partners with Arrow to help entrepreneurs bring ideas from concept to market. They’re also looking at applications for their machine-learning library beyond the operating room — for example, instrument-rental companies across a variety of industries that need to keep tabs on property in the field. And they see a future when the data could enhance robotic surgery by helping automatons find the right tools for the tasks at hand.

Up first, they plan to test their prototype in an actual hospital setting. Initial deployment will be in sterilization rooms, though long term they want to develop and place products in the operating room.