Wheelchair Detection for Mobile Robots

Bo, J, Van der Loos, HFM (2021). Detection of Wheelchair Orientation in Human-Robot Interactions. In proceedings of International Conference BIOMDLORE 2021.

Abstract:

Autonomous mobile robots are being introduced in human-populated environments with increasing frequency. Hence, ensuring safe and intuitive human-robot interaction (HRI) is becoming a growing need, especially for less common pedestrian groups like wheelchair users. Wheel- chair dynamics differ from those of foot pedestrians, so accurate characterization of a wheelchair’s orientation for trajectory prediction is crucial. The 2D laser scanner is a well-suited sensor for accurate distance measurements with fast computation speeds, but the sparsity of its data is often a hindrance to effective object detection. Despite so, we demonstrate that 2D range data from laser scanners can be used effectively to detect wheelchairs and estimate their orientation, even in cluttered environments. We pre-process the range data by using density-based clustering for object segmentation. Our two-step classification algorithm first identifies wheelchair candidates from segmented objects with random forest, then estimates the wheelchair’s orientation as one of six classes with a neural network.

Responsibilities:

• Proposed original research project in 3D vision and carried out experiments on a robotic platform
• Collected data for machine learning models and evaluated performance

Awards:

• UBC Multi-Disciplinary Undergraduate Research Conference - Top Oral Presentation

Adversarial Masking for ECGs

Accepted as non-arxival extended abstract to:

Bo, J, Huang HW, Chan A, Traverso G (2022). Pretraining ECG Data with Adversarial Masking Improves Model Generalizability for Data-Scarce Tasks. In accepted abstracts of Machine Learning for Health Symposium 2022.

Bo, J, Huang HW, Chan A, Traverso G (2022). Adversarial Masking for Pretraining ECG Data Improves Downstream Model Generalizability. In proceedings of Learning from Time Series for Health Workshop at NeurIPS 2022.

Abstract:

Medical datasets often face the problem of data scarcity, as ground truth labels must be generated by medical professionals. One mitigation strategy is to pretrain deep learning models on large, unlabelled datasets with self-supervised learning (SSL), but this introduces the issue of domain shift if the pretraining and task dataset distributions differ. Data augmentations are essential for improving the generalizability of SSL-pretrained models, but they tend to be either handcrafted or randomly applied. We use an adversarial model to generate masks as augmentations for 12-lead electrocardiogram (ECG) data, where masks learn to occlude diagnostically-relevant regions. Compared to random augmentations, models pretrained with adversarial masking reaches better accuracy under a domain shift condition and in data-scarce regimes on two diverse downstream tasks, arrhythmia classification and patient age estimation. Adversarial masking is competitive with and even reaches further improvements when combined with state-of-art ECG augmentation methods, 3KG and random lead masking (RLM), demonstrating the generalizability of our method.

Awards:

• IEEE Computational Intelligence Society Graduate Research Grant
• Zeno Karl Schindler Foundation Master Thesis Grant
• Swiss-European Mobility Scholarship
• Heyning-Roelli Foundation Mobility Grant

Worldloop Transfer: Embodied Perception

Abstract:

How do the perceptual abilities developed by embodied agents correlate to the active behaviours they execute in the 3D world? Parallels between computational actors, namely reinforcement learning (RL) agents, can be drawn with ecological agents through the principle of the action-perception-loop (APL), which describes the interlinked relationship between learning to see and learning to act.

We investigate how the APL manifests in RL agents trained on a variety of embodied tasks with different perceptual requirements, including motor movements, visually-guided navigation, and localization. In the absence of a holistic perceptual evaluation method, we use depth estimation as a substitute task to evaluate the agent's learned visual encoder post-training. Our preliminary results find that agents with smoother policies encode significantly better depth perception abilities than a scratch encoder, but performance differences between task variations are difficult to quantify given inconsistencies in training agents to convergence.

As a further step, we explore if APL can be applied to address technical challenges in RL, such as sample efficiency in training. We initialize agents with pretrained depth visual encoders and repeat their training to compare with the scratch agents. We find that some embodied tasks which require visual information saw benefits in the jumpstart reward performance, but definitive results are limited by agents overfitting to the training environments. We make a list of recommended changes to the technical setup of the experiments and suggest future directions.

HoloLens Mesh Alignment

Abstract:

In the field of 3D vision, registration is necessary to align two different point clouds with different reference frames. The motivation for this problem is to integrate similar features from different 3D registrations or time steps. Given a point cloud X and another Y , the goal is to find the transformation, f, that maps the reference frame of X to that of Y so as to minimise the transformation error and align the points as closely as possible.

Meshes generated by the Microsoft HoloLens and other similar recording devices register on different coordinate systems with each subsequent scan. As a result, it is a nontrivial problem to accurately, efficiently, and robustly align these meshes. Applications for mesh registration include Virtual Reality (VR) and Augmented Reality (AR), in which the stitching together of distinct meshes is required to match features taken at different points in time.

Our main objective is to implement a mesh registration pipeline based on the Guaranteed Outlier Removal (GORE) algorithm, developed by Bustos and Chin.We seek to refine GORE with the Iterative Closest Points (ICP) and RANSAC algorithms, and compare the performance of the GORE pre-processing step in relation to the subsequent registration algorithms.

QNX: Obstacle Detection with LiDAR

Motivation:

LiDAR (light detecting and ranging) devices are commonly used sensors in the automotive market, especially for the development of autonomous vehicles and ADAS (advanced driver-assistance systems). They are reliable for detecting distances of objects in the 3D world, which is useful for computing the location of obstacles and “free space” on the road. Using the speed of light as a reference, LiDARs emit a laser beam and measure the amount of time it takes to receive a return signal, which enables the sensor to perform highly accurate distance calculations. For the purposes of the QNX Advanced Development team, I investigated the usage of LiDAR data for real-time obstacle and free space detection.

Attentiv Catheter for IV Treatment

Bo, J, Ta, K, Nishida, R, Yeh, G, Tsang, V, Bolton, M, Ranger, M, Walus, K (2022). ATTENTIV: Instrumented Peripheral Catheter for the Detection of Catheter Dislodgement in IV Infiltration. In proceedings of 44th Annual International Conference on the IEEE Engineering in Medicine and Biology Society (EMBC).

Motivation:

90% of hospital patients get IV infusions, yet up to half of all IV procedures experience some form of failure. IV infiltration happens when IV fluids leak from the vein into tissues, causing damages that can require extended hospitalization, surgeries, and even amputation. Infiltration occurs in 23% of adults and 70% of newborn babies with IVs, and 4-6% of cases result in severe consequences. Hospitals face significant economic burden with $50k treatment costs/patient and the potential of $500k-$2M lawsuits.

The Attentiv catheter and infiltration monitoring system offers accurate and immediate detection of IV infiltration through its root cause, catheter dislodgement. Our sensor-embedded IV catheter provides real-time localization of the catheter tip and alerts the caretaker team if infiltration occurs. The monitoring system is seamlessly integrated into the existing nursing workflow and IV equipment. For hospitals, we reduce their economic expenditure and elevate their standards of care by preventing severe infiltration injuries.

Existing devices that aim to address the issue of IV infiltration use non-direct methods for detection . These devices are generally in the form of an external stick-on patch which requires additional dressings to secure and extra wiring, and is not ideal for premature babies with fragile skin. Our device is situated in-vivo and monitors the root cause of infiltration , allowing for quicker responses. Direct sensing allows us to expand into other types of catheters like central lines, where patch sensors cannot detect. Fewer dressings and reduction in protocol changes for nurses has the potential to lead to easier adoption and less costly retraining, which are both significant concerns in healthcare.

Awards:

• James Dyson Awards - National Winner (Canada)
• James Dyson Awards - International Top 20
• Medical Device Development Centre of British Columbia - Principle Design Award
• New Venture Design - Industry Award
• UBC APSC Design and Innovation Day - Faculty Award
• Microsoft Discover AI - Healthcare Category Winner
• UBC Innovation on Board Startup Competition - Runner Up
• RBC Get Seeded Pitch Award
• New Ventures British Columbia - Semi Finalist

Interactive Stroke Rehabilitation Wearable

Motivation:

Globally, about 6 million people die from strokes every year. Stroke survivors must undergo extensive physical rehabilitation exercises to recover their motor functions, some of which is done at home after hospital discharge. However, the effectiveness of the rehabilitation is often undermined by the repetitive and uninteresting nature of the exercises, leading to low motivation for the patients to continue the exercises in the long term. Nearly three quarters of stroke survivors report feeling a lack of confidence, and nearly half report a lack of motivation or anger. Over half of all stroke survivors are left with a disability, with a reported 85% suffering from loss of movement in upper limbs. This type of impairment negatively affects the patient’s ability to lead an independent life and perform daily tasks, lowering their quality of life.

There is a need to improve the effectiveness of at-home rehabilitation in order to cut back on acute care costs as well as to promote patient independence and decrease the reliance on caretakers. This can be done by combining intuitive ergonomic design with necessary rehabilitation exercises to create an interactive interface that patients will enjoy to use. We proposed and prototyped an interactive "glove" rehabilitation device for hand spasticity, a common outcome of stroke. Using Arduino, various sensors, and themoplastic, we constructed and clinically evaluated a functional prototype that provided real-time performance feedback through LEDs and allowed the patient to control a computer game through their therapy exercises.

FraXure: Femur Fracture Traction Device

Motivation:

Globally, between 20 and 50 million people suffer non-fatal injuries as a result of road traffic crashes. Ten percent of road injuries result in a femoral shaft fracture, with the young suffering a disproportionately higher incidence. Such injuries, particularly in low-resource settings, are associated with a high burden of handicap including poor alignment, limb shortening, and knee stiffness.

For femoral fractures, non-surgical treatment refers to traction, applied via the skin or the skeleton. Traction operates under the primary principle of applying two equal and opposite forces on two fragments of the fracture to achieve and maintain fracture position for healing. Skeletal traction is currently the standard of care for femoral shaft fractures in low and middle income countries (LIMCs). Skeletal traction is unfortunately associated with unwanted medical complications that can prolong treatment period and lowered quality of healing. In traction, insufficient reduction, slippage, or infections can cause malunions. Adverse effects of malunion include functional impairments to gait, limb shortening, and excessive stresses on other joints and body parts.

The goal of the project was to improve patient outcomes through the design of a cost effective, clinically viable device for the treatment of femur fractures. It was deemed critical that our device offers a non-surgical solution to help alleviate some of the demand for orthopaedic surgeons, whom are scarce in many LMICs as described previously. Without the need for surgical intervention, our hope is that the device will make femur fracture treatment more accessible and affordable for patients. As well, reducing the hospital stay for patients suffering from femur fractures may allow medical resources to be reallocated to the treatment of other injuries, furthe reducing the burden on the healthcare system. As well, mobilizing patients at earlier stages of recovery may present additional social and economic benefits as they can return to their families and return to the workforce earlier.

Through extensive literature research and in-country consulting with our hospital partners in Uganda and Kenya, we developed a pneumatic traction device that does not require external grounding to the hospital bed, allowing patients to return home with partial mobility. The cushioning system comprises of alternating pressurized airbags that distributes the contact force and reduces risk of deep tissue injuries.

Awards:

• International Conference on Engineering Design 2017 - Design Fair Runner Up
• Medical Device Design Center Excellence Awards - Finalist
• Rice University 360° Global Health Design Competition - Finalist

H4ptic: Sensory Feedback Wearable

Motivation:

There is limited usability in today's current prosthetic devices as they do not offer haptic sensory feedback. Modular EMG prosthetics are becoming increasingly common. Although the functionality of these prosthetics depends greatly on feedback to the user, this is an area where there has been limited development. To resolve this problem, we created H4ptic, a non-invasive device that captures haptic sensation from the prosthetic and transmits that sensation to the user’s body. H4ptic was designed using the principles of somatosensory cortex remapping, neuroplasticity, and sensory substitution.

Awards:

• Hatching Health Hackathon- Best Technical Innovation
• Speak Out for Engineering - Vancouver Regional Winner
• Speak Out for Engineering - Americas Winner

ReCrutch: Reimagined Crutches

Motivation:

This project was inspired after I witnessed a friend on crutches struggling to navigate flights of stairs. The rigid, vertical design of traditional crutches exert high pressure at contact points (palms and underarms) for users, making it an uncomfortable device to use for prolonged durations. Additionally, the inflexibility of the crutch hinders natural gait when traversing stairs. In my redesign, the ReCrutch features a retractable duel-handle design that aids the user while both ascending and descending stairs, simply by reversing the holding direction. The flexible S-shaped body also acts as a shock absorber. The elbow cradle is an ergonomic feature that helps relieve underarm contact pressure.

Electronical Medical Records Systems

Gwara, M, Tsang, VL, Thompson, CA, Smith, S, Bo, J, Fletcher, S, Janusz, N, Chew, SY, Janusz, M, Thompson, CK, Bertrand, M, Woods, H, Thompson, C (2018). Use of Centralized Electronic Medical Records System in Paediatric Care. In American Academy of Pediatrics 2017.

Abstract:

Centralized Electronic Medical Record systems (EMR) have potential to provide a variety of benefits to national healthcare systems worldwide. However, their implementation has proven to be challenging and raised several concerns. Although previous work has analyzed the successes and limitations of centralized EMR systems, statistical analysis to quantify user perspective has not yet been conducted. In order to gain a better understanding of the opinions of medical professionals and others in the field regarding their perceived support, benefits, and barriers for a centralized EMR system, a survey was conducted on the participants of the International Paediatrics Association (IPA) 2016 Conference in Vancouver, BC. The survey contained Likert Scale questions, asking participants to rate the importance of listed benefits and barriers, as well as indicate their overall support for centralized EMR systems on a scale of 1 to 5. The data was analyzed using a one-way ANOVA, with a focus on the countries with the greatest number of respondents (USA, Canada, Nigeria, and Mexico). The responses were also categorized and analyzed based on country development. The results indicated that Mexico, as well as other non-developed countries were shown to be most in favour of the EMR system. Fast record access and efficiency were rated as the most significant benefits of a centralized EMR system, while cost and implementation time were perceived as the largest barriers. Implementation time was also rated significantly higher as a barrier by developed countries than non-developed countries. These findings provide useful guidelines for consideration in the implementation of a centralized EMR system, and help in the drive towards improving national healthcare systems worldwide.

Automated Heel Raiser for Skiing

Project Summary:

In a class project, our team of six Mechanical Engineering students were challenged to provide a custom designed solution for a fictictious customer. "FreeBoots Enterprises" discusses their interest for the design of a device that is ideally lightweight, portable, and mechatronics-based to assist backcountry skiers with their uphill ascents. The device is expected to increase the number of ascents in a day or reduce the net effort required to ascend. This may be accomplished by reducing the amount of time required to ascend, reducing the metabolic energy required from the skier to ascend, or redirecting forces from the parts of the body that are used most when ascending to other areas. The device could also reduce the net effort to ascend by decreasing the overall weight of the gear to be carried by the skier by having the device replace other pieces of equipment, by decreasing the energy lost due to friction from the skins, or by lifting the skis out of the snow during strides. The device should not reduce the effectiveness of safety gear, increase the risk of an avalanche, and impede the motion of the skier should an avalanche occur.

After iterating through many concepts, we selected a design that is simple and light-weight to attach to existing ski equipment. The automated heel raiser is derived from the mechanical heel raiser used by many backcountry skiiers in their routine. However, it comes with the benefit of auto adjustments that raise the heel correspondingly to the incline of the slope. This results in optimal reduction in energy, as either an over elongation (not raised high enough) or over contraction (raised too high) in the calf muscles would diminish the energy efficiency of the muscles.

Remote-Controlled Fireboat

Project Summary:

The Marine Emergency Response Group (MERG) requires a fireboat to respond to fires in the coastal waters. Vessel requirements include high speed, agile maneuverability, accurate delivery of pressurized payload, and construction using sustainable materials. Bonus points are given to a boat with a well-designed aesthetic.

Our team choose a double catamaran boat design to optimize for speed. Pressurized water, reflective of the firefighting capacity of fireboats, is released through the control of servo motors and an Arduino. The boat is constructed from sustainable materials, including a cardboard based replica of a certain Pokemon. We simulated the water spray trajectories and boat speeds to best prepare for evaluation.

Regenerative Braking Car

Project Summary:

Green Vehicles Canada (GVC) requires an autonomous vehicle capable of recovering braking energy. Requirements include ability to stop and resume movement, controlled descending speed, and cargo storage while maximizing travel distance.

The primary energy storage is a flywheel that stores angular momentum through rotation of the rear axle. A gearbox pivots about axle to engage and disengage flywheel through a differential drivetrain. The secondary energy storage is a mass and pulley system which maintains potential energy by raising the mass through axle rotation during the "braking"

ORBIT Cubesat

Project Summary:

The Thunderbird Satellite's primary payload consists of two cameras intended for high-resolution imaging of Earth. The photos are evaluated for potential forest fire risks using on-board machine learning modules, trained on the ground, with capability to update these modules over the course of the mission. The use of two cameras enables multiple filters to be applied to the image sensor, allowing for infrared and visible light imaging.

HoloLabel: AR Scene Annotator

Agrawal, D, Lobsiger, J, Bo, J, Kaufmann, V, Armeni, I (2022). HoloLabel: Augmented Reality User-In-The-Loop Online Annotation Tool for As-Is Building Information. In proceedings of European Conference on Computing in Construction 2022.

Abstract:

As the popularity of supervised and semi-supervised learning tasks in embodied perception rises, so does the demand for large-scale annotated 3D scene datasets. Devices equipped with RGB cameras and depth sensors being readily available simplifies the task of capturing and reconstructing an environment as a 3D mesh. However, the task of manually labelling this 3D data is still non-trivial. To obtain semantic information about the scene, experts often have to work for many hours using non-intuitive and error-prone tools. To address this problem, we propose a HoloLens application to allow a user to directly annotate the scene with semantic information while simultaneously capturing the spatial environment mesh. Our tool focuses on 3D mesh-based semantic segmentation, i.e. annotating areas of the scene using a predefined set of classes. We leverage the HoloLens' Spatial Mapping feature to generate a 3D mesh of the scene and apply an automatic segmentation algorithm to generate segmentation proposals. The user can use a virtual paint brush to refine the segments or create new ones. Finally, we allow the option to add richer semantic descriptions using voice-to-text technology. We aim to lay the groundwork to integrate mixed reality devices in enabling intuitive 3D scene annotation in the real world.

Into the Forest: Immersive Game

Motivation:

Achieving full immersion in gaming is difficult but desirable for player engagement. Despite being limited by computation power and the lack of AR/VR hardware, we nevertheless wanted to create an immersive and artistic gaming experience. Moreover, we try to shift the player's emotions from the start to the end of the game by creating gradually crescendoing tension and urgency. We focused on creating the visual and sound systems, as 80% of environmental information comes from vision and 15% comes from hearing. This, "Into the Forest", a first-person escape game set in a cheerfully deceptive forest, was born:

You wake up in the middle of a forest, no memories of how you got here. But somehow it feels like you’ve been floating through these woods for a long time now. With birdsongs in the background and the sun high, the forest feels alive. Something in the air seems off, however. A tinge of mystery. Well, nothing to do but to venture further. There must be someway home. But soon the bird calls cease, a wispy mist floats, around and into you. Dark clouds blot the sky. Thick clouds infect your mind. Something the air seems off.

The forest feels alive.

MyoHome: Gesture-Controlled Smarthome

Motivation:

Modern smarthomes are typically voice-controlled, which presents a detriment to potential users with speech impairments or strong accents. We turned to the Myo Armband, an non-invasive EMG sensing device that comes with pre-built abilities to recognize simple hand gestures. Instead of speaking to Alexa or Google Home, users can control their smart home devices such as lights and fans at the flick of their wrist and arm. Based on how many devices there are, users can use the MyoHome web app to configure the gestures for each device, and save those customized gestures to a user profile. Although there are only 5 basic gestures built in to the Myo Connect app, there are essentially an infinite number of gestures that could be configured based on the raw data output from the accelerometer, EMG sensors, and gyroscope.

Awards:

• nwHacks 2018 Honourable Mention

Postcards & Sketches

Street in Otaru, Hokkaido, Japan

(43.194092, 141.002771)

Canal in Otaru, Hokkaido, Japan

(43.199914, 141.001594)

Underpass in Shinjuku City, Tokyo, Japan

(35.701311, 139.700744)

Seafood stall in Nishki Market, Kyoto, Japan

(35.004882, 135.764916)

Golden Pass Railway near Saanen, Bern, Switzerland

(46.491337, 7.280064)

Vieille Ville de Annecy, Haute-Savoie, France

(45.898848, 6.125080)

Passage des Panoramas in Paris, Île-de-France, France

(48.870608, 2.341582)

Akrobaten Bridge in Oslo, Norway

(59.910171, 10.760282)

Peek a Bloom in Athens, Greece

(37.976883, 23.731930)

Kittie Point Rock in Porto, Portugal

(41.139156, -8.607268)

Alfama District in Lisbon, Portugal

(38.71260, -9.135844)

Previous Next

If you're interested in commisioning a postcard sketch, contact me!

Handpokes

fable

marbled

spring sprig

minimal mountains

triangles

ups and downs

Previous Next

Misc. Artwork

sandstorm

"A Sublime Night Out With My Step-Parents"

Ponti del Salti (unfinished)

throwback to the 2010s

fable

fleur