On this day we welcomed the ACM SIGCHI Chapter – SydCHI here in Data61 for a series of practice talks from Rukshani, Owen, Jieshan and Don. The following are the details of these talks.

Rukshani Somarathna

Exploring User Engagement in Immersive Virtual Reality Games through Multimodal Body Movements

User engagement in Virtual Reality (VR) games is crucial for creating immersive and captivating gaming experiences that meet the expectations of players. However, understanding and measuring these levels in VR games presents a challenge for game designers, as current methods, such as self-reports, may be limited in capturing the full extent of user engagement. Additionally, approaches based on biological signals to measure engagement in VR games present complications and challenges, including signal complexity, interpretation difficulties, and ethical concerns. This study explores body movements, as a novel approach to measure user engagement in VR gaming. We employ E4, emteqPRO, and off-the-shelf IMUs to measure the body movements from diverse participants engaged in multiple VR games. Further, we examine the simultaneous occurrence of player motivation and physiological responses to explore potential associations with body movements. Our findings suggest that body movements hold promise as a reliable and objective indicator of user engagement, offering game designers valuable insights on generating more engaging and immersive experiences.

29th ACM Symposium on Virtual Reality Software and Technology (VRST 2023), Christchurch, New Zealand.

Rukshani Somarathna, Don Samitha Elvitigala, Yijun Yan, Aaron J Quigley, and Gelareh Mohammadi. 2023. Exploring User Engagement in Immersive Virtual Reality Games through Multimodal Body Movements. In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology (VRST ’23). Association for Computing Machinery, New York, NY, USA, Article 3, 1–8. https://doi.org/10.1145/3611659.3615687

Yongquan Hu (Owen)

MicroCam: Leveraging Smartphone Microscope Camera for Context-Aware Contact Surface Sensing

The primary focus of this research is the discreet and subtle everyday contact interactions between mobile phones and their surrounding surfaces. Such interactions are anticipated to facilitate mobile context awareness, encompassing aspects such as dispensing medication updates, intelligently switching modes (e.g., silent mode), or initiating commands (e.g., deactivating an alarm). We introduce MicroCam, a contact-based sensing system that employs smartphone IMU data to detect the routine state of phone placement and utilizes a built-in microscope camera to capture intricate surface details. In particular, a natural dataset is collected to acquire authentic surface textures in situ for training and testing. Moreover, we optimize the deep neural network component of the algorithm, based on continual learning, to accurately discriminate between object categories (e.g., tables) and material constituents (e.g., wood). Experimental results highlight the superior accuracy, robustness and generalization of the proposed method.

Ubicomp/ISWC 2023, Oct 8-12 Cancun, Mexico

Yongquan Hu, Hui-Shyong Yeo, Mingyue Yuan, Haoran Fan, Don Samitha Elvitigala, Wen Hu, and Aaron Quigley. 2023. MicroCam: Leveraging Smartphone Microscope Camera for Context-Aware Contact Surface Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 3, Article 98 (September 2023), 28 pages. https://doi.org/10.1145/3610921

Jieshan Chen

Unveiling the Tricks: Automated Detection of Dark Patterns in Mobile Applications

Mobile apps bring us many conveniences, such as online shopping and communication, but some use malicious designs called dark patterns to trick users into doing things that are not in their best interest. Many works have been done to summarize the taxonomy of these patterns and some have tried to mitigate the problems through various techniques. However, these techniques are either time-consuming, not generalisable or limited to specific patterns. To address these issues, we propose \tool{}, a knowledge-driven system that utilizes computer vision and natural language pattern matching to automatically detect a wide range of dark patterns in mobile UIs. Our system relieves the need for manually creating rules for each new UI/app and covers more types with superior performance. In detail, we integrated existing taxonomies into a single and consistent one, conducted a characteristic analysis and distilled knowledge from real-world examples and the integrated taxonomy. Based on this, our \tool{} consists of two components, UI element detection and knowledge-driven dark pattern checker. For evaluation, we utilise Rico dataset and its semantic labelling to train and test each DL module in our system. We also contribute a new dark pattern dataset, which contains 4,999 benign UIs and 1,353 malicious UIs of 1,660 instances spanning 1,023 mobile apps. Our system achieves a superior performance in detecting dark patterns, with an overall performance of 0.83 in precision, 0.82 in recall, and 0.82 in F1 score. Our ablation experiments demonstrates the validity and necessity of each module. A user study involving 58 participants further showed that \tool{} significantly increases users’ knowledge of dark patterns, increasing the recall rate of detecting dark patterns from 18.5\% to 57.8\%. Our work is beneficial to end-users, app providers, and regulators, and can serve as a training tool for raising awareness of dark patterns.

Dr. Don Samitha Elvitigala

RadarFoot: Fine-grain Ground Surface Context Awareness for Smart Shoes

Everyday, billions of people use footwear for walking, running, or exercise. Of emerging interest are `smart footwear'', which help users track gait, count steps or even analyse performance. However, such nascent footwear lack fine-grain ground surface context awareness, which could allow them to adapt to the conditions and create usable functions and experiences. Hence, this research aims to recognize the walking surface using a radar sensor embedded in a shoe, enabling ground context-awareness. Using data collected from 23 participants from an in-the-wild setting, we developed several classification models. We show that our model can detect five common terrain types with an accuracy of 80.0% and further ten terrain types with an accuracy of 66.3%, while moving. Also, it can detect the gait motion types such aswalking’, stepping up',stepping down’, `still’, with an accuracy of 90%. Finally, we present potential use cases and insights for future work based on such ground-aware smart shoes.

The ACM Symposium on User Interface Software and Technology, UIST 2023, San Fransisco, USA

Don Samitha Elvitigala, Yunfan Wang, Yongquan Hu, and Aaron J Quigley. 2023. RadarFoot: Fine-grain Ground Surface Context Awareness for Smart Shoes. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 87, 1–13. https://doi.org/10.1145/3586183.3606738

Boeing is the world’s largest aerospace company and leading manufacturer of commercial airplanes, and defence, space and security systems. The company supplies aircraft and information platform systems to Australia’s airlines and the Australian Department of Defence.

CSIRO has been a partner with Boeing for over 30 years and this year I joined the CSIRO – Boeing Steering Committee which included a deep dive into our joint projects, visits to partners, project teams and collaborators across California.

This year, Boeing named CSIRO a supplier of the year for the fifth time, as the organisations mark 34 years of a thriving strategic relationship. Boeing, has named CSIRO, Australia’s national science agency, a Supplier of the Year for the fifth time since 2010.

Professor Thomas is currently Emeritus Professor at the University of South Australia and he kindly agreed to our invitation to work with us a day a week to help us develop our Immersive AI and University engagements. It’s been a great pleasure working with Bruce and the entire Immersive AI cluster here in Data61.

His current research interests include the following: user interfaces, augmented reality, virtual reality, visualisation, wearable computers, CSCW, tabletop display interfaces, and the use of cognitive psychology in virtual environments research. He has served in many roles for the IEEE International Symposium on Mixed and Augmented Reality, IEEE Virtual Reality, and IEEE/ACM International Symposium on Wearable Computers, including program chair, general chair and on the steering committee. He also founded the ACM Interactive Surfaces and Spaces Conference (formerly IEEE Tabletop). He was awarded the ACM International Symposium on Wearable Computers (ISWC) 20-Year Impact Award. He delivered a keynote address at ISMAR 2023.

I was honoured to join Professor Toby Walsh who received the Celestino Eureka Prize for Promoting Understanding of Science. I was Toby’s head of school while I worked in UNSW so I was able to see his efforts, day in day out, in the promotion of science so this award is very well deserved.

I was able to interview Eleanor Huntington, our executive director for Wear it Putple day and what is means for CSIRO!

Wear it Purple Day is an annual LGBTIQA+ awareness day especially for young people, based in Australia. Supporters wear purple to celebrate diversity and young people from the LGBTIQA+ community. The Day is organised by Wear it Purple, a student run, not-for-profit organisation, providing presentations and workshops. 

This week we spent a few days meeting our colleagues and University collaborators in Adelaide!

On August 1st I joined CSIRO at a Parliamentary Tech Showcase in Mural Halls A & B, Australian Parliament House (APH). The evening before we presented a series of CSIRO’s Data61demos at a TCA & CSIRO Industry Networking event at the CSIRO Discovery Centre, Black Mountain – North-science Road.

The Next Generation Quantum Graduates Program (NGQGP) is an Australian Government supported program set to fund nationally competitive scholarships to attract and train Australia’s next generation of quantum technology specialists. The scholarships will be funded for study at Australian Qualifications Framework (AQF) Level 10 (Doctoral). Students will participate in collaborative projects between industry and university, including placements with partners to build career-ready skills, and to forge ongoing relationships and networks.

As part of my role as Science Director and Deputy Director of CSIRO’s Data61 I chair the steering committee for this program with oversight for the proposal, development and delivery of this program (from conception to launch and delivery over the year ahead)

My 50th year rolled around quite quickly and we celebrated! Here you can see my grandparents (top right) and my mother and aunt (bottom left) with younger versions of me.

Congratulations to all the finalists and winners from last evening’s Women in AI Awards 2023 (Asia Pacific). 

I was delighted to join my new colleagues from across CSIRO and CSIRO’s Data61 to celebrate their success and share with my former colleagues from UNSW Computer Science and Engineering theirs. 

Thank you to all the organisers, sponsors and panellists for all their hard work. They brought together an amazing experience to celebrate the accomplishments and impact of so many Women in AI.

We had a wonderful evening celebrating the success of Women in AI from across the Asia Pacific region. It was inspiring to hear the impact and motivations of all the finalists for the work they do.