Augmented  Vision

Time to step into a whole new realm of research at the HKUST-Deutsche Telekom System and Media Laboratory, where augmented reality points the way to practical transformation of people’s lives


Greeting their multinational research teammates and sitting down at their computers, it may appear just another day in the lab for HKUST computer science and engineering PhD students Carlos Bermejo Fernandez, from Spain, and Paul LH Lee, from Hong Kong. At the same time, no two days are the same when you study at the HKUST-Deutsche Telekom System and Media Laboratory (SyMLab) and you never need to be spurred to return there when you are devoted to devising amazing augmented reality (AR) technologies to change the actual world for the better.

Carlos and Paul are part of a highly active and expanding team of 16 researchers at SyMLab, established by Prof Pan Hui in 2013 and the first joint Deutsche Telekom lab at a university in East Asia. Among those set to benefit from the lab’s work are the visually impaired, through the development of AR glasses that can rapidly identify objects in their environment, making it more straightforward and much less time-consuming to carry out regular activities such as supermarket shopping. Further projects include a hand gesture system for smart glasses; an emotion-sensing system for mobile and wearable devices; and an interactive visual privacy system enabling users to blur their faces in photos taken by AR devices to stop them being identified by other people and machines.  

The common theme is cutting-edge research that connects with real needs to transform the future. SyMLab cooperates closely but not solely with Telekom Innovation Laboratories (T-Labs), the research and development arm of the German telecommunications giant that provides generous funding for the research unit at HKUST. Further partners to date include Microsoft Research Asia and China Mobile.

Since the lab was set up, 19 patents have been filed spanning different AR sectors. The aim of the research team is not only to build useful practical applications themselves but also to provide the tools to assist other researchers and industry players to do so. This encompasses development of AR hardware platforms, software frameworks and display devices, enabling technologies such as object tracking to speed up the development cycle, and product innovation. Recent successes include a mobile cloud computing technology (CloudRidAR) and in-cloud big data processing algorithms that enable real-time performance of AR mobile apps by offloading intensive computing operations to the cloud and co-located devices. Point at an object with a mobile device and relevant data such as a review can be instantly downloaded.

Paul LH Lee

Industry Links and Learning to Lead 

The Lab’s strong ties with industry were a particular draw for both Carlos and Paul, both of whom had working experience and were keen to make their studies at HKUST count in daily life or the business world. Carlos, who previously studied telecommunications, had first met Prof Hui when employed at T-Labs in Berlin. Paul, whose BEng and MPhil focused on logistics and industrial engineering respectively, had been a business analyst and started his own company. “I want to produce something that people can use,” Paul said.

To optimize such links, Prof Hui, a University of Cambridge PhD graduate, strongly encourages his team members to take up overseas internships or undertake company or institutional visits to extend their scope and expertise. One of his students, he noted, had just completed a seven-month visit to Cambridge, having previously been to EPFL in Switzerland and T-Labs in Germany: “My philosophy is simple,” Prof Hui said. “I want to help my team gain access to companies and other professors besides myself so they will build their own pathway. If I always tell them what to do, they will not be better than me in the future. And I want every one of my students to be better. Each student should be among the best in the world in one area when they finish their PhD here. It can be a small area but they should be a world leader in it.”

Leadership skills are enhanced through Prof Hui’s strategy of giving each team member the responsibility of heading at least one research project, ensuring the studies assigned are of interest to participants, and all have the potential to have significant impact. Carlos (glasses for the visually impaired) is looking after undergraduates and interns keen to take advantage of the opportunities that HKUST offers to gain early exposure to life as a researcher. Paul (hand gesture system) has two post-docs to help support his research. 

Prof Pan Hui

Multicultural Teamwork in Hong Kong

By becoming their own boss in this way, Prof Hui believes his PhD students will adapt to independent problem-solving as well as learn how to guide others. “My own role is to give advice and provide them with the freedom to think,” he said.  At the same time, each member’s individual strengths are harnessed on behalf of the team as a whole. Carlos, who speaks fluent German and formerly worked at T-Labs, serves as the liaison point with Deutsche Telekom. Paul, an extrovert and Cantonese/Putonghua speaker, is the HKUST Lab’s outreach officer, introducing the Lab’s work locally to companies and non-governmental organizations (NGOs).  

A multinational, multi-skilled team is regarded as a major plus by both Prof Hui and HKUST for synergizing different talents, perspectives and cultural vision and achieving research excellence, with innovative students around the world encouraged to apply to undertake postgraduate studies at the University. And apply they do. In addition to Spain and Hong Kong, SyMLab members currently come from France, Turkey, Mainland China, Greece, Poland, Pakistan, and Iran.  “This University is very well known for computer science. If you look at the relevant international lists online, it is always there,” Carlos noted.

Not all hail from a computer science background either. For Paul and Carlos, this meant undertaking an intensive series of courses at HKUST to ensure the academic quality of the University’s students is maintained. It was hard work. However, the University and students view such an option as worthwhile as it allows different perspectives to be brought into a research team, widening its collective scope and skills as well as individual career potential.    

For those arriving from overseas or Mainland China, settling in to HKUST is made easier by Hong Kong’s dynamic east-west blended lifestyle that helps both sets of students feel at ease quicker, enabling them to focus fully on their research training. Carlos had never been to Hong Kong before setting out for HKUST. Yet he rapidly felt at home on the University’s scenic campus and within cosmopolitan Hong Kong. “My first thoughts on the city? When you leave the airport and start seeing all the tall buildings next to the sea… Wow!  That is really impressive.”

Prof Hui, a keen hiker in the few hours he has over from the Lab and other academic responsibilities, was also happy to return home from Germany to his family, Hong Kong, and to take up work at HKUST, with its ready access to beautiful coastal and countryside walks.

Carlos Bermejo Fernandez

Creative research design and entrepreneurship

The Lab’s approach to its technologies’ research is as innovative as the projects it works on. Carlos is using a participatory design for his smart glasses for the visually impaired. This has involved liaising with the Hong Kong Society for the Blind, conducting interviews, and the key discovery that the research team’s initial ideas about what was most needed (devices to assist in situations such as crossing the road) was in fact not the case. For the interviewees, identifying the products in the supermarket to help them shop was a much more important priority.

Carlos is now working through this challenge to produce glasses that can help perform the task and at the same time keep costs to a minimum so that large numbers of people can afford them. Helping him generate ideas and overcome the hurdles are his fellow students and HKUST academics. “I think the most inspiring part of being here is the people,” he said. “You get really good feedback from your peers and in the Lab. I have gained many leads and paths to improve my research. The professors, too, really care about the learning process as well as being first-class researchers themselves.”

Meanwhile Paul, who is keen to become an entrepreneur, has found the PhD program valuable for identifying what basic research and developments are key for the field’s future and offer commercial potential. The University provides strong support and incubation opportunities for related start-ups. “Before I came to Prof Hui’s lab, I had some ideas but after I started my PhD, I found those directions trivial. Now I am finding out what is likely to be significant in the future. Studying at this University, which is one of the leaders in computer science in the world, can inspire me to do better in my business. In addition, the HKUST Entrepreneurship Center offers lots of relevant talks and activities to help me become successful.”

Driving the Future

That the overall approach of top students from different backgrounds, multinational vision and talents, leading-edge projects and dynamic research techniques is having an impact can be seen in Lab members’ growing list of published articles in leading journals and conferences, several of which have received over 1,000 citations, and feedback from industry. The key is to take a visionary approach tied into practical needs that can shape the lives we will lead in the years ahead, according to Prof Hui, recently elected a Distinguished Scientist by the Association for Computing Machinery (ACM), the world’s largest educational and scientific computing society.

“Our Lab’s strategy is always to be more innovative and pace-setting in what we do and how we look at our research,” he said. “This way we don’t compete. We determine the future.”