AI reasoning for situation understanding in human-robot interaction
This page might be updated more or less any time I think of more information to consider, however, if you have questions do not hesitate to contact me!
The group for Robotics and Semantic Systems (RSS) at the Computer Science Department, Faculty of Engineering at Lund University can offer one PhD project, funded for four years through the Wallenberg Autonomous Systems and Software Program (WASP), Sweden's up to date largest national research program in this area, and to be supervised by me, Elin A. Topp. More information on WASP in general can be found here, while this is the link to the official call and instructions for application. Some more details on how a PhD studentship looks like in Sweden, and specifically in the RSS-group, can be found below.
Human users can provide robots (and other systems) with an immense amount of knowledge, e.g. through demonstrations, immediate (classic) programming, instructions, dialogues, or combinations thereof. However, due to robots being different from humans regarding their physical and perceptual capabilities, it is not always obvious to a human user what - and sometimes even how - to communicate to make the robot or underlying system understand. This is to some extent due to the fact that the human user is not necessarily familiar with the actual capabilities and models the system can make use of.
Hence, we must assume, that when autonomous systems, or robots, interact with humans in an open-ended world (the real world), there will be situations where the system works according to its technical specifications, but its behaviour is incomprehensible for the user. This can be because user and system do not quite share the same understanding of the given context and situation.
For example, when one person tells another person "this is my office" when standing in front of an office door, the recipient of the utterance will most likely assume "the room beyond the door is the speaker's office" and not "the place where I am standing is the speaker's office". A very simple robot, however, that only perceives the utterance and knows that it is currently standing within boundaries making up something like a region or room (the hallway), will probably store the label "speaker's office" for the hallway it is placed in, simply because the utterance "this is my office" can be the same both when standing inside or outside the actual labeled area - it just would carry different meaning. As discussed in a recent publication ("Interaction Patterns in Human Augmented Mapping", contact me if you are interested in the full-text), such situations can be resolved by looking at the speaker's (unconsciously applied) behaviour, as this carries a lot of information of the communicative intention underlying the utterance. Simply understanding, that there is an ambiguity between the user's utterance and their behaviour, helps to generate clarification requests, and ultimately to resolve the situation.
The PhD project will thus investigate how a better situation understanding can be provided for systems that interact with humans in a mixed-initiative setting. Example problems include how such an interactive system can identify patterns in the user's behaviour that can be used to detect ambiguities in the situation (see the example given above), how it can in general assess unexpected and ambiguous situations, and how suitable responses to such detections can be computed. Such responses can mean to act autonomously, or to ask the human for help to resolve an ambiguous situation. It is also possible to consider direct support for the user in an instruction or programming setting, where the prediction of intentions and goals for a sequence of instructions or activities can be used to generate suggestions or reminders for the user to prevent problematic situations from occurring. Even settings in a command central / control room, e.g. on a marine vessel, where the overall situation might have to be assessed to adapt visualisations and suggestions given to an operator, can be of interest for the investigations carried out.
Machine learning and pattern classification techniques, including Bayesian methods, are assumed suitable for the identification of patterns in user behaviour, while the situation assessment and computation of appropriate actions and responses will be based on reasoning mechanisms, potentially including defeasible reasoning.
Testbeds and opportunities for the demonstration of results include different mobile platforms, classic industrial manipulators, and collaborative robots available in the LTH Robot Lab, as well as the WASP demonstrator arena for Public Safety (WARA-PS).
Apart from the formal requirements specified in the official call, the applicant should match the following criteria. The ideal candidate has a solid background in computer science or a related field (also including, e.g., cognitive science) with very good mathematical and programming skills. Experience in at least one of the following areas is beneficial: Machine Learning, AI Reasoning, Robotics, Human-Robot Interaction, User Studies, Cognitive Science, and Programming Languages suitable for Robotics Research and ROS. Besides technical and mathematical skills, the candidate is expected to be curious and ambitious, with strong motivation to conduct research, but also willing to implement for and experiment with actual robotic systems. The applicant should be used to a well-structured work style, and have the ability to work both individually and in teams. Very good communication skills in both oral and written English are required. A short research proposal showing the applicant's interests in relation to the project description should be included in the application material.
In addition to the obvious interest in mixed-initiative interaction with both mobile (service) and industrial robot systems and the underlying approaches for situation understanding, intention recognition and general interaction monitoring represented by myself, the prospective supervisor for the project, the research group for Robotics and Semantic Systems has a well established track record in knowledge representation and management as well as integrated end-to-end software systems for (mainly, but not exclusively) industrial robotic applications, and is or has been involved in a series of respective EU-funded projects (SIARAS, ROSETTA, PRACE, SMErobotics, SARAFun).
A second, very strong and well established line of research is pursued by Pierre Nugues in the area of natural language processing (NLP), specifically semantic parsing, role labelling, and entity recognition. Some rather recent efforts of integrating NLP into work on intuitive programming of industrial robots have led to the establishment of an end-to-end support chain from high-level (spoken) commands formulated in natural language through levels of task and skill representation and abstraction down to executable robot code for industrial, both native and external, robot controllers (see Maj Stenmark’s respective publications).
The group is collaborating closely with the robotics researchers within the Dept of Automatic Control, enabling this type of research on all system levels, from investigations of high-level programming tools to sensor-based force-controlled manipulation approaches.
In addition to the opportunities for collaboration through the WASP Graduate School and WASP research topics, the prospective candidate has the opportunity to benefit from the different topics, perspectives, and core competencies offered within RSS and the RobotLab.
Currently, the group is involved in the Wallenberg Autonomous Systems and Software Program by hosting one industrial PhD project (Mårten Lager, "Digital Cognitive Companion for Marine Vessels" within the WASP project "Interaction and Collaboration in Sensor Rich Environments"), supervised by Jacek Malec and myself (Elin A. Topp). This project might give opportunities to cooperate regarding the observation and consideration of different situations in a command central / control room, as was mentioned in the project description. Both Jacek and I have also been involved as content developers and lecturers in the first edition of the WASP core course "Autonomous Systems". In addition to that, there are currently two other WASP PhD projects hosted at the Department of Computer Science. Collaboration across the respective research topics would be encouraged.
Further there are currently two recruitment processes for (senior) WASP funded positions running at the Department of Computer Science, both entailing additional PhD and PostDoc opportunities, which will prospectively give new possibilities for collaboration within the group, especially as one of the senior positions is assumed to be topic wise easily associated with the RSS group and the RobotLab.
The situation of a PhD student in Sweden is two-fold: While being a student with a certain course work load (currently at the Department of Computer Science this corresponds to 90 credits of in total 240, i.e., 37.5% of the nominal load), the candidate will also be fully employed by the University to carry out both course work and research. The employment entails a salary usually sufficient to sustain a small household, which is raised during the studentship according to certain milestones (in terms of credits) being achieved.
The studentship / employment in this particular case is fully funded for four years through WASP, however, it is common for PhD students to also take on up to 20% of full working time of departmental duties, most often as teaching assistant. These commitments are then funded by the department, which means that the overall time for the PhD studentship is spread out over five years with on average 80% of a full time work load per year assigned to the PhD studies (course work and research), and 20% dedicated to teaching. About half way through the project, i.e. after approximately 2.5 years, it is common for PhD candidates to obtain a Licentiate degree. This is not compulsory, however, it can be a good opportunity to consolidate the achievements of the first part of the project, and helps to shape the direction of further research.