LLM-Guided Robot Learning from Demonstration

Large Language Models (LLMs) are rapidly becoming a key ingredient in AI and robotics, enabling agents to follow natural language instructions, decompose tasks, and plan sequences of actions. However, most LLM-based robot systems still rely heavily on hand-designed skills and lack explicit representations of task-specific constraints (e.g., safety, collision avoidance, forbidden regions, task-specific “do-not” rules). As a result, LLM-driven robots may behave unsafely or unexpectedly in some tasks.

While our previous work [1] solves the problem of learning such task constraints from human demonstrations, this approach still requires users to manually specify the constraint types or objective functions, which remains a significant burden for non-expert users.

To bridge this gap, this project explores a novel LLM-guided learning from demonstration method: using LLMs to propose structured constraint hypotheses from language, and then using data-driven constraint learning to identify the actual constraint function from human demonstrations (including failed attempts). For example, a human demonstrates how to carry a cup and tells the robot what the task is. Ideally, the robot should realize that the cup must not tilt and automatically represent this as a quantitative constraint, cup angle < 10. The key idea of the framework is to combine:

• LLMs as a source of high-level structural priors (e.g., identifying potential constraints type, relevant features, and appropriate parameterization).

• Constraint learner that learns an explicit, quantitative constraint function that can be used for planning.

This provides a promising pathway toward safer and more reliable LLM-powered robots, where constraints are learned as explicit models that can be checked, enforced, and used for planning/control.

Project Goals

The student will develop a prototype framework that:

1. Uses an LLM to generate structured constraint hypotheses (e.g., constraint templates, predicates, or relations). This involves designing the constraint structure and prompting the LLM to generate hypotheses that adhere to this structure.
2. Uses the constraint learning pipeline to learn an explicit constraint function from human demonstration.
3. Implements a constrained planner (e.g., MPC) to generate trajectories that satisfy the learned constraints.

Evaluates the resulting constraint-aware planner in simulation, with metrics such as feasibility, safety violations, and data efficiency.

Expected Outcome

We expect this project to lead to a publishable result, targeting top robotics venues such as IROS / ICRA / CoRL, or AI conferences NIPS/ICLR/ICML

Candidate Requirements

Time Commitment

We expect a commitment of 20+ hours per week. This project is ideally suited for a Master’s Thesis or an Exchange Student Project.

For students with a strong relevant background, a Semester Project may be considered, provided a minimum commitment of 12 hours per week is maintained.

Must-have:

• Strong Python skills
  
• Good knowledge in machine learning, especially neural networks and classification methods.

Nice-to-have (plus): knowledge and experience in 

• Robot control / motion planning
• LLMs (prompting, tool use, structured generation)
• Reinforcement learning, representation learning

 

Personal expectation:

Strong self-motivation, collaborative, and genuine interest in the research direction.

 

Feel free to get in touch and discuss more details with me 🙂  at [email protected]

References

[1] B. Peng and A. Billard, “Positive-Unlabeled Constraint Learning for Inferring Nonlinear Continuous Constraints Functions From Expert Demonstrations,” in IEEE Robotics and Automation Letters, vol. 10, no. 2, pp. 1593-1600, Feb. 2025, doi: 10.1109/LRA.2024.3522756.