The goal of my research is to enable the safe operation of autonomous systems, such as unmanned aircraft, autonomous cars, and smart infrastructure.  My research focuses on modeling, building, and validating intelligent safety-critical systems.  My research combines ideas from decision theory, reinforcement learning, machine learning, game theory, and control theory.

Adaptive Stress Testing

Adaptive Stress Testing (AST) is a novel framework that frames finding the most likely failure scenario as a sequential decision-making problem, allowing existing algorithms such as Monte Carlo tree search (MCTS) and deep reinforcement learning (DRL) to solve them.  AST has been used as part of a Federal Aviation Administration (FAA) program to discover scenarios of near mid-air collisions (NMACs) in prototypes of the next-generation Airborne Collision Avoidance System (ACAS X).  AST has also been applied to analyze trajectory planning systems for unmanned aircraft and autonomous cars at pedestrian crossings.

Related Publications:

  • R. Lee, M. J. Kochenderfer, O. J. Mengshoel, G. P. Brat, and M. P. Owen, “Adaptive Stress Testing of Airborne Collision Avoidance Systems,” in Digital Avionics Systems Conference (DASC), 2015.
    [Bibtex]
    @InProceedings{Lee2015,
    author = {Ritchie Lee and Mykel J. Kochenderfer and Ole J. Mengshoel and Guillaume P. Brat and Michael P. Owen},
    title = {Adaptive Stress Testing of Airborne Collision Avoidance Systems},
    booktitle = dasc,
    organization = {AIAA/IEEE},
    year = {2015},
    note = {Best Paper of Session},
    owner = {rcnlee},
    timestamp = {2015.05.20},
    }
  • M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer, “Adaptive Stress Testing for Autonomous Vehicles,” in Intelligent Vehicles Symposium (IV), 2018.
    [Bibtex]
    @InProceedings{Koren2018,
    author = {Mark Koren and Saud Alsaif and Ritchie Lee and Mykel J. Kochenderfer},
    title = {Adaptive Stress Testing for Autonomous Vehicles},
    booktitle = iv,
    Organization = {IEEE},
    year = {2018}
    }
  • R. Lee, O. J. Mengshoel, A. K. Agogino, D. Giannakopoulou, and M. J. Kochenderfer, “Adaptive Stress Testing of Trajectory Planning Systems,” in SciTech, Intelligent Systems Conference (IS), 2019.
    [Bibtex]
    @InProceedings{Lee2019a,
    author = {Lee, Ritchie and Mengshoel, Ole J. and Agogino, Adrian K. and Giannakopoulou, Dimitra and Kochenderfer, Mykel J.},
    title = {Adaptive Stress Testing of Trajectory Planning Systems},
    booktitle = aiaa_is,
    year = {2019},
    Organization = {AIAA},
    Owner = {rcnlee}
    }
  • R. Lee, O. J. Mengshoel, Saksena Anshu, R. Gardner, D. Genin, J. Silbermann, M. Owen, and M. J. Kochenderfer, “Adaptive Stress Testing: Finding Failure Events with Reinforcement Learning,” ArXiv e-prints, 2018.
    [Bibtex]
    @article{Lee2019b,
    author = {Lee, Ritchie and Mengshoel, Ole J. and Saksena, Anshu, and Gardner, Ryan and Genin, Daniel and Silbermann, Joshua and Owen, Michael and Kochenderfer, Mykel J.},
    title = {Adaptive Stress Testing: Finding Failure Events with Reinforcement Learning},
    journal = arxiv,
    archivePrefix = {arXiv},
    eprint = {1811.02188},
    year = {2018},
    Owner = {rcnlee}
    }
 

Differential Adaptive Stress Testing (DAST) extends the AST approach to analyze differences in failure behavior between two systems.  DAST finds failure scenarios where the test system fails but the baseline system does not, making it very useful for comparing two candidate systems or regression testing of successive prototypes.  DAST has been used to compare scenarios of near mid-air collisions between ACAS X and the existing Traffic Alert and Collision Avoidance System (TCAS).

Related Publications:

  • R. Lee, O. J. Mengshoel, A. Saksena, R. Gardner, D. Genin, J. Brush, and M. J. Kochenderfer, “Differential Adaptive Stress Testing of Airborne Collision Avoidance Systems,” in SciTech, Modeling and Simulation Technologies Conference (MST), 2018.
    [Bibtex]
    @InProceedings{Lee2018a,
    author = {Lee, Ritchie and Mengshoel, Ole J. and Saksena, Anshu and Gardner, Ryan and Genin, Daniel and Brush, Jeffrey and Kochenderfer, Mykel J.},
    title = {Differential Adaptive Stress Testing of Airborne Collision Avoidance Systems},
    Booktitle = aiaa_mst,
    year = {2018},
    Organization = {AIAA},
    note = {Best Paper Award},
    Owner = {rcnlee}
    }
 

Automatic Categorization

Grammar-Based Decision Tree (GBDT) is a machine learning model for automatically categorizing and explaining failure scenarios to help domain experts diagnose the underlying issues of failures.  GBDT combines a context-free grammar, temporal logic, expression optimization, and decision tree into a single framework that provides human-interpretability and support for high-dimensional heterogeneous time series data.  Decision rules in the tree are Boolean expressions derived from a grammar, allowing much flexibility and user customization.  We take each leaf node to be a separate category and the conjunction of branch expressions to be its explanation.  GBDT learning relies on existing expression optimization algorithms, such as genetic programming and grammatical evolution.  GBDT has been applied to categorize scenarios of near mid-air collision in ACAS X.

Related Publications:

  • R. Lee, M. J. Kochenderfer, O. J. Mengshoel, and J. Silbermann, “Interpretable Categorization of Heterogeneous Time Series Data,” in International Conference on Data Mining (SDM), 2018.
    [Bibtex]
    @InProceedings{Lee2018b,
    author = {Lee, Ritchie and Kochenderfer, Mykel J. and Mengshoel, Ole J. and Silbermann, Joshua},
    title = {Interpretable Categorization of Heterogeneous Time Series Data},
    booktitle = sdm,
    year = {2018},
    Organization = {SIAM},
    Owner = {rcnlee}
    }
 

Autonomous Vehicle Technologies

We have multiple autonomous vehicle platforms with various computing and sensing capabilities, including the MAX unmanned ground vehicle, X-SCAV half-scale unmanned aircraft, and SWIFT full-scale unmanned electric glider.  We also have vehicle and mission simulation capabilities using the NASA Reflection software framework.  The vehicles have been deployed on a variety of research and scientific missions.   Payload-directed flight (PDF) aims to close the control loop around unconventional sensors such as scientific payloads in order to maximize the return of scientific missions.  PDF integrates sensor measurements,  performs estimation over models of physical processes, and performs intelligent real-time trajectory planning to maximize information gathering and mission value.  In a collaboration with United States Geological Survey (USGS), we have created both hardware and software for conducting autonomous geomagnetic surveys using an unmanned ground vehicle and an unmanned aircraft.

Related Publications:

  • R. Lee and C. Ippolito, “A Perception and Mapping Approach for Plume Detection in Payload Directed Flight,” in Infotech@Aerospace Conference, 2009.
    [Bibtex]
    @InProceedings{Lee2009,
    author = {Lee, Ritchie and Ippolito, Corey},
    title = {A Perception and Mapping Approach for Plume Detection in Payload Directed Flight},
    booktitle = aiaa_info,
    organization = {AIAA},
    year = {2009},
    month = {April},
    owner = {rcnlee},
    }
  • R. Lee, C. Ippolito, Y. Yeh, J. Spritzer, and G. Phelps, “Payload-Directed Control of Geophysical Magnetic Surveys,” in Infotech@Aerospace Conference, 2010.
    [Bibtex]
    @InProceedings{Lee2010a,
    Title = {Payload-Directed Control of Geophysical Magnetic Surveys},
    Author = {Lee, Ritchie and Ippolito, Corey and Yeh, Yoo-Hsiu and Spritzer, John and Phelps, Geoffrey},
    Booktitle = aiaa_info,
    organization = {AIAA},
    Year = {2010},
    Month = {April},
    Owner = {rcnlee}
    }
  • G. A. Phelps, C. Ippolito, R. Lee, J. Spritzer, and Y. Yeh, “Investigations into Near-Real-Time Surveying for Geophysical Data Collection Using an Autonomous Ground Vehicle,” United States Geological Survey (USGS) 2014.
    [Bibtex]
    @TechReport{Phelps2014,
    author = {Phelps, Geoffrey A and Ippolito, Corey and Lee, Ritchie and Spritzer, John and Yeh, Yoo-Hsiu},
    title = {Investigations into Near-Real-Time Surveying for Geophysical Data Collection Using an Autonomous Ground Vehicle},
    institution = usgs,
    year = {2014},
    owner = {rcnlee},
    }
 

Modeling Multiple Human Decision-Makers

Network-Form Game (NFG) is a novel framework for modeling stochastic systems with multiple human actors.  The approach combines ideas from game theory, probabilistic models, and reinforcement learning to capture stochastic, bounded rational, and strategic decision-making behavior.  NFG models scenarios over a Bayesian network but adds special decision nodes for the actors.  The decision nodes are aware of the structure of the scenario but receive limited information from their observation.  They behave strategically using level-K thinking.  The NFG approach has been applied to model aircraft collision avoidance, human-in-the-loop experiments, and cyber-security of computer networks and smart power grids.

Related Publications:

  • R. Lee and D. H. Wolpert, “Game Theoretic Modeling of Pilot Behavior During Mid-Air Encounters,” in Decision Making with Imperfect Decision Makers, Springer, 2012, vol. 28, pp. 75-111.
    [Bibtex]
    @InCollection{Lee2012,
    author = {Lee, Ritchie and Wolpert, David H.},
    title = {Game Theoretic Modeling of Pilot Behavior During Mid-Air Encounters},
    booktitle = {Decision Making with Imperfect Decision Makers},
    publisher = {Springer},
    year = {2012},
    volume = {28},
    series = {Intelligent Systems Reference Library},
    chapter = {4},
    pages = {75-111},
    owner = {rcnlee},
    }
  • E. J. Schlicht, R. Lee, D. H. Wolpert, M. J. Kochenderfer, and B. Tracey, “Predicting the Behavior of Interacting Humans by Fusing Data from Multiple Sources,” in Conference on Uncertainty in Artificial Intelligence (UAI), 2012.
    [Bibtex]
    @InProceedings{Schlicht2012,
    Title = {Predicting the Behavior of Interacting Humans by Fusing Data from Multiple Sources},
    Author = {Erik J. Schlicht and Ritchie Lee and David H. Wolpert and Mykel J. Kochenderfer and Brendan Tracey},
    Booktitle = uai,
    Organization = {AUAI},
    Year = {2012},
    Owner = {rcnlee},
    }
  • G. Yan, R. Lee, A. Kent, and D. H. Wolpert, “Towards a Bayesian Network Game Framework for Evaluating DDoS Attacks and Defense,” in Conference on Computer and Communications Security (CCS), 2012, p. 553–566.
    [Bibtex]
    @InProceedings{Yan2012,
    Title = {Towards a {B}ayesian Network Game Framework for Evaluating {DDoS} Attacks and Defense},
    Author = {Yan, Guanhua and Lee, Ritchie and Kent, Alex and Wolpert, David H.},
    Booktitle = acmccs,
    Year = {2012},
    Organization = {ACM},
    Pages = {553--566},
    Owner = {rcnlee}
    }
  • Y. Yildiz, R. Lee, and G. Brat, “Using Game Theoretic Models to Predict Pilot Behavior in NextGen Merging and Landing Scenario,” in SciTech, Modeling and Simulation Technologies Conference (MST), 2012.
    [Bibtex]
    @InProceedings{Yildiz2012,
    Title = {Using Game Theoretic Models to Predict Pilot Behavior in {N}ext{G}en Merging and Landing Scenario},
    Author = {Yildiz, Yildiray and Lee, Ritchie and Brat, Guillaume},
    Organization = {AIAA},
    booktitle = aiaa_mst,
    Year = {2012},
    Owner = {rcnlee}
    }
  • R. Lee, D. H. Wolpert, J. Bono, S. Backhaus, R. Bent, and B. Tracey, “Counter-Factual Reinforcement Learning: How to Model Decision-Makers That Anticipate the Future,” in Decision Making and Imperfection, Springer, 2013, vol. 474, pp. 101-128.
    [Bibtex]
    @InCollection{Lee2013,
    author = {Lee, Ritchie and Wolpert, David H. and Bono, James and Backhaus, Scott and Bent, Russell and Tracey, Brendan},
    title = {Counter-Factual Reinforcement Learning: How to Model Decision-Makers That Anticipate the Future},
    booktitle = {Decision Making and Imperfection},
    publisher = {Springer},
    year = {2013},
    volume = {474},
    series = {Studies in Computational Intelligence},
    chapter = {4},
    pages = {101-128},
    owner = {rcnlee},
    }
  • S. Backhaus, R. Bent, J. Bono, R. Lee, B. Tracey, D. H. Wolpert, D. Xie, and Y. Yildiz, “Cyber-Physical Security: A Game Theory Model of Humans Interacting Over Control Systems,” IEEE Transactions on Smart Grid, vol. 4, iss. 4, p. 2320–2327, 2013.
    [Bibtex]
    @Article{Backhaus2013,
    author = {Backhaus, Scott and Bent, Russell and Bono, James and Lee, Ritchie and Tracey, Brendan and Wolpert, David H. and Xie, Dongping and Yildiz, Yildiray},
    title = {Cyber-Physical Security: A Game Theory Model of Humans Interacting Over Control Systems},
    journal = ieeesg,
    year = {2013},
    volume = {4},
    number = {4},
    pages = {2320--2327},
    owner = {rcnlee},
    publisher = {IEEE},
    }
 

Multi-Fidelity Air Traffic Simulator

Multi-fidelity simulator (MFSim) is a fast-time air traffic simulator of the national airspace that simulates aircraft at different levels of abstraction in order to gain computational savings.  The simulator is used as a research platform for multi-agent reinforcement learning of air traffic management policies.

Related Publications:

  • A. Agogino, A. Iscen, R. Lee, D. Bowers, K. Tumer, and G. Brat, “Scalable Hierarchical Multifidelity Simulation and Multiagent Optimization of Air Traffic,” in AAMAS Workshop on Massive Multiagent Systems, 2015.
    [Bibtex]
    @InProceedings{Agogino2015,
    Title = {Scalable Hierarchical Multifidelity Simulation and Multiagent Optimization of Air Traffic},
    Author = {Adrian Agogino and Atil Iscen and Ritchie Lee and Devin Bowers and Kagan Tumer and Guillaume Brat},
    Booktitle = {{AAMAS} Workshop on Massive Multiagent Systems},
    Year = {2015},
    Owner = {rcnlee}
    }