A new tutorial has emerged, showcasing a safety-critical reinforcement learning pipeline that operates solely on fixed, offline data. This approach is especially important in environments where live exploration could lead to dangerous outcomes. The tutorial guides users through creating a custom environment, generating a dataset from a constrained policy, and training two types of agents: a Behavior Cloning baseline and a Conservative Q-Learning agent using the d3rlpy library.
The process begins with setting up the environment, which involves installing necessary libraries and fixing random seeds to ensure consistent results. The tutorial emphasizes the importance of configuring the computation device to maintain uniformity across different systems. A utility function is also provided to create configuration objects safely across various versions of d3rlpy.
Central to the tutorial is the design of a safety-critical GridWorld environment. This environment includes hazards, terminal states, and stochastic transitions, which reflect real-world safety constraints. The tutorial details how to encode penalties for unsafe states and rewards for successfully reaching the goal.
To gather data, a constrained behavior policy is implemented, which generates offline data without engaging in risky exploration. This policy is rolled out to collect trajectories, which are then structured into episodes and converted into a format that d3rlpy can utilize for offline learning.
The tutorial also provides utilities to iterate through episodes effectively and visualize the dataset. This helps users understand state visitation and data bias, as well as analyze reward distributions to assess the learning signal available to the agent.
Evaluation routines are included to measure the performance of the trained policies without uncontrolled exploration. Metrics such as returns, hazard rates, and goal rates are computed to gauge the effectiveness of the agents. Additionally, a diagnostic tool is introduced to quantify how often the learned actions deviate from the actions taken in the dataset.
The tutorial culminates in training both the Behavior Cloning and Conservative Q-Learning agents using the collected offline data. The results are compared through controlled evaluations, demonstrating that Conservative Q-Learning tends to yield more reliable policies in safety-sensitive environments.
Overall, this tutorial offers a comprehensive and reproducible workflow for offline reinforcement learning. It highlights the potential for extending these methods to more complex fields like robotics, healthcare, or finance, all while prioritizing safety. For those interested in the full codes and implementation details, a link is provided to access the complete tutorial.