5-2. Windy Frozen Lake 구현

김성훈 교수님의 강의내용을 정리한 내용입니다.

출처 : http://hunkim.github.io/ml/

모두를 위한 머신러닝/딥러닝 강의

---

 

새로운 Q-learning 알고리즘

번경된 업데이트 식을 그대로 적어 구현하자!

FrozenLake-v0을 사용하여 (is_slippery = True) 구현하자.

 

빠른 학습을위해 학습률을 0.85로 설정하자.

이전보다 더 나은 결과를 보인다!

 

이유?

-> 가끔일어나는 slippery때문에 Q값이 적절하지 않은 값으로 설정될 수 있다.

이를 방지하기 위해 이전의 학습을 어느정도 반영하는 방법이 필요하다!

---

구현 코드 (환경: ubuntu:16.04 python 3.6)

import time
 
import gym
import numpy as np
 
import utils.prints as print_utils
 
N_ACTIONS = 4
N_STATES = 16
 
LEARNING_RATE = .5
DISCOUNT_RATE = .98
 
N_EPISODES = 2000
 
def main():
    """Main"""
    frozone_lake_env = gym.make("FrozenLake-v0")
 
    # Initialize table with all zeros
    Q = np.zeros([N_STATES, N_ACTIONS])
 
    # Set learning parameters
 
    # create lists to contain total rewards and steps per episode
    rewards = []
 
    for i in range(N_EPISODES):
        # Reset environment and get first new observation
        state = frozone_lake_env.reset()
        episode_reward = 0
        done = False
 
        # The Q-Table learning algorithm
        while not done:
            # Choose an action by greedily (with noise) picking from Q table
            noise = np.random.randn(1, N_ACTIONS) / (i + 1)
            action = np.argmax(Q[state, :] + noise)
 
            # Get new state and reward from environment
            new_state, reward, done, _ = frozone_lake_env.step(action)
 
            reward = -1 if done and reward < 1 else reward
 
            # Update Q-Table with new knowledge using learning rate
            Q[state, action] = (
                1 - LEARNING_RATE) * Q[state, action] + LEARNING_RATE * (
                    reward + DISCOUNT_RATE * np.max(Q[new_state, :]))
 
            episode_reward += reward
            state = new_state
 
        rewards.append(episode_reward)
 
    print("Score over time: " + str(sum(rewards) / N_EPISODES))
    print("Final Q-Table Values")
 
    for i in range(10):
        # Reset environment and get first new observation
        state = frozone_lake_env.reset()
        episode_reward = 0
        done = False
 
        # The Q-Table learning algorithm
        while not done:
            # Choose an action by greedily (with noise) picking from Q table
            action = np.argmax(Q[state, :])
 
            # Get new state and reward from environment
            new_state, reward, done, _ = frozone_lake_env.step(action)
            print_utils.clear_screen()
            frozone_lake_env.render()
            time.sleep(.1)
 
            episode_reward += reward
            state = new_state
 
            if done:
                print("Episode Reward: {}".format(episode_reward))
                print_utils.print_result(episode_reward)
 
        rewards.append(episode_reward)
 
    frozone_lake_env.close()
 
 
if __name__ == '__main__':
    main()

utils.prints 함수 참고 : https://github.com/hunkim/ReinforcementZeroToAll