Action Selection Strategy Heuristics for Cart Pole

Problem

The Cart Pole problem is a classic reinforcement learning task in OpenAI Gym, aiming to maximize the duration that a pole remains balanced on a moving cart within specific position and angle constraints.

• Given: A cart with its position uniformly randomly assigned within the range (-0.05, 0.05), a pole stand on the cart.

• Objective: Maximize the total number of iterations during which the pole angle remains within the range (-0.2095, 0.2095) (or ±12°).

• Constraints:

  • The position of the cart along the x-axis must be between (-2.4, 2.4).

  • The pole angle must between (-.2095, .2095).

  • The possible actions are:

    1. Push cart to the left

    2. Push cart to the right

Algorithm Design Task

• Action selection strategy heuristics: Push the cart to move left and right iteratively, keeping the pole angle within the specified range. The task is to design the heuristic for selecting the action in each iteration.

  • Inputs: Cart position, cart velocity, pole angle, pole angular velocity, last selected action.

  • Outputs: Next action.

Evaluation

• Dataset: Each designed algorithm is evaluated on gym environment.

• Fitness:

  • If the pole can’t stand within the iteration limit: - final_iteration / max_iteration + 1 + 2.4, where 1 is ratio bias and 2.4 is position bias.

  • If the pole stands within the iteration limit: cart_position.

Template:

template_program = '''
import numpy as np
def choose_action(cp: float, cv: float, pa: float, pav: float, last_action: int) -> int: 
    """
    Design a novel algorithm to select the action in each step.

    Args:
        cp: cart position, float between [-2.4, 2.4].
        cv: cart velocity, float between [-inf, inf].
        pa: pole angle, float between [-0.2095, 0.2095].
        pav: pole angular velocity, float between [-inf, inf].
        last_action: cart's next move, a int ranges between [0, 1, 2].

    Return:
         An integer representing the selected action for the cart.
         0: push cart to the left
         1: push cart to the right

    """
    # this is a placehold, replace it with your algorithm
    action =  np.random.randint(2)

    return action
'''

task_description = "I need help designing an innovative heuristic strategy function to prevent a pole from toppling over a cart, step by step. At each step, the function should select a specific action based on the pole's current state to move the cart, aiming to keep the pole balanced and upright without moving the cart too far from the center."