Specifying your LLM sampler
============================

Use 'https' to request your LLM
---------------------------------

.. note::
    The `LLM` class (an abstract class) defines how to access the LLM.
    You can either deploy an LLM locally on your own device/server or use an LLM API.
    The user should create a new child class of the `LLM` class (extend `LLM`) and implement (override) the `draw_sample` function.

Initialization of the user-defined sampler class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There is a keyword argument `auto_trim` in the `LLM` class, with a default value of `True`. This means that regardless of whether the user chooses a code completion model (such as StarCoder, CodeLlama-Python, etc.) or a chat model (GPT series, Llama series, etc.), we can automatically identify the “useful part” without descriptions and truncated code.

.. tip::
    Therefore, unless there is a special issue, please **always leave 'auto_trim' default**.

Implementation of the draw_sample function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The `draw_sample` function decides how to obtain the generated content from the LLM and return the `str`-typed content

.. note::
    feel free to return the answer generated by LLM, which may incorporate some useless descriptions, as they will be trimmed automatically by our trimmer).

Here, we show a brief example of using LLM API.

.. code:: python

    from llm4ad.tools.llm.llm_api_https import HttpsApi

    # note that the 'host' has no 'https'
    sampler = HttpsApi(host='api.bltcy.ai', key='Your API key', model='gpt-3.5-turbo', timeout=30)

You can also implement your own sampler.

.. code:: python

    import llm4ad
    import time
    import http.client
    import json

    class MyLLM(llm4ad.base.LLM):
        def __init__(self):
            super().__init__()

        def draw_sample(self, prompt: str | Any, *args, **kwargs) -> str:
            while True:
                try:
                    conn = http.client.HTTPSConnection(f'{api_endpoint}', timeout=30)
                    payload = json.dumps({
                        'max_tokens': 512,
                        'model': 'gpt-3.5-turbo',
                        'messages': [{'role': 'user', 'content': prompt}]
                    })
                    headers = {
                        'Authorization': f'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                        'User-Agent': 'Apifox/1.0.0 (https://apifox.com)',
                        'Content-Type': 'application/json'
                    }
                    conn.request('POST', '/v1/chat/completions', payload, headers)
                    res = conn.getresponse()
                    data = res.read().decode('utf-8')
                    data = json.loads(data)
                    response = data['choices'][0]['message']['content']
                    return response
                except Exception as e:
                    continue


Deploy your own LLM
-----------------------

It is also encouraged to deploy your own LLM.
As we have tested diverse existing open-source LLMs, and they work well in our platform.

.. important::
    Please be noted that not all LLMs can follow our prompt well.
    In addition, the behaviors of different LLMs may be inconsistent.
    An brief example of different behaviors of LLM is shown below:

    - Prompt content:
        .. code:: python

            def f_v1(a, b):
                return a + b

            def f_v2():
                """improved version of `f_v1`"""

    - Output content example 1 (GPT-4o may perform like this):
        .. code:: python

            Here is an implementation of f_v2:

            ```python
            def f_v2(a, b):
                return a * (a+b)
            ```

    - Output content example 2 (DeepSeek-v2 may perform like this):
        .. code:: python

            Okay, the improved version is shown as follows:

            ```python
            def f_v1(a, b):
                return a + b

            def f_v2(a, b):
                return a * (a+b)
            ```

    Since the platform preserves the first function and trim out the rest functions by default,
    we can successfully extract the generated function in example 1.
    However, we cannot get the newest version in example 2.


Here we show a brief way to tackle the situation in example 2, the code is shown below.
In this code, we find the last function in the generated content as our target function.

.. code:: python

    class LLM4Example2(llm4ad.base.LLM):

    def __init__(self):
        super().__init__()

        def draw_sample(self, prompt: str | Any, *args, **kwargs) -> str:
            while True:
                try:
                    conn = http.client.HTTPSConnection(f'{api_endpoint}', timeout=30)
                    payload = json.dumps({
                        'max_tokens': 512,
                        'model': 'gpt-3.5-turbo',
                        'messages': [{'role': 'user', 'content': prompt}]
                    })
                    headers = {
                        'Authorization': f'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                        'User-Agent': 'Apifox/1.0.0 (https://apifox.com)',
                        'Content-Type': 'application/json'
                    }
                    conn.request('POST', '/v1/chat/completions', payload, headers)
                    res = conn.getresponse()
                    data = res.read().decode('utf-8')
                    data = json.loads(data)
                    response = data['choices'][0]['message']['content']
                    return self._trim_response(response)
                except Exception as e:
                    continue

    def _trim_response(self, response: str):
        from llm4ad.base import TextFunctionProgramConverter as TFPC
        import re
        match = re.search(r'```python\n(.*?)\n```', response, re.DOTALL)
        if match:
            extracted_text = match.group(1)
            prog = TFPC.text_to_program(extracted_text)
            last_func = prog.functions[-1]
            return str(last_func)
        else:
            extracted_text = response
        return extracted_text