Specifying your LLM sampler#

Use ‘https’ to request your LLM#

Note

The LLM class (an abstract class) defines how to access the LLM. You can either deploy an LLM locally on your own device/server or use an LLM API. The user should create a new child class of the LLM class (extend LLM) and implement (override) the draw_sample function.

Initialization of the user-defined sampler class#

There is a keyword argument auto_trim in the LLM class, with a default value of True. This means that regardless of whether the user chooses a code completion model (such as StarCoder, CodeLlama-Python, etc.) or a chat model (GPT series, Llama series, etc.), we can automatically identify the “useful part” without descriptions and truncated code.

Tip

Therefore, unless there is a special issue, please always leave ‘auto_trim’ default.

Implementation of the draw_sample function#

The draw_sample function decides how to obtain the generated content from the LLM and return the str-typed content

Note

feel free to return the answer generated by LLM, which may incorporate some useless descriptions, as they will be trimmed automatically by our trimmer).

Here, we show a brief example of using LLM API.

from llm4ad.tools.llm.llm_api_https import HttpsApi

# note that the 'host' has no 'https'
sampler = HttpsApi(host='api.bltcy.ai', key='Your API key', model='gpt-3.5-turbo', timeout=30)

You can also implement your own sampler.

import llm4ad
import time
import http.client
import json

class MyLLM(llm4ad.base.LLM):
    def __init__(self):
        super().__init__()

    def draw_sample(self, prompt: str | Any, *args, **kwargs) -> str:
        while True:
            try:
                conn = http.client.HTTPSConnection(f'{api_endpoint}', timeout=30)
                payload = json.dumps({
                    'max_tokens': 512,
                    'model': 'gpt-3.5-turbo',
                    'messages': [{'role': 'user', 'content': prompt}]
                })
                headers = {
                    'Authorization': f'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                    'User-Agent': 'Apifox/1.0.0 (https://apifox.com)',
                    'Content-Type': 'application/json'
                }
                conn.request('POST', '/v1/chat/completions', payload, headers)
                res = conn.getresponse()
                data = res.read().decode('utf-8')
                data = json.loads(data)
                response = data['choices'][0]['message']['content']
                return response
            except Exception as e:
                continue

Deploy your own LLM#

It is also encouraged to deploy your own LLM. As we have tested diverse existing open-source LLMs, and they work well in our platform.

Important

Please be noted that not all LLMs can follow our prompt well. In addition, the behaviors of different LLMs may be inconsistent. An brief example of different behaviors of LLM is shown below:

Prompt content:

def f_v1(a, b):
    return a + b

def f_v2():
    """improved version of `f_v1`"""

Output content example 1 (GPT-4o may perform like this):

Here is an implementation of f_v2:

```python
def f_v2(a, b):
    return a * (a+b)
```

Output content example 2 (DeepSeek-v2 may perform like this):

Okay, the improved version is shown as follows:

```python
def f_v1(a, b):
    return a + b

def f_v2(a, b):
    return a * (a+b)
```

Since the platform preserves the first function and trim out the rest functions by default, we can successfully extract the generated function in example 1. However, we cannot get the newest version in example 2.

Here we show a brief way to tackle the situation in example 2, the code is shown below. In this code, we find the last function in the generated content as our target function.

class LLM4Example2(llm4ad.base.LLM):

def __init__(self):
    super().__init__()

    def draw_sample(self, prompt: str | Any, *args, **kwargs) -> str:
        while True:
            try:
                conn = http.client.HTTPSConnection(f'{api_endpoint}', timeout=30)
                payload = json.dumps({
                    'max_tokens': 512,
                    'model': 'gpt-3.5-turbo',
                    'messages': [{'role': 'user', 'content': prompt}]
                })
                headers = {
                    'Authorization': f'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                    'User-Agent': 'Apifox/1.0.0 (https://apifox.com)',
                    'Content-Type': 'application/json'
                }
                conn.request('POST', '/v1/chat/completions', payload, headers)
                res = conn.getresponse()
                data = res.read().decode('utf-8')
                data = json.loads(data)
                response = data['choices'][0]['message']['content']
                return self._trim_response(response)
            except Exception as e:
                continue

def _trim_response(self, response: str):
    from llm4ad.base import TextFunctionProgramConverter as TFPC
    import re
    match = re.search(r'```python\n(.*?)\n```', response, re.DOTALL)
    if match:
        extracted_text = match.group(1)
        prog = TFPC.text_to_program(extracted_text)
        last_func = prog.functions[-1]
        return str(last_func)
    else:
        extracted_text = response
    return extracted_text