Do you want to operate an LLM locally in your own company? Well the simple solution is to use OpenLLM api as shown in this simple and quick demo. There is built in support for including StableLM, Falcon, Dolly, Flan-T5, ChatGLM, StarCoder and many more.

Commands Used:

- Create notebook 

- Install pre-reqs 

   !pip install einops

   !pip install xformers

   !pip install safetensors

- !pip install openllm

- !openllm -h

- !openllm start opt 

- !openllm query --endpoint <NOTEBOOK url> "hi"

Through Python:

import openllm

client = openllm.client.HTTPClient('<notebook URL>')

response = client.query('who are you?')

print (response)

Errors:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[9], line 2
      1 import openllm
----> 2 client = openllm.client.HTTPClient('https://openllm.notebook.ap-southeast-2.sagemaker.aws/proxy/3000/')
      3 response = client.query('who are you?')
      4 print (response)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/openllm_client/runtimes/http.py:77, in HTTPClient.__init__(self, address, timeout)
     75 def __init__(self, address: str, timeout: int = 30):
     76     address = address if "://" in address else "http://" + address
---> 77     self._host, self._port = urlparse(address).netloc.split(":")
     78     super().__init__(address, timeout)

ValueError: not enough values to unpack (expected 2, got 1)

Solution: Make sure your endpoint URL and proxy is correct.