This video shows a step-by-step process to install AutoCoder in GGUF format on Google Colab for free.
Code:
!wget https://huggingface.co/ukung/AutoCoder_S_6.7B-GGUF/resolve/main/AutoCoder_S_6.7B-q5_k_m.gguf?download=true
import torch
!pip install llama-cpp-python
from llama_cpp import Llama
llm = Llama(
model_path="/content/AutoCoder_S_6.7B-q5_k_m.gguf",
n_gpu_layers=32,
seed=1337,
n_ctx=2048,
)
output = llm(
"Q: write a script to say hello? A: ",
max_tokens=1500,
stop=["Q:", "\
"],
echo=True
)
print(output['choices'][0])