informaticker/vllm-fly
1
0
Fork 0
mirror of https://github.com/informaticker/vllm-fly.git synced 2024-11-21 17:21:58 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
elia 2024-10-03 16:26:09 +02:00 committed by GitHub
parent 69206fd301
commit 0073f43af8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
README.md
View File

@ -1,43 +1 @@
# [vLLM](https://github.com/vllm-project/vllm) demo app for Fly.io schmerzen
First deploy with:
```
fly launch
```
from there update by running: `fly deploy`
Once deploy, interact with the API at https://$APPNAME.fly.dev/
```
❯ curl https://vllm-demo.fly.dev/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "facebook/opt-125m",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}' -s |jq .
{
"id": "cmpl-b4b03ec33d794a50ba5cf2801d807025",
"object": "text_completion",
"created": 1716250075,
"model": "facebook/opt-125m",
"choices": [
{
"index": 0,
"text": " great place to live. I",
"logprobs": null,
"finish_reason": "length",
"stop_reason": null
}
],
"usage": {
"prompt_tokens": 5,
"total_tokens": 12,
"completion_tokens": 7
}
}
```