mirror of
https://github.com/informaticker/vllm-fly.git
synced 2024-10-18 20:25:03 +02:00
44 lines
883 B
Markdown
44 lines
883 B
Markdown
# [vLLM](https://github.com/vllm-project/vllm) demo app for Fly.io
|
||
|
||
|
||
First deploy with:
|
||
```
|
||
fly launch
|
||
```
|
||
|
||
from there update by running: `fly deploy`
|
||
|
||
Once deploy, interact with the API at https://$APPNAME.fly.dev/
|
||
|
||
|
||
```
|
||
❯ curl https://vllm-demo.fly.dev/v1/completions \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "facebook/opt-125m",
|
||
"prompt": "San Francisco is a",
|
||
"max_tokens": 7,
|
||
"temperature": 0
|
||
}' -s |jq .
|
||
{
|
||
"id": "cmpl-b4b03ec33d794a50ba5cf2801d807025",
|
||
"object": "text_completion",
|
||
"created": 1716250075,
|
||
"model": "facebook/opt-125m",
|
||
"choices": [
|
||
{
|
||
"index": 0,
|
||
"text": " great place to live. I",
|
||
"logprobs": null,
|
||
"finish_reason": "length",
|
||
"stop_reason": null
|
||
}
|
||
],
|
||
"usage": {
|
||
"prompt_tokens": 5,
|
||
"total_tokens": 12,
|
||
"completion_tokens": 7
|
||
}
|
||
}
|
||
```
|