Lm Eval

post https://api.monsterapi.ai/v1/deploy/evaluation/llm/lm_eval

Use Eluether Evaluation Harness to evaluate on lm_eval engine. Supported evals are as follow for engine:

lm_eval: mmlu, gsm8k, hellaswag, arc, truthfulqa, winogrande

Models >8B and context more than 8k are not currently supported. Support will be added shortly.

Body Params

deployment_name

string

Defaults to Null

Unique deployment for the instance, auto-generated if not provided.

basemodel_path

string

Defaults to mistralai/Mistral-7B-v0.1

loramodel_path

string

Lora adapter HF path or HTTPS link to download the model.

eval_engine

const

required

task

string

Defaults to gsm8k,hellaswag

command seperated supported tasks. For supported see route description above.

Responses

Language

Credentials

Bearer

JWT


xxxxxxxxxx
11
1
curl --request POST \
2
     --url https://api.monsterapi.ai/v1/deploy/evaluation/llm/lm_eval \
3
     --header 'accept: application/json' \
4
     --header 'content-type: application/json' \
5
     --data '
6
{
7
  "deployment_name": "Null",
8
  "basemodel_path": "mistralai/Mistral-7B-v0.1",
9
  "task": "gsm8k,hellaswag"
10
}
11
'

Click Try It! to start a request and see the response here! Or choose an example:

application/json

200Success response

402Not enough tokens to launch an instance

403User has crossed the maximum number of running deployments

405User ID or user email not found in the request

422Validation Error

500Failure response