OpenMark
MCP server by OpenMark · mcp
OpenMark benchmarks LLMs on the user's own task with real API calls and deterministic scoring. Typical flow: call get_task_format, write a task YAML from the user's real prompt and a few realistic inputs with expected outputs, check it with validate_task (free; fix what it reports), pick 3-8 models with list_models, check the price with estimate_cost (or pass max_credits to run_benchmark; both take the app's run settings such as max_tokens, preset and temperature_discovery), call run_benchmark, then poll get_results with the job_id until it is completed (usually 1-5 minutes). get_results ranks by accuracy by default (highest score; ties go to the cheaper, then the faster model) and also lists the best pick for cost, speed and consistency. Ask the user what matters most and pass it as optimize_for; if a much cheaper model is within a couple of points of the winner, say so. Explain losses with the misses get_results lists. For images or documents, call prepare_attachments first. Offer w…
Endpoint: https://openmark.ai/mcp
Verification: observed. Source: crawl.
Capabilities (10)
- get_task_format
- validate_task
- list_models
- prepare_attachments
- estimate_cost
- run_benchmark
- get_results
- export_openclaw_csv
- get_account
- watch_task