Local LLM for VS code copilot - Revision history

Davrot@uni-bremen.de: /* vllm-qwen-coder.service */

2025-12-09T19:55:13Z

vllm-qwen-coder.service

← Older revision		Revision as of 19:55, 9 December 2025
Line 24:		Line 24:
	Qwen3 Coder falls in the category "strange".		Qwen3 Coder falls in the category "strange".

	You '''really really''' want to make a new custom agent setting in VS Code Insider Copilot. You can get inspired by other model setting:		<big>You '''really really''' want to make a new custom agent setting in VS Code Insider Copilot. You can get inspired by other model setting:</big>

	https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/VSCode%20Agent/gpt-5-mini.txt		<big>https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/VSCode%20Agent/gpt-5-mini.txt</big>

	=== vllm-qwen-coder.service ===		=== vllm-qwen-coder.service ===

Davrot@uni-bremen.de: /* VS Code JSON */

2025-12-09T19:46:03Z

VS Code JSON

← Older revision		Revision as of 19:46, 9 December 2025
Line 23:		Line 23:

	Qwen3 Coder falls in the category "strange".		Qwen3 Coder falls in the category "strange".

			You '''really really''' want to make a new custom agent setting in VS Code Insider Copilot. You can get inspired by other model setting:

			https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/VSCode%20Agent/gpt-5-mini.txt

	=== vllm-qwen-coder.service ===		=== vllm-qwen-coder.service ===

Davrot@uni-bremen.de: /* How to install vLLM */

2025-12-09T17:20:56Z

How to install vLLM

@@ Line 69: / Line 69: @@
 <code>--tool-parser-plugin</code> can ruin your day.  Look under https://docs.vllm.ai/en/latest/features/tool_calling/#hermes-models-hermes what the correct value for your model is.
-Also we need to follow this equation:<syntaxhighlight>maxInputTokens = max-model-len - maxOutputTokens - safety_buffer</syntaxhighlight>max-model-len is from the model page (256k = 256 *1024) = 262144
+==== maxInputTokens ====
 https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
 == How to install vLLM ==

Davrot@uni-bremen.de: /* vllm-qwen-coder.service */

2025-12-09T17:13:25Z

vllm-qwen-coder.service

← Older revision		Revision as of 17:13, 9 December 2025
Line 69:		Line 69:
	<code>--tool-parser-plugin</code> can ruin your day. Look under https://docs.vllm.ai/en/latest/features/tool_calling/#hermes-models-hermes what the correct value for your model is.		<code>--tool-parser-plugin</code> can ruin your day. Look under https://docs.vllm.ai/en/latest/features/tool_calling/#hermes-models-hermes what the correct value for your model is.

	Also we need to follow this equation:<syntaxhighlight>		Also we need to follow this equation:<syntaxhighlight>maxInputTokens = max-model-len - maxOutputTokens - safety_buffer</syntaxhighlight>max-model-len is from the model page (256k = 256 *1024) = 262144
	maxInputTokens = max-model-len - maxOutputTokens - safety_buffer
	</syntaxhighlight>		https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8

	== How to install vLLM ==		== How to install vLLM ==

Davrot@uni-bremen.de: /* vllm-qwen-coder.service */

2025-12-09T17:01:50Z

vllm-qwen-coder.service

← Older revision		Revision as of 17:01, 9 December 2025
Line 8:		Line 8:

	=== VS Code JSON ===		=== VS Code JSON ===
	<syntaxhighlight lang="json">		<syntaxhighlight lang="json">"github.copilot.chat.customOAIModels": {
	"github.copilot.chat.customOAIModels": {
	"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": {		"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": {
	"name": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8",		"name": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8",
Line 19:		Line 18:
	"maxOutputTokens": 8192,		"maxOutputTokens": 8192,
	"requiresAPIKey": false		"requiresAPIKey": false
	}		}</syntaxhighlight>Ollama is not a good choice for an agent model. This has something to do with the format the agent commands are delivered by the model.
	</syntaxhighlight>Ollama is not a good choice for an agent model. This has something to do with the format the agent commands are delivered by the model.

	There are at least three modes: XML, JSON, and something strange.		There are at least three modes: XML, JSON, and something strange.
Line 53:		Line 51:

	[Install]		[Install]
	WantedBy=multi-user.target</syntaxhighlight>To be installed and activated via<syntaxhighlight lang="bash">		WantedBy=multi-user.target</syntaxhighlight>Environment="HF_HOME=/ollama_coder/huggingface_cache" tell the tool where to store the model data (HF -> huggingface).

			To be installed and activated via<syntaxhighlight lang="bash">
	cp -f vllm-qwen-coder.service /etc/systemd/system/vllm-qwen-coder.service		cp -f vllm-qwen-coder.service /etc/systemd/system/vllm-qwen-coder.service
	systemctl daemon-reload		systemctl daemon-reload
Line 61:		Line 61:
	systemctl status vllm-qwen-coder		systemctl status vllm-qwen-coder

	</syntaxhighlight>This ~~look~~ harmless but		</syntaxhighlight>This looks harmless but (https://docs.vllm.ai/en/latest/features/tool_calling/#none-function-calling)

			* <code>--enable-auto-tool-choice</code> -- '''mandatory''' Auto tool choice. It tells vLLM that you want to enable the model to generate its own tool calls when it deems appropriate.

			* <code>--tool-parser-plugin</code> -- '''optional''' tool parser plugin used to register user defined tool parsers into vllm, the registered tool parser name can be specified in <code>--tool-call-parser</code>.

			<code>--tool-parser-plugin</code> can ruin your day. Look under https://docs.vllm.ai/en/latest/features/tool_calling/#hermes-models-hermes what the correct value for your model is.

			Also we need to follow this equation:<syntaxhighlight>
			maxInputTokens = max-model-len - maxOutputTokens - safety_buffer
			</syntaxhighlight>

	== How to install vLLM ==		== How to install vLLM ==

Davrot@uni-bremen.de: /* How to install vLLM */

2025-12-09T16:52:14Z

How to install vLLM

← Older revision		Revision as of 16:52, 9 December 2025
Line 3:		Line 3:
	Add in a custom model to copilot in vs code insiders is easy. Let it do agent stuff is really hard.		Add in a custom model to copilot in vs code insiders is easy. Let it do agent stuff is really hard.

	Here an example: Qwen3 Coder		Here an example: Qwen3 Coder

			Also change the directories if you need to.

	=== VS Code JSON ===		=== VS Code JSON ===
Line 18:		Line 20:
	"requiresAPIKey": false		"requiresAPIKey": false
	}		}
	</syntaxhighlight>Ollama is not a good choice for an agent model. This has something to do with the format the agent commands are delivered by the model. There are at least three modes. XML, JSON, something strange. Qwen3 Coder falls in the category strange.		</syntaxhighlight>Ollama is not a good choice for an agent model. This has something to do with the format the agent commands are delivered by the model.

			There are at least three modes: XML, JSON, and something strange.

			Qwen3 Coder falls in the category "strange".

	=== vllm-qwen-coder.service ===		=== vllm-qwen-coder.service ===
Line 61:		Line 67:
	cd /ollama_coder		cd /ollama_coder
	uv init vllm-project		uv init vllm-project
	</syntaxhighlight>~~Put~~ pyproject.toml into /ollama_coder/vllm-project<syntaxhighlight lang="bash">		</syntaxhighlight>Copy pyproject.toml into /ollama_coder/vllm-project<syntaxhighlight lang="bash">
	uv sync		uv sync
	</syntaxhighlight>		</syntaxhighlight>

Davrot@uni-bremen.de at 16:50, 9 December 2025

2025-12-09T16:50:30Z

← Older revision		Revision as of 16:50, 9 December 2025
Line 5:		Line 5:
	Here an example: Qwen3 Coder		Here an example: Qwen3 Coder

	VS Code JSON:<syntaxhighlight lang="json">		=== VS Code JSON ===
			<syntaxhighlight lang="json">
	"github.copilot.chat.customOAIModels": {		"github.copilot.chat.customOAIModels": {
	"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": {		"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": {
Line 19:		Line 20:
	</syntaxhighlight>Ollama is not a good choice for an agent model. This has something to do with the format the agent commands are delivered by the model. There are at least three modes. XML, JSON, something strange. Qwen3 Coder falls in the category strange.		</syntaxhighlight>Ollama is not a good choice for an agent model. This has something to do with the format the agent commands are delivered by the model. There are at least three modes. XML, JSON, something strange. Qwen3 Coder falls in the category strange.

	vllm-qwen-coder.service<syntaxhighlight lang="bash">[Unit]		=== vllm-qwen-coder.service ===
			<syntaxhighlight lang="bash">[Unit]
	Description=vLLM Qwen3 Coder Service		Description=vLLM Qwen3 Coder Service
	After=network.target		After=network.target
Line 61:		Line 63:
	</syntaxhighlight>Put pyproject.toml into /ollama_coder/vllm-project<syntaxhighlight lang="bash">		</syntaxhighlight>Put pyproject.toml into /ollama_coder/vllm-project<syntaxhighlight lang="bash">
	uv sync		uv sync
	</syntaxhighlight>~~The~~ pyproject.toml : <syntaxhighlight lang="toml">		</syntaxhighlight>

			=== pyproject.toml ===
			<syntaxhighlight lang="toml">
	[project]		[project]
	name = "vllm-project"		name = "vllm-project"

Davrot@uni-bremen.de at 16:49, 9 December 2025

2025-12-09T16:49:36Z

@@ Line 17: / Line 17: @@
        "requiresAPIKey": false
+     }
 </syntaxhighlight>

Davrot@uni-bremen.de at 16:33, 9 December 2025

2025-12-09T16:33:25Z

← Older revision		Revision as of 16:33, 9 December 2025
Line 1:		Line 1:
	In [[Config VS Code (Insiders)]] we see how we can add a custom LLM via the "OpenAI compatible API".		In [[Config VS Code (Insiders)]] we see how we can add a custom LLM via the "OpenAI compatible API".

			Add in a custom model to copilot in vs code insiders is easy. Let it do agent stuff is really hard.

			Here an example: Qwen3 Coder

			VS Code JSON:<syntaxhighlight lang="json">
			"github.copilot.chat.customOAIModels": {
			"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": {
			"name": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8",
			"url": "http://gate0.neuro.uni-bremen.de:8000/v1",
			"toolCalling": true,
			"vision": false,
			"thinking": true,
			"maxInputTokens": 256000,
			"maxOutputTokens": 8192,
			"requiresAPIKey": false
			}
			</syntaxhighlight>

Davrot@uni-bremen.de: Created page with "In Config VS Code (Insiders) we see how we can add a custom LLM via the "OpenAI compatible API"."

2025-12-09T16:30:31Z

Created page with "In Config VS Code (Insiders) we see how we can add a custom LLM via the "OpenAI compatible API"."

New page

In [[Config VS Code (Insiders)]] we see how we can add a custom LLM via the "OpenAI compatible API".