convert.py : handle special tokens

[ggerganov]

Here we need to start handling special tokens in convert.py:

https://github.com/ggerganov/llama.cpp/blob/e4324cbd4d367fb9ddaaef98c0929acebbf22954/convert.py#L790-L800

An example is shown in convert-llama-7b-pth-to-gguf.py:

https://github.com/ggerganov/llama.cpp/blob/e4324cbd4d367fb9ddaaef98c0929acebbf22954/convert-llama-7b-pth-to-gguf.py#L186-L239

[KerfuffleV2]

I can look at this after #2753 (hopefully) gets merged. I was planning on doing some cleanup work and fixing the type annotations, seems like the kind of thing that would be reasonable to throw into that kind of pull as well.

[ggerganov]

I think we need to use a model that utilizes special tokens to test this with. I see people mentioning "OpenChat V2 x OpenOrca" when they need to handle special tokens - maybe we can try to make those work

[KerfuffleV2]

I think we need to use a model that utilizes special tokens to test this with.

Using a model with special tokens to test handling special tokens is an idea just crazy enough to work!

[klosax]

For BPE to work with llama models (Aquila?) convert.py should also add the merges like it is done in the falcon conversion script.

[KerfuffleV2]

In progress over here: #2842

[ggerganov]

The next step is using the special tokens in llama.cpp - any ideas what needs to be done?

My guess is we need to just update the id_to_token and token_to_id maps:

https://github.com/ggerganov/llama.cpp/blob/dc07dc492ef9640bbb82904d7c7679f7bdcf6d76/llama.cpp#L947-L950

[KerfuffleV2]

I'm not sure where discussion about this should be.

For BPE to work with llama models (Aquila?)

I've been doing some testing with https://huggingface.co/BAAI/Aquila-7B and https://huggingface.co/kfkas/Llama-2-ko-7b-Chat trying to get the BPE stuff to work.

First, it seems like all these BPE models just die in llama.cpp without #2889. Little surprised that pull has gotten no attention so far.

It also seems like the stuff in convert.py is still pretty far off even with merges being handled now. I started to try to fix some stuff in #2938

[rajveer43]

Is it available for working?

[ggerganov]

We now have to use these special tokens in llama.cpp

Can somebody confirm that the following is correct:

• we load the following special tokens (e.g. open llama):

{
	"bos_token": {
		"content": "<s>",
		"lstrip": false,
		"normalized": true,
		"rstrip": false,
		"single_word": false
	},
	"eos_token": {
		"content": "</s>",
		"lstrip": false,
		"normalized": true,
		"rstrip": false,
		"single_word": false
	},
	"unk_token": {
		"content": "<unk>",
		"lstrip": false,
		"normalized": true,
		"rstrip": false,
		"single_word": false
	}
}

• we now tokenize the following string <s>hello world</s>
• the result is that <s> and </s> are no longer tokenized as strings, but instead they are tokenized to the special tokens BOS and EOS. So we get for example the tokens: [1, 22172, 3186, 2]

[KerfuffleV2]

I don't know what behavior is considered correct, but it seems like in that particular case it means you can't talk about HTML strikethrough tags anymore. I.E. a prompt like "Dear LLaMA model, people make a list according to such and such rules. Surround elements that meet a certain criteria with strikethrough like <s>item</s>." You'll pretty much immediately get nonsense if <s> and </s> are tokenized to BOS/EOS — same thing when the special tokens can conflict with something else that could plausibly be in a prompt.

It's less of an issue when the special tokens are like <|endoftext|> or whatever since it's less like to be something a user would write.

[klosax]

Normally the special token strings should not be recognized as special tokens in the user prompt. Better to have a CLI parameter for users who need to use them. Instead of using the model vocab these tokens should be user-configurable. Something like --bos-token "<|my-bos-token|>" should work.

[KerfuffleV2]

What if it was something like --set-token "1=<|my-bos-token|>"? Then there would be a general facility to override any token id, sort of like setting the logit overrides. (Maybe --set-token isn't great, could be --override-token, --assign-token, whatever.)