Implement MosiacML's 7B model.

[rjb7731]

Comparative to Llama in results I believe and also commercially available for use!

https://huggingface.co/mosaicml/mpt-7b

https://www.mosaicml.com/blog/mpt-7b

[ggerganov]

Lets start with a basic inference example in the ggml repo.

If it lives up to the hype, we can think about also integrating it in llama.cpp so we get all the infrastructure benefits or maybe something better depending on the results.

[sbsce]

Licensed as Apache 2.0, and a context length of 65k! Yes, would be great to have this supported in llama.cpp.

[Green-Sky]

MPT-7B-StoryWriter-65k+ is a model designed to read and write stories with super long context lengths. It was built by finetuning MPT-7B with a context length of 65k tokens on a filtered fiction subset of the books3 dataset. At inference time, thanks to ALiBi, MPT-7B-StoryWriter-65k+ can extrapolate even beyond 65k tokens, and we have demonstrated generations as long as 84k tokens on a single node of A100-80GB GPUs.

that's a looooong context.

[DannyDaemonic]

As someone who likes to occasionally use LLMs as an aid in writing, the idea of a 65k token context is exciting. Only 7B parameters is a little disappointing but one thing we're learning is not to judge a model by it's parameter count.

One of the first things I did when I found this project was to hack my own custom context reset to restart on a sentence boundary and leave only ~10% free space for context generation instead of 50%, just to keep the context more relevant. It was terribly inefficient but that's how bad I wanted a longer context length. There's really no substitute to having more (relevant) text in the context.

[jploski]

I also opened an issue for it here: mosaicml/llm-foundry#60

[drbh]

@jploski thanks for starting this conversation in ggml and llm-foundry! I agree that adding Mosaic 7B is a great idea! I happen to see you mentioned that you started some work but ran into tensor formatting issue.

Would you be open to sharing that branch of ggml? Mostly because I'm eager to learn more about the quantization process and even if it is not the full implementation, it may be helpful to see others starting points. Thanks!

[jploski]

@jploski thanks for starting this conversation in ggml and llm-foundry! I agree that adding Mosaic 7B is a great idea! I happen to see you mentioned that you started some work but ran into tensor formatting issue.

Would you be open to sharing that branch of ggml? Mostly because I'm eager to learn more about the quantization process and even if it is not the full implementation, it may be helpful to see others starting points. Thanks!

FWIW:

https://github.com/jploski/ggml/tree/mpt-experiment/examples/mpt

See commit comments and "TODO" in source code and README.md in examples/mpt for things that I do not understand. The main challenge seems to be that MPT uses a transformer model with customized code (found in their llm-foundry repository), so it is probably silly to expect the stablelm code to just work. All I did was some (rather uninformed) guessing.

Also note that the inference will not even start for mpt-7b-storywriter using the default context length of 65535 - it will just complain about "not enough space in the context's memory pool" and segfault. But this can be worked around by specifying a smaller n_ctx (rather than letting it load from GGML/model config).

Please do not let this first attempt raise false hopes or stand in the way of an actual implementation. I do not feel qualified enough (wrt to understanding transformers) to implement it. At best my branch can save someone some typing for the boilerplate (and at worst it can mislead, too).