Commit 400b824

authored

feat(prompt): use tokenizer from @xenova/transformers (#38)

1 parent ccb6ec0 commit 400b824Copy full SHA for 400b824

7 files changed

+1260

-654

lines changed

package-lock.json
package.json
src
- common
  - completion
    - index.ts
  - prompt
    - promptCompletion.ts
    - tokenizer.ts
- extension.ts
webpack.config.js

7 files changed

+1260

-654

lines changed

Comments

(0)