]> git.sondrewold.no Git - reference_transformer.git/commit
projects / reference_transformer.git / commit
? search:
summary | shortlog | log | commit | commitdiff | tree
(parent: 6bd163e) | patch
Add StupidTokenizer and test
authorSondre Wold <[email protected]>
Mon, 17 Jun 2024 11:45:29 +0000 (13:45 +0200)
committerSondre Wold <[email protected]>
Mon, 17 Jun 2024 11:45:29 +0000 (13:45 +0200)
commit7a9053858bc7b118309e67b689a32e5ac65c7193
treea51fc69ac1f6e5f6ea2e2763f191d032296880fctree | snapshot
parent6bd163ebf8f59139d66bcd45a7e30e71f02e441fcommit | diff
Add StupidTokenizer and test
reference_transformer/reference_transformer/__init__.py [new file with mode: 0644] blob
reference_transformer/reference_transformer/dataset.py [new file with mode: 0644] blob
reference_transformer/reference_transformer/tokenizer.py [new file with mode: 0644] blob
reference_transformer/tests/.test_tokenizers.py.swp [new file with mode: 0644] blob
reference_transformer/tests/__init__.py [new file with mode: 0644] blob
reference_transformer/tests/test_tokenizers.py [new file with mode: 0644] blob
Reference implementation for the Transformer architecture.