word2vec explanation with drawings

1. Drawing word2vec
Kai Sasaki

2. Training Algorithm
CBOW
Skip-gram
Hierarchical Softmax
Omit Negative Sampling this time :(

3. CBOW
Continuous Bag of Words
Disregard grammar and work order
Share the weight of each words
Training around words

4. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]

5. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
neu1

6. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
neu1
syn1
Use word on the path to
syn0[i] in huffman tree as
network weight
f

7. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
neu1
syn1
Use word on the path to
syn0[i] in huffman tree as
network weight
f g
g is the gradient that is
calculated with
(1 - syn1.code -f) * alpha
This is error of output

8. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
neu1
syn1
Use word on the path to
syn0[i] in huffman tree as
network weight
f g
neu1e
g
syn1
Backpropagate to hidden layer

9. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
f g
neu1e
g
syn1
Backpropagate to hidden layer
Adding error back to each
words in window

10. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
f g
neu1e
g
syn1
Backpropagate to hidden layer
Adding error back to each
words in window

11. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
f g
neu1e
g
syn1
Backpropagate to hidden layer
Adding error back to each
words in window

12. CBOW
syn0[i-1]
syn0[i]
syn0[i+1]
They are the conclusive
continuous representations
that are available through
training

13. Skip-gram
Reverse format of CBOW
Predict representations of word that is
put around the target words

14. Skip-gram
syn0[i]
syn1
Use word on the path to
syn0[i] in huffman tree as
network weight
f

15. Skip-gram
syn0[i]
syn1
Use word on the path to
syn0[i] in huffman tree as
network weight
f g
g is the gradient that is
calculated with
(1 - syn1.code -f) * alpha
This is error of output

16. Skip-gram
syn0[i]
syn1
Use word on the path to
syn0[i] in huffman tree as
network weight
f g
neu1e
g
syn1
Backpropagate to hidden layer

17. Skip-gram
syn0[i]
f g
neu1e
g
syn1
Backpropagate to hidden layer
Adding error back to each
words in window

18. Reference
https://code.google.com/p/word2vec/