ICLR2017@DeNA（https://connpass.com/event/57631/）の論文紹介資料

1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
June 17, 2017 ICLR @DeNA
Eiji Sekiya
AI Research and Development Gr.
AI System Dept.
DeNA Co., Ltd.
1
h"ps://openreview.net/pdf?id=SJU4ayYgl

2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
•
• (@eratostennis)
•
• Master
• DeNA (2014)
• : (~2016)
• Hadoop, Vertica
• : (2016~)
•
• GameAI
2

3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Abstract
• Semi-supervised learning on graph-structured data
• Localized first-order approximation of spectral graph
• :
•
•
3

4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Agenda
• Introduction
• Related Work
• Fast Approximate Convolutions on Graphs
• Semi-Supervised Node Classification
• Experiments
• Results
• Conclusion
4

5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Relation to Weisfeiler-Lehman Algorithm
• WL-1
• Hash Neural Network
5
hash

6. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Node Embeddings with Random Weights
6
output 2

7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Semi-Supervised Node Embeddings
• 4 1
7
: h"p://tkipf.github.io/graph-convoluDonal-networks/

8. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Related Work
• Graph-Based Semi-Supervised Learning
• Graph Laplacian regularization
• Label Propagation (Zhu et al., 2003)
• Manifold Regularization (Belkin et al., 2006)
• Graph embedding-based
• DeepWalk (Perozzi et al., 2014)
• Planetoid (Yang et al., 2016)
8

9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Related Work (DeepWalk: Perozzi et al., 2014)
•
• SkipGram embedding
• Random Walk
9

10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Related Work
• Graph-Based Semi-Supervised Learning
• Graph Laplacian regularization
• Label Propagation (Zhu et al., 2003)
• Manifold Regularization (Belkin et al., 2006)
• Graph embedding-based
• DeepWalk (Perozzi et al., 2014)
• Planetoid (Yang et al., 2016)
• Neural Networks on Graphs ( )
• Spectral graph convolutional neural network (Bruna et al., 2014)
• Fast localized convolution (Defferrard et al., 2016)
10

11. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Graph-Based Semi-Supervised Learning
• (citation network ) (documents)
•
•
•
• encode
•
11
Neural Networks on Graph

12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Multi-layer Graph Convolutional Network (GCN)
•
•
•
•
12
Contribution

13. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Spectral Graph Convolutions
• Convolution with filter
• :
•
• …(Hammond at el., 2011)
•
• ( )
13

14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Spectral Graph Convolutions (Defferrad at el., 2016)
• Graph Convolution ( , )
• K K
• :
•
14

15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Layer-Wise Linear Model (Contribution)
• K=1 …
• k k
•
• Renormalization Trick ( )
• Input C, F …
•
15

16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Semi-Supervised Node Classification
• Forward Model ( )
• Cross-Entropy:
16

17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Experiments (Dataset)
• (Citeseer, Cora, Pubmed)
• Sparse bag-of-words feature vectors for each document
• a list of citation links between documents
• (NELL)
• A set of entities connected with directed, labeled edges
• Random Graph
• A dataset with N nodes we create a random graph assigning 2N edges
uniformly at random
17

18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Experiments (Setup)
•
• 2 GCN
•
• Prediction Accuracy
• 1000
•
• 500
• Dropout rate, L2 regularization factor, Number of hidden units
• Cora Citeseer Pubmed
18

19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Experiments (Baseline)
• Baseline
• Label Propagation (LP)
• Semi-supervised embedding (SemiEmb)
• Manifold regularization (ManiReg)
• Skip-gram based graph embeddings (DeepWalk)
• Iterative classification algorithm (ICA)
• Planetoid
19

20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Results (Semi-Supervised Node Classification)
• Results for all other baseline methods are taken from the Planetoid paper
• Environment
• The same hardware as planetoid
20
Training Dme
in seconds
unDl convergence

21. • Compare different variants of our proposed per-layer propagation model
on the citation network datasets
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Results (Evaluation Propagation Model)
21

22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Results (Training Time per Epoch)
• The mean training time per epoch for 100 epochs on simulated random
graphs
22

23. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Experiments on Model Depth
• Residual connections
• 2, 3
• 7 …
• residual connections
•
23

24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Discussion
• Keypoints
• Overcome both limitations
• Graph-Laplacian regularization
• Edges encode mere similarity of nodes
• Skip-gram based methods
• Limited by the fact that they are based on a multi-step pipeline
which difficult to optimize
• Findings
• Propagation from neighboring nodes improves classification
performance
• Proposed renormalized propagation model offers both improved
efficiency
24

25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Future Work
• Memory Requirement
• Full-batch gradient descent
• Grows linearly in the size of the dataset
• Mini-batch stochastic gradient descent can alleviate this issue
• Directed edges and edge features
• Limited to undirected graph
• NELL show it is possible to handle
• By representing the original directed graph as an undirected bipartite
graph
• Limiting assumptions
• Locality
• dependence on the Kth-order neighborhood for a GCN with K layers
• Equal importance of self-connections vs. edges to neighboring nodes 25

26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Reference
• Graph Semi-Supervised Learning
• Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using
gaussian fields and harmonic functions. In International Conference on Machine Learning
(ICML), volume 3, pp. 912–919, 2003.
• Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric
frame- work for learning from labeled and unlabeled examples. Journal of machine learning
research (JMLR), 7(Nov):2399–2434, 2006.
• JasonWeston,Fre ́de ́ricRatle,HosseinMobahi,andRonanCollobert. Deeplearningviasemi-
supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639–655. Springer,
2012.
• Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed
repre- sentations of words and phrases and their compositionality. In Advances in neural
information processing systems (NIPS), pp. 3111–3119, 2013.
• Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social
repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on
Knowledge discovery and data mining, pp. 701–710. ACM, 2014.
• Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning
with graph embeddings. In International Conference on Machine Learning (ICML), 2016.
26

27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Reference
• Dataset & Pre-Processing
• Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina
Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93, 2008.
• Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr, and
Tom M. Mitchell. Toward an architecture for never-ending language learning. In AAAI,
volume 5, pp. 3, 2010.
• Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning
with graph embeddings. In International Conference on Machine Learning (ICML), 2016.
• Website
• http://tkipf.github.io/graph-convolutional-networks/
• Implementation
• https://github.com/tkipf/gcn
27