Daniele GrattarolaArtificial intelligence scientist
https://danielegrattarola.github.io/
Fri, 07 Jun 2024 06:12:52 +0000Fri, 07 Jun 2024 06:12:52 +0000Jekyll v3.9.5My second interview on Machine Learning Street Talk<div class="video-container">
<iframe src="https://www.youtube-nocookie.com/embed/v5NysEyZkl0" frameborder="0" allowfullscreen=""></iframe>
</div>
<p>I was featured for the second time on <a href="https://www.youtube.com/channel/UCMLtBahI5DMrt0NPvDSoIRQ">Machine Learning Street Talk</a>.</p>
<p>This interview was shot at NeurIPS 2023 last year, where I was presenting our work on
<a href="https://arxiv.org/abs/2205.15674">generalized implicit neural representations</a> from my time at EPFL.</p>
<p>Cheers!</p>
Sat, 16 Dec 2023 00:00:00 +0000
/posts/2023-12-16/MLST-2.html
updatepostsMy interview on Machine Learning Street Talk<div class="video-container">
<iframe src="https://www.youtube-nocookie.com/embed/MDt2e8XtUcA" frameborder="0" allowfullscreen=""></iframe>
</div>
<p>I had the pleasure of being a guest on <a href="https://www.youtube.com/channel/UCMLtBahI5DMrt0NPvDSoIRQ">Machine Learning Street Talk</a> to chat about cellular automata, emergence, life, the universe, and my own work on <a href="https://danielegrattarola.github.io/posts/2021-11-08/graph-neural-cellular-automata.html">graph neural cellular automata</a>.</p>
<p>I had a great time with Tim and Keith, they are doing an incredible work with the podcast and it’s really an honor to having been a part of it.</p>
<p>Enjoy!</p>
<p><sup style="font-size: 10px;">P.S. I was so nervous and hyper-excited that I lost my own train of thought a couple of times, please be patient :D</sup></p>
Fri, 29 Apr 2022 00:00:00 +0000
/posts/2022-04-29/MLST.html
updatepostsGraph Neural Cellular Automata<p><img src="https://danielegrattarola.github.io/images/2021-11-08/fixed_target_animation.gif" alt="Graph Neural Cellular Automata for morphogenesis" class="centered" /></p>
<p><a href="https://en.wikipedia.org/wiki/Cellular_automaton">Cellular automata</a> (or CA for short) are a fascinating computational model.
They consist of a lattice of stateful cells and a transition rule that updates the state of each cell as a function of its neighbourhood configuration.
By applying this local rule synchronously over time, we see interesting dynamics emerge.</p>
<p>For example, here is the transition table of <a href="https://en.wikipedia.org/wiki/Rule_110">Rule 110</a> in a 1-dimensional binary CA:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/Rule110-rule.png" alt="Rule 110, transition table" class="threeq-width" /></p>
<p>And here is the corresponding evolution of the states starting from a random initialization (time goes downwards):</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/Rule110rand.png" alt="Rule 110, evolution of the states" class="quarter-width" /></p>
<p>By changing the rule, we get different dynamics, some of which can be extremely interesting. One example of this is the 2-dimensional <a href="https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life">Game of Life</a>, with its complex patterns that replicate and move around the grid.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/glider_gun.gif" alt="Gosper glider gun" class="half-width" /></p>
<p>We can also bring this idea of locality to the extreme, by keeping it as the only requirement and making everything else more complicated.</p>
<p>For example, if we make the states continuous and change the size of the neighbourhood, we get <a href="https://arxiv.org/abs/1812.05433">the mesmerizing Lenia CA</a> with its <em>insanely</em> life-like creatures that move around smoothly, reproduce, and even organize themselves into higher-order organisms.</p>
<div class="video-container">
<iframe src="https://www.youtube-nocookie.com/embed/iE46jKYcI4Y" frameborder="0" allowfullscreen=""></iframe>
</div>
<p>By this principle, we can also derive an even more general version of CA, in which the neighbourhoods of the cells no longer have a fixed shape and size. Instead, the cells of the CA are organized in an arbitrary graph.</p>
<p>Note that the central idea of locality that characterizes CA does not change at all: we’re just extending it to account for these more general neighbourhoods.</p>
<p>The super-general CA are usually called <strong>Graph Cellular Automata (GCA)</strong>.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/gca_transition.png" alt="Example of GCA transition" class="half-width" /></p>
<p>The general form of GCA transition rules is a map from a cell and its neighbourhood to the next state, and we can also make it <strong>anisotropic</strong> by introducing edge attributes that specify a relation between the cell and each neighbour.</p>
<h2 id="learning-ca-rules">Learning CA rules</h2>
<p>The world of CA is fascinating, but unfortunately, they are almost always considered simply pretty things.</p>
<p><strong>But can they be also useful? Can we design a rule to solve an interesting problem using the decentralized computation of CA?</strong></p>
<p>The answer is yes, but manually designing such a rule may be hard. However, being AI scientists, we can try to learn the rule.</p>
<p>This is not a new idea.</p>
<p>We can go back to NeurIPS 1992 to find a seminal work on <a href="https://papers.nips.cc/paper/1992/hash/d6c651ddcd97183b2e40bc464231c962-Abstract.html">learning CA rules with neural networks</a> (they use convolutional neural networks, although back then they were called “sum-product networks with shared weights”).</p>
<p>Since then, we’ve seen other approaches to learn CA rules, like these papers using <a href="https://mobile.aau.at/~welmenre/papers/elmenreich-iwsos2011.pdf">genetic algorithms</a> or <a href="https://ieeexplore.ieee.org/abstract/document/8004527">compositional pattern-producing networks</a> to find rules that lead to a desired configuration of states, a task called <strong>morphogenesis</strong>.<sup style="font-size: 10px;"> <a href="https://sci-hub.se/">Papers not on arXiv, sorry</a></sup></p>
<p>More recently, convolutional networks have been shown to be extremely versatile in learning CA rules.
<a href="https://arxiv.org/abs/1809.02942">This work by William Gilpin</a>, for example, shows that we can implement any desired transition rule with CNNs by smartly setting their weights.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/planarian.jpg" alt="A planarian flatworm" class="half-width" /></p>
<p>CNNs have also been used for morphogenesis. Inspired by the regenerative abilities of the flatworm (pictured above), <a href="https://distill.pub/2020/growing-ca/">in this visually-striking paper</a> they train a CNN to grow into a desired image and to regenerate the image if it is perturbed.</p>
<h2 id="learning-gca-rules">Learning GCA rules</h2>
<p>So, can we do something similar in the more general setting of GCA?</p>
<p>Well, let’s start with the model.
Similar to how CNNs are the natural family of models to implement typical grid-based CA rules, the more general family of graph neural networks is the natural choice for GCA.</p>
<p>We call this setting the <strong>Graph Neural Cellular Automata (GNCA)</strong>.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/thumbnail_cut.png" alt="Graph Neural Cellular Automata" class="threeq-width" /></p>
<p>We propose an architecture composed of a pre-preprocessing MLP, a message-passing layer, and a post-processing MLP, which we use as transition function.</p>
<p>This model is universal to represent GCA transition rules. We can prove this by making an argument similar to the one for CNNs that I mentioned above.</p>
<p>I won’t go into the specific details here, but in short, we need to implement two operations:</p>
<ol>
<li>One-hot encoding of the states;</li>
<li>Pattern-matching for the desired rule.</li>
</ol>
<p>The first two blocks in our GNCA are more than enough to achieve this.
The pre-processing MLP can compute the one-hot encoding, and by using edge attributes and <a href="https://arxiv.org/abs/1704.02901">edge-conditioned convolutions</a> we can implement pattern matching easily.</p>
<h2 id="experiments">Experiments</h2>
<p>However, regardless of what the theory says, we want to know whether we can learn a rule in practice. Let’s try a few experiments.</p>
<h3 id="voronoi-gca">Voronoi GCA</h3>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/voronoi.png" alt="Voronoi GCA" class="half-width" /></p>
<p>We can start from the simplest possible binary GCA, inspired by the 1992 NeurIPS paper I mentioned before. The difference is that our CA cells are the <a href="https://en.wikipedia.org/wiki/Voronoi_diagram">Voronoi tasselletion</a> of some random points.
Alternatively, you can think of this GCA as being defined on the <a href="https://en.wikipedia.org/wiki/Delaunay_triangulation">Delaunay triangulation</a> of the points.</p>
<p>We use an <a href="https://en.wikipedia.org/wiki/Life-like_cellular_automaton.">outer-totalistic rule</a> that swaps the state of a cell if the density of its alive neighbours exceeds a certain threshold, not too different from the Game of Life.</p>
<p>We try to see if our model can learn this kind of transition rule. In particular, we can train the model to approximate the 1-step dynamics in a supervised way, given that we know the true transition rule.</p>
<div style="text-align: center">
<img src="https://danielegrattarola.github.io/images/2021-11-08/learn_gca_loss_v_epoch.svg" width="30%" style="display: inline-block; margin:auto;" />
<img src="https://danielegrattarola.github.io/images/2021-11-08/learn_gca_acc_v_epoch.svg" width="30%" style="display: inline-block; margin:auto;" />
</div>
<p>The results are encouraging. We see that the GNCA achieves 100% accuracy with no trouble and, if we let it evolve autonomously, it does not diverge from the real trajectory.</p>
<h3 id="boids">Boids</h3>
<div style="text-align: center">
<img src="https://danielegrattarola.github.io/images/2021-11-08/alignment.png" width="30%" style="display: inline-block; margin:auto;" />
<img src="https://danielegrattarola.github.io/images/2021-11-08/cohesion.png" width="30%" style="display: inline-block; margin:auto;" />
<img src="https://danielegrattarola.github.io/images/2021-11-08/separation.png" width="30%" style="display: inline-block; margin:auto;" />
</div>
<p>For our second experiment, we keep a similar setting but make the target GCA much more complicated.
We consider the <a href="https://en.wikipedia.org/wiki/Boids">Boids</a> algorithm, an agent-based model designed to simulate the flocking of birds. This can be still seen as a kind of GCA because the state of each bird (its position and velocity) is updated only locally as a function of its closest neighbours.
However, this means that the states of the GCA are continuous and multi-dimensional, and also that the graph changes over time.</p>
<p>Again, we can train the GNCA on the 1-step dynamics. We see that, although it’s hard to approximate the exact behaviour, we get very close to the true system.
The GNCA (yellow) can form the same kind of flocks as the true system (purple), even if their trajectories diverge.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/boids_animation.gif" alt="Boids GCA and trained GNCA" class="centered" /></p>
<h3 id="morphogenesis">Morphogenesis</h3>
<p>The final experiment is also the most interesting, and the one where we actually design a rule.
Like previously in the literature, here too we focus on morphogenesis. Our task is to find a GNCA rule that, starting from a given initial condition, converges to a desired point cloud (like a bunny) where the connectivity of the cells has a geometrical/spatial meaning.</p>
<p>In this case, we don’t know the true rule, so we must train the model differently, by teaching it to arrive at the target state when evolving autonomously.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/gnca_training.png" alt="Training scheme for GNCA" class="threeq-width" /></p>
<p>To do so, we let the model evolve for a given number of steps, then we compute the loss from the target, and we update the weights with backpropagation through time.
To stabilise training, and to ensure that the target state becomes a stable attractor of the GNCA, we use a cache. This is a kind of replay memory from which we sample the initial conditions, so that we can reuse the states explored by the GNCA during training.
Crucially, this teaches the model to remain at the target state when starting from the target state.</p>
<p>And the results are pretty amazing… have you seen the gif at the <a href="#">top of the post</a>? Let’s unroll the first few frames here.</p>
<p>A 2-dimensional grid:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/clouds/Grid_10-20/evolution.png" alt="" /></p>
<p>A bunny:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/clouds/Bunny_10-20/evolution.png" alt="" /></p>
<p>The <a href="https://pygsp.readthedocs.io/en/stable/">PyGSP</a> logo:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/clouds/Logo_20/evolution.png" alt="" /></p>
<p>We see that the GNCA has no trouble in finding a stable rule that converges quickly at the target and then remains there.</p>
<p>Even for complex and seemingly random graphs, like the Minnesota road network, the GNCA can learn a rule that quickly and stably converges to the target:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/clouds/Minnesota_20/anim.gif" alt="" class="third-width" /></p>
<p>However, this is not the full story. Sometimes, instead of converging, the GNCA learns to remain in an orbit around the target state, giving us these oscillating point clouds.</p>
<p>Grid:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/clouds/Grid_10/evolution.png" alt="" /></p>
<p>Bunny:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/clouds/Bunny_10/evolution.png" alt="" /></p>
<p>Logo:</p>
<p><img src="https://danielegrattarola.github.io/images/2021-11-08/clouds/Logo_10/evolution.png" alt="" /></p>
<h2 id="now-what">Now what?</h2>
<p>So, where do we go from here?</p>
<p>We have seen that GNCA can reach global coherence through local computation, which is not that different from what we do in graph representation learning. In fact, <a href="https://www.researchgate.net/profile/Franco_Scarselli/publication/4202380_A_new_model_for_earning_in_raph_domains/links/0c9605188cd580504f000000.pdf">the first GNN paper</a>, back in 2005, already contained this idea.</p>
<p>But moving forward, it’s easy to see that the idea of emergent computation on graphs could apply to many scenarios, including swarm optimization and control, modelling epidemiological transmission, and it could even improve our understanding of complex biological systems, like the brain.</p>
<p>GNCA enable the design of GCA transition rules, unlocking the power of decentralised and emergent computation to solve real-world problems.</p>
<p>The code for the paper is available <a href="https://github.com/danielegrattarola/GNCA">on Github</a> and feel free to reach out via email if you have any questions or comments.</p>
<h2 id="read-more">Read more</h2>
<p>This blog post is the short version of our NeurIPS 2021 paper:</p>
<p><a href="https://arxiv.org/abs/2110.14237">Learning Graph Cellular Automata</a><br />
<em>D. Grattarola, L. Livi, C. Alippi</em></p>
<p>You can cite the paper as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@inproceedings{grattarola2021learning,
title={Learning Graph Cellular Automata},
author={Grattarola, Daniele and Livi, Lorenzo and Alippi, Cesare},
booktitle={Neural Information Processing Systems},
year={2021}
}
</code></pre></div></div>
Mon, 08 Nov 2021 00:00:00 +0000
/posts/2021-11-08/graph-neural-cellular-automata.html
GNNcellular-automatapostsA practical introduction to GNNs - Part 2<p><em>This is Part 2 of an introductory lecture on graph neural networks that I gave for the “Graph Deep Learning” course at the University of Lugano.</em></p>
<p><em>After a practical introduction to GNNs in <a href="https://danielegrattarola.github.io/posts/2021-03-03/gnn-lecture-part-1.html">Part 1</a>, here I show how we can formulate GNNs in a much more flexible way using the idea of message passing.</em></p>
<p><em>First, I introduce message passing. Then, I show how to implement message-passing networks in Jax/pseudocode using a paradigm called “gather-scatter”. Finally, I show how to implement a couple of more advanced GNN models.</em></p>
<p><a href="https://danielegrattarola.github.io/files/talks/2021-03-01-USI_GDL_GNNs.pdf">The full slide deck is available here</a>.</p>
<hr />
<p>In <a href="https://danielegrattarola.github.io/posts/2021-03-03/gnn-lecture-part-1.html">Part 1</a> of this series we constructed our first kind of GNN by replicating the behavior of conventional CNNs on data supported by graphs.</p>
<p>The core building block that we used in our simple GNNs looked like this:</p>
\[\mathbf{X}' = \mathbf{R}\mathbf{X}\mathbf{\Theta}\]
<p>which, as we saw, has two effects:</p>
<ol>
<li>All node attributes \(\mathbf{X}\) are transformed using the learnable matrix \(\mathbf{\Theta}\);</li>
<li>The attribute of each node gets replaced with a weighted sum of its neighbors via the reference operator \(\mathbf{R}\) (also, sometimes we can include the node itself in the sum);</li>
</ol>
<p>By combining these two ideas we were able to get a very good approximation of a CNN for graphs.</p>
<p>In this part of the lecture, we will take these two ideas and describe them a little more formally, distilling the essential role that they have in a GNN.</p>
<p>We will see a general framework called <strong>message passing</strong>, which will allow us to describe more complex GNNs than those we have seen so far.</p>
<h2 id="message-passing-networks">Message Passing Networks</h2>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-14.svg" width="100%" style="border: solid 1px;" /></p>
<p>The idea of message passing networks was introduced in a paper by <a href="">Gilmer et al.</a> in 2017 and it essentially boils GNN layers down to three main steps:</p>
<ol>
<li>Every node in the graph computes a <strong>message</strong> for each of its neighbors. Messages are a function of the node, the neighbor, and the edge between them.</li>
<li>Messages are sent, and every node <strong>aggregates</strong> the messages it receives, using a permutation-invariant function (i.e., it doesn’t matter in which order the messages are received). This function is usually a sum or an average, but it can be anything.</li>
<li>After receiving the messages, each node <strong>updates</strong> its attributes as a function of its current attributes and the aggregated messages.</li>
</ol>
<p>This procedure happens synchronously for all nodes in the graph, so that at each message passing step all nodes are updated.</p>
<p>If we look back at our super-simple GNN formulation \(\mathbf{X}' = \mathbf{R}\mathbf{X}\mathbf{\Theta}\), we can easily see the three message-passing steps:</p>
<ol>
<li><strong>Message</strong> - Each node \(i\) will receive the same kind of message \(\mathbf{\Theta}^\top\mathbf{x}_j\) from all its neighbors \(j \in \mathcal{N}(i)\).</li>
<li><strong>Aggregate</strong> - Messages are aggregated with a weighted sum, where weights are defined by the reference operator \(\mathbf{R}\).</li>
<li><strong>Update</strong> - Each node simply replaces its attributes with the aggregated messages. <br />
If \(\mathbf{R}\) has a non-zero diagonal, then each node also computes a message “from itself to itself” using \(\mathbf{\Theta}\).</li>
</ol>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-15.svg" width="100%" style="border: solid 1px;" /></p>
<p>Message passing is usually formalized with the equation in the slide above.</p>
<p>While it may look complicated at first, the formula simply describes the three steps that we saw before, and if you wanted to write it in pseudo-Python it would look something like this:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># For every node in the graph
</span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_nodes</span><span class="p">):</span>
<span class="c1"># Compute messages from neighbors
</span> <span class="n">messages</span> <span class="o">=</span> <span class="p">[</span><span class="n">message</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">x</span><span class="p">[</span><span class="n">j</span><span class="p">],</span> <span class="n">e</span><span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">])</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">neighbors</span><span class="p">(</span><span class="n">i</span><span class="p">)]</span>
<span class="c1"># Aggregate messages
</span> <span class="n">aggregated</span> <span class="o">=</span> <span class="n">aggregate</span><span class="p">(</span><span class="n">messages</span><span class="p">)</span>
<span class="c1"># Update node attributes
</span> <span class="n">x</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">update</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">aggregated</span><span class="p">)</span>
</code></pre></div></div>
<p>As long as <code class="language-plaintext highlighter-rouge">message</code>, <code class="language-plaintext highlighter-rouge">aggregate</code>, and <code class="language-plaintext highlighter-rouge">update</code> are differentiable functions, we can train any neural network to transforms its inputs like this. <br />
In fact, this framework is so general that virtually all libraries that implement GNNs are based on it.</p>
<p>For example, <a href="https://graphneural.network">Spektral</a>, <a href="https://pytorch-geometric.readthedocs.io/">Pytorch Geometric</a>, and <a href="https://www.dgl.ai/">DGL</a> all have a <code class="language-plaintext highlighter-rouge">MessagePassing</code> class which looks like this:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MessagePassing</span><span class="p">(</span><span class="n">Layer</span><span class="p">):</span> <span class="c1"># Or `Module`
</span>
<span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inputs</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> <span class="c1"># Or `forward`
</span> <span class="c1"># This is the actual message-passing step
</span> <span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">propagate</span><span class="p">(</span><span class="o">*</span><span class="n">inputs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">propagate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">e</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c1"># process arguments and create *_kwargs
</span> <span class="p">...</span>
<span class="c1"># Message
</span> <span class="n">messages</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">message</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="o">**</span><span class="n">msg_kwargs</span><span class="p">)</span>
<span class="c1"># Aggregate
</span> <span class="n">aggregated</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">aggregate</span><span class="p">(</span><span class="n">messages</span><span class="p">,</span> <span class="o">**</span><span class="n">agg_kwargs</span><span class="p">)</span>
<span class="c1"># Update
</span> <span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">update</span><span class="p">(</span><span class="n">aggregated</span><span class="p">,</span> <span class="o">**</span><span class="n">upd_kwargs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">output</span>
<span class="k">def</span> <span class="nf">message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">aggregate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">messages</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">aggregated</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="p">...</span>
</code></pre></div></div>
<h2 id="gather-scatter">Gather-Scatter</h2>
<p>The cool thing about message passing is that it lets us define the operations that our GNN computes, without necessarily resorting to matrix multiplication.</p>
<p>In fact, the only thing that we specify is how the GNN acts on a generic node \(i\) as a function of its generic neighbors \(j \in \mathcal{N}(i)\).</p>
<p>For instance, let’s say that we wanted to implement the “Edge Convolution” operator from the paper <a href="https://arxiv.org/abs/1801.07829">“Dynamic Graph CNN for Learning on Point Clouds”</a>.</p>
<p>In the message-passing framework, we write its effect as:</p>
\[\mathbf{x}_i' = \sum\limits_{j \in \mathcal{N}(i)} \textrm{MLP}\big( \mathbf{x}_i \| \mathbf{x}_j - \mathbf{x}_i \big)\]
<p>If we wanted to implement this as a simple matrix multiplication we would have some troubles, because GNNs of the form \(\mathbf{R}\mathbf{X}\mathbf{\Theta}\) assume that every node sends the same message to each of its neighbors. Here, instead, messages are a function of edges \(j \rightarrow i\).</p>
<p>In fact, this is a limitation of every GNN with edge-dependent messages.</p>
<p>We could still implement our Edge Convolution using broadcasting operations, but it would not be efficient at all. Here’s one way we could do it:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">jax</span><span class="p">,</span> <span class="n">jax</span><span class="p">.</span><span class="n">numpy</span> <span class="k">as</span> <span class="n">jnp</span>
<span class="n">x</span> <span class="o">=</span> <span class="p">...</span> <span class="c1"># Node attributes of shape [n, f]
</span><span class="n">a</span> <span class="o">=</span> <span class="p">...</span> <span class="c1"># Adjacency matrix of shape [n, n]
</span>
<span class="c1"># Compute all pairwise differences between nodes
</span><span class="n">x_diff</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="p">:,</span> <span class="p">:]</span> <span class="o">-</span> <span class="n">x</span><span class="p">[:,</span> <span class="bp">None</span><span class="p">,</span> <span class="p">:]</span> <span class="c1"># shape: (n, n, f)
</span>
<span class="c1"># Repeat the nodes so that we can concatenate them to the differences
</span><span class="n">x_repeat</span> <span class="o">=</span> <span class="n">jnp</span><span class="p">.</span><span class="n">repeat</span><span class="p">(</span><span class="n">x</span><span class="p">[:,</span> <span class="bp">None</span><span class="p">,</span> <span class="p">:],</span> <span class="n">n</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># shape: (n, n, f)
</span>
<span class="c1"># Concatenate the attributes so that, for each edge, we have x_i || (x_i - x_j)
</span><span class="n">x_all</span> <span class="o">=</span> <span class="n">jnp</span><span class="p">.</span><span class="n">concatenate</span><span class="p">([</span><span class="n">x_repeat</span><span class="p">,</span> <span class="n">x_diff</span><span class="p">],</span> <span class="n">axis</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># shape: (n, n, 2 * f)
</span>
<span class="c1"># Give x_i || (x_i - x_j) as input to an MLP
</span><span class="n">messages</span> <span class="o">=</span> <span class="n">mlp</span><span class="p">(</span><span class="n">x_all</span><span class="p">)</span> <span class="c1"># shape: (n, n, channels)
</span>
<span class="c1"># Broadcast-multiply `a` to keep only "real" messages
</span><span class="n">output</span> <span class="o">=</span> <span class="n">a</span><span class="p">[...,</span> <span class="bp">None</span><span class="p">]</span> <span class="o">*</span> <span class="n">messages</span> <span class="c1"># shape: (n, n, channels)
</span>
<span class="c1"># Sum along the "neighbors" axis.
</span><span class="n">output</span> <span class="o">=</span> <span class="n">output</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># shape: (n, channels)
</span></code></pre></div></div>
<p><strong>This is not ideal</strong>, because it cost us \(O(N^2)\) to do something that should have a cost linear in the number of edges (which is a big difference when working with real-world graphs, which are usually very sparse).</p>
<p>In general, using broadcasting to define edge-dependent GNNs means that we have to compute the messages for <strong>all possible edges</strong> and then simply multiply some of the messages by zero by broadcasting <code class="language-plaintext highlighter-rouge">a</code>.</p>
<p>This is because broadcasting is a “dense” operation.</p>
<p>A much better way to achieve our goal is to exploit the advanced indexing features offered by all libraries for tensor manipulation, using a technique called <strong>gather-scatter</strong>.</p>
<p>The gather-scatter technique requires us to think a bit differently, using node indices to access <strong>only the nodes that we are interested in</strong>, in a sparse way.</p>
<p>This is much easier done than said, so let’s see an example.</p>
<p>Let us consider an adjacency matrix <code class="language-plaintext highlighter-rouge">a</code>:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">a</span> <span class="o">=</span> <span class="p">[[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">]]</span>
</code></pre></div></div>
<p>This matrix is equivalently represented in the sparse COOrdinate format:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">row</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">]</span> <span class="c1"># Nodes that are sending a message
</span><span class="n">col</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span> <span class="c1"># Nodes that are receiving a message
</span></code></pre></div></div>
<p>which simply tells us the indices of the non-zero entries of <code class="language-plaintext highlighter-rouge">a</code> (we usually also have an extra array that tells us the actual values of the entries, but we won’t need it for now).</p>
<p>It’s easy to see, now, that if we look at all edges of the form \(j \rightarrow i\), then the attributes of all nodes that are sending a message can be retrieved with <code class="language-plaintext highlighter-rouge">x[row]</code>.
Similarly, the attributes of nodes that are receiving a message can be retrieved with <code class="language-plaintext highlighter-rouge">x[col]</code>.</p>
<p>This is called <strong>gathering</strong> the nodes.</p>
<p>In our case, if we want to take the difference of the nodes at the opposite side of an edge, we can simply do <code class="language-plaintext highlighter-rouge">x[row] - x[col]</code>.
Instead of computing the difference <code class="language-plaintext highlighter-rouge">x[j] - x[i]</code> for all possible pairs <code class="language-plaintext highlighter-rouge">j, i</code>, like we did before, now we only compute the differences that we are really interested in.</p>
<p>All these operations will give us matrices that have as many rows as there are edges. So for instance, <code class="language-plaintext highlighter-rouge">x[row]</code> will look like this:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span>
<span class="n">x</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span>
<span class="n">x</span><span class="p">[</span><span class="mi">2</span><span class="p">]]</span> <span class="c1"># shape: (n_edges, f)
</span></code></pre></div></div>
<p>The other half of this story tells us how to aggregate the messages after we have gathered them. We call this <strong>scattering</strong>.</p>
<p>For all nodes \(i\), we want to aggregate all messages that are being sent via edges that have index \(i\) on the <strong>receiving</strong> end, i.e., all edges of the form \(j \rightarrow i\).
For instance, in the small example above we know that node 2 will receive a message from nodes 0 and 1.</p>
<p>We can do this using some special operations available more or less in all libraries for tensor manipulation:</p>
<ul>
<li>In TensorFlow, we have <code class="language-plaintext highlighter-rouge">tf.math.segment_[sum|prod|mean|max|min]</code>.</li>
<li>For PyTorch, we have the <a href="https://github.com/rusty1s/pytorch_scatter">Torch Scatter</a> library by Matthias Fey.</li>
<li>In Jax, we only have <code class="language-plaintext highlighter-rouge">jax.ops.segment_sum</code>.</li>
</ul>
<p>These operations apply a reduction to “segments” of a tensor, where the segments are defined by integer indices. Something like this:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Example: segment sum
</span><span class="n">data</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="c1"># A tensor that we want to reduce
</span><span class="n">segments</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="c1"># Segment indices (we have 4 segments)
</span>
<span class="n">output</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="nb">max</span><span class="p">(</span><span class="n">segments</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="c1"># One result for each segment
</span><span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">s</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">segments</span><span class="p">):</span>
<span class="n">output</span><span class="p">[</span><span class="n">s</span><span class="p">]</span> <span class="o">+=</span> <span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="c1"># It could be a product, max, etc...
</span>
<span class="o">>>></span> <span class="n">output</span>
<span class="p">[</span><span class="mi">13</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
</code></pre></div></div>
<p>So for instance, if we want to sum all messages based on their intended recipient, we can do:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># recipients = col
</span><span class="n">aggregated</span> <span class="o">=</span> <span class="n">jax</span><span class="p">.</span><span class="n">ops</span><span class="p">.</span><span class="n">segment_sum</span><span class="p">(</span><span class="n">messages</span><span class="p">,</span> <span class="n">recipients</span><span class="p">)</span>
</code></pre></div></div>
<p>Now we can put all of this together to create our Edge Convolution layer with a gather-scatter implementation:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">x</span> <span class="o">=</span> <span class="p">...</span> <span class="c1"># Node attributes of shape [n, f]
</span><span class="n">a</span> <span class="o">=</span> <span class="p">...</span> <span class="c1"># Adjacency matrix of shape [n, n]
</span>
<span class="c1"># Get indices of the non-zero entries of the adjacency matrix
</span><span class="kn">import</span> <span class="nn">scipy</span>
<span class="n">senders</span><span class="p">,</span> <span class="n">recipients</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">scipy</span><span class="p">.</span><span class="n">sparse</span><span class="p">.</span><span class="n">find</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="c1"># Calculate difference of nodes for each edge j -> i
</span><span class="n">x_diff</span> <span class="o">=</span> <span class="n">x</span><span class="p">[</span><span class="n">senders</span><span class="p">]</span> <span class="o">-</span> <span class="n">x</span><span class="p">[</span><span class="n">recipients</span><span class="p">]</span> <span class="c1"># shape: (n_edges, f)
</span>
<span class="c1"># Concatenate x_i with (x_i - x_j) for each edge j -> i
</span><span class="n">x_all</span> <span class="o">=</span> <span class="n">jnp</span><span class="p">.</span><span class="n">concatenate</span><span class="p">([</span><span class="n">x</span><span class="p">[</span><span class="n">recipients</span><span class="p">],</span> <span class="n">x_diff</span><span class="p">],</span> <span class="n">axis</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># shape: (n_edges, 2 * f)
</span>
<span class="c1"># Give x_i || (x_i - x_j) as input to an MLP
</span><span class="n">messages</span> <span class="o">=</span> <span class="n">mlp</span><span class="p">(</span><span class="n">x_all</span><span class="p">)</span> <span class="c1"># shape: (n_edges, channels)
</span>
<span class="c1"># Aggregate all messages according to their intended recipient
</span><span class="n">output</span> <span class="o">=</span> <span class="n">jax</span><span class="p">.</span><span class="n">ops</span><span class="p">.</span><span class="n">segment_sum</span><span class="p">(</span><span class="n">messages</span><span class="p">,</span> <span class="n">recipients</span><span class="p">)</span> <span class="c1"># shape: (n, channels)
</span></code></pre></div></div>
<p>Wrap this up in a layer and we’re done!</p>
<p>Here’s what it looks like <a href="https://github.com/danielegrattarola/spektral/blob/master/spektral/layers/convolutional/edge_conv.py">in Spektral</a> and <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/nn/conv/edge_conv.html#EdgeConv">in Pytorch Geometric</a>.</p>
<h2 id="methods">Methods</h2>
<p>Since now we’ve moved past the simple models based on reference operators and edge-independent messages that we saw in the first part of this series, we can look at some more advanced methods.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-17.svg" width="100%" style="border: solid 1px;" /></p>
<p>For instance, the popular <a href="https://arxiv.org/abs/1710.10903">Graph Attention Networks</a> by Veličković et al. can be implemented as a message-passing network using gather-scatter:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Transform node attributes with a dense layer (defined elsewhere)
</span><span class="n">h</span> <span class="o">=</span> <span class="n">dense</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="c1"># Concatenate attributes of recipients/senders
</span><span class="n">h_cat</span> <span class="o">=</span> <span class="n">jnp</span><span class="p">.</span><span class="n">concatenate</span><span class="p">([</span><span class="n">h</span><span class="p">[</span><span class="n">recipients</span><span class="p">],</span> <span class="n">h</span><span class="p">[</span><span class="n">senders</span><span class="p">]],</span> <span class="n">axis</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Compute attention logits w/ a dense layer (single output, LeakyReLU)
</span><span class="n">logits</span> <span class="o">=</span> <span class="n">dense</span><span class="p">(</span><span class="n">h_cat</span><span class="p">)</span>
<span class="c1"># Apply softmax only to the logits in the same segment, as defined by recipients
# i.e., normalize the scores only among the neighbors of each node.
#
# Note that segment_softmax does **not** reduce the tensor: `coef` has the same
# size as `logits`.
#
# This function is available in Spektral and PyG.
</span><span class="n">coef</span> <span class="o">=</span> <span class="n">segment_softmax</span><span class="p">(</span><span class="n">logits</span><span class="p">,</span> <span class="n">recipients</span><span class="p">)</span>
<span class="c1"># Now we aggregate with a weighted sum (weights given by coef)
</span><span class="n">output</span> <span class="o">=</span> <span class="n">jax</span><span class="p">.</span><span class="n">ops</span><span class="p">.</span><span class="n">segment_sum</span><span class="p">(</span><span class="n">coef</span> <span class="o">*</span> <span class="n">h</span><span class="p">[</span><span class="n">senders</span><span class="p">],</span> <span class="n">recipients</span><span class="p">)</span>
</code></pre></div></div>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-18.svg" width="100%" style="border: solid 1px;" /></p>
<p>And, easily enough, we can also define a message-passing network that includes edge attributes in the computation of messages. One of my favorite models is the <a href="https://arxiv.org/abs/1704.02901">Edge-Conditioned Convolution</a> by Simonovsky & Komodakis, of which I’ve summarized the math in the slide above.</p>
<p>To implement it with gather-scatter we can do:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Use a Filter-Generating Network to create the weights (defined elsewhere)
</span><span class="n">kernel</span> <span class="o">=</span> <span class="n">fgn</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="c1"># Reshape the weights so that we have a matrix of shape (f, f_) for each edge
</span><span class="n">kernel</span> <span class="o">=</span> <span class="n">jnp</span><span class="p">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">kernel</span><span class="p">,</span> <span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">f</span><span class="p">,</span> <span class="n">f_</span><span class="p">))</span>
<span class="c1"># Multiply the node attribute of each neighbor by the associated edge-dependent
# kernel.
# We can use einsum to do this efficiently
</span><span class="n">messages</span> <span class="o">=</span> <span class="n">jnp</span><span class="p">.</span><span class="n">einsum</span><span class="p">(</span><span class="s">"ab,abc->ac"</span><span class="p">,</span> <span class="n">x</span><span class="p">[</span><span class="n">senders</span><span class="p">],</span> <span class="n">kernel</span><span class="p">)</span>
<span class="c1"># Aggergate with a sum
</span><span class="n">output</span> <span class="o">=</span> <span class="n">jax</span><span class="p">.</span><span class="n">ops</span><span class="p">.</span><span class="n">segment_sum</span><span class="p">(</span><span class="n">messages</span><span class="p">,</span> <span class="n">recipients</span><span class="p">)</span>
</code></pre></div></div>
<p>Once you get the hang of it, building GNNs becomes so intuitive that you’ll never want to go back to the matrix-multiplication-based implementations.
Although, sometimes, it makes sense to do it. But that’s a story for another day.</p>
<hr />
<p>With the first two parts of this blog series in your toolbelt, you should be able to go a long way in the world of GNNs.</p>
<p>The next and final part will take a more historical and mathematical journey in the world of GNNs. We’ll cover spectral graph theory and how we can define the operation of <strong>convolution</strong> on graphs.</p>
<p>I have left this for last because it is not <em>essential</em> to understand and use GNNs in practice, although I think that understanding the historical perspective that led to the creation of modern GNNs is very important.</p>
<p>Stay tuned.</p>
Fri, 12 Mar 2021 00:00:00 +0000
/posts/2021-03-12/gnn-lecture-part-2.html
GNNlecturepostsA practical introduction to GNNs - Part 1<p><em>This is Part 1 of an introductory lecture on graph neural networks that I gave for the “Graph Deep Learning” course at the University of Lugano.</em></p>
<p><em>At this point in the course, the students had already seen a high-level overview of GNNs and some of their applications. My goal was to give them a practical understanding of GNNs.</em></p>
<p><em>Here I show that, starting from traditional CNNs and changing a few underlying assumptions, we can create a neural network that processes graphs.</em></p>
<p><a href="https://danielegrattarola.github.io/files/talks/2021-03-01-USI_GDL_GNNs.pdf">The full slide deck is available here</a>.</p>
<hr />
<p>My goal for this lecture is to show you how Graph Neural Networks (GNNs) can be obtained as a generalization of traditional convolutional neural networks (CNNs), where instead of images we have graphs as input.</p>
<p><strong>But what does it mean that a CNN can be made more general? Why are graphs a more general version of images?</strong></p>
<p>We know that CNNs are designed to process data that describe the world through a collection of discrete data points: time steps in a time series, pixels in an image, pixels in a video, etc.</p>
<p>However, one aspect of images and time series that we rarely (if at all) consider explicitly is the fact that the collection of data points alone is not enough. The order in which pixels are arranged to form an image is possibly more important than the pixels themselves. <br />
An image can be in color or in grayscale but, as long as the arrangement of pixels is the same, we’ll likely be able to recognize the image for what it is.</p>
<p>We could go as far as saying that an image is only an image because its pixels are arranged in a particular structure: pixels that represent points close in space or time should also be next to each other in the collection. Change this structure, and the image loses meaning.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-4.svg" width="100%" style="border: solid 1px;" /></p>
<p>CNNs are designed to take this <strong>locality</strong> into account. They are designed to transform the value of each pixel, not as a function of the whole image (like a MLP would do), but as a function of the pixel’s immediate surroundings. Its neighbors.</p>
<p>Since <strong>locality is a kind of relation</strong> between pixels, it is natural to represent the underlying structure of an image using a graph.
And, by requiring that each pixel is related only to the few other pixels that are closer to it, our graph will be a <strong>regular grid</strong>. Every pixel has 8 neighbors (give or take boundary conditions), and the CNN uses this fact to compute a localized transformation.</p>
<p>You can also interpret it the other way around. The kind of processing that the CNN does means that the transformation of each pixel will only depend on the few pixels that fall under the convolutional kernel. We can say that the grid structure emerges as a consequence of the CNN’s inductive bias.</p>
<p>In any case, the important thing to note is that the grid structure does not depend on the specific pixel values. <strong>We separate the values of the data points from the underlying structure that supports them.</strong></p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-5.svg" width="100%" style="border: solid 1px;" /></p>
<p>With this perspective in mind, the question of “how to make CNNs work on graphs” becomes:</p>
<p><em>Can we create a neural network in which the structure of the data is no longer a regular grid, but an arbitrary graph that we give as input?</em></p>
<p>In other words, since we know that data and structure are different things, can we change the structure as we please?</p>
<p>The only thing that we require is that the CNN does the same kind of local processing as it did for the regular grid: transform each node as a function of its neighbors.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-6.svg" width="100%" style="border: solid 1px;" /></p>
<p>If we look at what this request entails, we immediately see some problems:</p>
<ol>
<li>
<p>In the “regular grid” case, the learnable kernel of the CNN is compact and has a fixed size: one set of weights for each possible neighbor of a pixel, plus one set for the pixel itself. In other words, the kernel is supported by a smaller grid.
We can’t do that easily for an arbitrary graph. Since nodes can have a variable number of neighbors, we also need a kernel that varies in size. Possible, but not straightforward.</p>
</li>
<li>
<p>In the regular grids processed by CNNs, we have an implicit notion of directionality. We always know where up, down, left and right are. When we move to an arbitrary graph, we might not be able to define a direction. Direction is, in essence, a kind of attribute that we assign to the edges, but in our case we also allow graphs that have no edge attributes at all. Ask yourself: do you have an up-and-to-the-left follower on Twitter?</p>
</li>
</ol>
<p>To go from CNN to GNN we need to solve these problems.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-7.svg" width="100%" style="border: solid 1px;" /></p>
<p><em>[I recall notation here because the students had already seen most of these things anyway, but the concept of “reference operator” gave me a nice segue into the next slide.]</em></p>
<p>All this talking about edge attributes also made me remember that now is a good time to do a notation check. Briefly:</p>
<ul>
<li>
<p>We define a graph as a collection of nodes and edges.</p>
</li>
<li>
<p>Nodes can have vector attributes, which we represent in a neatly packed matrix \(\mathbf{X} \in \mathbb{R}^{N \times F}\) (sometimes called a <em>graph signal</em>).
Same thing for edges, with attributes \(\mathbf{e}_{ij} \in \mathbb{R}^S\) for edge i-j.</p>
</li>
</ul>
<p>Then there are the characteristic matrices of a graph:</p>
<ul>
<li>
<p>The adjacency matrix \(\mathbf{A}\) is binary and has a 1 in position i-j if there exists an edge from node i to node j. All entries are 0 otherwise.</p>
</li>
<li>
<p>The degree matrix \(\mathbf{D}\) counts the number of neighbors of each node. It’s a diagonal matrix so that the degree of node i is in position i-i.</p>
</li>
<li>
<p>The Laplacian, which we will use a lot later, is defined as \(\mathbf{L} = \mathbf{D} - \mathbf{A}\).</p>
</li>
<li>
<p>Finally, the normalized adjacency matrix is \(\mathbf{A}_n = \mathbf{D}^{-1/2} \mathbf{A} \mathbf{D}^{-1/2}\).</p>
</li>
</ul>
<p>Note that \(\mathbf{A}\), \(\mathbf{L}\), and \(\mathbf{A}_n\) <strong>share the same sparsity pattern, if you don’t count the diagonal</strong>. Their only non-zero entries are in position i-j only if edge i-j exists.</p>
<p>Since we’re more interested in this specific property than in the actual values that are stored in the non-zero entries, let’s give it a name: we call any matrix that has the same sparsity pattern of \(\mathbf{A}\) a <strong>reference operator</strong> (sometimes a <em>structure</em> operator, sometimes a <em>graph shift</em> operator, it’s not important).</p>
<p>Also note: so far we are considering graphs with undirected edges. This means that all reference operators will be symmetric (if edge i-j exists, then edge j-i exists).</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-8.svg" width="100%" style="border: solid 1px;" /></p>
<p>Reference operators are nice.</p>
<p>First of all, they are operators. You multiply them by a graph signal and you get a new graph signal in return. Let’s look at the “shape” of the multiplication: N-by-N times N-by-F equals N-by-F. Checks out.</p>
<p>But not only that. By their own definition, multiplying a reference operator by a graph signal will compute a weighted sum of each node’s neighborhood. Let’s expand the matrix multiplication from the slide above to see what happens to node 1 when we apply a reference operator.</p>
<p>All values \(\mathbf{r}_{ij}\) that are not associated with an edge are 0, so we have:</p>
\[(\mathbf{R}\mathbf{X})_1 = \mathbf{r}_{12}\cdot\mathbf{x}_2 + \mathbf{r}_{13}\cdot\mathbf{x}_3 + \mathbf{r}_{14}\cdot\mathbf{x}_4\]
<p>Look at that: with a simple matrix multiplication we can now do the same kind of local processing that the CNN does.</p>
<p>Since applying a reference operator results in a simple sum-product, the result will not depend on the particular order in which we consider the nodes. As long as row \(i\) of the reference operator describes the connections of the node with attributes \(\mathbf{x}_i\), the result will be the same. We say that this kind of operation is <strong>equivariant to permutations of the nodes</strong>. <br />
This is good, because the particular order with which we consider the nodes is not important. Remember: we’re only interested in the structure – which nodes are connected to which.</p>
<p>Now that we are able to aggregate information from a node’s neighborhood, we only need to solve the issue of how to create the learnable kernel and we will have a good first approximation of a CNN for graphs. Remember the two issues that we have:</p>
<ol>
<li>Neighborhoods vary in size;</li>
<li>We don’t know how to orient the kernel (i.e., we may not have attributes that allow us to distinguish a node’s neighbors);</li>
</ol>
<p>These problems are also related to our request that the GNN must be equivariant to permutations. We cannot simply assign a different weight to each neighbor because we would need to train the GNN on all possible permutations of the nodes in order to make it equivariant.</p>
<p>However, there is a simple solution: <strong>use the same set of weights for each node in the neighborhood.</strong><br />
Let our weights be a matrix \(\mathbf{\Theta} \in \mathbb{R}^{F \times F'}\), so that the output will have \(F'\) “feature maps”.</p>
<p>Now, we simply use \(\mathbf{\Theta}\) to transform the node attributes, then sum them over using a reference operator.</p>
<p>Let’s check the shapes to make sure that it works out: N-by-N times N-by-F times F-by-F’ equals N-by-F’.<br />
We went from graph signal to graph signal, with new node attributes that we obtain as a local, learnable, and differentiable transformation.</p>
<p>Done! We have our first GNN: \(\mathbf{X}' = \mathbf{R} \mathbf{X} \mathbf{\Theta}\).</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-9.svg" width="100%" style="border: solid 1px;" /></p>
<p>One thing that is still missing from our relatively simple implementation is the ability to have kernels that span more than the immediate neighborhood of a node. In fact, in a CNN this is usually a hyperparameter. Also, depending on the reference operator that we use, we may or may not consider a node itself when computing its transformation: it depends on whether \(\mathbf{R}\) has a non-zero diagonal.</p>
<p>Luckily we can generalize the idea of a bigger kernel to the graph domain: we simply process each node as a function of its neighbors up to \(K\) steps away from it.</p>
<p>We can achieve this by considering that applying a reference operator to a graph signal has the effect of making node attributes <em>flow</em> through the graph.
Apply a reference operator once, and all nodes will “read” from their immediate neighbors to update themselves. Apply it again, and all nodes will read again from their neighbors, except that this time the information that they read will be whatever the neighbors computed at the previous step.</p>
<p>In other words: if we multiply a graph signal by \(\mathbf{R}^{K}\), each node will update itself with the node attributes of nodes \(K\) steps away.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-10.svg" width="100%" style="border: solid 1px;" /></p>
<p>In a CNN, this would be equivalent to having a kernel shaped like an empty square.
To make the kernel full, we simply sum all “empty square” kernels up to the desired size. In our case, instead of considering \(\mathbf{R}^{K}\), we consider a polynomial of \(\mathbf{R}\) up to order \(K\).</p>
<p>This is called a <strong>polynomial graph filter</strong>, and we will see a different interpretation of it in Part 3 of this series.</p>
<p>Note that this filter solves both problems that we had before, and also makes our GNN more expressive:</p>
<ol>
<li>The value of a node itself is always included in the transformation, since \(\mathbf{R}^{0} = \mathbf{I}\);</li>
<li>The sum of polynomials up to order \(K\) will necessarily cover all neighbors in a radius of \(K\) steps;</li>
<li>Since we can treat neighborhoods separately, we can also have different weights \(\mathbf{\Theta}^{(k)}\) for each \(k\)-hop neighborhood. This is like having a radial filter, a function that only depends on the radius from the origin.</li>
</ol>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-11.svg" width="100%" style="border: solid 1px;" /></p>
<p>This idea of using a polynomial filter to create a GNN was first introduced in a paper by <a href="https://arxiv.org/abs/1606.09375">Defferrard et al.</a>, which can be seen as the first scalable and practical implementation of a GNN ever proposed.</p>
<p>In that paper they used a particular choice of polynomial, namely one for which different powers are defined in a recursive manner, called a <strong>Chebyshev polynomial</strong>.</p>
<p>In particular, as reference operator they use a version of the graph Laplacian that is first normalized and then rescaled so that its eigenvalues are between -1 and 1.
Then, using the recursive formulation of Chebyshev polynomials, they build a polynomial graph filter.</p>
<p>The reason why they use these polynomials and not the simple ones we saw above is not important, for now. Let us just say: they have some desirable properties and they are fast to compute.</p>
<p><img src="https://danielegrattarola.github.io/images/2021-03-03/presentation-12.svg" width="100%" style="border: solid 1px;" /></p>
<p>Just a few months after the paper by Defferrard et al. was published on ArXiv, a new paper by <a href="">Kipf & Welling</a> also appeared online.</p>
<p>In that paper, the authors looked at the Chebyshev filter proposed by Defferrard et al. and introduced a few key changes to make the layer more simple and more scalable.</p>
<ol>
<li>They changed the reference operator. Instead of the rescaled and normalized Laplacian, they assumed that \(\lambda_{max} = 2\) so that the whole formulation of the operator was simplified to \(-\mathbf{A}_n\).</li>
<li>They proposed to use polynomials of order 1, following the intuition that \(K\) layers of order 1 would be equivalent to 1 layer of order \(K\). In particular, they also added non-linearities between each successive layer, leading to more complex transformations of the nodes at each propagation step.</li>
<li>They observed that the same set of weights could be used both for a node itself and its neighbors. No need to have \(\mathbf{\Theta}^{(0)}\) and \(\mathbf{\Theta}^{(1)}\) as different weights.</li>
<li>After simplifying the layer down to
\(\mathbf{X}' = ( \mathbf{I} + \mathbf{A}_n) \mathbf{X} \mathbf{\Theta},\)
they observed that a more stable behavior could be obtained by instead using \(\mathbf{R} = \mathbf{D}^{-1/2} (\mathbf{I} + \mathbf{A}) \mathbf{D}^{-1/2}\) as reference operator.</li>
</ol>
<p>Putting this all together, we get to what is commonly known as the Graph Convolutional Network (GCN):</p>
\[\mathbf{X}' = \mathbf{D}^{-1/2} (\mathbf{I} + \mathbf{A}) \mathbf{D}^{-1/2} \mathbf{X} \mathbf{\Theta}\]
<hr />
<p>What we have seen so far is a very simple construction that takes the general concepts behind CNNs and, by changing a few assumptions, extends them to the case in which the input is an arbitrary graph instead of a grid.</p>
<p>This is far from the whole story, but it should give you a good starting point to learn about GNNs.</p>
<p>In the <a href="https://danielegrattarola.github.io/posts/2021-03-12/gnn-lecture-part-2.html">next part of this series</a> we will see:</p>
<ol>
<li>How to describe what we just saw as a general algorithm that allows us to describe a much richer family of operations on graphs.</li>
<li>How to throw edge attributes in the mix and create GNNs that can treat neighbors differently.</li>
<li>How to make the entries of a reference operator a learnable function.</li>
<li>A general recipe for a GNN that <em>should</em> work well for many problems.</li>
</ol>
<p>Stay tuned.</p>
Wed, 03 Mar 2021 00:00:00 +0000
/posts/2021-03-03/gnn-lecture-part-1.html
GNNlecturepostsTelestrations Neural Networks<p><img src="https://danielegrattarola.github.io/images/2020-01-21/telestrations.jpg" alt="" /></p>
<p>Yesterday, it was board game day at <a href="http://www.neurontobrainlaboratory.ca/">the lab</a> where I have been working recently.
Everyone got together for lunch at Snakes & Lattes, a Torontonian board game cafè chain, and we spent a couple of hours laughing and chatting and, obviously, playing board games.</p>
<p>The lab has a go-to traditional game for the occasion: <a href="https://en.wikipedia.org/wiki/Telestrations">Telestrations</a>.
The game is inspired by the classic childhood’s game of <a href="https://en.wikipedia.org/wiki/Chinese_whispers">Chinese whispers</a> (or <em>Telephone</em>, or <em>Wireless phone</em>, or <em>Gossip</em>, there’s a bunch of different names for different countries) and its rules are pretty simple.</p>
<p>Everyone gets a booklet, an erasable sharpie, and a list of random terms like “flamingo” or “pipe dream” or “treehouse”. Everyone picks a word and writes it on the first page of the booklet: that’s the secret source word.</p>
<p>At each turn, players pass their booklet to the person on their right, and the rules are as follows:</p>
<ul>
<li>When you see a word, you turn the page and you have sixty seconds to <em>draw</em> whatever the word is;</li>
<li>When you see a drawing, you turn the page and you write your best guess for what is pictured.</li>
</ul>
<p>Players keep alternating between guessing, drawing, and passing down the booklets until every booklet has done a full round of the table and is back in the hands of the original owner.
For extra fun, everybody gets to draw their secret source word at the very beginning.</p>
<p>In other words, it’s a written game of Chinese whispers where every other word is drawn instead of written.</p>
<p>There are some rules to decide who wins at the end, but the obvious source of entertainment is the complete chaos that ensues as information gets corrupted drawing after drawing. At the end of a round, not one of the original secret words ever survives.</p>
<p>So now the obvious, rational, almost trivial question is: what happens when you use a GAN to draw, and an image classifier to guess? <br />
Well, here I am to show you!</p>
<!--more-->
<h2 id="how-to-in-three-paragraphs">How-to in three paragraphs</h2>
<p><a href="https://openreview.net/forum?id=B1xsqj09Fm">BigGAN</a> can generate images conditioned on an ImageNet label. So if you give it label 1, it will generate goldfish, if you give it label 42, it will generate an agama, and so on.</p>
<p>ResNet does the opposite: if you show it a goldfish, it will try to guess what it is. To make things more interesting, I added a bit of noise to the guessing procedure, so that sometimes we get a random one out of the top-5 guesses. If you think that this is unreasonable, try and play a game with real humans, I dare you.</p>
<p>The idea now is to play the game using BigGAN to draw, and ResNet to guess: you start with a label, you have BigGAN generate an image of that label, you classify that image to get a new label, and so on.</p>
<h2 id="results">Results</h2>
<p>I’ll start with my favourite sequence: honeycomb to cheeseburger.
The images below are read top-to-bottom, left-to-right. At the very top you see the source class, then the first generated image, then what that image was classified as, then the next generated image, etc..</p>
<p>The first image is generated from class 599 of ImageNet, “honeycomb”. It looked a lot like a bagel, I guess because of that bright spot in the middle (?), so the ResNet classified it as such. From that classification, we get a couple of bagel-y looking pieces of bread, which soon become French loafs, then dough.<br />
Then, that perfect-looking dough in image 6 gets classified as a wooden spoon (probably because of the extra noise that I mentioned).
Finally, the green spot on the wooden spoon confuses ResNet into thinking it’s a cheeseburger, and we get juicy burgers until the end. That burger generation is impressive, not gonna lie.</p>
<p><img src="https://danielegrattarola.github.io/images/2020-01-21/bagel.png" alt="" /></p>
<p>Moving on: trilobite to long-horned beetle.
The first two trilobites look really good, but then get classified as isopods after two turns (curiously, isopods and trilobites look a lot similar but are not that closely related according <a href="https://www.reddit.com/r/geology/comments/lt9so/how_closely_related_are_isopods_to_trilobites/">to Reddit</a>).
From the isopod label, we get what is clearly a marine creature (look at the background), which unfortunately gets classified as a cockroach. From there, we stay on dry land and just get more and more specialized bugs until the end.</p>
<p><img src="https://danielegrattarola.github.io/images/2020-01-21/trilobite.png" alt="" /></p>
<p>The next one is <strong>REALLY</strong> good because it’s remarkably similar to a real game of Telestrations. It could happen. Hell, it probably happened.</p>
<p>We start with a coffeepot. At image three, the coffeepot is a bit ambiguous and becomes a teapot. Understandable, I would probably have made that mistake myself. Then we get a proper teapot, that gets recognized as such.
The next image, however, is half-assed by the player and it’s not clear at all what it is. The next player guesses that it’s a pitcher. The next guy tries his best but eventually, the pitcher becomes a vase.</p>
<p>Nothing more to say, I can see this happening in real life.</p>
<p><img src="https://danielegrattarola.github.io/images/2020-01-21/coffeepot.png" alt="" /></p>
<p>Our next and last one is also a likely sequence.</p>
<p>A volcano. Easy. We get two perfect volcano drawings. Except that the last one gets classified as a type of tent.</p>
<p>The next player over-does it, and draws a full camping spot with caravans instead of a tent. Curiously, we still have a volcano-looking thing in the background, but that’s just a coincidence (no information from previous images or labels is preserved between turns).</p>
<p>The camp is seen as a bee house. Next thing we know, there’s a weird-looking BigGAN human harvesting honey.
But ResNet doesn’t care about the human and focuses on the crate in the middle, instead.</p>
<p>We get a good-looking crate, that becomes a chest, and we stay with chests until the end.</p>
<p>The yurt and the apiary are the only weird ones in this sequence, and the least likely to appear in a human game. I can see someone drawing a full camping spot instead of a single yurt, and I can see how one would mistake a poorly-drawn volcano for a tent, but no human would ignore the beekeeper in image 4.</p>
<p><img src="https://danielegrattarola.github.io/images/2020-01-21/volcano.png" alt="" /></p>
<p>I have generated a bunch of these sequences on my laptop, and these are just four random ones that I got. It’s really easy to get fun sequences. So here’s how I did it.</p>
<h2 id="code">Code</h2>
<p>First of all, I was not going to spend a single € to train anything involved in this project because, like, let’s be real…</p>
<p>So I turned to Google and I found:</p>
<ul>
<li><a href="https://github.com/huggingface/pytorch-pretrained-BigGAN">A pre-trained BigGAN</a></li>
<li><a href="https://pytorch.org/docs/stable/torchvision/index.html">Torchvision’s pre-trained ResNet50</a></li>
</ul>
<p>I usually write my stuff in TensorFlow but whatever, let’s PyTorch this one.</p>
<p>We start with some essential imports:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="kn">import</span> <span class="nn">torch</span>
<span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="kn">from</span> <span class="nn">pytorch_pretrained_biggan</span> <span class="kn">import</span> <span class="n">BigGAN</span><span class="p">,</span> <span class="n">one_hot_from_int</span><span class="p">,</span> <span class="n">truncated_noise_sample</span><span class="p">,</span> <span class="n">convert_to_images</span>
<span class="kn">from</span> <span class="nn">spektral.utils</span> <span class="kn">import</span> <span class="n">init_logging</span>
<span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">torchvision</span> <span class="kn">import</span> <span class="n">transforms</span>
</code></pre></div></div>
<p>and we define a couple of useful variables:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iterations</span> <span class="o">=</span> <span class="mi">8</span> <span class="c1"># How many players there are
</span><span class="n">standard_noise</span> <span class="o">=</span> <span class="mf">0.3</span> <span class="c1"># Some random noise because people are not perfect
</span><span class="n">current_class</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1001</span><span class="p">)</span> <span class="c1"># The secret source word is random
</span>
<span class="c1"># Load ImageNet class list
</span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'imagenet_classes.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">labels</span> <span class="o">=</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">.</span><span class="n">readlines</span><span class="p">()]</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">imagenet_classes.txt</code> <a href="https://github.com/Lasagne/Recipes/blob/master/examples/resnet50/imagenet_classes.txt">can be found online</a>, it’s just a list of ImageNet class names.</p>
<p>Now, let’s create the models that we will use. First we create the GAN:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gan</span> <span class="o">=</span> <span class="n">BigGAN</span><span class="p">.</span><span class="n">from_pretrained</span><span class="p">(</span><span class="s">'biggan-deep-256'</span><span class="p">)</span>
<span class="n">gan</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="s">'cuda'</span><span class="p">)</span>
</code></pre></div></div>
<p>Then, we create the ResNet50 ImageNet classifier:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">classifier</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="n">resnet50</span><span class="p">(</span><span class="n">pretrained</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">classifier</span><span class="p">.</span><span class="nb">eval</span><span class="p">()</span> <span class="c1"># Do this to set the model to inference mode
</span></code></pre></div></div>
<p>and its image pre-processor:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">transform</span> <span class="o">=</span> <span class="n">transforms</span><span class="p">.</span><span class="n">Compose</span><span class="p">([</span>
<span class="n">transforms</span><span class="p">.</span><span class="n">CenterCrop</span><span class="p">(</span><span class="mi">224</span><span class="p">),</span>
<span class="n">transforms</span><span class="p">.</span><span class="n">ToTensor</span><span class="p">(),</span>
<span class="n">transforms</span><span class="p">.</span><span class="n">Normalize</span><span class="p">(</span>
<span class="n">mean</span><span class="o">=</span><span class="p">[</span><span class="mf">0.485</span><span class="p">,</span> <span class="mf">0.456</span><span class="p">,</span> <span class="mf">0.406</span><span class="p">],</span>
<span class="n">std</span><span class="o">=</span><span class="p">[</span><span class="mf">0.229</span><span class="p">,</span> <span class="mf">0.224</span><span class="p">,</span> <span class="mf">0.225</span><span class="p">]</span>
<span class="p">)])</span>
</code></pre></div></div>
<p>We will be drawing and guessing images of 256 x 256 pixels (cropped to 224 x 244 for ResNet50). The hard-coded normalization is just something that you have to do for Torchvision models, no biggie.</p>
<p>So now we have loaded the networks. Let’s define some helper functions that will compute the main steps of the game for us:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">draw</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="n">truncation</span><span class="o">=</span><span class="mf">1.</span><span class="p">):</span>
<span class="c1"># Create the inputs for the GAN
</span> <span class="n">class_vector</span> <span class="o">=</span> <span class="n">one_hot_from_int</span><span class="p">([</span><span class="n">label</span><span class="p">],</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">class_vector</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">from_numpy</span><span class="p">(</span><span class="n">class_vector</span><span class="p">)</span>
<span class="n">class_vector</span> <span class="o">=</span> <span class="n">class_vector</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="s">'cuda'</span><span class="p">)</span>
<span class="n">noise_vector</span> <span class="o">=</span> <span class="n">truncated_noise_sample</span><span class="p">(</span><span class="n">truncation</span><span class="o">=</span><span class="n">truncation</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">noise_vector</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">from_numpy</span><span class="p">(</span><span class="n">noise_vector</span><span class="p">)</span>
<span class="n">noise_vector</span> <span class="o">=</span> <span class="n">noise_vector</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="s">'cuda'</span><span class="p">)</span>
<span class="c1"># Generate image
</span> <span class="k">with</span> <span class="n">torch</span><span class="p">.</span><span class="n">no_grad</span><span class="p">():</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">gan</span><span class="p">(</span><span class="n">noise_vector</span><span class="p">,</span> <span class="n">class_vector</span><span class="p">,</span> <span class="n">truncation</span><span class="p">)</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">output</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="s">'cpu'</span><span class="p">)</span>
<span class="c1"># Get a PIL image from a Torch tensor
</span> <span class="n">img</span> <span class="o">=</span> <span class="n">convert_to_images</span><span class="p">(</span><span class="n">output</span><span class="p">)</span>
<span class="k">return</span> <span class="n">img</span>
<span class="k">def</span> <span class="nf">guess</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="n">top</span><span class="o">=</span><span class="mi">5</span><span class="p">):</span>
<span class="c1"># Pre-process image
</span> <span class="n">img</span> <span class="o">=</span> <span class="n">transform</span><span class="p">(</span><span class="n">img</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="c1"># Classify image
</span> <span class="n">classification</span> <span class="o">=</span> <span class="n">classifier</span><span class="p">(</span><span class="n">img</span><span class="p">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span>
<span class="n">_</span><span class="p">,</span> <span class="n">indices</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">sort</span><span class="p">(</span><span class="n">classification</span><span class="p">,</span> <span class="n">descending</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">percentage</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">nn</span><span class="p">.</span><span class="n">functional</span><span class="p">.</span><span class="n">softmax</span><span class="p">(</span><span class="n">classification</span><span class="p">,</span> <span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># Get the global ImageNet class, labels, and the predicted probabilities
</span> <span class="n">idxs</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">([</span><span class="n">idx</span> <span class="k">for</span> <span class="n">idx</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">[</span><span class="mi">0</span><span class="p">]][:</span><span class="n">top</span><span class="p">])</span>
<span class="n">labs</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">([</span><span class="n">labels</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span> <span class="k">for</span> <span class="n">idx</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">[</span><span class="mi">0</span><span class="p">]][:</span><span class="n">top</span><span class="p">])</span>
<span class="n">probs</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">([</span><span class="n">percentage</span><span class="p">[</span><span class="n">idx</span><span class="p">].</span><span class="n">item</span><span class="p">()</span> <span class="k">for</span> <span class="n">idx</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">[</span><span class="mi">0</span><span class="p">]][:</span><span class="n">top</span><span class="p">])</span>
<span class="k">return</span> <span class="n">idxs</span><span class="p">,</span> <span class="n">labs</span><span class="p">,</span> <span class="n">probs</span>
</code></pre></div></div>
<p>Now we can start playing!</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">output_imgs</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># Stores the drawings
</span><span class="n">output_labels</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># Stores the guesses
</span><span class="n">output_labels</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">labels</span><span class="p">[</span><span class="n">current_class</span><span class="p">])</span>
<span class="c1"># Main game loop
</span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">iterations</span><span class="p">):</span>
<span class="c1"># Draw an image
</span> <span class="n">img</span> <span class="o">=</span> <span class="n">draw</span><span class="p">(</span><span class="n">current_class</span><span class="p">)</span>
<span class="n">output_imgs</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">img</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="c1"># Guess what the image is
</span> <span class="n">idxs</span><span class="p">,</span> <span class="n">labs</span><span class="p">,</span> <span class="n">probs</span> <span class="o">=</span> <span class="n">guess</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="n">top</span><span class="o">=</span><span class="n">top</span><span class="p">)</span>
<span class="c1"># Add noise
</span> <span class="n">probs</span> <span class="o">+=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">standard_noise</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">probs</span><span class="p">.</span><span class="n">shape</span><span class="p">)</span>
<span class="n">probs</span> <span class="o">/=</span> <span class="n">probs</span><span class="p">.</span><span class="nb">sum</span><span class="p">()</span> <span class="c1"># Re-normalize because of noise
</span>
<span class="c1"># Choose from the predictions
</span> <span class="n">choice</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">choice</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">labs</span><span class="p">)),</span> <span class="n">p</span><span class="o">=</span><span class="n">probs</span><span class="p">)</span>
<span class="n">current_class</span> <span class="o">=</span> <span class="n">idxs</span><span class="p">[</span><span class="n">choice</span><span class="p">]</span>
<span class="n">output_labels</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">labs</span><span class="p">[</span><span class="n">choice</span><span class="p">])</span>
</code></pre></div></div>
<p>At the end of the game, we will have the generated drawings in <code class="language-plaintext highlighter-rouge">output_imgs</code> and the guesses in <code class="language-plaintext highlighter-rouge">output_labels</code>.</p>
<p>Here, instead of copy-pasting from the cells above you can just look at <a href="https://gist.github.com/danielegrattarola/8296b9fd29116443da74d0aa2519d7c3">the full gist</a>.</p>
<h2 id="conclusions">Conclusions</h2>
<p>What can I say? It’s neural networks playing Telestrations.</p>
<p>“No new knowledge can be extracted from my telling. This confession has meant nothing.”</p>
<p>Cheers!</p>
<hr />
<p>In case you didn’t know: agamas (label 42 of ImageNet) are extra-fucking-cool lizards.</p>
<p><img src="https://danielegrattarola.github.io/images/2020-01-21/agama.jpg" alt="" /></p>
Tue, 21 Jan 2020 00:00:00 +0000
/posts/2020-01-21/telestrations.html
AIrandomcodepostsPitfalls of Graph Neural Network Evaluation 2.0<p>In this post, I’m going to summarize some conceptual problems that I have found when comparing different graph neural networks (GNNs) between them.</p>
<p>I’m going to argue that it is extremely difficult to make an objectively fair comparison between structurally different models and that the experimental comparisons found in the literature are not always sound.</p>
<p>I will try to suggest reasonable solutions whenever possible, but the goal of this post is simply to make these issues appear on your radar and maybe spark a conversation on the matter.</p>
<p>Some of the things that I’ll say are also addressed in the original <a href="https://arxiv.org/abs/1811.05868">Pitfalls of Graph Neural Network Evaluation (Shchur et al., 2018)</a>, which I warmly suggest you read.</p>
<!--more-->
<h2 id="neighbourhoods">Neighbourhoods</h2>
<p>The first source of inconsistency when comparing GNNs comes from the fact that different layers are designed to take into account neighborhoods of different sizes.<br />
We usually have that a layer either looks at the 1-neighbours of each node, or it has a hyperparameter K that controls the size of the neighbourhood. Some examples of popular methods (implemented both in Spektral and Pytorch Geometric) in either category:</p>
<ul>
<li>1-hop: <a href="https://arxiv.org/abs/1609.02907">GCN</a>, <a href="https://arxiv.org/abs/1710.10903">GAT</a>, <a href="https://arxiv.org/abs/1706.02216">GraphSage</a>, <a href="https://arxiv.org/abs/1810.00826">GIN</a>;</li>
<li>K-hop: <a href="https://arxiv.org/abs/1606.09375">Cheby</a>, <a href="https://arxiv.org/abs/1901.01343">ARMA</a>, <a href="https://arxiv.org/abs/1810.05997">APPNP</a>, <a href="https://arxiv.org/abs/1902.07153">SGC</a>.</li>
</ul>
<p>A fair evaluation should keep these differences into account and allow each GNN to look at the same neighborhoods, but at the same time, it could be argued that a layer designed to operate on larger neighborhoods is more expressive. How can we tell what is better?</p>
<p>Let’s say we are comparing GCN with Cheby. The equivalent of a 2-layer GCN could be a 2-layer Cheby with K=1, or a 1-layer Cheby with K=2. In the GCN paper, they use a 2-layer Cheby with K=3. Should they have compared with a 6-layer GCN?</p>
<p>Moreover, this difference between methods may have an impact on the number of parameters, nonlinearity, and overall amount of regularization in a GNN. <br />
For instance, a GCN that reaches a neighborhood of order 3 may have 3 dropout layers, while the equivalent Cheby with K=3 will have only one. <br />
Another example: an SGC architecture can reach any neighborhood with a constant number of parameters, while other methods can’t.</p>
<p>We’re only looking at one simple issue, and it is already difficult to say how to fairly evaluate different methods. It gets worse.</p>
<h2 id="regularization-and-training">Regularization and training</h2>
<p>Regularization is an aspect that is particularly essential in GNNs, because the community uses very small benchmark datasets and most GNNs tend to overfit like crazy (more on this later).
For these reasons, the performance of a GNN can vary wildly depending on how the model is regularized. This is true for all other hyperparameters in general, because things like the learning rate and batch size can be a form of implicit regularization.</p>
<p>The literature is largely inconsistent with how regularization is applied across different papers, making it difficult to say whether the performance improvements reported for a model are due to the actual contribution or to a different regularization scheme.</p>
<p>The following are often found in the literature:</p>
<ul>
<li>High learning rates;</li>
<li>High L2 penalty;</li>
<li>Extremely high dropout rates on node features and adjacency matrix;</li>
<li>Low number of training epochs;</li>
<li>Low patience for early stopping.</li>
</ul>
<p>I’m going to focus on a few of these.</p>
<p>First, I argue that setting a fixed number of training epochs is a form of alchemy that should be avoided if possible, because it’s incredibly task-specific. Letting a model train to convergence is almost always a better approach, because it’s less dependent on the initialization of the weights. If the validation performance is not indicative of the test performance and we need to stop the training without a good criterion, then something is probably wrong.</p>
<p>A second important aspect that I feel gets overlooked often is dropout. <br />
In particular, when dropout is applied to the adjacency matrix it leads to big performance improvements, because the GNN is exposed to very noisy instances of the graphs at each training step and is forced to generalize well. <br />
When comparing different models, if one is using dropout on the adjacency matrix then all the others should do the same. However, the common practice of comparing methods using the “same architecture from the original paper” means that some methods will be tested with dropout on A, and some without, as if the dropout is a particular characteristic of only some methods.</p>
<p>Finally, the remaining key factors in training are the learning rate and weight decay.
These are often given as-is in the literature, but it is a good idea to tune them whenever possible. For what it’s worth, I can personally confirm that searching for a good learning rate, in particular, can lead to unexpected results, even for well-established methods (if the model is trained to convergence).</p>
<h2 id="parallel-heads">Parallel heads</h2>
<p><em>Heads</em> are parallel computational units that perform the same calculation with different weights and then merge the results to produce the output. To give a sense of the problems that one may encounter when comparing methods that use heads, I will focus on two methods: GAT and ARMA.</p>
<p>Having parallel attention heads is fairly common in NLP literature, from where the very concept of attention comes, and therefore it was natural to do the same in GAT.</p>
<p>In ARMA, using parallel <em>stacks</em> is theoretically motivated by the fact that ARMA filters of order H can be computed by summing H ARMA filters of order 1. While similar in practice to the heads in GAT, in this case having parallel heads is key to the implementation of this particular graph filter.</p>
<p>Because of these fundamental semantic differences, it is impossible to say whether a comparison between GAT with H heads and an ARMA layer of order H is fair.</p>
<p>Extending to the other models as well, it is not guaranteed that having parallel heads would necessarily lead to any practical improvements for a given model. Some methods can, in fact, benefit from a simpler architecture.
It is therefore difficult to say whether a comparison between monolithic and parallel architectures is fair.</p>
<h2 id="datasets">Datasets</h2>
<p>Finally, I’m going to spend a few words on datasets, because there is no chance of having a fair evaluation if the datasets on which we test our models are not good. And in truth, the benchmark datasets that we use for evaluating GNNs are not that good.</p>
<p>Cora, CiteSeer, PubMed, and the Dortmund benchmark datasets for graph kernels: these are, collectively, the Iris dataset of GNNs, and should be treated carefully. While a model should work on these in order to be considered usable, they cannot be the only criterion to run a fair evaluation.</p>
<p>Recently, the community has moved towards a more sensible use of the datasets (ok, maybe I was exaggerating a bit about Iris), thanks to papers like <a href="https://arxiv.org/abs/1811.05868">this</a> and <a href="https://arxiv.org/abs/1910.12091">this</a>. However, many experiments in the literature still had to be repeated hundreds of times in order to give significant results, and that is bad for three reasons: time, money, and the environment, in no particular order.<br />
Especially if running a grid search of hyperparameters, it just doesn’t make sense to be using datasets that require that much computation to give reliable outcomes, more so if we consider that these are supposed to be <em>easy</em> datasets.</p>
<p>Personally, I find that there are better alternatives out there, that however are not considered often. For node classification, the GraphSage datasets (PPI and Reddit) are significantly better benchmarks than the citation networks (although they’re inductive tasks).
For graph-level learning, QM9 has 134k small graphs, of variable order, and will lead to minuscule uncertainty about the results after a few runs. I realize that it is a dataset for regression, but it still is a better alternative to PROTEINS.
For classification, Filippo Bianchi, with whom I’ve recently worked a lot, released a dataset that simply cannot be classified without using a GNN. You can find it <a href="https://github.com/FilippoMB/Benchmark_dataset_for_graph_classification">here</a>.</p>
<p>I will admit that I am as guilty as the next person when it comes to using the “bad” datasets mentioned above. One reason is that it is easy to not move away from what everybody else is doing. One reason is that reviewers outright ask for them if you don’t include them, caring little for anything else.</p>
<p>I think we can do better, as a community.</p>
<h2 id="in-conclusion">In conclusion</h2>
<p>I started thinking seriously about these issues as I was preparing a paper that required me to compare several models for the experiments.
I am not sure whether the few solutions that I have outlined here are definitive, or even correct, but I feel that this is a conversation that needs to be had in the field of GNNs.</p>
<p>Many of the comparisons that are found in the wild do not take any of this stuff into account, and I think that this may ultimately slow the progress of GNN research and its propagation to other fields of science.</p>
<p>If you want to continue this conversation, or if you have any ideas that could complement this post, shoot me an email or look for me on <a href="https://twitter.com/riceasphait">Twitter</a>.</p>
<p>Cheers!</p>
Fri, 13 Dec 2019 00:00:00 +0000
/posts/2019-12-13/pitfalls.html
AIGNNpostsImplementing a Network-based Model of Epilepsy with Numpy and Numba<p><img src="https://danielegrattarola.github.io/images/2019-10-03/2_nodes_complex_plane.png" alt="" class="full-width" /></p>
<p>Mathematically modeling how epilepsy acts on the brain is one of the major topics of research in neuroscience.
Recently I came across <a href="https://mathematical-neuroscience.springeropen.com/articles/10.1186/2190-8567-2-1">this paper</a> by Oscar Benjamin et al., which I thought that it would be cool to implement and experiment with.</p>
<p>The idea behind the paper is simple enough. First, they formulate a mathematical model of how a seizure might happen in a single region of the brain. Then, they expand this model to consider the interplay between different areas of the brain, effectively modeling it as a network.</p>
<!--more-->
<h2 id="single-system">Single system</h2>
<p>We start from a complex dynamical system defined as follows:</p>
\[\dot{z} = f(z) = (\lambda - 1 + i \omega)z + 2z|z|^2 - z|z|^4\]
<p>where \( z \in \mathbb{C} \) and \(\lambda\) controls the possible attractors of the system.
For \( 0 < \lambda < 1 \), the system has two stable attractors: one fixed point and one attractor that oscillates with an angular velocity of \(\omega\) rad/s.<br />
We can consider the stable attractor as a simplification of the brain in its resting state, while the oscillating attractor is taken to be the <em>ictal</em> state (i.e., when the brain is having a seizure).</p>
<p>We can also consider a <em>noise-driven</em> version of the system:</p>
\[dz(t) = f(z)\,dt + \alpha\,dW(t)\]
<p>where \( W(t) \) is a Wiener process rescaled by a factor \( \alpha \).<br />
A Wiener process \( W(t)_{t\ge0} \), sometimes called <em>Brownian motion</em>, is a stochastic process with the following properties:</p>
<ul>
<li>\(W(0) = 0\);</li>
<li>the increments between two consecutive observations are normally distributed with a variance equal to the time between the observations:</li>
</ul>
\[W(t + \tau) - W(t) \sim \mathcal{N}(0, \tau).\]
<p>In the noise-driven version of the system, it is guaranteed that the system will eventually <em>escape</em> any region of phase space, moving from one attractor to the other.</p>
<p>In short, we have a system that due to external, unpredictable inputs (the noise), will randomly switch from a state of rest to a state of oscillation, which we consider as a seizure.</p>
<p>The two figures below show an example of the system starting from the stable attractor and then moving to the oscillator.
Since the system is complex, we can observe its dynamics in phase space:</p>
<p><img src="https://danielegrattarola.github.io/images/2019-10-03/1_nodes_complex_plane.png" alt="" class="centered" /></p>
<p>Or we can observe the real part of \( f(t) \) as if we were reading an EEG of brain activity:</p>
<p><img src="https://danielegrattarola.github.io/images/2019-10-03/1_nodes_re_v_time.png" alt="" class="centered" /></p>
<p>See how the change of attractor almost looks like an epileptic seizure?</p>
<h2 id="network-model">Network model</h2>
<p>While this simple model of seizure initiation is interesting on its own, we can also take our modeling a step further and explicitly represent the connections between different areas of the brain (or sub-systems, if you will) and how they might affect the propagation of seizures from one area to the other.</p>
<p>We do this by defining a connectivity matrix \( A \) where \( A_{ij} = 1 \) if sub-system \( i \) has a direct influence on sub-system \( j \), and \( A_{ij} = 0 \) otherwise. In practice, we also normalize the matrix by dividing each row element-wise by the product of the square roots of the node’s out-degree and in-degree.</p>
<p>Starting from the system described above, the dynamics of one node in the networked system are described by:</p>
\[dz_{i}(t) = \big( f(z_i) + \beta \sum\limits_{j \ne i} A_{ji} (z_j - z_i) \big) + \alpha\,dW_{i}(t)\]
<p>If we look at the individual nodes, their behavior may not seem different than what we had with the single sub-system, but in reality, the attractors of these networked systems are determined by the connectivity \( A \) and the coupling strength \( \beta \).</p>
<p><img src="https://danielegrattarola.github.io/images/2019-10-03/4_graph.png" alt="" class="centered" /></p>
<p>Here’s what the networked system of 4 nodes pictured above looks like in phase space:</p>
<p><img src="https://danielegrattarola.github.io/images/2019-10-03/4_nodes_complex_plane.png" alt="" class="centered" /></p>
<p>And again we can also look at the real part of each node:</p>
<p><img src="https://danielegrattarola.github.io/images/2019-10-03/4_nodes_re_v_time.png" alt="" class="centered" /></p>
<p>If you want to have more details on how to control the different attractors of the system, I suggest you look at the <a href="https://mathematical-neuroscience.springeropen.com/articles/10.1186/2190-8567-2-1">original paper</a>. They analyze in depth the attractors and <em>escape times</em> of all possible 2-nodes and 3-nodes networks, as well as giving an overview of higher-order networks.</p>
<h2 id="implementing-the-system-with-numpy-and-numba">Implementing the system with Numpy and Numba</h2>
<p>Now that we got the math sorted out, let’s look at how to translate this system in Numpy.</p>
<p>Since the system is so precisely defined, we only need to convert the mathematical formulation into code. In short, we will need:</p>
<ol>
<li>The core functions to compute the complex dynamical system;</li>
<li>The main loop to compute the evolution of the system starting from an initial condition.</li>
</ol>
<p>While developing this, I quickly realized that my original, kinda straightforward implementation was painfully slow and that it would have required some optimization to be usable.</p>
<p>This was the perfect occasion to use <a href="http://numba.pydata.org/">Numba</a>, a JIT compiler for Python that claims to yield speedups of up to two orders of magnitude.<br />
Numba can be used to JIT compile any function implemented in pure Python, and natively supports a vast number of Numpy operations as well.
The juicy part of Numba consists of compiling functions in <code class="language-plaintext highlighter-rouge">nopython</code> mode, meaning that the code will run without ever using the Python interpreter.
To achieve this, it is sufficient to decorate your functions with the <code class="language-plaintext highlighter-rouge">@njit</code> decorator and then simply run your script as usual.</p>
<h2 id="code">Code</h2>
<p>At the very start, let’s deal with imports and define a couple of helper functions that we are going to use only once:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="kn">from</span> <span class="nn">numba</span> <span class="kn">import</span> <span class="n">njit</span>
<span class="k">def</span> <span class="nf">degree_power</span><span class="p">(</span><span class="n">adj</span><span class="p">,</span> <span class="nb">pow</span><span class="p">):</span>
<span class="s">"""
Computes D^{p} from the given adjacency matrix.
:param adj: rank 2 array.
:param pow: exponent to which elevate the degree matrix.
:return: the exponentiated degree matrix.
"""</span>
<span class="n">degrees</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">power</span><span class="p">(</span><span class="n">adj</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="nb">pow</span><span class="p">).</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">degrees</span><span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">isinf</span><span class="p">(</span><span class="n">degrees</span><span class="p">)]</span> <span class="o">=</span> <span class="mf">0.</span>
<span class="n">D</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">diag</span><span class="p">(</span><span class="n">degrees</span><span class="p">)</span>
<span class="k">return</span> <span class="n">D</span>
<span class="k">def</span> <span class="nf">normalized_adjacency</span><span class="p">(</span><span class="n">adj</span><span class="p">):</span>
<span class="s">"""
Normalizes the given adjacency matrix using the degree matrix as
D^{-1/2}AD^{-1/2} (symmetric normalization).
:param adj: rank 2 array.
:return: the normalized adjacency matrix.
"""</span>
<span class="n">normalized_D</span> <span class="o">=</span> <span class="n">degree_power</span><span class="p">(</span><span class="n">adj</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.5</span><span class="p">)</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">normalized_D</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">adj</span><span class="p">).</span><span class="n">dot</span><span class="p">(</span><span class="n">normalized_D</span><span class="p">)</span>
<span class="k">return</span> <span class="n">output</span>
</code></pre></div></div>
<p>The code for these functions was copy-pasted from <a href="https://danielegrattarola.github.io/spektral/">Spektral</a> and slightly adapted so that we don’t need to import the entire library just for two functions. Note that there’s no need to JIT compile these two functions because they will run only once, and in fact, it is not guaranteed that compiling them will be less expensive than simply executing them with Python. Especially because both functions are heavily Numpy-based already, so they should run at C-like speed.</p>
<p>Moving forward to implementing the actual system. Let’s first define the fixed hyper-parameters of the model:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">omega</span> <span class="o">=</span> <span class="mi">20</span> <span class="c1"># Frequency of oscillations in rad/s
</span><span class="n">alpha</span> <span class="o">=</span> <span class="mf">0.2</span> <span class="c1"># Intensity of the noise
</span><span class="n">lamb</span> <span class="o">=</span> <span class="mf">0.5</span> <span class="c1"># Controls the possible attractors of each node
</span><span class="n">beta</span> <span class="o">=</span> <span class="mf">0.1</span> <span class="c1"># Coupling strength b/w nodes
</span><span class="n">N</span> <span class="o">=</span> <span class="mi">4</span> <span class="c1"># Number of nodes in the system
</span><span class="n">seconds_to_generate</span> <span class="o">=</span> <span class="mi">1</span> <span class="c1"># Number of seconds to evolve the system for
</span><span class="n">dt</span> <span class="o">=</span> <span class="mf">0.0001</span> <span class="c1"># Time interval between consecutive states
</span>
<span class="c1"># Random connectivity matrix
</span><span class="n">A</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="p">(</span><span class="n">N</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
<span class="n">np</span><span class="p">.</span><span class="n">fill_diagonal</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">A_norm</span> <span class="o">=</span> <span class="n">normalized_adjacency</span><span class="p">(</span><span class="n">A</span><span class="p">).</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">complex128</span><span class="p">)</span>
</code></pre></div></div>
<p>The core of the dynamical system is the update function \( f(z) \), that in code looks like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">njit</span>
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="n">lamb</span><span class="o">=</span><span class="mf">0.</span><span class="p">,</span> <span class="n">omega</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="s">"""The deterministic update function of each node.
:param z: complex, the current state.
:param lamb: float, hyper-parameter to control the attractors of each node.
:param omega: float, frequency of oscillations in rad/s.
"""</span>
<span class="k">return</span> <span class="p">((</span><span class="n">lamb</span> <span class="o">-</span> <span class="mi">1</span> <span class="o">+</span> <span class="nb">complex</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">omega</span><span class="p">))</span> <span class="o">*</span> <span class="n">z</span>
<span class="o">+</span> <span class="p">(</span><span class="mi">2</span> <span class="o">*</span> <span class="n">z</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="nb">abs</span><span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="o">-</span> <span class="p">(</span><span class="n">z</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="nb">abs</span><span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="o">**</span> <span class="mi">4</span><span class="p">))</span>
</code></pre></div></div>
<p>There’s not much to say here, except that using <code class="language-plaintext highlighter-rouge">complex</code> instead of <code class="language-plaintext highlighter-rouge">np.complex</code> seems to be slightly faster (157 ns vs. 178 ns), although the performance impact on the overall function is clearly negligible.</p>
<p>To compute the noise-driven system, we need to define the increment function of a complex Wiener process. We can start by implementing the increment function of a simple Wiener process, first:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">njit</span>
<span class="k">def</span> <span class="nf">delta_wiener</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">dt</span><span class="p">):</span>
<span class="s">"""Returns the random delta between two consecutive steps of a Wiener
process (Brownian motion).
:param size: tuple, desired shape of the output array.
:param dt: float, time increment in seconds.
:return: numpy array with shape 'size'.
"""</span>
<span class="k">return</span> <span class="n">np</span><span class="p">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">dt</span><span class="p">)</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randn</span><span class="p">(</span><span class="o">*</span><span class="n">size</span><span class="p">)</span>
</code></pre></div></div>
<p>At the time of writing this, Numba <a href="https://numba.pydata.org/numba-doc/dev/reference/numpysupported.html#distributions">does not support</a> the <code class="language-plaintext highlighter-rouge">size</code> argument in <code class="language-plaintext highlighter-rouge">np.random.normal</code> but it does support <code class="language-plaintext highlighter-rouge">np.random.randn</code>. Instead of setting the <code class="language-plaintext highlighter-rouge">scale</code> parameter explicitly, we simply multiply the sampled values by the scale.<br />
Since we are using the scale, and not the variance, we have to take the square root of the time increment <code class="language-plaintext highlighter-rouge">dt</code>.</p>
<p>Finally, we can compute the increment of a complex Wiener process as \( U(t) + jV(t) \), where both \( U \) and \( V \) are simple Wiener processes:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">njit</span>
<span class="k">def</span> <span class="nf">complex_delta_wiener</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">dt</span><span class="p">):</span>
<span class="s">"""Returns the random delta between two consecutive steps of a complex
Wiener process (Brownian motion). The process is calculated as u(t) + jv(t)
where u and v are simple Wiener processes.
:param size: tuple, the desired shape of the output array.
:param dt: float, time increment in seconds.
:return: numpy array of np.complex128 with shape 'size'.
"""</span>
<span class="n">u</span> <span class="o">=</span> <span class="n">delta_wiener</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">dt</span><span class="p">)</span>
<span class="n">v</span> <span class="o">=</span> <span class="n">delta_wiener</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">dt</span><span class="p">)</span>
<span class="k">return</span> <span class="n">u</span> <span class="o">+</span> <span class="n">v</span> <span class="o">*</span> <span class="mf">1j</span>
</code></pre></div></div>
<p>Now that we have all the necessary components to define the noise-driven system, let’s implement the main step function:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">njit</span>
<span class="k">def</span> <span class="nf">step</span><span class="p">(</span><span class="n">z</span><span class="p">):</span>
<span class="s">"""
Compute one time step of the system, s.t. z[t+1] = z[t] + step(z[t]).
:param z: numpy array of np.complex128, the current state.
:return: numpy array of np.complex128.
"""</span>
<span class="c1"># Matrix with pairwise differences of nodes
</span> <span class="n">delta_z</span> <span class="o">=</span> <span class="n">z</span><span class="p">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">-</span> <span class="n">z</span><span class="p">.</span><span class="n">reshape</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Compute diffusive coupling
</span> <span class="n">diffusive_coupling</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">diag</span><span class="p">(</span><span class="n">A_norm</span><span class="p">.</span><span class="n">T</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">delta_z</span><span class="p">))</span>
<span class="c1"># Compute change in state
</span> <span class="n">update_from_self</span> <span class="o">=</span> <span class="n">f</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="n">lamb</span><span class="o">=</span><span class="n">lamb</span><span class="p">,</span> <span class="n">omega</span><span class="o">=</span><span class="n">omega</span><span class="p">)</span>
<span class="n">update_from_others</span> <span class="o">=</span> <span class="n">beta</span> <span class="o">*</span> <span class="n">diffusive_coupling</span>
<span class="n">noise</span> <span class="o">=</span> <span class="n">alpha</span> <span class="o">*</span> <span class="n">complex_delta_wiener</span><span class="p">(</span><span class="n">z</span><span class="p">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">dt</span><span class="p">)</span>
<span class="n">dz</span> <span class="o">=</span> <span class="p">(</span><span class="n">update_from_self</span> <span class="o">+</span> <span class="n">update_from_others</span><span class="p">)</span> <span class="o">*</span> <span class="n">dt</span> <span class="o">+</span> <span class="n">noise</span>
<span class="k">return</span> <span class="n">dz</span>
</code></pre></div></div>
<p>Originally, I had implemented the following line</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">delta_z</span> <span class="o">=</span> <span class="n">z</span><span class="p">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">-</span> <span class="n">z</span><span class="p">.</span><span class="n">reshape</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
</code></pre></div></div>
<p>as</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">delta_z</span> <span class="o">=</span> <span class="n">z</span><span class="p">[...,</span> <span class="bp">None</span><span class="p">]</span> <span class="o">-</span> <span class="n">z</span><span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="p">...]</span>
</code></pre></div></div>
<p>but Numba does not support adding new axes with <code class="language-plaintext highlighter-rouge">None</code> or <code class="language-plaintext highlighter-rouge">np.newaxis</code>.</p>
<p>Also, when computing <code class="language-plaintext highlighter-rouge">diffusive_coupling</code>, a more efficient way of doing</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">np</span><span class="p">.</span><span class="n">diag</span><span class="p">(</span><span class="n">A</span><span class="p">.</span><span class="n">T</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">B</span><span class="p">))</span>
</code></pre></div></div>
<p>would have been</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">np</span><span class="p">.</span><span class="n">einsum</span><span class="p">(</span><span class="s">'ij,ij->j'</span><span class="p">,</span> <span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">)</span>
</code></pre></div></div>
<p>for reasons which I still fail to understand (3.48 µs vs. 2.57 µs, when <code class="language-plaintext highlighter-rouge">A</code> and <code class="language-plaintext highlighter-rouge">B</code> are 3 by 3 float matrices). However, Numba does not support <code class="language-plaintext highlighter-rouge">np.einsum</code>.</p>
<p>Finally, we can implement the main loop function that starts from a given initial state <code class="language-plaintext highlighter-rouge">z0</code> and computes <code class="language-plaintext highlighter-rouge">steps</code> number of updates at time intervals of <code class="language-plaintext highlighter-rouge">dt</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">njit</span>
<span class="k">def</span> <span class="nf">evolve_system</span><span class="p">(</span><span class="n">z0</span><span class="p">,</span> <span class="n">steps</span><span class="p">):</span>
<span class="s">"""
Evolve the system starting from the given initial state (z0) for a given
number of time steps (steps).
:param z0: numpy array of np.complex128, the initial state.
:param steps: int, number of steps to evolve the system for.
:return: list, the sequence of states.
"""</span>
<span class="n">steps_in_percent</span> <span class="o">=</span> <span class="n">steps</span> <span class="o">/</span> <span class="mi">100</span>
<span class="n">z</span> <span class="o">=</span> <span class="p">[</span><span class="n">z0</span><span class="p">]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">steps</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">i</span> <span class="o">%</span> <span class="n">steps_in_percent</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">i</span> <span class="o">/</span> <span class="n">steps_in_percent</span><span class="p">,</span> <span class="s">'%'</span><span class="p">)</span>
<span class="n">dz</span> <span class="o">=</span> <span class="n">step</span><span class="p">(</span><span class="n">z</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="n">z</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">z</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">dz</span><span class="p">)</span>
<span class="k">return</span> <span class="n">z</span>
</code></pre></div></div>
<p>I had originally wrapped the loop in a <code class="language-plaintext highlighter-rouge">tqdm</code> progress bar, but an old-fashioned <code class="language-plaintext highlighter-rouge">if</code> and <code class="language-plaintext highlighter-rouge">print</code> can reduce the overhead by 50% (2.29s vs. 1.23s, tested on a simple <code class="language-plaintext highlighter-rouge">for</code> loop with 1e7 iterations). Pre-computing <code class="language-plaintext highlighter-rouge">steps_in_percent</code> also reduces the overhead by 30% compared to computing it every time.<br />
(You’ll notice that at some point it just became a matter of optimizing every possible aspect of this :D)</p>
<p>The only thing left to do is to evolve the system starting from a given intial state:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">z0</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">N</span><span class="p">).</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">complex128</span><span class="p">)</span> <span class="c1"># Starting conditions
</span><span class="n">steps</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">seconds_to_generate</span> <span class="o">/</span> <span class="n">dt</span><span class="p">)</span> <span class="c1"># Number of steps to generate
</span>
<span class="n">timesteps</span> <span class="o">=</span> <span class="n">evolve_system</span><span class="p">(</span><span class="n">z0</span><span class="p">,</span> <span class="n">steps</span><span class="p">)</span>
<span class="n">timesteps</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">(</span><span class="n">timesteps</span><span class="p">)</span>
</code></pre></div></div>
<p>You can now run any analysis on <code class="language-plaintext highlighter-rouge">timesteps</code>, which will be a Numpy array of <code class="language-plaintext highlighter-rouge">np.complex128</code>. Note also how we had to cast the initial conditions <code class="language-plaintext highlighter-rouge">z0</code> to this <code class="language-plaintext highlighter-rouge">dtype</code>, in order to have strict typing in the JIT-compiled code.</p>
<p><a href="https://gist.github.com/danielegrattarola/c663346b529e758f0224c8313818ad77">I published the full code as a Gist, including the code I used to make the plots.</a></p>
<h2 id="general-notes-on-performance">General notes on performance</h2>
<p>My original implementation was based on a <code class="language-plaintext highlighter-rouge">Simulator</code> class that implemented all the same methods in a compact abstraction:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Simulator</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">N</span><span class="p">,</span> <span class="n">A</span><span class="p">,</span> <span class="n">dt</span><span class="o">=</span><span class="mf">1e-4</span><span class="p">,</span> <span class="n">omega</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.05</span><span class="p">,</span> <span class="n">lamb</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span><span class="mf">0.1</span><span class="p">):</span>
<span class="p">...</span>
<span class="o">@</span><span class="nb">staticmethod</span>
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="n">lamb</span><span class="o">=</span><span class="mf">0.</span><span class="p">,</span> <span class="n">omega</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="p">...</span>
<span class="o">@</span><span class="nb">staticmethod</span>
<span class="k">def</span> <span class="nf">delta_weiner</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">dt</span><span class="p">):</span>
<span class="p">...</span>
<span class="o">@</span><span class="nb">staticmethod</span>
<span class="k">def</span> <span class="nf">complex_delta_weiner</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">dt</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">step</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">z</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">evolve_system</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">z0</span><span class="p">,</span> <span class="n">steps</span><span class="p">):</span>
<span class="p">...</span>
</code></pre></div></div>
<p>There were some issues with this implementation, the biggest one being that it is much more messy to JIT compile an entire class with Numba (the substance of the code did not change much, and I’ve explicitly highlighted all implementation changes above).</p>
<p>Having moved to a more functional style feels cleaner and it honestly looks more elegant (opinions, I know). Crucially, it also allowed me to optimize each function to work flawlessly with Numba.</p>
<p>After optimizing all that was optimizable, I tested the old code against the new one and the speedup was about 31x, going from ~8k iterations/s to ~250k iterations/s.</p>
<p>Most of the improvement came from Numba and removing the overhead of Python’s interpreter, but it must be said that the true core of the system is dealt with by Numpy. In fact, as we increase the number of nodes the bottleneck becomes the matrix multiplication in Numpy, eventually leading to virtually no performance difference between using Numba or not (verified for <code class="language-plaintext highlighter-rouge">N=1000</code> - the 31x speedup was for <code class="language-plaintext highlighter-rouge">N=2</code>).</p>
<p><br />
I hope that you enjoyed this post and hopefully learned something new, be it about models of the epileptic brain or Python optimization.</p>
<p>Cheers!</p>
Thu, 03 Oct 2019 00:00:00 +0000
/posts/2019-10-03/epilepsy-model.html
tutorialcodeepilepsypostsMinCUT Pooling in Graph Neural Networks<p><img src="https://danielegrattarola.github.io/images/2019-07-25/horses.png" alt="Embeddings" class="full-width" /></p>
<p>In <a href="https://arxiv.org/abs/1907.00481">our latest paper</a>, we presented a new pooling method for GNNs, called <strong>MinCutPool</strong>, which has a lot of desirable properties as far as pooling goes:</p>
<ol>
<li>It’s based on well-understood theoretical techniques for node clustering;</li>
<li>It’s fully differentiable and learnable with gradient descent;</li>
<li>It depends directly on the task-specific loss on which the GNN is being trained, but …</li>
<li>It can be trained on its own without a task-specific loss if needed;</li>
<li>It’s fast;</li>
</ol>
<p>The method is based on the minCUT optimization problem, which consists of finding a cut on a weighted graph in such a way that the overall weight of the cut is minimized. We considered a continuous relaxation of the minCUT problem and implemented it as a neural network layer to provide a sound pooling method for GNNs.</p>
<p>In this post, I’ll describe the working principles of minCUT pooling and show some applications of the layer.</p>
<!--more-->
<h2 id="background">Background</h2>
<p><img src="https://danielegrattarola.github.io/images/2019-07-25/mincut_problem.png" alt="Embeddings" /></p>
<p>The <a href="https://en.wikipedia.org/wiki/Minimum_k-cut">K-way normalized minCUT</a> is an optimization problem to find K clusters on a graph by minimizing the overall intra-cluster edge weight. This is equivalent to solving:</p>
\[\text{maximize} \;\; \frac{1}{K} \sum_{k=1}^K \frac{\sum_{i,j \in \mathcal{V}_k} \mathcal{E}_{i,j} }{\sum_{i \in \mathcal{V}_k, j \in \mathcal{V} \backslash \mathcal{V}_k} \mathcal{E}_{i,j}},\]
<p>where \(\mathcal{V}\) is the set of nodes, \(\mathcal{V_k}\) is the \(k\)-th cluster of nodes, and \(\mathcal{E_{i, j}}\) indicates a weighted edge between two nodes.</p>
<p>If we define a <strong>cluster assignment matrix</strong> \(C \in \{0,1\}^{N \times K}\), which maps each of the \(N\) nodes to one of the \(K\) clusters, the problem can also be re-written as:</p>
\[\text{maximize} \;\; \frac{1}{K} \sum_{k=1}^K \frac{C_k^T A C_k}{C_k^T D C_k}\]
<p>where \(A\) is the adjacency matrix of the graph, and \(D\) is the diagonal degree matrix.</p>
<p>While finding the optimal minCUT is an NP-hard problem, there exist relaxations that can find near-optimal solutions in polynomial time. These relaxations, however, are still very expensive and are not able to generalize to unseen samples.</p>
<h2 id="mincut-pooling">MinCUT pooling</h2>
<p><img src="https://danielegrattarola.github.io/images/2019-07-25/GNN_pooling.png" alt="Embeddings" /></p>
<p>The idea behind minCUT pooling is to take a continuous relaxation of the minCUT problem and implement it as a GNN layer with a custom loss function. By minimizing the custom loss, the GNN learns to find minCUT clusters on any given graph and aggregates the clusters to reduce the graph’s size. <br />
At the same time, because the layer can be used as a part of a larger architecture, any other loss that is being minimized during training will influence the clusters found by MinCutPool, making them optimal for the particular task at hand.</p>
<p>At the core of minCUT pooling there is a MLP, which maps the node features \(\mathbf{X}\) to a <strong>continuous</strong> cluster assignment matrix \(\mathbf{S}\) (of size \(N \times K\)):</p>
\[\mathbf{S} = \textrm{softmax}(\text{ReLU}(\mathbf{X}\mathbf{W}_1)\mathbf{W}_2)\]
<p>We can then use the MLP to generate \(\mathbf{S}\) on the fly, and reduce the graphs with simple multiplications as:</p>
\[\mathbf{A}^{pool} = \mathbf{S}^T \mathbf{A} \mathbf{S}; \;\;\; \mathbf{X}^{pool} = \mathbf{S}^T \mathbf{X}.\]
<p>At this point, we can already make a couple of considerations:</p>
<ol>
<li>Nodes with similar features will likely belong to the same cluster because they will be “classified” similarly by the MLP. This is especially true when using message-passing layers before pooling, since they will cause the node features of connected nodes to become similar;</li>
<li>Because of the MLP, \(\mathbf{S}\) is pretty fast to compute and the layer can generalize to new graphs once it has been trained.</li>
</ol>
<p>This is already pretty good, and it covers some of the main desiderata of a GNN layer, but we also want to explicitly account for the connectivity of the graph in order to pool it.</p>
<p>This is where the minCUT optimization comes in.</p>
<p>By slightly adapting the minCUT formulation above, we can design an auxiliary loss to train the MLP, so that it will learn to solve the minCUT problem in an unsupervised way. <br />
In practice, our unsupervised regularization loss encourages the MLP to cluster together nodes that are strongly connected with each other and weakly connected with the nodes in the other clusters.</p>
<p>The full unsupervised loss that we minimize in order to achieve this is:</p>
\[\mathcal{L}_u = \mathcal{L}_c + \mathcal{L}_o =
\underbrace{- \frac{Tr ( \mathbf{S}^T \mathbf{A} \mathbf{S} )}{Tr ( \mathbf{S}^T\mathbf{D} \mathbf{S})}}_{\mathcal{L}_c} +
\underbrace{\bigg{\lVert} \frac{\mathbf{S}^T\mathbf{S}}{\|\mathbf{S}^T\mathbf{S}\|_F} - \frac{\mathbf{I}_K}{\sqrt{K}}\bigg{\rVert}_F}_{\mathcal{L}_o},\]
<p>where \(\mathbf{A}\) is the <a href="https://danielegrattarola.github.io/spektral/utils/convolution/#normalized_adjacency">normalized</a> adjacency matrix of the graph.</p>
<p>Let’s break this loss down and see how it works.</p>
<h3 id="cut-loss">Cut loss</h3>
<p>The first term, \(\mathcal{L}_c\), encourages the MLP to find cluster assignments that solve the minCUT problem (to see why, compare it with the minCUT maximization that I described above). We refer to this loss as the <strong>cut loss</strong>.</p>
<p>In particular, minimizing the numerator leads to clustering together nodes that are strongly connected on the graph, while the denominator prevents any of the clusters to be too small.</p>
<p>The cut loss is bounded between -1 and 0, which are <strong>ideally</strong> reached in the following situations:</p>
<ul>
<li>\(\mathcal{L}_c = 0\) when all pairs of connected nodes are assigned to different clusters;</li>
<li>\(\mathcal{L}_c = -1\) when there are \(K\) disconnected components in the graph, and \(\mathbf{S}\) exactly maps the \(K\) components to the \(K\) clusters;</li>
</ul>
<p>The figure below shows what these situations might look like. Note that both cases can only happen if \(\mathbf{S}\) is binary.</p>
<p><img src="/images/2019-07-25/loss_bounds.png" alt="L_c bounds" /></p>
<p>However, because of the continuous relaxation, \(\mathcal{L}_c\) is non-convex and there are spurious minima that can be found by SGD.<br />
For example, for \(K = 4\), the uniform assignment matrix</p>
\[\mathbf{S}_i = (0.25, 0.25, 0.25, 0.25) \;\; \forall i,\]
<p>would cause the numerator and the denominator of \(\mathcal{L}_c\) to be equal, and the loss to be \(-1\).<br />
A similar situation occurs when all nodes in the graph are assigned to the same cluster.</p>
<p>This can be easily verified with Numpy:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="c1"># Adjacency matrix
</span> <span class="p">...:</span> <span class="n">A</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span>
<span class="p">...:</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span>
<span class="p">...:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]])</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="c1"># Degree matrix
</span> <span class="p">...:</span> <span class="n">D</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">diag</span><span class="p">(</span><span class="n">A</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="c1"># Perfect cluster assignment
</span> <span class="p">...:</span> <span class="n">S</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">]])</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">np</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">S</span><span class="p">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">A</span> <span class="o">@</span> <span class="n">S</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">S</span><span class="p">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">D</span> <span class="o">@</span> <span class="n">S</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="mf">1.0</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="c1"># All nodes uniformly distributed
</span> <span class="p">...:</span> <span class="n">S</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span> <span class="o">/</span> <span class="mi">2</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">np</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">S</span><span class="p">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">A</span> <span class="o">@</span> <span class="n">S</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">S</span><span class="p">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">D</span> <span class="o">@</span> <span class="n">S</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="mf">1.0</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="c1"># All nodes in the same cluster
</span> <span class="p">...:</span> <span class="n">S</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">]])</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">np</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">S</span><span class="p">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">A</span> <span class="o">@</span> <span class="n">S</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">S</span><span class="p">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">D</span> <span class="o">@</span> <span class="n">S</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="mf">1.0</span>
</code></pre></div></div>
<h3 id="orthogonality-loss">Orthogonality loss</h3>
<p>The second term, \(\mathcal{L}_o\), helps to avoid such degenerate minima of \(\mathcal{L}_c\) by encouraging the MLP to find clusters that are orthogonal between each other. We call this the <strong>orthogonality loss</strong>.</p>
<p>In other words, \(\mathcal{L}_o\) encourages the MLP to “make a decision” about which nodes belong to which clusters, avoiding those degenerate solutions where \(\mathbf{S}\) assigns one \(K\)-th of a node to each cluster.</p>
<p>Moreover, we can see that the perfect minimizer of \(\mathcal{L}_o\) is only reached if we have \(N \le K\) nodes, because in general, given a \(K\) dimensional vector space, we cannot find more than \(K\) mutually orthogonal vectors.
The only way to minimize \(\mathcal{L}_o\) given \(N\) assignment vectors is, therefore, to distribute the nodes between the \(K\) clusters. This causes the MLP to avoid the other type of spurious minima of \(\mathcal{L}_c\), where all nodes are in a single cluster.</p>
<h2 id="interaction-of-the-two-losses">Interaction of the two losses</h2>
<p><img src="/images/2019-07-25/cora_mc_loss+nmi.png" alt="Loss terms" /></p>
<p>We can see how the two loss terms interact with each other to find a good solution to the cluster assignment problem.
The figure above shows the evolution of the unsupervised loss as the network is trained to cluster the nodes of Cora (plot on the left). We can see that as the network is trained, the normalized mutual information (NMI) between the cluster assignments and the true labels improves, meaning that the layer is learning to find meaningful clusters (plot on the right).</p>
<p>Note how \(\mathcal{L}_c\) starts from a trivial assignment (-1) due to the random initialization and then moves away from the spurious minima as the orthogonality loss forces the MLP towards more sensible solutions.</p>
<h3 id="pooled-graph">Pooled graph</h3>
<p>As a further consideration, we can take a closer look at the pooled adjacency matrix \(\mathbf{A}^{pool}\). <br />
First of all, we can see that it is a \(K \times K\) matrix that contains the number of links connecting each cluster. For example, the entry \(\mathbf{A}^{pool}_{1,\;2}\) contains the number of links between the nodes in cluster 1 and cluster 2.
We can also see that the trace of \(\mathbf{A}^{pool}\) is being maximized in \(\mathcal{L}_c\). Therefore, we can expect the diagonal elements \(\mathbf{A}^{pool}_{i,\;i}\) to be much larger than the other entries of \(\mathbf{A}^{pool}\).</p>
<p>For this reason, \(\mathbf{A}^{pool}\) will represent a graph with very strong self-loops, and the message-passing layers after pooling will have a hard time propagating information on the graph (because the self-loops will keep sending the information of a node back onto itself, and not its neighbors).</p>
<p>To address this problem, a solution is to remove the diagonal of \(\mathbf{A}^{pool}\) and renormalize the matrix by its degree, before giving it as output of the pooling layer:</p>
\[\hat{\mathbf{A}} = \mathbf{A}^{pool} - \mathbf{I}_K \cdot diag(\mathbf{A}^{pool}); \;\; \tilde{\mathbf{A}}^{pool} = \hat{\mathbf{D}}^{-\frac{1}{2}} \hat{\mathbf{A}} \hat{\mathbf{D}}^{-\frac{1}{2}}\]
<p>In the paper, we combined minCUT with message-passing layers that have a built-in skip connection, in order to bring each node’s information forward (e.g., Spektral’s <a href="https://danielegrattarola.github.io/spektral/layers/convolution/#graphconvskip">GraphConvSkip</a>).
However, if your GNN is based on the <a href="https://danielegrattarola.github.io/spektral/layers/convolution/#graphconv">graph convolutional networks (GCN)</a> of <a href="https://arxiv.org/abs/1609.02907">Kipf & Welling</a>, you may want to manually add the self-loops back after pooling.</p>
<h3 id="notes-on-gradient-flow">Notes on gradient flow</h3>
<p><img src="/images/2019-07-25/mincut_layer.png" alt="mincut scheme" /></p>
<p>The unsupervised loss \(\mathcal{L}_u\) can be optimized on its own, adapting the weights of the MLP to compute an \(\mathbf{S}\) that solves the minCUT problem under the orthogonality constraint.</p>
<p>However, given the multiplicative interaction between \(\mathbf{S}\) and \(\mathbf{X}\), the gradient of the task-specific loss (i.e., whatever the GNN is being trained to do) can flow through the MLP. We can see in the picture above how there is a path going from the input \(\mathbf{X}^{(t+1)}\) to the output \(\mathbf{X}_{\textrm{pool}}^{(t+1)}\), directly passing through the MLP.</p>
<p>This means that the overall solution found by the GNN will keep into account both the graph structure (to solve minCUT) and the final task.</p>
<h2 id="code">Code</h2>
<p>Implementing minCUT in TensorFlow is fairly straightforward. Let’s start from some setup:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="n">tf</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.layers</span> <span class="kn">import</span> <span class="n">Dense</span>
<span class="n">A</span> <span class="o">=</span> <span class="p">...</span> <span class="c1"># Adjacency matrix (N x N)
</span> <span class="n">X</span> <span class="o">=</span> <span class="p">...</span> <span class="c1"># Node features (N x F)
</span> <span class="n">n_clusters</span> <span class="o">=</span> <span class="p">...</span> <span class="c1"># Number of clusters to find with minCUT
</span></code></pre></div></div>
<p>First, the layer computes the cluster assignment matrix <code class="language-plaintext highlighter-rouge">S</code> by applying a softmax MLP to the node features:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">H</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'relu'</span><span class="p">)(</span><span class="n">X</span><span class="p">)</span>
<span class="n">S</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">'softmax'</span><span class="p">)(</span><span class="n">H</span><span class="p">)</span> <span class="c1"># Cluster assignment matrix
</span></code></pre></div></div>
<p>The cut loss is then implemented as:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Cut loss
</span><span class="n">A_pool</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span>
<span class="n">tf</span><span class="p">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">S</span><span class="p">)),</span> <span class="n">S</span>
<span class="p">)</span>
<span class="n">num</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">A_pool</span><span class="p">)</span>
<span class="n">D</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">reduce_sum</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">axis</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">D_pooled</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span>
<span class="n">tf</span><span class="p">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">D</span><span class="p">,</span> <span class="n">S</span><span class="p">)),</span> <span class="n">S</span>
<span class="p">)</span>
<span class="n">den</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">trace</span><span class="p">(</span><span class="n">D_pooled</span><span class="p">)</span>
<span class="n">mincut_loss</span> <span class="o">=</span> <span class="o">-</span><span class="p">(</span><span class="n">num</span> <span class="o">/</span> <span class="n">den</span><span class="p">)</span>
</code></pre></div></div>
<p>And the orthogonality loss is implemented as:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Orthogonality loss
</span><span class="n">St_S</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">S</span><span class="p">),</span> <span class="n">S</span><span class="p">)</span>
<span class="n">I_S</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">eye</span><span class="p">(</span><span class="n">n_clusters</span><span class="p">)</span>
<span class="n">ortho_loss</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">norm</span><span class="p">(</span>
<span class="n">St_S</span> <span class="o">/</span> <span class="n">tf</span><span class="p">.</span><span class="n">norm</span><span class="p">(</span><span class="n">St_S</span><span class="p">)</span> <span class="o">-</span> <span class="n">I_S</span> <span class="o">/</span> <span class="n">tf</span><span class="p">.</span><span class="n">norm</span><span class="p">(</span><span class="n">I_S</span><span class="p">)</span>
<span class="p">)</span>
</code></pre></div></div>
<p>Finally, the full unsupervised loss of the layer is obtained as the sum of the two auxiliary losses:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">total_loss</span> <span class="o">=</span> <span class="n">mincut_loss</span> <span class="o">+</span> <span class="n">ortho_loss</span>
</code></pre></div></div>
<p>The actual pooling step is simply implemented as a simple multiplication of <code class="language-plaintext highlighter-rouge">S</code> with <code class="language-plaintext highlighter-rouge">A</code> and <code class="language-plaintext highlighter-rouge">X</code>, then we zero-out the diagonal of <code class="language-plaintext highlighter-rouge">A_pool</code> and re-normalize the matrix. Since we already computed <code class="language-plaintext highlighter-rouge">A_pool</code> for the numerator of \(\mathcal{L}_c\), we only need to do:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Pooling node features
</span><span class="n">X_pool</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">S</span><span class="p">),</span> <span class="n">X</span><span class="p">)</span>
<span class="c1"># Zeroing out the diagonal
</span><span class="n">A_pool</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">linalg</span><span class="p">.</span><span class="n">set_diag</span><span class="p">(</span><span class="n">A_pool</span><span class="p">,</span> <span class="n">tf</span><span class="p">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">tf</span><span class="p">.</span><span class="n">shape</span><span class="p">(</span><span class="n">A_pool</span><span class="p">)[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span> <span class="c1"># Remove diagonal
</span>
<span class="c1"># Normalizing A_pool
</span><span class="n">D_pool</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">reduce_sum</span><span class="p">(</span><span class="n">A_pool</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">D_pool</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">D_pool</span><span class="p">)[:,</span> <span class="bp">None</span><span class="p">]</span> <span class="o">+</span> <span class="mf">1e-12</span> <span class="c1"># Add epsilon to avoid division by 0
</span><span class="n">A_pool</span> <span class="o">=</span> <span class="p">(</span><span class="n">A_pool</span> <span class="o">/</span> <span class="n">D_pool</span><span class="p">)</span> <span class="o">/</span> <span class="n">tf</span><span class="p">.</span><span class="n">transpose</span><span class="p">(</span><span class="n">D_pool</span><span class="p">)</span>
</code></pre></div></div>
<p>Wrap this up in a layer, and use the layer in a GNN. Done.</p>
<p>You can find minCUT pooling implementations both in <a href="https://danielegrattarola.github.io/spektral/layers/pooling/#mincutpool">Spektral</a> and <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#module-torch_geometric.nn.dense.mincut_pool">Pytorch Geometric</a>.</p>
<h2 id="experiments">Experiments</h2>
<h3 id="unsupervised-clustering">Unsupervised clustering</h3>
<p>Because the core of MinCutPool is an unsupervised loss that does not require labeled data in order to be minimized, we can optimize \(\mathcal{L}_u\) on its own to test the clustering ability of minCUT.</p>
<p>A good first test is to check whether the layer is able to cluster a grid (the size of the clusters should be the same) and to isolate communities in a network.
We see in the figure below that minCUT was able to do this perfectly.</p>
<p><img src="/images/2019-07-25/regular_clustering.png" alt="Clustering with minCUT pooling" /></p>
<p>To make things more interesting, we can also test minCUT on the task of graph-based image segmentation. We can build a <a href="https://scikit-image.org/docs/dev/auto_examples/segmentation/plot_rag.html">region adjacency graph</a> from a natural image, and cluster its nodes in order to see if regions with similar colors are clustered together. <br />
The results look nice, and remember that this was obtained by only optimizing \(\mathcal{L}_u\)!</p>
<p><img src="/images/2019-07-25/horses.png" alt="Horse segmentation with minCUT pooling" /></p>
<p>Finally, we also checked the clustering abilities of MinCutPool on the popular citations datasets: Cora, Citeseer, and Pubmed.
As mentioned before, we used the NMI score to see whether the layer was clustering together nodes of the same class. Note that the layer did not have access to the labels during training.</p>
<p>You can check <a href="https://arxiv.org/abs/1907.00481">the paper</a> to see how minCUT fared in comparison to other methods, but in short: it did well, sometimes by a full order of magnitude better than other methods.</p>
<h3 id="autoencoder">Autoencoder</h3>
<p>Another interesting unsupervised test that we did was to check how much information is preserved in the coarsened graph after pooling.
To do this, we built a simple graph autoencoder with the structure pictured below:</p>
<p><img src="/images/2019-07-25/ae.png" alt="unsupervised reconstruction with AE" /></p>
<p>The “Unpool” layer is simply obtained by transposing the same \(\mathbf{S}\) found by minCUT, in order to upscale the graph instead of downscaling it:</p>
\[\mathbf{A}^\text{unpool} = \mathbf{S} \mathbf{A}^\text{pool} \mathbf{S}^T; \;\; \mathbf{X}^\text{unpool} = \mathbf{S}\mathbf{X}^\text{pool}.\]
<p>We tested the graph AE on some very regular graphs that should have been easy to reconstruct after pooling. Surprisingly, this turned out to be a difficult problem for some pooling layers from the GNN literature. MinCUT, on the other hand, was able to defend itself quite nicely.</p>
<p><img src="/images/2019-07-25/reconstructions.png" alt="unsupervised reconstruction with AE" /></p>
<h3 id="supervised-inductive-tasks">Supervised inductive tasks</h3>
<p>Finally, we tested whether minCUT provides an improvement on the usual graph classification and graph regression tasks. <br />
We picked a fixed GNN architecture and tested several pooling strategies by swapping the pooling layers in the network.</p>
<p>The dataset that we used were:</p>
<ol>
<li><a href="https://ls11-www.cs.tu-dortmund.de/staff/morris/graphkerneldatasets">The Benchmark Data Sets for Graph Kernels</a>;</li>
<li><a href="https://github.com/FilippoMB/Benchmark_dataset_for_graph_classification">A synthetic dataset created by F. M. Bianchi to test GNNs</a>;</li>
<li><a href="http://quantum-machine.org/datasets/">The QM9 dataset for the prediction of chemical properties of molecules</a>.</li>
</ol>
<p>I’m not gonna report the comparisons with other methods, but I will highlight an interesting sanity check that we performed in order to see whether using GNNs and graph pooling even made sense at all.</p>
<p>Among the various methods that we tested, we also included:</p>
<ol>
<li>A simple MLP which did not exploit the relational information carried by the graphs;</li>
<li>The same GNN architecture without pooling layers.</li>
</ol>
<p>We were once again surprised to see that, while minCUT yielded a consistent improvement over such simple baselines, other pooling methods did not.</p>
<h2 id="conclusions">Conclusions</h2>
<p>Working on minCUT pooling was an interesting experience that deepened my understanding of GNNs, and allowed me to see what is really necessary for a GNN to work.</p>
<p>We have put the paper <a href="https://arxiv.org/abs/1907.00481">on arXiv</a>, and you can check the official implementations of the method in <a href="https://danielegrattarola.github.io/spektral/layers/pooling/#mincutpool">Spektral</a> and <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#module-torch_geometric.nn.dense.mincut_pool">Pytorch Geometric</a>.</p>
<p>If you want to use MinCutPool in your own work, you can cite us with:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@article{bianchi2019mincut,
title={Spectral Clustering with Graph Neural Networks for Graph Pooling},
author={Filippo Maria Bianchi and Daniele Grattarola and Cesare Alippi},
booktitle={Proceedings of the 37th International Conference on Machine learning (ICML)},
year={2020}
}
</code></pre></div></div>
<p>Cheers!</p>
Thu, 25 Jul 2019 00:00:00 +0000
/posts/2019-07-25/mincut-pooling.html
AIGNNpoolingpostsDetecting Hostility from Skeletal Graphs Using Non-Euclidean Embeddings<p>The first paper on which I worked during my PhD is about <a href="https://arxiv.org/abs/1805.06299">detecting changes in sequences of graphs using non-Euclidean geometry and adversarial autoencoders</a>. As a real-world application of the method presented in the paper, we showed that we could detect epileptic seizures in the brain, by monitoring a stream of functional connectivity brain networks.</p>
<p>In general, the methodology presented in the paper can work for any data that:</p>
<ol>
<li>can be represented as graphs;</li>
<li>has a temporal dimension;</li>
<li>has a change that you want to identify somewhere along the stream of data;</li>
<li>has i.i.d. samples.</li>
</ol>
<p>There are <a href="https://icon.colorado.edu/#!/networks">a lot</a> of temporal networks that can be found in the wild, but not many datasets respect all the requirements at the same time. What’s more, many public datasets have very little samples along the temporal axis. <!--more-->
Recently, however, I was looking for some nice graph classification dataset on which to test <a href="https://danielegrattarola.github.io/spektral">Spektral</a>, and I stumbled upon the <a href="http://rose1.ntu.edu.sg/datasets/actionrecognition.asp">NTU RGB+D</a> dataset released by the Nanyang Technological University of Singapore.<br />
The dataset consists of about 60 thousand video clips of people performing everyday actions, including mutual actions and some health-related ones. The reason why I found this dataset is that it contains skeletal annotations for each frame of each video clip, meaning lots and lots of graphs that <a href="https://arxiv.org/abs/1801.07455">can be used for graph classification</a>.</p>
<h2 id="ntu-rgbd-for-change-detection">NTU RGB+D for change detection</h2>
<p><img src="https://danielegrattarola.github.io/images/2019-04-13/graphs.svg" alt="graphs" title="Figure 1: examples of hugging and punching graphs." class="threeq-width" /></p>
<p>While reading through the website, however, I realized that this dataset could actually be a good playground for our change detection methodology as well, because it respects almost all requirements:</p>
<ol>
<li>it has graphs;</li>
<li>it has a temporal dimension;</li>
<li>it has classes, which can be easily converted to what we called the <em>regimes</em> of our graph streams;</li>
</ol>
<p>The fourth requirement of having i.i.d. samples is due to the nature of the change detection test that we adopted in the paper. The test is able to detect changes in stationarity of a stochastic process, which means that it can tell whether the samples coming from the process have been drawn from a different distribution than the one observed during training. <br />
In order to do so, the test needs to estimate whether a window of observations from the process is significantly different than what observed in the nominal regime. This requires having i.i.d. samples in each window.</p>
<p>By their very nature, however, the graphs in NTU RGB+D are definitely not i.i.d. (they would have been, had the subjects been recorded under a strobe light – dammit!).<br />
There are several ways of converting a heavily autocorrelated signal to a stationary one, with the simplest one being randomizing along the time axis.
The piece-wise stationarity requirement is a very strong one, and we are looking into relaxing it, but for testing the method on NTU RGB+D we had to stick with it.</p>
<h2 id="setting">Setting</h2>
<p>Defining the change detection problem is easy: have a nominal regime of neutral or positive actions like walking, reading, taking a selfie, or being at the computer, and try to detect when the regime changes to a negative action like falling down, getting in fights with people, or feeling sick (there are at least 5 action classes of people acting hurt or sick in NTU RGB+D).</p>
<p>Applications of this could include:</p>
<ul>
<li>monitoring children and elderly people when they are alone;</li>
<li>detecting violence in at-risk, crowded situations;</li>
<li>detecting when a driver is distracted;</li>
</ul>
<p>In all of these situations, you might have a pretty good idea of what you <em>want</em> to be happening at a given time, but have no way of knowing how things could go wrong.</p>
<p>We chose the “hugging” action for the nominal, all-is-well regime, and we took the “punching/slapping” class to symbolize any unexpected, undesirable behaviour that deviates from our concept of nominal.
Then, we trained our adversarial autoencoder to represent points on an ensemble of constant-curvature manifolds, and we ran the change detection test.
At this point, it would probably help if one was familiar with the details of <a href="https://arxiv.org/abs/1805.06299">the paper</a>. In short, what we do is:</p>
<ol>
<li>take an adversarial graph autoencoder (AAE);</li>
<li>train the AAE on the nominal samples that you have at training time;</li>
<li>impose a geometric regularization onto the latent space of the AAE, so that the embeddings will lie on a Riemannian constant-curvature manifold (CCM).<br />
This happens in one of two ways:
<ol>
<li>use a prior distribution with support on the CCM to train the AAE;</li>
<li>make the encoder maximise the membership of its embeddings to the CCM (this is the one we use for this experiment);</li>
</ol>
</li>
<li>use the trained AAE to represent incoming graphs on the CCM;</li>
<li>run the change detection test on the CCM;</li>
</ol>
<p><img src="https://danielegrattarola.github.io/images/2019-04-13/embeddings.svg" alt="embeddings" title="Figure 2: embeddings produced by the AAE on the three different CCMs. Blue for hugging, orange for punching." class="full-width" /></p>
<p>This procedure can be adapted to learn a representation on more than one CCM at a time, by having parallel latent spaces for the AAE. This worked pretty well in the paper, so we tried the same here.
We also chose one of the two types of change detection tests that we introduced in the paper, namely the one we called <em>Riemannian</em>, because it gave us the best results on the seizure detection problem.</p>
<h2 id="results">Results</h2>
<p>Running the whole method on the stream of graphs gave us very nice results. We were able to recognize the change from friendly to violent interactions in most experiments, although sometimes the autoencoder failed to capture the differences between the two regimes (and consequently, the CDT couldn’t pick up the change).</p>
<p><img src="https://danielegrattarola.github.io/images/2019-04-13/accumulator.svg" alt="accumulator" title="Figure 3: accumulators of R-CDT (see the paper) for the three CCMs. The change is marked with the red line, the decision threshold with the green line. " class="full-width" /></p>
<p>An interesting thing that we observed is that when using an ensemble of three different geometries, namely spherical, hyperbolic, and Euclidean, the change would only show up in the spherical CCM.
This was a consistent result that gave us yet another confirmation of two things:</p>
<ol>
<li>assuming Euclidean geometry for the latent space is not always a good idea;</li>
<li>our idea of learning a representation on multiple CCMs at the same time worked as expected. Originally, we suggested this trick to potential adopters of our CDT methodology, in order to not having to guess the best geometry for the representation. Now, we have the confirmation that it is indeed a good idea, because the AAE will choose the best geometry for the task on its own.</li>
</ol>
<p>Figure 2 above (hover over the images to see the captions) shows the embeddings produced by the encoder on the test stream of graphs. Figure 3 shows the three <em>accumulators</em> used in the change detection test to decide whether or not to raise an alarm indicating that a change occurred.
In both pictures, the decision for raising an alarm is informed almost exclusively by the spherical CCM.</p>
<h2 id="conclusions">Conclusions</h2>
<p>That’s all, folks!<br />
This was a pretty little experiment to run, and it gave us further insights into the world of non-Euclidean neural networks. We have actually <a href="https://arxiv.org/abs/1805.06299">updated the paper</a> with the findings of this new experiment, and you can also try and play with our algorithm using the <a href="https://github.com/danielegrattarola/cdt-ccm-aae">code on Github</a> (the code there is for the synthetic experiments of the paper, but you can adapt it to any dataset easily).</p>
<p>If you want to mention our CDT strategy in your work, you can cite:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@article{grattarola2018change,
title={Change Detection in Graph Streams by Learning Graph Embeddings on Constant-Curvature Manifolds},
author={Grattarola, Daniele and Zambon, Daniele and Livi, Lorenzo and Alippi, Cesare},
journal={IEE Transactions on Neural Networks and Learning Systems},
year={2019},
doi={10.1109/TNNLS.2019.2927301}
}
</code></pre></div></div>
<p>Cheers!</p>
Sat, 13 Apr 2019 00:00:00 +0000
/posts/2019-04-13/hostility-detection.html
AIexperimentnon-euclideanposts