# Quelle est la complexité temporelle pour former un réseau de neurones en utilisant la rétropropagation?

17

Supposons qu'un NN contient $$nnn$$ couches masquées, $$mmm$$ exemples d'apprentissage, $$xxx$$ entités et $$ninin_i$$ nœuds dans chaque couche. Quelle est la complexité temporelle pour former ce NN en utilisant la rétropropagation?

J'ai une idée de base sur la façon dont ils trouvent la complexité temporelle des algorithmes, mais ici, il y a 4 facteurs différents à considérer ici, à savoir les itérations, les couches, les nœuds dans chaque couche, les exemples de formation et peut-être plus de facteurs. J'ai trouvé une réponse ici mais elle n'était pas assez claire.

Y a-t-il d'autres facteurs, à part ceux que j'ai mentionnés ci-dessus, qui influencent la complexité temporelle de l'algorithme d'apprentissage d'un NN?

Voir également https://qr.ae/TWttzq .
nbro

Réponses:

11

Je n'ai pas vu de réponse d'une source fiable, mais j'essaierai d'y répondre moi-même, avec un exemple simple (avec mes connaissances actuelles).

En général, notez que la formation d'un MLP utilisant la rétropropagation est généralement implémentée avec des matrices.

### Complexité temporelle de la multiplication matricielle

La complexité temporelle de la multiplication matricielle pour $$Mij∗MjkMij∗MjkM_{ij} * M_{jk}$$ est simplement $$O(i∗j∗k)O(i∗j∗k)\mathcal{O}(i*j*k)$$ .

Notez que nous supposons ici l'algorithme de multiplication le plus simple: il existe d'autres algorithmes avec une complexité temporelle un peu meilleure.

### Algorithme de passage direct

L'algorithme de propagation à action directe est le suivant.

Tout d'abord, pour passer de la couche $$iii$$ à $$jjj$$ , vous



${S}_{j}={W}_{ji}\ast {Z}_{i}$

Ensuite, vous appliquez la fonction d'activation



${Z}_{j}=f\left({S}_{j}\right)$

Si nous avons $$NNN$$ couches (y compris les couches d'entrée et de sortie), cela s'exécutera $$N−1N−1N-1$$ fois.

### Exemple

À titre d'exemple, calculons la complexité temporelle de l'algorithme de passage direct pour un MLP à $$444$$ couches, où $$iii$$ désigne le nombre de nœuds de la couche d'entrée, $$jjj$$ le nombre de nœuds dans la deuxième couche, $$kkk$$ le nombre de nœuds dans le troisième couche et $$lll$$ le nombre de nœuds dans la couche de sortie.

Puisqu'il y a $$444$$ couches, vous avez besoin de $$333$$ matrices pour représenter les poids entre ces couches. Notons-les par $$WjiWjiW_{ji}$$ , $$WkjWkjW_{kj}$$ et $$WlkWlkW_{lk}$$ , où $$WjiWjiW_{ji}$$ est une matrice à $$jjj$$ lignes et $$iii$$ colonnes ( $$WjiWjiW_{ji}$$ contient donc les poids allant de la couche $$iii$$ à la couche $$jjj$$ ).

Supposons que vous avez $$ttt$$ exemples de formation. Pour la propagation de la couche $$iii$$ à $$jjj$$ , nous avons d'abord



${S}_{jt}={W}_{ji}\ast {Z}_{it}$

et cette opération (c'est-à-dire la multiplication matricielle) a une complexité temporelle $$O(j∗i∗t)O(j∗i∗t)\mathcal{O}(j*i*t)$$ . Ensuite, nous appliquons la fonction d'activation



${Z}_{jt}=f\left({S}_{jt}\right)$

et cela a une complexité temporelle $$O(j∗t)O(j∗t)\mathcal{O}(j*t)$$ , car il s'agit d'une opération élément par élément.

Donc, au total, nous avons



$\mathcal{O}\left(j\ast i\ast t+j\ast t\right)=\mathcal{O}\left(j\ast t\ast \left(t+1\right)\right)=\mathcal{O}\left(j\ast i\ast t\right)$

En utilisant la même logique, pour aller $$j→kj→kj \to k$$ , nous avons $$O(k∗j∗t)O(k∗j∗t)\mathcal{O}(k*j*t)$$ , et, pour $$k→lk→lk \to l$$ , nous avons $$O(l∗k∗t)O(l∗k∗t)\mathcal{O}(l*k*t)$$ .

In total, the time complexity for feedforward propagation will be



$\mathcal{O}\left(j\ast i\ast t+k\ast j\ast t+l\ast k\ast t\right)=\mathcal{O}\left(t\ast \left(ij+jk+kl\right)\right)$

I'm not sure if this can be simplified further or not. Maybe it's just $$O(t∗i∗j∗k∗l)O(t∗i∗j∗k∗l)\mathcal{O}(t*i*j*k*l)$$, but I'm not sure.

### Back-propagation algorithm

The back-propagation algorithm proceeds as follows. Starting from the output layer $$l→kl→kl \to k$$, we compute the error signal, $$EltEltE_{lt}$$, a matrix containing the error signals for nodes at layer $$lll$$



${E}_{lt}={f}^{\prime }\left({S}_{lt}\right)\odot \left({Z}_{lt}-{O}_{lt}\right)$

$$⊙⊙\odot$$$$EltEltE_{lt}$$$$lll$$ rows and $$ttt$$ columns: it simply means each column is the error signal for training example $$ttt$$.

$$Dlk∈Rl×kDlk∈Rl×kD_{lk} \in \mathbb{R}^{l \times k}$$ (between layer $$lll$$ and layer $$kkk$$)



${D}_{lk}={E}_{lt}\ast {Z}_{tk}$

$$ZtkZtkZ_{tk}$$ is the transpose of $$ZktZktZ_{kt}$$.



${W}_{lk}={W}_{lk}-{D}_{lk}$

$$l→kl→kl \to k$$$$O(lt+lt+ltk+lk)=O(l∗t∗k)O(lt+lt+ltk+lk)=O(l∗t∗k)\mathcal{O}(lt + lt + ltk + lk) = \mathcal{O}(l*t*k)$$.

Now, going back from $$k→jk→jk \to j$$. We first have



${E}_{kt}={f}^{\prime }\left({S}_{kt}\right)\odot \left({W}_{kl}\ast {E}_{lt}\right)$

Then



${D}_{kj}={E}_{kt}\ast {Z}_{tj}$

And then



${W}_{kj}={W}_{kj}-{D}_{kj}$

where $$WklWklW_{kl}$$ is the transpose of $$WlkWlkW_{lk}$$. For $$k→jk→jk \to j$$, we have the time complexity $$O(kt+klt+ktj+kj)=O(k∗t(l+j))O(kt+klt+ktj+kj)=O(k∗t(l+j))\mathcal{O}(kt + klt + ktj + kj) = \mathcal{O}(k*t(l+j))$$.

And finally, for $$j→ij→ij \to i$$, we have $$O(j∗t(k+i))O(j∗t(k+i))\mathcal{O}(j*t(k+i))$$. In total, we have



$\mathcal{O}\left(ltk+tk\left(l+j\right)+tj\left(k+i\right)\right)=\mathcal{O}\left(t\ast \left(lk+kj+ji\right)\right)$

which is same as feedforward pass algorithm. Since they are same, the total time complexity for one epoch will be 

$O\left(t\ast \left(ij+jk+kl\right)\right).$

This time complexity is then multiplied by number of iterations (epochs). So, we have 

$O\left(n\ast t\ast \left(ij+jk+kl\right)\right),$
where $$nnn$$ is number of iterations.

### Notes

Note that these matrix operations can greatly be paralelized by GPUs.

### Conclusion

We tried to find the time complexity for training a neural network that has 4 layers with respectively $$iii$$, $$jjj$$, $$kkk$$ and $$lll$$ nodes, with $$ttt$$ training examples and $$nnn$$ epochs. The result was $$O(nt∗(ij+jk+kl))O(nt∗(ij+jk+kl))\mathcal{O}(nt*(ij + jk + kl))$$.

We assumed the simplest form of matrix multiplication that has cubic time complexity. We used batch gradient descent algorithm. The results for stochastic and mini-batch gradient descent should be same. (Let me know if you think the otherwise: note that batch gradient descent is the general form, with little modification, it becomes stochastic or mini-batch)

Also, if you use momentum optimization, you will have same time complexity, because the extra matrix operations required are all element-wise operations, hence they will not affect the time complexity of the algorithm.

I'm not sure what the results would be using other optimizers such as RMSprop.

### Sources

The following article http://briandolhansky.com/blog/2014/10/30/artificial-neural-networks-matrix-form-part-5 describes an implementation using matrices. Although this implementation is using "row major", the time complexity is not affected by this.

http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4

Your answer is great..I could not find any ambiguity till now, but you forgot the no. of iterations part, just add it...and if no one answers in 5 days i'll surely accept your answer
DuttaA

@DuttaA I tried to put every thing I knew. it may not be 100% correct so feel free to leave this unaccepted :) I'm also waiting for other answers to see what other points I missed.
M.kazem Akhgary

4

For the evaluation of a single pattern, you need to process all weights and all neurons. Given that every neuron has at least one weight, we can ignore them, and have $$O(w)O(w)\mathcal{O}(w)$$ where $$www$$ is the number of weights, i.e., $$n∗nin∗nin * n_i$$, assuming full connectivity between your layers.

The back-propagation has the same complexity as the forward evaluation (just look at the formula).

So, the complexity for learning $$mmm$$ examples, where each gets repeated $$eee$$ times, is $$O(w∗m∗e)O(w∗m∗e)\mathcal{O}(w*m*e)$$.

The bad news is that there's no formula telling you what number of epochs $$eee$$ you need.

From the above answer don't you think itdepends on more factors?
DuttaA

1
@DuttaA No. There's a constant amount of work per weight, which gets repeated e times for each of m examples. I didn't bother to compute the number of weights, I guess, that's the difference.
maaartinus

1
I think the answers are same. in my answer I can assume number of weights w = ij + jk + kl. basically sum of n * n_i between layers as you noted.
M.kazem Akhgary

1

A potential disadvantage of gradient-based methods is that they head for the nearest minimum, which is usually not the global minimum.

This means that the only difference between these search methods is the speed with which solutions are obtained, and not the nature of those solutions.

An important consideration is time complexity, which is the rate at which the time required to find a solution increases with the number of parameters (weights). In short, the time complexities of a range of different gradient-based methods (including second-order methods) seem to be similar.

Six different error functions exhibit a median run-time order of approximately O(N to the power 4) on the N-2-N encoder in this paper:

Lister, R and Stone J "An Empirical Study of the Time Complexity of Various Error Functions with Conjugate Gradient Back Propagation" , IEEE International Conference on Artificial Neural Networks (ICNN95), Perth, Australia, Nov 27-Dec 1, 1995.

Summarised from my book: Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning.

Hi J. Stone. Thanks for trying to contribute to the site. However, please, note that this is not a place for advertising yourself. Anyway, you can surely provide a link to your own books if they are useful for answering the questions and provided you're not just trying to advertise yourself.
nbro

@nbro If James Stone can provide an insightful answer - and it seems so - then i'm fine with him also mentioning some of his work. Having experts on this network is a solid contribution to the quality and level.