View source for BCPNN

{{distinguish|text=[[Bayesian neural network|Bayesian Neural Network (BNN)]]}}
{{Short description|Artificial neural network}}
A '''Bayesian Confidence Propagation Neural Network (BCPNN)''' is an [[artificial neural network]] inspired by [[Bayes' theorem]], which regards neural computation and processing as probabilistic inference. Neural unit activations represent probability ("confidence") in the presence of input features or categories, synaptic weights are based on estimated correlations and the spread of activation corresponds to calculating posterior probabilities. It was originally proposed by Anders Lansner and Örjan Ekeberg at [[KTH Royal Institute of Technology|KTH]] Royal Institute of Technology.<ref>{{cite journal| vauthors = Lansner A, Ekeberg Ö |year=1989|title=A one-layer feedback artificial neural network with a Bayesian learning rule|journal=International Journal of Neural Systems|volume=1|issue=1|pages=77–87|doi=10.1142/S0129065789000499}}</ref> This probabilistic neural network model can also be run in generative mode to produce spontaneous activations and temporal sequences.

The basic model is a [[feedforward neural network]] comprising neural units with continuous activation, having a bias representing prior, and being connected by Bayesian weights in the form of [[Second-order co-occurrence pointwise mutual information|point-wise mutual information]]. The original network has been extended to a modular structure of [[Cortical minicolumn|minicolumns]] and [[Cortical column|hypercolumns]], representing discrete coded features or attributes.<ref name = "Lansner_1996">{{cite journal | vauthors = Lansner A, Holst A | title = A higher order Bayesian neural network with spiking units | journal = International Journal of Neural Systems | volume = 7 | issue = 2 | pages = 115–28 | date = May 1996 | pmid = 8823623 | doi = 10.1142/S0129065796000816 }}</ref><ref name = "Sandberg_2002">{{cite journal | vauthors = Sandberg A, Lansner A, Petersson KM, Ekeberg O | title = A Bayesian attractor network with incremental learning | journal = Network | volume = 13 | issue = 2 | pages = 179–94 | date = May 2002 | pmid = 12061419 | doi = 10.1080/net.13.2.179.194 | s2cid = 218898276 }}</ref> The units can also be connected as a [[Recurrent neural networks|recurrent neural network]] (losing the strict interpretation of their activations as probabilities)<ref>{{cite conference | vauthors = Lansner A | chapter = A recurrent bayesian ANN capable of extracting prototypes from unlabeled and noisy examples. | title = Artificial Neural Networks. Proceedings of the 1991 International Conference on Artificial Neural Networks (ICANN-91) | volume = 1–2 | date = June 1991 | location = Espoo, Finland | publisher = Elsevier }}</ref> but becoming a possible abstract model of biological neural networks and [[associative memory (psychology)|associative memory]].<ref>{{Cite book|last=Lansner|first=Anders|url=http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-292426|title=INVESTIGATIONS INTO THE PATIERN PROCESSING CAPABILITIES OF ASSOCIATIVE NETS|date=1986|publisher=KTH Royal Institute of Technology}}</ref><ref>{{cite journal| vauthors = Fransén E, Lansner A |date= January 1998 |title=A model of cortical associative memory based on a horizontal network of connected columns |journal=Network: Computation in Neural Systems|volume=9|issue=2|pages=235–264|doi=10.1088/0954-898x_9_2_006|pmid= 9861988 |issn=0954-898X}}</ref><ref>{{cite journal| vauthors = Lansner A, Fransen E |date= January 1992 |title=Modelling Hebbian cell assemblies {{sic|comprised |hide=y|of}} cortical neurons |journal=Network: Computation in Neural Systems|volume=3|issue=2|pages=105–119|doi=10.1088/0954-898x_3_2_002|issn=0954-898X}}</ref><ref>{{cite journal | vauthors = Lansner A | title = Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations | journal = Trends in Neurosciences | volume = 32 | issue = 3 | pages = 178–86 | date = March 2009 | pmid = 19187979 | doi = 10.1016/j.tins.2008.12.002 | s2cid = 11912288 | doi-access = free }}</ref><ref>{{cite journal | vauthors = Lundqvist M, Herman P, Lansner A | title = Theta and gamma power increases and alpha/beta power decreases with memory load in an attractor network model | journal = Journal of Cognitive Neuroscience | volume = 23 | issue = 10 | pages = 3008–20 | date = October 2011 | pmid = 21452933 | doi = 10.1162/jocn_a_00029 | s2cid = 2044858 }}</ref>

BCPNN has been used for machine learning classification<ref name="Ravichandran_2020">{{cite book | vauthors = Ravichandran NB, Lansner A, Herman P | title = 2020 International Joint Conference on Neural Networks (IJCNN) | chapter = Learning representations in Bayesian Confidence Propagation neural networks | journal = IJCNN | date = 2020 | pages = 1–7 | publisher = IEEE | doi =10.1109/IJCNN48605.2020.9207061 | arxiv = 2003.12415 | isbn = 978-1-7281-6926-2 | s2cid = 214692985 }}</ref> and [[data mining]], for example for discovery of adverse drug reactions.<ref name="Orre_2020">{{cite journal| vauthors = Orre R, Lansner A, Bate A, Lindquist M |year=2000|title=Bayesian neural networks with confidence estimations applied to data mining|journal=Computational Statistics & Data Analysis|volume=34|issue=4|pages=473–493|doi=10.1016/S0167-9473(99)00114-0}}</ref>  The BCPNN learning rule has also been used to model biological synaptic plasticity and intrinsic excitability in large-scale spiking neural network ([[Spiking neural network|SNN]]) models of cortical associative memory<ref name="Fiebig_2017">{{cite journal | vauthors = Fiebig F, Lansner A | title = A Spiking Working Memory Model Based on Hebbian Short-Term Potentiation | journal = The Journal of Neuroscience | volume = 37 | issue = 1 | pages = 83–96 | date = January 2017 | pmid = 28053032 | pmc = 5214637 | doi = 10.1523/JNEUROSCI.1989-16.2016 }}</ref><ref name="Fiebig_2020">{{cite journal | vauthors = Fiebig F, Herman P, Lansner A | title = An Indexing Theory for Working Memory Based on Fast Hebbian Plasticity | journal = eNeuro | volume = 7 | issue = 2 | pages = ENEURO.0374–19.2020 | date = March 2020 | pmid = 32127347 | pmc = 7189483 | doi = 10.1523/ENEURO.0374-19.2020 }}</ref> and reward learning in Basal ganglia.<ref name="Berthet_2012">{{cite journal | vauthors = Berthet P, Hellgren-Kotaleski J, Lansner A | title = Action selection performance of a reconfigurable basal ganglia inspired model with Hebbian-Bayesian Go-NoGo connectivity | language = English | journal = Frontiers in Behavioral Neuroscience | volume = 6 | pages = 65 | date = 2012 | pmid = 23060764 | pmc = 3462417 | doi = 10.3389/fnbeh.2012.00065 | doi-access = free }}</ref>

== Network architecture ==
The BCPNN network architecture is modular in terms of [[hypercolumns]] and [[Cortical minicolumn|minicolumns]]. This modular structure is inspired by and generalized from the modular structure of the mammalian [[Cerebral cortex|cortex]]. In abstract models, the [[Cortical minicolumn|minicolumns]] serve as the smallest units and they typically feature a membrane time constant and adaptation. In spiking models of [[Cerebral cortex|cortex]], a layer 2/3 [[Cortical minicolumn|minicolumn]] is typically represented by some 30 pyramidal cells and one double bouquet cell.<ref>{{cite journal | vauthors = Chrysanthidis N, Fiebig F, Lansner A | title = Introducing double bouquet cells into a modular cortical associative memory model | journal = Journal of Computational Neuroscience | volume = 47 | issue = 2–3 | pages = 223–230 | date = December 2019 | pmid = 31502234 | pmc = 6879442 | doi = 10.1007/s10827-019-00729-1 }}</ref> The latter turns the negative BCPNN-weights formed between neurons with anti-correlated activity into di-synaptic inhibition.

Lateral inhibition within the [[Cortical column|hypercolumn]] makes it a soft [[Winner-take-all (computing)|winner-take-all]] module. Looking at real [[Cerebral cortex|cortex]], the number of [[Cortical minicolumn|minicolumns]] within a [[Cortical column|hypercolumn]] is on the order of a hundred, which makes the activity sparse, at the level of 1% or less, given that [[Cortical column|hypercolumns]] can also be silent.<ref>{{cite journal | vauthors = Meli C, Lansner A | title = A modular attractor associative memory with patchy connectivity and weight pruning | journal = Network: Computation in Neural Systems| volume = 24 | issue = 4 | pages = 129–50 | date = December 2013 | pmid = 24251411 | doi = 10.3109/0954898X.2013.859323 | s2cid = 14848878 | url = http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-124654 }}</ref> A BCPNN network with a size of the human [[neocortex]] would have a couple of million [[Cortical column|hypercolumns]], partitioned into some hundred areas. In addition to sparse activity, a large-scale BCPNN would also have very sparse connectivity, given that the real [[Cerebral cortex|cortex]] is sparsely connected at the level of 0.01 - 0.001% on average.

== Bayesian-Hebbian learning rule ==

The BCPNN learning rule was derived from Bayes rule and is Hebbian such that neural units with activity correlated over time get excitatory connections between them whereas anti-correlation generates inhibition and lack of correlation gives zero connections. The independence assumptions are the same as in naïve Bayes formalism. BCPNN represents a straight-forward way of deriving a neural network from Bayes rule.<ref name = "Lansner_1996" /><ref name = "Sandberg_2002" /><ref name="Tully_2014" /> In order to allow the use the standard equation for propagating activity between neurons, transformation to log space was necessary. The basic equations for postsynaptic unit intrinsic excitability <math>\beta_j</math> and synaptic weight between pre- and postsynaptic units, <math>w_{ij}</math>, are:

<math>\beta_j = \log P_{j}</math>

<math>w_{ij} = \log P_{ij}/P_{i}P_{j}</math>
[[File:Spiking BCPNN learning rule.png|thumb|555x555px|'''Schematic flow of BCPNN update equations reformulated as spike-based plasticity. (A)''' The <math>S_i</math> pre- '''(A–D''', red''')''' and <math>S_j</math> postsynaptic '''(A–D''', blue''')''' neuron spike trains are presented as arbitrary example input patterns. Each subsequent row '''(B–D)''' corresponds to a single stage in the exponentially weighted moving average (EWMA) estimate of the terms used in the incremental Bayesian weight update. '''(B)''' ''<math>Z</math>'' traces low pass filter input spike trains. '''(C)''' ''<math>E</math>'' traces compute a low pass filtered representation of the ''<math>Z</math>'' traces at slower time scale. Co-activity now enters in a mutual trace '''(C,D''', black''')'''. '''(D)''' ''<math>E</math>'' traces feed into <math>P</math> traces that have the slowest plasticity and longest memory. <math>\kappa</math> represent a "print-now" signal that modulates learning rate.]]
where the activation and co-activation probabilities <math>P_*</math> are estimated from the training set, which can be done e.g. by exponentially weighted moving averages (see Figure).

There has been proposals for a biological interpretation of the BCPNN learning rule.  <math>Z_i</math> may represent binding of glutamate to [[NMDA receptor]]s, whereas <math>Z_j</math> could represent a [[Neural backpropagation|back-propagating action potential]] reaching the synapse. The conjunction of these events lead to <math>Ca^{2+}</math> influx via NMDA channels, [[CAMKII|CaMKII]] activation, [[AMPA receptor|AMPA]] channel phosphorylation, and eventually enhanced synaptic conductance.

The <math>Z</math> traces are further filtered  into the <math>E</math> traces, which serve as temporal buffers, [[Reinforcement learning|eligibility traces]] or [[Synaptic tagging|synaptic tags]] necessary to allow delayed reward to affect synaptic parameters. E traces are subsequently filtered into the P traces that finally determine the values of bias and weight values. This summarizes many complex protein and non-protein dependent synaptic processes behind [[LTP induction|LTP]], exhibiting highly variable timescales, from several seconds up to potentially days or months. The parameter κ ∈ [0, ∞] regulates the degree of plasticity or learning rate and is supposed to represent the release and action of some endogenous neuromodulator, e.g., [[Dopamine#Functions in the brain|dopamine]] activating D1R-like receptors, triggered by some unexpected emotionally salient situation, and resulting in [[Neuromodulation|neuromodulated]] [[activity dependent plasticity]] and learning.

== Models of brain systems and functions ==
The [[Cerebral cortex|cortex]] inspired modular architecture of BCPNN has been the basis for several spiking neural network models of [[Cerebral cortex|cortex]] aimed at studying its associative memory functions. In these models, [[Cortical minicolumn|minicolumns]] comprise about 30 model pyramidal cells and a [[Cortical column|hypercolumn]] comprises ten or more such [[Cortical minicolumn|minicolumns]] and a population of basket cells that mediate local feedback inhibition. A modelled network is composed of about ten or more such [[Cortical column|hypercolumns]]. Connectivity is excitatory within [[Cortical minicolumn|minicolumns]] and support feedback inhibition between [[Cortical minicolumn|minicolumns]] in the same [[Cortical column|hypercolumn]] via model basket cells. Long-range connectivity between [[Cortical column|hypercolumns]] is sparse and excitatory and is typically set up to form number of distributed [[Hebbian theory|cell assemblies]] representing earlier encoded memories. Neuron and synapse properties have been tuned to represent their real counterparts in terms of e.g. spike frequency adaptation and fast non-Hebbian synaptic plasticity.

These cortical models have mainly been used to provide a better understanding of the mechanisms underlying cortical dynamics and oscillatory structure associated with different activity states.<ref>{{cite journal|vauthors=Lundqvist M, Compte A, Lansner A|date=June 2010|title=Bistable, irregular firing and population oscillations in a modular attractor memory network|journal=PLOS Computational Biology|volume=6|issue=6|pages=e1000803|bibcode=2010PLSCB...6E0803L|doi=10.1371/journal.pcbi.1000803|pmc=2880555|pmid=20532199|veditors=Morrison A |doi-access=free }}</ref> Cortical oscillations in the range from theta, over alpha and beta to gamma are generated by this model. The embedded memories can be recalled from partial input and when activated they show signs of fixpoint attractor dynamics, though neural adaptation and synaptic depression terminates activity within some hundred milliseconds. Notably, a few cycles of [[Gamma wave|gamma oscillations]] are generated during such a brief memory recall. Cognitive phenomena like attentional blink and its modulation by benzodiazepine has also been replicated in this model.<ref>{{cite journal|vauthors=Silverstein DN, Lansner A|date=2011|title=Is attentional blink a byproduct of neocortical attractors?|journal=Frontiers in Computational Neuroscience|volume=5|pages=13|doi=10.3389/fncom.2011.00013|pmc=3096845|pmid=21625630|doi-access=free}}</ref>

In recent years, Hebbian plasticity has been incorporated into this [[Cerebral cortex|cortex]] model and simulated with abstract non-spiking as well as spiking neural units.<ref name="Tully_2014">{{cite journal | vauthors = Tully PJ, Hennig MH, Lansner A | title = Synaptic and nonsynaptic plasticity approximating probabilistic inference | journal = Frontiers in Synaptic Neuroscience | volume = 6 | pages = 8 | date = April 2014 | pmid = 24782758 | pmc = 3986567 | doi = 10.3389/fnsyn.2014.00008 | doi-access = free }}</ref> This made it possible to demonstrate online learning of temporal sequences<ref>{{cite journal | vauthors = Tully PJ, Lindén H, Hennig MH, Lansner A | title = Spike-Based Bayesian-Hebbian Learning of Temporal Sequences | journal = PLOS Computational Biology | volume = 12 | issue = 5 | pages = e1004954 | date = May 2016 | pmid = 27213810 | pmc = 4877102 | doi = 10.1371/journal.pcbi.1004954 | bibcode = 2016PLSCB..12E4954T | veditors = Morrison A | doi-access = free }}</ref> as well as one-shot encoding and immediate recall in human word list learning.<ref name="Fiebig_2017" /> These findings further lead to the proposal and investigation of a novel theory of [[working memory]] based on fast Hebbian synaptic plasticity.<ref name="Fiebig_2020" />

A similar approach was applied to model reward learning and [[Behavior selection algorithm|behavior selection]] in a Go-NoGo connected non-spiking and spiking neural network models of the Basal ganglia.<ref name="Berthet_2012" /><ref>{{cite journal | vauthors = Berthet P, Lindahl M, Tully PJ, Hellgren-Kotaleski J, Lansner A | title = Functional Relevance of Different Basal Ganglia Pathways Investigated in a Spiking Model with Reward Dependent Plasticity | journal = Frontiers in Neural Circuits | volume = 10 | pages = 53 | date = July 2016 | pmid = 27493625 | pmc = 4954853 | doi = 10.3389/fncir.2016.00053 | doi-access = free }}</ref>

== Machine learning applications ==
The point-wise mutual information weights of BCPNN is since long one of the standard methods for detection of drug adverse reactions.<ref name="Orre_2020" />

BCPNN has recently been successfully applied to Machine Learning classification benchmarks, most notably the hand written digits of the [[MNIST database]]. The BCPNN approach uses biologically plausible learning and structural plasticity for unsupervised generation of a sparse hidden representation, followed by a one-layer classifier that associates this representation to the output layer.<ref name="Ravichandran_2020" /> It achieves a classification performance on the full MNIST test set around 98%, comparable to other methods based on unsupervised representation learning.<ref>{{cite arXiv | vauthors = Ravichandran NB, Lansner A, Herman P | title = Brain-like approaches to unsupervised learning of hidden representations--a comparative study. | eprint = 2005.03476 | date = May 2020 | class = cs.NE }}</ref> The performance is notably slightly lower than that of the best methods that employ end-to-end error back-propagation. However, the extreme performance comes with a cost of lower biological plausibility and higher complexity of the learning machinery. The BCPNN method is also quite well suited for semi-supervised learning.

== Hardware designs for BCPNN ==
The structure of BCPNN with its [[Cerebral cortex|cortex]]-like modular architecture and massively parallel correlation based Hebbian learning makes it quite hardware friendly. Implementation with reduced number of bits in synaptic state variables have been shown to be feasible.<ref>{{cite journal | vauthors = Vogginger B, Schüffny R, Lansner A, Cederström L, Partzsch J, Höppner S | title = Reducing the computational footprint for real-time BCPNN learning | journal = Frontiers in Neuroscience | volume = 9 | pages = 2 | date = January 2015 | pmid = 25657618 | pmc = 4302947 | doi = 10.3389/fnins.2015.00002 | doi-access = free }}</ref> BCPNN has further been the target for parallel simulators on cluster computers and GPU:s. It was recently implemented on the [[SpiNNaker]]  compute platform<ref>{{cite journal | vauthors = Knight JC, Tully PJ, Kaplan BA, Lansner A, Furber SB | title = Large-Scale Simulations of Plastic Neural Networks on Neuromorphic Hardware | journal = Frontiers in Neuroanatomy | volume = 10 | pages = 37 | date = April 2016 | pmid = 27092061 | pmc = 4823276 | doi = 10.3389/fnana.2016.00037 | doi-access = free }}</ref> as well as in a series of dedicated neuromorphic VLSI designs.<ref>{{cite book| vauthors = Farahini N, Hemani A, Lansner A, Clermidy F, Svensson C |title= 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC) |chapter= A scalable custom simulation machine for the Bayesian Confidence Propagation Neural Network model of the brain |date= January 2014 |location=Singapore|publisher=IEEE|pages=578–585|doi=10.1109/ASPDAC.2014.6742953|isbn=978-1-4799-2816-3|s2cid=15069505}}</ref><ref>{{cite book| vauthors = Lansner A, Hemani A, Farahini N |title= 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC) |chapter= Spiking brain models: Computation, memory and communication constraints for custom hardware implementation |date= January 2014 |location=Singapore|publisher=IEEE|pages=556–562|doi=10.1109/ASPDAC.2014.6742950|isbn=978-1-4799-2816-3|s2cid=18476370}}</ref><ref>{{cite journal| vauthors = Stathis D, Sudarshan C, Yang Y, Jung M, Weis C, Hemani A, Lansner A, Wehn N |date= November 2020 |title=eBrainII: a 3 kW Realtime Custom 3D DRAM Integrated ASIC Implementation of a Biologically Plausible Model of a Human Scale Cortex |journal=Journal of Signal Processing Systems|language=en|volume=92|issue=11|pages=1323–1343|doi=10.1007/s11265-020-01562-x|s2cid=207870792|issn=1939-8018|doi-access=free|arxiv=1911.00889|bibcode= 2020JSPSy..92.1323S }}</ref><ref>{{cite journal | vauthors = Yang Y, Stathis D, Jordão R, Hemani A, Lansner A | title = Optimizing BCPNN Learning Rule for Memory Access | journal = Frontiers in Neuroscience | volume = 14 | pages = 878 | date = August 2020 | pmid = 32982673 | pmc = 7487417 | doi = 10.3389/fnins.2020.00878 | doi-access = free }}</ref> From these it has been estimated that a human [[Cerebral cortex|cortex]] sized BCPNN with continuous learning could be executed in real time with a power dissipation on the order of few kW.

== References ==
{{Reflist}}

[[Category:Artificial neural networks]]