07-03-2011, 09:30 AM
[attachment=9642]
Artificial Neural Networks
Introduction to Neural Networks
Principles of Brain Processing
Brain Computer: What is it?
Biological Neurons
Brain-like Computer
• ANN as a Brain-Like Computer
Applications of Artificial Neural Networks
Image Recognition: Decision Rule and Classifier
• Is it possible to formulate (and formalize!) the decision rule, using which we can classify or recognize our objects basing on the selected features?
• Can you propose the rule using which we can definitely decide is it a tiger or a rabbit?
• Once we know our decision rule, it is not difficult to develop a classifier, which will perform classification/recognition using the selected features and the decision rule.
• However, if the decision rule can not be formulated and formalized, we should use a classifier, which can develop the rule from the learning process
• In the most of recognition/classification problems, the formalization of the decision rule is very complicated or impossible at all.
• A neural network is a tool, which can accumulate knowledge from the learning process.
• After the learning process, a neural network is able to approximate a function, which is supposed to be our decision rule
Why neural network?
• Mathematical Interpretation of Classification in Decision Making
• Intelligent Data Analysis in Engineering Experiment
• Learning via Self-Organization Principle
• Symbol Manipulation or Pattern Recognition ?
Artificial Neuron
A Neuron
• Neurons’ functionality is determined by the nature of its activation function, its main properties, its plasticity and flexibility, its ability to approximate a function to be learned
• Artificial Neuron:
Classical Activation Functions
Principles of Neurocomputing
Threshold Neuron (Perceptron)
• Output of a threshold neuron is binary, while inputs may be either binary or continuous
• If inputs are binary, a threshold neuron implements a Boolean function
• The Boolean alphabet {1, -1} is usually used in neural networks theory instead of {0, 1}. Correspondence with the classical Boolean alphabet {0, 1} is established as follows:
Threshold Boolean Functions
• The Boolean function is called a threshold (linearly separable) function, if it is possible to find such a real-valued weighting vector that equation
holds for all the values of the variables x from the domain of the function f.
• Any threshold Boolean function may be learned by a single neuron with the threshold activation function.
• Threshold Boolean Functions: Geometrical Interpretation
“OR” (Disjunction) is an example of the threshold (linearly separable) Boolean function:
“-1s” are separated from “1” by a line
• 1 1à 1
• 1 -1à -1
• -1 1à -1
• -1 -1à -1
• XOR is an example of the non-threshold (not linearly separable) Boolean function: it is impossible separate “1s” from “-1s” by any single line
• 1 1 à 1
• 1 -1à -1
• -1 1à -1
• -1 -1à 1
Threshold Neuron: Learning
• A main property of a neuron and of a neural network is their ability to learn from its environment, and to improve its performance through learning.
• A neuron (a neural network) learns about its environment through an iterative process of adjustments applied to its synaptic weights.
• Ideally, a network (a single neuron) becomes more knowledgeable about its environment after each iteration of the learning process.
• Let us have a finite set of n-dimensional vectors that describe some objects belonging to some classes (let us assume for simplicity, but without loss of generality that there are just two classes and that our vectors are binary). This set is called a learning set:
• Learning of a neuron (of a network) is a process of its adaptation to the automatic identification of a membership of all vectors from a learning set, which is based on the analysis of these vectors: their components form a set of neuron (network) inputs.
• This process should be utilized through a learning algorithm.
• Let T be a desired output of a neuron (of a network) for a certain input vector and Y be an actual output of a neuron.
• If T=Y, there is nothing to learn.
• If T≠Y, then a neuron has to learn, in order to ensure that after adjustment of the weights, its actual output will coincide with a desired output
• Error-Correction Learning
• If T≠Y, then is the error .
• A goal of learning is to adjust the weights in such a way that for a new actual output we will have the following:
• That is, the updated actual output must coincide with the desired output.
Error-Correction Learning
• The error-correction learning rule determines how the weights must be adjusted to ensure that the updated actual output will coincide with the desired output:
• α is a learning rate (should be equal to 1 for the threshold neuron, when a function to be learned is Boolean)
Learning Algorithm
• Learning algorithm consists of the sequential checking for all vectors from a learning set, whether their membership is recognized correctly. If so, no action is required. If not, a learning rule must be applied to adjust the weights.
• This iterative process has to continue either until for all vectors from the learning set their membership will be recognized correctly or it will not be recognized just for some acceptable small amount of vectors (samples from the learning set).
When we need a network
• The functionality of a single neuron is limited. For example, the threshold neuron (the perceptron) can not learn non-linearly separable functions.
• To learn those functions (mappings between inputs and output) that can not be learned by a single neuron, a neural network should be used.
A simplest network
• Solving XOR problem using the simplest network
• Solving XOR problem using the simplest network
• Threshold Functions and
Threshold Neurons
• Threshold (linearly separable) functions can be learned by a single threshold neuron
• Non-threshold (nonlinearly separable) functions can not be learned by a single neuron. For learning of these functions a neural network created from threshold neurons is required (Minsky-Papert, 1969)
• The number of all Boolean functions of n variables is equal to , but the number of the threshold ones is substantially smaller. Really, for n=2 fourteen from sixteen functions (excepting XOR and not XOR) are threshold, for n=3 there are 104 threshold functions from 256, but for n>3 the following correspondence is true (T is a number of threshold functions of n variables):
• For example, for n=4 there are only about 2000 threshold functions from 65536
• Is it possible to learn XOR, Parity n and other non-linearly separable functions
using a single neuron?
• Any classical monograph/text book on neural networks claims that to learn the XOR function a network from at least three neurons is needed.
• This is true for the real-valued neurons and real-valued neural networks.
• However, this is not true for the complex-valued neurons !!!
• A jump to the complex domain is a right way to overcome the Misky-Papert’s limitation and to learn multiple-valued and Boolean nonlinearly separable functions using a single neuron.
• XOR problem
• Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN
• Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN
• I. Aizenberg, D. Paliy, J. Zurada, and J. Astola, "Blur Identification by Multilayer Neural Network based on Multi-Valued Neurons", IEEE Transactions on Neural Networks, vol. 19, No 5, May 2008, pp. 883-898.
Problem statement: capturing
• Mathematically a variety of capturing principles can be described by the Fredholm integral of the first kind
• where x,t ℝ2, v(t) is a point-spread function (PSF) of a system, y(t) is a function of a real object and z(x) is an observed signal.
Image deblurring: problem statement
• Mathematically blur is caused by the convolution of an image with the distorting kernel.
• Thus, removal of the blur is reduced to the deconvolution.
• Deconvolution is an ill-posed problem, which results in the instability of a solution. The best way to solve it is to use some regularization technique.
• To use any kind of regularization technique, it is absolutely necessary to know the distorting kernel corresponding to a particular blur: so it is necessary to identify the blur.
Blur Identification
• We use multilayer neural network based on multi-valued neurons (MLMVN) to recognize Gaussian, motion and rectangular (boxcar) blurs.
• We aim to identify simultaneously both blur and its parameters using a single neural network.
Artificial Neural Networks
Introduction to Neural Networks
Principles of Brain Processing
Brain Computer: What is it?
Biological Neurons
Brain-like Computer
• ANN as a Brain-Like Computer
Applications of Artificial Neural Networks
Image Recognition: Decision Rule and Classifier
• Is it possible to formulate (and formalize!) the decision rule, using which we can classify or recognize our objects basing on the selected features?
• Can you propose the rule using which we can definitely decide is it a tiger or a rabbit?
• Once we know our decision rule, it is not difficult to develop a classifier, which will perform classification/recognition using the selected features and the decision rule.
• However, if the decision rule can not be formulated and formalized, we should use a classifier, which can develop the rule from the learning process
• In the most of recognition/classification problems, the formalization of the decision rule is very complicated or impossible at all.
• A neural network is a tool, which can accumulate knowledge from the learning process.
• After the learning process, a neural network is able to approximate a function, which is supposed to be our decision rule
Why neural network?
• Mathematical Interpretation of Classification in Decision Making
• Intelligent Data Analysis in Engineering Experiment
• Learning via Self-Organization Principle
• Symbol Manipulation or Pattern Recognition ?
Artificial Neuron
A Neuron
• Neurons’ functionality is determined by the nature of its activation function, its main properties, its plasticity and flexibility, its ability to approximate a function to be learned
• Artificial Neuron:
Classical Activation Functions
Principles of Neurocomputing
Threshold Neuron (Perceptron)
• Output of a threshold neuron is binary, while inputs may be either binary or continuous
• If inputs are binary, a threshold neuron implements a Boolean function
• The Boolean alphabet {1, -1} is usually used in neural networks theory instead of {0, 1}. Correspondence with the classical Boolean alphabet {0, 1} is established as follows:
Threshold Boolean Functions
• The Boolean function is called a threshold (linearly separable) function, if it is possible to find such a real-valued weighting vector that equation
holds for all the values of the variables x from the domain of the function f.
• Any threshold Boolean function may be learned by a single neuron with the threshold activation function.
• Threshold Boolean Functions: Geometrical Interpretation
“OR” (Disjunction) is an example of the threshold (linearly separable) Boolean function:
“-1s” are separated from “1” by a line
• 1 1à 1
• 1 -1à -1
• -1 1à -1
• -1 -1à -1
• XOR is an example of the non-threshold (not linearly separable) Boolean function: it is impossible separate “1s” from “-1s” by any single line
• 1 1 à 1
• 1 -1à -1
• -1 1à -1
• -1 -1à 1
Threshold Neuron: Learning
• A main property of a neuron and of a neural network is their ability to learn from its environment, and to improve its performance through learning.
• A neuron (a neural network) learns about its environment through an iterative process of adjustments applied to its synaptic weights.
• Ideally, a network (a single neuron) becomes more knowledgeable about its environment after each iteration of the learning process.
• Let us have a finite set of n-dimensional vectors that describe some objects belonging to some classes (let us assume for simplicity, but without loss of generality that there are just two classes and that our vectors are binary). This set is called a learning set:
• Learning of a neuron (of a network) is a process of its adaptation to the automatic identification of a membership of all vectors from a learning set, which is based on the analysis of these vectors: their components form a set of neuron (network) inputs.
• This process should be utilized through a learning algorithm.
• Let T be a desired output of a neuron (of a network) for a certain input vector and Y be an actual output of a neuron.
• If T=Y, there is nothing to learn.
• If T≠Y, then a neuron has to learn, in order to ensure that after adjustment of the weights, its actual output will coincide with a desired output
• Error-Correction Learning
• If T≠Y, then is the error .
• A goal of learning is to adjust the weights in such a way that for a new actual output we will have the following:
• That is, the updated actual output must coincide with the desired output.
Error-Correction Learning
• The error-correction learning rule determines how the weights must be adjusted to ensure that the updated actual output will coincide with the desired output:
• α is a learning rate (should be equal to 1 for the threshold neuron, when a function to be learned is Boolean)
Learning Algorithm
• Learning algorithm consists of the sequential checking for all vectors from a learning set, whether their membership is recognized correctly. If so, no action is required. If not, a learning rule must be applied to adjust the weights.
• This iterative process has to continue either until for all vectors from the learning set their membership will be recognized correctly or it will not be recognized just for some acceptable small amount of vectors (samples from the learning set).
When we need a network
• The functionality of a single neuron is limited. For example, the threshold neuron (the perceptron) can not learn non-linearly separable functions.
• To learn those functions (mappings between inputs and output) that can not be learned by a single neuron, a neural network should be used.
A simplest network
• Solving XOR problem using the simplest network
• Solving XOR problem using the simplest network
• Threshold Functions and
Threshold Neurons
• Threshold (linearly separable) functions can be learned by a single threshold neuron
• Non-threshold (nonlinearly separable) functions can not be learned by a single neuron. For learning of these functions a neural network created from threshold neurons is required (Minsky-Papert, 1969)
• The number of all Boolean functions of n variables is equal to , but the number of the threshold ones is substantially smaller. Really, for n=2 fourteen from sixteen functions (excepting XOR and not XOR) are threshold, for n=3 there are 104 threshold functions from 256, but for n>3 the following correspondence is true (T is a number of threshold functions of n variables):
• For example, for n=4 there are only about 2000 threshold functions from 65536
• Is it possible to learn XOR, Parity n and other non-linearly separable functions
using a single neuron?
• Any classical monograph/text book on neural networks claims that to learn the XOR function a network from at least three neurons is needed.
• This is true for the real-valued neurons and real-valued neural networks.
• However, this is not true for the complex-valued neurons !!!
• A jump to the complex domain is a right way to overcome the Misky-Papert’s limitation and to learn multiple-valued and Boolean nonlinearly separable functions using a single neuron.
• XOR problem
• Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN
• Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN
• I. Aizenberg, D. Paliy, J. Zurada, and J. Astola, "Blur Identification by Multilayer Neural Network based on Multi-Valued Neurons", IEEE Transactions on Neural Networks, vol. 19, No 5, May 2008, pp. 883-898.
Problem statement: capturing
• Mathematically a variety of capturing principles can be described by the Fredholm integral of the first kind
• where x,t ℝ2, v(t) is a point-spread function (PSF) of a system, y(t) is a function of a real object and z(x) is an observed signal.
Image deblurring: problem statement
• Mathematically blur is caused by the convolution of an image with the distorting kernel.
• Thus, removal of the blur is reduced to the deconvolution.
• Deconvolution is an ill-posed problem, which results in the instability of a solution. The best way to solve it is to use some regularization technique.
• To use any kind of regularization technique, it is absolutely necessary to know the distorting kernel corresponding to a particular blur: so it is necessary to identify the blur.
Blur Identification
• We use multilayer neural network based on multi-valued neurons (MLMVN) to recognize Gaussian, motion and rectangular (boxcar) blurs.
• We aim to identify simultaneously both blur and its parameters using a single neural network.
