What is a deep neural network?

A Deep Neural Network (DNN) is a machine learning technique that allows a computer, by training it, to do tasks that would be very difficult to do using conventional programming techniques. Neural network algorithms were inspired by the human brain and its functions: like our human mind, it is designed to work not only by following a preset list of rules, but by predicting solutions and drawing conclusions based on previous iterations and experiences.

What is neural network architecture?

A neural network is composed of multiple layers of nodes that receive input from other layers and produce an output until a final result is reached. Neural networks can have any number of hidden layers: the more layers of nodes are in the network, the higher the complexity. Here are different neural network architectures:

1. Traditional neural networks, usually composed of 2 or 3 hidden layers;
2. Deep learning network, that can have up to 150 hidden layers;

What is the difference between neural networks and deep neural networks?

A deep neural network is a much more complicated system than a “simple” neural system. A neural network is comparable to a chess game, and behaves according to algorithms: different tactics will be suggested according to the opponent’s moves and actions. This neural network will be limited to what the programmer’s input:

• How do the chess pieces move;
• The size of the chessboard;
• Different strategies for different situations;

A neural network that goes beyond the input data and can learn from previous experiences becomes a deep neural network

On the same computer you could, for example, train a neural network, then play with it against other people and let it learn as it played. From there, if a neural network can learn from different players, it may become difficult, or literally impossible, to defeat a deep neural network, even for chess masters.

Deep neural networks can recognize voice commands, identify voices, recognize sounds and graphics and do much more than a neural network. Deep learning networks utilize "Big Data'' along with algorithms in order to solve a problem, and these deep neural networks can solve problems with limited or no human input.

How to visualize the work of a deep neural network?

A deep neural network’s process is best understood by looking at an example. Imagine you had hundreds of thousands of images, some of which had dogs in them, and you decided you wanted to write a computer program to recognize dogs in pictures.

You have two choices. You can write a program to explicitly identify dogs, or you can write a program that “learns” how to identify dogs.

You unwisely decide to try to do the former.

Using conventional programming techniques is long and difficult, and results aren't always accurate

In order to identify dog pictures, you create a software program using “if” and “then” statements where the likelihood that you are looking at a dog is programmed to increase every time you identify a doglike attribute such as fur, floppy ears and a tail. However, this type of neural system is difficult on many levels:

For example if a clump of pixels resembles a tail increase the likelihood that you are looking at a dog. Your deep neural network needs to identify groups of pixels that correspond to the doglike attributes. Even if you manage to do that, there are issues for your algorithm:

• Many photographed objects share some of the dog-like attributes, especially photographs of similar animals. You will need to add rules;
• Sometimes the attributes are there but obscured. Your algorithms will not pick them up;
• Sometimes attributes are only of importance when other attributes are present. More decision rules will be needed;

Your classification fails. You realize you cannot manually identify the complete set of attributes let alone devise all the rules needed to deal with all these special cases.

You wisely give up and decide to try the latter approach. To use a neural network or, even better, a deep neural network.

A deep learning model can save you coding time and offer better results

The neural network is so named because there is a similarity between this programming approach and the way the brain works.

Just like the brain, the neural net algorithms use a network of neurons or nodes. And like the brain, these neurons are discrete functions (or little machines if you like) that take in inputs and generate outputs. These nodes are arranged in layers whereby the outputs of neurons in one layer become the inputs to neurons in the next layer until the neurons on the outer layer of the network generate the final result.

There are therefore layers of neurons with each individual neuron receiving very limited inputs and generating very limited outputs just like in the brain. The first layer (or input layer) of neurons takes in the inputs and the last layer of neurons (or output layer) in the network outputs the result.

Is it accurate to call this type of algorithm a “neural network”?

The human brain is far more complex and powerful than a neural network of course. Naming the algorithm a “deep neural network” was a branding coup but it may create unrealistic expectations about what is achievable with these techniques. That said there are people trying to re-engineer the brain, using a very complex neural network, in the hope that by doing this they will be able to replicate general, human-like intelligence in bot development. So how does a neural net and machine learning techniques help us with our dog recognition problem?

Well, instead of manually defining dog-like attributes, a deep neural network algorithm can identify the important attributes and deal with all the special cases without programming.

How does a deep neural network function?

It does this as follows:

Each neuron on the input layer receives a bit of information from the image as an input and then randomly weights (between zero and one) whether that information suggests a dog or not. A low weight (less than 0.5) means it's less likely that the information is associated with a dog and a high weight means it's more likely the information is associated with a dog. This multilayer neural network approach is called deep learning. Neural networks and deep learning are very powerful techniques for achieving computer comprehension.

Deep neural networks are composed of multiple layers of nodes behaving like neurons in our brain

So to continue on deep neural networks, the weights of these neurons are then fed as inputs into the other layers of neurons which also randomly assign weights and pass them on as inputs to yet more neurons in the network. This continues until the output layer of neurons gives a binary verdict. If the average of the weights passed to them is greater than 0.5, it's a dog otherwise it's not. These connections between and activation of neurons across a multiple layers of nodes are what give deep neural network applications their power.

How does a deep neural network know if it produces the right answer?

Relevant questions at this point are: has the deep neural network guessed correctly or not and what happens if it has or has not guessed correctly? And how does a neural network know if it's guessed correctly or not?

One way it would know is if you undertook the extremely time consuming classification task of labelling all the photographs “dog” or “not dog” depending as to whether there is a dog in the photo or not. The neural net will simply look at the label to see if it correctly identified the dog or not.

And of course we are not interested in whether it got the “Dog or not” question right on a single dog picture. We are interested in whether it got the question right for every photo, or at least to find out what percentage of the time it was accurate in assessing if there was a dog or not in the picture.

How do deep neural networks use training to learn?

A neural network learns from each iteration through data to improve its accuracy

For a given set of weights across all neurons in the network, the neural net will make guesses for all photos and then determine how accurate it was. What percentage of the time did it get the right result i.e. say the dog was in the photo when it was in the photo, and how many times did it get the wrong result, say the dog was in the photo when it wasn't or say the dog wasn't in the photo when it was. This indication as to how accurate the AI algorithm is essential feedback for the neural network model.

Once it has run through all the photos once, it can randomly (or otherwise) adjust some of the weights and then do the whole exercise of guessing what is in the photo again. If the result from the second run is better, it will keep instead of reverting back to the previous set of weights. If the result from the second run was worse it may revert to the previous set of weights and then try different modifications to those weights.

This process will carry on in this way until the neural net becomes good at identifying dogs in photos (hopefully).

When the algorithm can accurately identify dogs it is said to have converged. It has been successfully “trained” to identify dogs.

What are the different types of neural networks?

Convolutional neural networks

Convolutional neural networks (CNN) are a type of artificial intelligence designed to process, or learn from, large data sets. Convolutional neural network is a newly coined term specifically describing this type of network, or AI technology in general.

CNNs are powerful image recognition AI tools that use deep learning to perform not only generative tasks, but also descriptive tasks. Examples of generative tasks include auto-cropping, caption writing, videography, mimeograph, and image overlays. A convolutional neural network contains what are called convolutional layers. Each neuron in these layers only processes information from a small part of a visual field. Each neuron’s inputs are lined up in a checksum-like fashion to generate a feature map.

Artificial neural networks

An artificial neural network (ANN) is a network of many perceptrons at different depths or layers that can be understood as a Logistic Regression. ANN is often called Feed Forward Neural Network because inputs are processed only in the forward direction: a layer receives input and sends an output in a linear fashion.

Artificial neural networks are also known as universal function approximators. Pure neural network algorithms such as ANNs and mapping functions that can be implemented as deep learning allow computers to learn any function. One of the reasons why universal approximation is important is the activation function. Activation functions introduce nonlinear properties into the network while learning any complex relationship between input and output. It helps different kinds of the network learn from each other.

How does a deep neural network improve over time?

One way to imagine what the algorithm is doing is to imagine each neuron as a kind of certainty test. Instead of coding all those if-then statements to identify dogs, each neuron is calibrated to add or take away from the final judgment that the object in the photo is a dog. It's as though the judgment (such as dog or not) is split into a large number of connected judgements that contribute in aggregate to a final judgment.

Of course the primary objective is to achieve convergence if that is possible. It's also an important objective to do this in a reasonable amount of time, preferably in a short time.

Deep learning algorithms have a learning process that makes them difficult to understand for humans

What is interesting is that the logic that allows neural networks to identify the dogs in the picture is not human understandable. Deep learning models have hidden logic, essentially a black box of hidden layers of nodes creating its own deep network. That said, there have been some attempts to try to visually represent the logic behind neural networks for image recognition tasks. For other cases, it is not possible to see what the algorithm is doing behind the scenes and the deep learning methods remain hidden.

Neural networks and machine learning are popular now but many of these algorithms were known around 50 years ago.

Why are deep neural networks increasingly popular in various industries?

One of the primary reasons that neural nets are much more popular now than when they were first invented, is that processing power is faster and cheaper than it was. Computing power has made all the difference in achieving fast convergence. The other reason is that data is now ubiquitous which increases the value of algorithms that can make use of the data like chatbots for business

Advanced neural networks require high processing power and a lot of data

Deep learning neural networks are data and processor-hungry techniques that can achieve results that would be impossible for programmers using programming techniques to achieve. They are ideally suited to certain problems where ubiquitous data is available and it is easy to categorize or rank preferable outcomes.

Without having hundreds of thousands or preferably millions of photos of dogs it would be impossible to train the algorithm. These techniques only work when a lot of data is available. This is fairly obvious as all the special cases are unlikely to be represented in a set of just 1000 photos.

Neural networks can be faced with structured data or unstructured data

One problem in the above example is that much manual work is involved in labelling all the photos. It is easier for the algorithms to use data that is labelled in a structured way. Neural network machine learning that uses structured data is called supervised learning.

This brings us to a central question: is it possible to avoid all of that tagging work? That would be good because not only could you avoid a lot of manual work but also most of the data available on the internet is unstructured i.e. is not carefully labelled or structured.

Can neural networks work with unstructured data?

Artificial neural networks and machine learning that works with unstructured data are called unsupervised learning. Of course, this is the holy grail of machine learning and is more analogous to how humans learn. However, even unsupervised learning by machines requires much more data to “learn” than humans do and machines cannot easily extrapolate to examples that are outside what they have been trained on.

Many deep learning models try to reproduce the human brain processes

Some people believe that these types of algorithms can be developed, perhaps by re-engineering the brain, to the point that the algorithms start to approach a human-level “understanding”. They believe that it will be possible to use sophisticated scanning technology of the brain to give us insights into how the neural networks of the brain actually work. By copying these designs and patterns we can replicate human-level intelligence.

While the techniques are no doubt ingenious and very useful, especially where a large data set is available, it is hard to imagine that such simple algorithms could be the basis of a highly creative human-like intelligence.

‍