MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_NextPart_01CE649F.28501E40" This document is a Single File Web Page, also known as a Web Archive file. If you are seeing this message, your browser or editor doesn't support Web Archive files. Please download a browser that supports Web Archive, such as Microsoft Internet Explorer. ------=_NextPart_01CE649F.28501E40 Content-Location: file:///C:/518A1270/NeurApp.htm Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="us-ascii"
by
Artificial Neural Networks
by Igor Grešovnik & Tadej Kodelja



Contents:
2.2 One Parametric
Approximations
2.2.1  =
; Defining
the Approximated Function and Training Data.
2.2.2  =
; Training
the Neural Network
2.2.4  =
; Checking
Results
2.3 Functions of 2 Variab=
les
with 3D Surface Plots
2.3.1  =
; Defining
the Approximated Function and Training Data.
2.3.2  =
; Network
Architecture and Training Parameters.
2.3.3  =
; Showing
Training Data and Results
2.4 Accessing Help and Web
Pages

Figure 1: GUI
for approximation of functions of one variable.
While approximation of functions of a single variable is the least interesting problem in ANN-based modeling, playing with such approximations is instruct= ive because very clear graphic representations of results can be made. <= /p>
Explor= ing 1D ANN-based approximations in NeurApp is intuitive and can be learned in a minute. Clic= king on the “1D Approximation” tab on the top of the application’s form activates the user interface for 1D approximations= . To see how it works, click first on the “Generate data” but= ton and then on the “Start” button, and that’s it.
Clicki= ng on the “Generate data” button samples the function that you wan= t to approximate and is specified next to the “f(x) =3D ” lab= el, in order to generate function values used to train the neural network. After t= he training data is generated, it is immediately shown on the graph on the right-hand side (together with the approximated function on the interval wh= ere values are sampled), and the “Start” button gets active. Pressing the start button trains the neural network on the data that has be= en generated.
During= the training procedure, the neural network tries to produce an intrinsic functi= on that fits well the training data, without knowing anything about the origin= al function used to generate that data. After training is done, the graph on t= he top right of the form shows the approximation generated by the neural netwo= rk, the training data used to generate this approximation, and the original function used to generated the data. In this way you can visualize how much= the approximated function deviates from the original one.
You ca= n vary the number of sampled function values in the training data, change interval on which training data is generated, modify the approximation function itself,= and change neural network architecture and training parameters and observe how = any of these circumstances affect the accuracy of the approximation (i.e., how closely the approximation fits the original function). In further subsectio= ns it is explained how to adjust specific things that influence the approximat= ion.
When s= tarting the application, there is already a pre-set default function to be approximated. You can change the approximated function by using the input f= ield next to the “f(x) =3D” label, and this can be done in two ways. If the checkbox “Function defined by user” is unchecked, you can choose one of the pre-defined functions from the drop-do= wn list that opens when clicking on the triangular arrow positioned on the right-hand side of the text field.
In ord= er to define at own will an arbitrary function to be approximated, check the R= 20;Function defined by user” checkbox. When the checkbox is checked, you can insert expressions defining an arbitrary function in a standard symbolic wa= y by using “x” for independent variables, standard operators = such as +, -, * (multiplication), / (division), a number of standard mathematical functions, and parentheses for grouping expressions and overriding operator precedence in order to define order of evaluation. There is no operator for powers, and you must used the function “pow”, e.g. the expression “pow(x,3/2)” means the independent variable x raised= to the power of 3/2 (which is equivalent to calculating square root of x and raising the obtained value to the third power).
Warnin= g: You can not insert your own function definition when the “Function defined= by user” checkbox is not checked. The text field for inserting funct= ions is green when user defined functions are allowed. If you insert an invalid expression that defines the approximated function (e.g. forget a bracket or= use a mathematical function that is not defined, the text field will turn orange when you attempt to generate the data.
In the= “Limits” box, you can define the interval on which the approximated function is samp= led in order to generate the training data. Number of function values to be generated and used in training is defined next to “Number of training samples”. If the “Random training data” is checked then function will be evaluated in randomly chosen points. Otherwise, it is sampled in equidistant points on the selected interval.
Genera= tion of function values used for training (sampling) is triggered by clicking on the “Generate Data” button. Sampled function values are shown as po= ints on the graph where the original function is shown, too.
The AN= N-based approximation of the original function is calculated by training the neural network on the sampled values generated before. In this procedure, the inte= rnal state of the neural network is continuously updated in such a way that the discrepancy between the training data and the outputs generated by the netw= ork are minimized. Training (generation of the approximation) begins by pressing the “Start” button and finishes either when the prescrib= ed accuracy is reached or when the maximal prescribed training time (measured = in epochs, i.e. in iterations of the training algorithm) elapses.
The us= er can define the required accuracy by setting a tolerance (a small positive numbe= r) on the RMS (root mean square) error, which is inserted next to the “<= i>RMS” label. The training procedure will stop when the error measure drops below = the user defined tolerance. In order to prevent infinite looping when the toler= ance can not be achieved, the maximal number of epochs can be set next to the ”Max. Epochs” label. If the training algorithm performs = that number of epochs, it will stop even if the required accuracy is not reached. The box next to “Epochs in bundle” just defines granular= ity of convergence checks, which are not performed after every epoch (iteration= ) of the training procedure, but only after each specified number of iterations defined by this parameter. This has no significant effect on results in most cases, but speed ups computation by a small factor.
Network architecture can be defined next to the “Neurons in hidden layers<= /i>” label. Networks with one or two layers of hidden neurons can be used, and t= he number of neurons in each hidden layer can be independently defined in the corresponding field. The second number can be 0 meaning that a neural netwo= rk with a single hidden layer is be used. In general, more neurons in hidden layers means more connections between neurons, thus more free parameters (weights) to be adjusted and better fitting capabilities of the network (but also increased probability for overfitting and longer computation times). Therefore, approximation of more complex functions (e.g. with wild oscillat= ions or sharp transitions) requires more neurons to achieve comparable accuracy = as with simple functions, provided of course that the number of training data = is also appropriately larger. In a two layer network, each neuron of the first hidden layer is connected to each neuron in the second layer, meaning that = the number of connection increases quadratically with simultaneously increasing number of neurons in both layers.
Two pa= rameters of the training procedure, namely the learning rate and the momen= tum, can be adjusted in the corresponding text fields. These parameters can be b= oth set to positive values lesser than 1. Increasing the learning rate means th= at the weights and biases are more quickly adjusted in the opposite direction = of the error gradient in order to decrease the error, which means faster decre= ase of errors at the beginning, but can cause more numerical instability. Momen= tum adds inertia to the training procedure, and simply adds a fraction of the previous weight update to the current one calculated by error back propagat= ion.
In neu= ral networks used, output signals of neurons are transformed by a sigmoid activation function that has a range between 0 and 1. In order to approxima= te different ranges of values, output of the neural network are scaled by an affine function that maps the [-1, 1] interval to some other interval, say = [a, b]. Only values within that interval are attainable by the neural netwo= rk approximation. The question that arises is how to appropriately choose the interval. An obvious strategy would be to choose a very large interval, lar= ge enough to include any value that could be possibly expected to be generated= by the approximated function on the interval of interest. However, making this= interval too large can reduce the approximation capabilities of the neural network d= ue to wasting large portions of the output interval that would never contain realistic values. The interval attainable by the outputs (and thus the mapp= ing function) should therefore be set such that there is some space beyond the values ever expected to be attained by the original (approximated) function, yet not too big.
It is = obvious that the interval should at least contain all the values generated by sampl= ing, i.e. all function values contained in the training data. However, since it = is likely that some function values on the sampled interval exceed the interval that contains all sampled values, some extension of the interval should be provided beyond that, and this is what we call the safety factor. User can = vary the output safety factor from 1 upwards by inserting the factor into the te= xt field next to the “Output Safety Factor” label. <= /p>
As far= as inputs are concerned, any input form minus to plus infinity can be fed to the neur= al network to produce the appropriate output. Inputs are transformed by the sigmoid function before they are sent to the neurons in the next layer. However, since the slope of the transforming function approaches zero far f= rom 0, it is numerically desirable that most of the inputs that will be fed to = the network fall on the interval [-1, 1]. Input values are therefore also scaled in a similar way as outputs, = and the mapping function is specified in such a way that all inputs from the tr= aining data are mapped close to the interval [-1, 1] before they are fed to the ne= ural network. The input safety factor defines how much smaller from the reference [-1, 1] interval is the interval to which all inputs from the training data fall when scaled before being fed to the network.
When t= he training procedure is completed (after clicking the “Generate Data= 221; and then the “Start” buttons), results are shown automatically.= The original function, its neural network approximation calculated by the train= ing procedure, and the training data on basis of which the model is calculated = are plotted on the graph located on the top-right portion of the form. <= /p>
On the= graph below, inside the “Approximation error” box, the course = of different error measures during the training procedure can be shown, plotted against the number of epochs. By clicking on the appropriate radio button y= ou can choose which error measure is shown on the graph, while final values for all error measures are written in the appropriate boxes.

Figure 4:
Convergence of different types of errors.
Max
Training Error
Maximum absolute error calculated on complete training data set after each set of epochs (Epochs in Bundle) and shown on graph
Rms
Training Error
Root mean square error calculated on complete training data set after each set of epochs (Epochs in Bundle) and shown on graph
Max
Verification Error
Maximum absolute error calculated on complete verification data set after each set of epochs (Epochs in Bundle) = and shown on graph. Verification data are not used in training procedure, but a= re separated from training data and used just for verification the accuracy of= ANN during and after training.
Rms
Verification Error
Root mean square error calculated on complete verification data set after each set of epochs (Epochs in Bundle) = and shown on graph.

Figure 2:=
span> GUI for approximation of functio=
ns of
two variables.
Explor= ing 2D approximations is just as simple as exploring 2D approximations. You can st= art without any special knowledge, just click on the “2D Approximation= ” tab in order to display the appropriate panel (Figure 2= ), and then the “Generate Data”= and then the “Start” button. Before clicking on the ̶= 0;Start” button, you should wait until the graphic window showing the approximated function and sampled data is displayed, and close the window. Graphs are shown in a separate windows in order to allow better quality and interaction with 3D graphs.
When t= he training procedure completes, the graph showing the result is open immediat= ely in a new window. You can interact with the graph (rotate it, zoom in or out, and translate it) in order to view interesting details from the best perspective, and you should close the graph before performing other actions= .
The fu= nction to be approximated is defined similarly as for 1D approximation (see Section <= span style=3D'mso-field-code:" REF _Ref349776721 \\n \\h "'>2.2.1). In the case of user defined function, the expression defining the function can contain two independent variables deno= ted by x and y, respectively. Since there are two variables, four interval bounds must be defined in the “Limits” box.
Everyt= hing is similar rather similar than in 1D approximation, and some differences will = be outlined below. Also the number of sampled values is defined by two values instead of one. If you clear the “Random Training Data” checkbox then values are sampled on a regular grid of points, where the text field beside “Samples on X axis” defines number of point= s on the grid in x direction, and “Samples on Y axis” defines number of points on the grid in the perpendicular (y) direction. If the “Random Training Data” checkbox is checked then sampling points= are generated randomly (with uniform distribution) and the two directions are n= ot distinguished what concerns the generated points. The total number of the generated sampling points equals to the product of both numbers, but their order is interchangeable.
Things=
related
to the network architecture and parameters of the training algorithm are
defined in the same way as in the case of 1D approximation (see Section
Just t= ake into account that in general, in the case of two independent variables, more deg= rees of freedom are needed for the same quality of approximation as in 1D. This means that the number of neurons in each hidden layer should usually be hig= her than in the 1D case, but consequently also the number of sampling points sh= ould be higher. To be precise, this is a bit simplified and overly generalized statement since the optimal number of neurons varies from case to case and depends primarily on the complexity of the function to be approximated. However, on average over a large range of functions, the statement is valid= .

Figure 3:=
span> Results of a 2 parametric approx=
imation
are shown as 3D graph. Right-hand side: window that shows the original func=
tion
sand training data. 3D graphics is rendered by the VTK graphical engine.
Traini= ng data and the approximated function are shown, after clicking on the “Gener= ate data”, in a separate interactive 3D window. In a similar way, the resulting approximation is shown, together with the approximated function a= nd training data, after clicking the “Start” button (Figure 3). You user can interact with 3D graphs by mouse a= nd keyboard in order to adjust the viewing angles, position and zoom as desire= d. Close the window after viewing is done, since failing to do so may result in undesirable effects, depending on the operating system.
You ca= n interact with the graphs by using mouse and keyboards. Click on the window in order = to make it active and interact with it. The relevant commands are listed below= :
· r – resets the viewpoint along the viewing directi= on such that all graphic objects are within the viewing areal. This centers and zooms the view appropriately. If the graphics window opens and nothing is visible, press the r key because objects may just be out of scope.= span>
· j / t – toggles between joystick or trackball mode. In joystick mode, motion occurs continuously as long as mouse button = is pressed. In trackball mode, motion occurs only during dragging (when the mo= use button is pressed and the mouse cursor moves). The trackball mode is more suitable for finely adjusting the view.
· left m= ouse button – rotates the camera (in the cam= era mode, switched on by pressing the c key) or graphic objects (in the actor mode, switched on by pressing a). The camera is rotated around= the focal point (which can be repositioned by pressing the f key over a graphic primitive). In trackball mode, you must press the button and move t= he mouse cursor, while in joystick mode pressing the button is enough (switch between the two modes by pressing the t or the j key).=
· Ctrl += left mouse button – rotates the camera (in c= amera mode, switched on by presing the c key) or graphic objects (or actor= s, switched on by pressing the a keys) around the axis perpendicular to= the projection plane.
· Shift = + left mouse button – pans the camera (moves it parallel to the projection plane) in camera mode or translates the graphic objects (actors) in actor mode. Does the same as the middle mouse button.= span>
· middle= mouse button – pans the camera (moves it para= llel to the projection plane) in camera mode (switched on by pressing the c= i> key) or translates the graphic objects (actors) in actor mode (the a key). In joystick mode, translation is from the center of viewport toward t= he mouse cursor position (toggled by pressing j for joystick and t= i> for trackball mode). Camera mode is switched on by pressing the c key and actor mode by pressing the a key.
· right = mouse button – zooms the camera (in camera mo= de, switched on by pressing the c key) or the graphic objects (actors). = In joystick mode (switched on by pressing the j key) the direction of z= oom depends on whether the mouse cursor is located in the top half of the windo= w or viewport (zooms in) or in the lower half (zooms out) while the zoom speed depends on the distance from the horizontal centerline.
· mouse = wheel – zooms in or out, dependent on the direction of rotati= on.
·
f – flies to the point under the mouse cursor, and=
sets
the focal point on it (rotations then occur around that point). This command
works only when the mouse cursor is located closely enough over a graphic
primitive (auxiliary objects such as axes do not count). It centers the gra=
ph
around the point under the mouse cursor. If later on the graph is translate=
d,
the point of rotation still remains on the same point until the next ”
fly to” operation is performed.
· c / a – toggles between camera and a= ctor modes. In camera mode, user interaction affects the camera position and foc= al point, while in actor mode it affects the object (“actor” in VTK terminology) under the mouse pointer. Therefore in actor mode one can rotate and translate different objects independently. This enables e.g. separation= of different graphic objects. You can press a, then click on the graph = of the original function with the middle mouse button and drag it away from ot= her objects, then press c and center the view around the graphs of the approximation and sampled points in order to view them together, without the original function.
· p – pick operation. This shows a wire frame around the ob= ject under the mouse cursor. This may turn useful in rare occasions in order to = see to which object the graphic primitive under the cursor belongs, e.g. when separating objects in actor mode.
· s / w – all objects become represented as surfaces= / in wire frame. You will seldom need this command.
· e / q – exits / quits the graphic window. <= /span>
User i=
nteraction
capabilities can be used to precisely position the view in order to make
interesting details appear more obvious, but also for achieving some special
effect. Figure
In the= camera mode (switched on by pressing the c key), all graphic objects stand still and the camera rotates and translates around. In the actor mode (swit= ched on by pressing the a key), camera remains still and the selected gra= phic object (which may consist of many graphic primitives, such as the whole sur= face in the above described case) moves and rotates (thus it can also change position with respect to other objects).

Figure 4:=
span> Separation of the graph of the o=
riginal
function and sampling points (left-hand side) and approximation (right-hand
side) by using the actor mode. Case is the same as shown in Fig=
ure 3.
= &nb= sp; = &nb= sp; = &nb= sp; =
This is equivalent as in 1D case (Section 2.2.4). Final values of different error measures are sh= own in the “Approximation error” box and course of the selec= ted error measure is shown on the graph on the right of error values. Radio but= tons are used to select the error measure to be shown on the graph ().

Figure 4:
Convergence of different types of errors.
Max
Training Error
Maximum absolute error calculated on complete training data set after each set of epochs (Epochs in Bundle) and shown on graph
Rms
Training Error
Root mean square error calculated on complete training data set after each set of epochs (Epochs in Bundle) and shown on graph
Max
Verification Error
Maximum absolute error calculated on complete verification data set after each set of epochs (Epochs in Bundle) = and shown on graph. Verification data are not used in training procedure, but a= re separated from training data and used just for verification the accuracy of= ANN during and after training.
Rms
Verification Error
Root mean square error calculated on complete verification data set after each set of epochs (Epochs in Bundle) = and shown on graph.

Figure 5:
Accessing help and other information.

Figure 6: Neur=
App has a very intuitive user interface. All you have=
to
remember is to press the “Generate Data” and then the
“Start” button. Other things are for changing parameters of the
training procedure, bounds for the training data, and the function to be
approximated.

Figure 7: After generation of data, you can immediately s=
ee the
graph of the original function and data that will be used for training.

Figure 8: After pressing the “Start” button,
training of the neural network begins, and the result is shown immediately
after training is complete. It is easy to compare the generated ANN model w=
ith
the original function. Smaller graph shows how error was reduced during the
training procedure, making possible to correct some parameters such as
tolerances and maximal number of epochs if results are not as expected.

Figure 9: Example 3D plot of the original function of 2
variables and the generated training data on the surface of the function gr=
aph.

Figure 10: After training of the neural network is complet=
e, a
surface plot of the approximated function is shown together with the origin=
al
function and with the generated training data. Error magnitude is shown in
color scale.

Figure 11: One can interactively adjust the viewing angle =
and
zoom in order to inspect properties of the approximation in detail.<=
span
lang=3DEN-US style=3D'font-weight:normal'>
Softwa= re was developed by Igor Grešovnik and Tadej Kodelja as an easy to use and intuitive tool for exploring features of approximation by the artificial ne= ural networks (ANN). Authors have developed the software while working on ANN-ba= sed model of process parameters in material production in the Centre of Excelle= nce for Biosensors, Instrumentation and Process Control (COBIK). In the scope of h= is work at COBIK, Tadej Kodelja was also working on his Ph.D. thesis under supervis= ion of Prof. Božidar Šarler and Igor Grešovnik.
The so= ftware is based on the Investigative Generic Library (IGLib), a framewo= rk library for development of technical applications. The library itself is ba= sed on the .NET framework and relies on a number of third party libraries. It h= as been used by several commercial software projects as well as in research projects. Among the others, it serves as code base for the software for industrial ANN-based modeling, which is developed by the authors of NeurApp. This is a complex software used as support to ta= ckle most challenging ANN modeling problems in industry. It requires some expert= ise and skills to use that software, but with the benefit of a powerful and flexible problem solving tool. The software is used internally for performi= ng research and providing services to industry, but authors are open for all k= inds of collaboration where the software can be utilized.
Motiva= tion for development of the software came partially from authors’ personal nee= ds, since authors wanted to gain better insight in features and function of the ANN-based approximations, gained from practical experience and on fast and intuitive way. Therefore they devised a software where one can conceive arbitrary functions of one and two variables, input these functions through= a graphical user interface and set them as the reference functions for ANN-ba= sed models, sample function values on a regular grid or in randomly distributed values, train the neural network to obtain an approximate model, and inspect properties of the obtained model by observing various statistics and by vis= ual comparison of the model and the original reference function. In this way, it is easy to develop an intuitive feeling for the capabilities and limitations of the ANN-based modeling.
Additi= onal motivation was in creating a tool that could be easily utilized to demonstr= ate some features of the artificial neural networks to our colleagues. Often pe= ople have wrong expectations about the ANN-based modeling. Many times they heavi= ly overestimate the capabilities of the neural network modeling, considering t= his tool as a kind of Panacea, an almighty remedy for all problems. It is somet= imes difficult to explain that ANNs are just an ordinary (though very strong in = some aspects) utility for black box modeling. As such, it crucially depends on t= he availability and quality of data, thus being susceptible to many kinds of d= ata deficiency and absolutely adherent to the “garbage in, garbage out= 221; principle. By a tool such as NeurApp, it is easier to plastically demonstrate how ANN-based modeling works.
Below =
there is a
collection of selected references and links that are in some way related to=
the
NeurApp software. They are mainly related to the
authors’ past work that brought to the situation where NeurApp was created. References [1] - [3] are some papers published within the scope of our
work at the . [
Refere= nces [12] and [13] are links to the institution Centre of Excellence for biosensors, instrumentation and process control (= COBIK) and the laboratory where the authors currently work. Within COBIK, authors = are developing methodologies for application of artificial neural networks to modeling of industrial processes. The problem solving software developed in= the scope of this work is based on the IGLib ([10]). This is also the case with NeurApp, which is just a side product of the above work initially developed as a simple tool for authors’ training in basic f= eatures of ANN-based approximation.
The mo= re professional software used for actual research & industrial work is qui= te complex and requires a high level of expertise from the user, and it is interpreter - centered rather than a GUI - based. It is used internally for performing research tasks and to offer services to industry, with interface= s to a number of partner software and with some specialized interfaces for users= in partner institutions. This software is designed to be highly flexible and quickly adaptable to new problems and customer requirements. It does not targeted any specific range of tasks, but is a good base for a wide range of specialized modules or applications for specific and well defined tasks. The software is not available for purchase, but authors will welcome any sugges= tion for collaboration on research, development or industrial projects or request for development of software needed, where they can provide reliable and professional service.
[1=
] Grešovnik,
[2]  =
; Grešovnik,
[3] Grešovnik,
[4] Gresovnik, =
I.
(2000): A General Purpose Computational Shell for Solving Inverse and
Optimisation Problems - Applications to Metal Forming Processes, Ph. D. the=
sis,
University of Wales Swansea, U.K.
[5] Grešovnik,
[6] Grešovnik,=
[7] Grešovnik,=
[8] Grešovnik,=
[9] Grešovnik,
[10] Grešovnik,
[11] Aforge.Net. (2012): Artifi=
cial
intelligence library. Available at: http://www.aforgenet.com/=
.
[12] Centre of Excellence for b=
iosensors,
instrumentation and process control (COBIK) http://www.cobik.si/index?lang=3Deng.
[13] Laboratory for Advanced Materials Systems, COBIK: http://www.cobik.si/laboratoriji/laboratorij-za-sisteme-z-napr= ednimi-materiali?lang=3Deng
|
1. Introduction &nbs= p; &= nbsp; &nbs= p; &= nbsp; &nbs= p; &= nbsp; &nbs= p; &= nbsp; 2.1. Installation and Use |
Igor
Grešovnik & Tadej=
span> Kodelja
NeurApp
About<=
span
lang=3DSL style=3D'mso-ansi-language:SL;mso-no-proof:yes'> the Software &n=
bsp;  =
; &n=
bsp;  =
; &n=
bsp;  =
; &n=
bsp;  =
; &n=
bsp; NeurApp=
References &n=
bsp;  =
; &n=
bsp;  =
; &n=
bsp;  =
; &n=
bsp;  =
; &n=
bsp;  =
; NeurApp
Sandbox &nb=
sp; =
&nb=
sp; =
&nb=
sp; =
&nb=
sp; =
&nb=
sp;