Estude fácil! Tem muito documento disponível na Docsity
Ganhe pontos ajudando outros esrudantes ou compre um plano Premium
Prepare-se para as provas
Estude fácil! Tem muito documento disponível na Docsity
Prepare-se para as provas com trabalhos de outros alunos como você, aqui na Docsity
Encontra documentos específicos para os exames da tua universidade
Prepare-se com as videoaulas e exercícios resolvidos criados a partir da grade da sua Universidade
Responda perguntas de provas passadas e avalie sua preparação.
Ganhe pontos para baixar
Ganhe pontos ajudando outros esrudantes ou compre um plano Premium
The concept of hybrid recurrent neural networks (hrnns), which combine the strengths of recurrent neural networks (rnns) and hidden markov models (hmms) for modeling time-varying sequences. The theoretical foundations of rnns, the hybrid architecture of hrnns, and their applications. It also mentions the use of evolutionary neural learning for training rnns and the encoding of symbolic knowledge in recurrent neural networks.
Tipologia: Teses (TCC)
1 / 142
Esta página não é visível na pré-visualização
Não perca as partes importantes!
Modelling of Deterministic, Fuzzy and Probablistic Dynamical Systems
Rohitash Chandra
A Thesis in the Field of Computing Science for the Degree of Master of Science in Computing Science
The University of Fiji
August, 2007
Supervisor: Prof. Christian Omlin.
iv
Author's Biographical Sketch The author is from the Fiji islands. He was born in Nausori and attended Saraswati Primary and Saraswati College for his primary and secondary school education. The author graduated with a Bachelor of Science Degree from the University of the South Pacific in April 2006. He joined the University of Fiji as a tutor in computing science in early 2007. His research work has been published in numerous international conference proceedings in the field of artificial intelligence. Apart from his contribution to the field of computing science, the author has published poetry in a number of international literary journals. He has released two books of poetry in the years of 2006 and 2007: “Barefoot on Soft River Sand” and “A Hot Pot of Roasted Poems”. He is also the editor and publisher of The Blue Fog Journal.
v
To Roni.
Table of Contents ..............................................................................................................vii
1.2 Premises Recurrent neural networks on their own can very well represent dynamical systems such as finite automaton. They have been applied to a wide range of real world problems with dynamical characteristics including speech, signature and gesture recognition [1, 2, and 3]. Hidden Markov models, on the other hand, have been popular tools for modelling speech sequences [6]. Finite state automata have been a useful paradigm for studying recurrent neural networks and their dynamical characteristics. Hybrid systems combine useful features of at least two paradigms. Examples of hybrid systems in machine learning include: evolutionary neural learning, neural expert systems, neuro-fuzzy systems, and symbolic connectionist learning.
1.2.1 Recurrent Neural Networks
Neural networks are loosely modelled on the brain. They learn by training from past experience and can demonstrate good generalization performance when presented with data not initially included in the training process. Neural networks can be divided into two classes: feedforward and recurrent neural networks. Feedforward networks are used in applications where the data does not contain time variant information while recurrent neural networks model time series sequences and possesses dynamical characteristics. Recurrent neural networks contain feedback connections. They have the ability to maintain information from past states for the computation of future state outputs. It has been shown that recurrent neural networks can model non-linear dynamical systems. Recurrent neural networks have been successfully applied to a wide range of applications including speech, gesture and signature recognition [1, 2, and 3]. One limitation of neural networks is the difficulty to train them using gradient descent learning; a network may get trapped in the local minima resulting in poor training and generalization performance.
1.2.2 Hidden Markov Models
Hidden Markov models have been popular tools for automatic speech recognition [6]. In a regular Markov model, the state is directly visible to the observer. Therefore, the state transition probabilities are the only parameters. In a hidden Markov model, the state is not directly visible, however, the variables influenced by the states are visible. Each state has a probability distribution over the possible output tokens. The sequence of tokens generated by a hidden Markov model gives some information about the sequence of states. In a first order hidden Markov model, the state at time t+1 depends only on state at time t , regardless of the states in the previous times [12]. This first-order assumption is generally inappropriate for speech signals where dependencies often extend through several states, however, hidden Markov models have been very successful for certain types of speech recognition [13].
1.2.3 Finite-State Automata and Knowledge Representation
Finite-state automata represent dynamical behaviour and are a useful framework for studying recurrent neural networks as no feature extraction is necessary. A deterministic finite automaton is a finite automaton where one transition to the next state exists for each pair of state and input signal. A deterministic finite automaton reads in a string of input symbols. For each input symbol, it performs a state transition. When the last input symbol has been received, the automaton will either accept or reject the string depending on the output of the state. A fuzzy finite automaton is a finite-state automata where for each pair of state and input signal, there exists a set of possible successor states.
Symbolic or expert knowledge can be inserted into neural networks prior to training for better training and generalization performance. It has been shown that deterministic finite-state automata can be directly encoded into recurrent neural networks
Compared to gradient descent training of neural networks, evolutionary neural learning tends to drive the network out of the local minima resulting in better generalization performance.
1.2.5 Speech Recognition
A speech sequence contains huge amount of irrelevant information. Feature extraction reduces speech to salient which is then used for modelling. Recurrent neural networks and hidden Markov models have been successfully applied to modelling speech sequences [1, 6]. They have been applied to recognize words and phonemes. The performance of speech recognition system can be measured in terms of accuracy and speed. Recurrent neural networks are capable of modelling complicated sequences. They have shown more accuracy in recognition in cases of low quality, noisy data compared to hidden Markov models. However, hidden Markov models have shown to perform better when it comes to large vocabularies. Extensive research on the application of speech recognition has been done for more than forty years, however, scientists are unable to implement systems which can show excellent performance in environments with background noise which come anywhere near human recognition performance.
1.3 Research Hypothesis Modelling of real world time varying sequences such as speech, signature and gesture is difficult. These sequences have dynamical characteristics which can be modelled by recurrent neural networks and hidden Markov models. In Section 1.2, the limitations of both these systems have been discussed. Hybrid systems aim at combining the strengths of different paradigms while, at the same, alleviating respective weaknesses. The combination of recurrent neural networks and hidden Markov models may yield a powerful structure which may deal with the individual limitations of these systems.
Finite state automata represent dynamical behaviour and are useful models for studying recurrent neural networks. Recurrent neural networks can learn and represent finite state automata in their internal states. These issues are addressed through the following hypothesis:
1.3.1 Learning Finite-State Automata
In Section 1.2.1, it has been discussed that recurrent neural networks can represent dynamical systems. Finite-state automata represent dynamical behaviour and are useful frameworks for studying recurrent neural networks as no feature extraction is necessary. Recurrent neural networks can represent deterministic finite automaton in their internal structure upon training from sample strings which represent such automaton.
The hypothesis is that recurrent neural networks can also learn and represent fuzzy finite automaton despite the fact that computation in recurrent neural networks is deterministic.
