There are three key issues in HHMMs: 1. For Problem 2, it is the one in which we attempt to uncover the hidden part of the model, i. In many practical situations, we use an optimality criteria to solve the problem as good as possible. For a complete tutorial on hidden Markov model, we refer readers to the paper by Rabiner [] and the book by MacDonald and Zucchini [].

Similar to the case of Markov chain, MDP is a system that can move from one distinguished state to any other possible states. The obvious problem that the decision maker facing is to determine a suitable plan of actions so that the overall gain is optimized. The process of MDP is summarized as follows: i At time t, a certain state i of the Markov chain is observed.

A policy D is a rule of taking actions. It prescribes all the decisions that should be made throughout the process.

For the case of one-period remaining, i. The model can be easily extended to a more general situation, the process having n transitions remained. The queueing system is a classical application of continuous Markov chain. We then present an important numerical algorithm based on computation of Markov chain for ranking the webpages in the Web.

This is a modern applications of Markov though the numerical methods used are classical. The main assumptions of a Markovian queueing system are the Poisson arrival process and exponential service time. The one-server system discussed in the previous section is a queueing system without waiting space. In the following sections, we will introduce some more Markovian queueing systems. Queueing system is a classical application of continuous time Markov chain.

- Markov chains. Models, algorithms and applications.
- Markov Chains: models, algorithms and applications — Macquarie University.
- Similar titles.
- Models, Algorithms and Applications.
- Markov Chains - Models, Algorithms and Applications | Wai-Ki Ching | Springer.
- Shakespeare: Poet and Citizen?

We will further discuss its applications in re-manufacturing systems in Chapter 3. For more details about numerical solutions for queueing system and Markov chain, we refer the read to the books by Ching [52], Leonard [], Neuts [, ] and Stewart []. Otherwise, the customer has to leave the queueing system. To describe the queueing system, we use the number of customers in the queue to represent the state of the system.

There are n states, namely 38 2 Queueing Systems and the Web 0, 1,. The Markov chain for the queueing system is given in Fig.

- Link Prediction and Path Analysis Using Markov Chains?
- Ng / Ching | Markov Chains: Models, Algorithms and Applications | | !
- Top Authors!
- Twilight of the Idols: Hollywood and the Human Sciences in 1920s America!
- MINERALS AND COAL PROCESS CALCULATIONS.
- Bibliographic Information!
- ADVERTISEMENT;

The number of customers in the system is used to represent the states in the Markov chain. Clearly it is an irreducible Markov chain. The Markov chain for the one-queue system. Example 2. Otherwise, the customer has to leave the system.

## Markov Chains: Models, Algorithms and Applications - Wai-Ki Ching, Michael K. Ng - Google книги

To apply the continuous time Markov chain for model this queueing system, one has to obtain the waiting for one departure of customer when there are more than one customer let us say k customers in the queueing system. We need the following lemma Lemma 2. Suppose that X1 , X2 ,. Then X 1 is again exponentially distributed with mean the original random variables.

If we use the number of customers in the queue to represent the state of the system. There are n states, namely 0, 1,. Suppose that there are two one-queue systems as discussed in Section 2. Thus this is a two-dimensional queueing model. In this case, the generator matrix is given by the following matrix: 2.

Here en2 is the unit vector 0, 0,. Unfortunately, there is no analytical solution for the generator matrix A4. In fact, there are a lot applications related to queueing systems whose problem size are very large [34, 35, 36, 43, 44, 52, 80]. Direct methods for solving the the probability distribution such as the Gaussian elimination and LU factorization can be found in [, ]. Another popular method is called the matrix analytic methods [].

Apart from the direct methods, another class of popular numerical methods is called the iterative methods. Sometimes when the generator matrix has block structure, block Jacobi method, block Gauss-Seidel method and block SOR method are also popular methods []. A hybrid numerical algorithm which combines both SOR and genetic algorithm has been also introduced by Ching et al [] for solving queueing systems. Therefore iterative method such as CG method can be a good solver with a suitable preconditioner. Circulant-based Preconditioners In this subsection, we illustrate how to get a circulant preconditioner from a generator matrix of a queueing system.

The generator matrices of the queueing 44 2 Queueing Systems and the Web networks can be written in terms of the sum of tensor products of matrices. Moreover, c Q can be diagonalized by the discrete Fourier Transformation and closed form solution of its eigenvalues can be easily obtained. This is important in the convergence rate analysis of CG method. A number of related applications can be found in [43, 44, 48, 50, 52, 55]. Toeplitz-Circulant-based Preconditioners Another class of queueing systems with batch arrivals have been discussed by Chan and Ching in [43].

It is clear that the matrix is dense and the method of circulant approximation does not work directly in this case. A Toeplitz-circulant type of preconditioner was proposed to solve this queueing system Chan and Ching [43]. The idea is that the generator matrix is close to a Toeplitz matrix whose generating function has a zero on the unit circle of order one. By factoring the zero, the quotient has no zero on the unit circle. Using this fact, a Toeplitz-circulant preconditioner is then constructed for the queueing system. Both the construction cost and the preconditioner system can be solved in n log n operations.

Moreover, the preconditioned system was proved to have singular values clustered around one. Hence very fast convergence rate is expected when CG method is applied to solving the preconditioned system. This idea was further applied to queueing systems with batch arrivals and negative customers Ching [54]. Here the role of a negative customer is to remove a number of customers waiting in the queueing system.

## Markov Chains: Models, Algorithms and Applications

For example, one may consider a communication network in which messages are transmitted in a packet-switching mode. When a server fails this corresponds to an arrival of a negative customer during a transmission, part of the messages will be lost. One may also consider a manufacturing system where a negative customer represents a cancellation of a job.

These lead to many practical applications in the modelling of physical systems. Here pi is the probability that an arrived batch is of size i. Furthermore, if the arrived negative customer is supposed to kill i customers in the system but the number of customers in the system is less than i, then the queueing system will become empty. The killing strategy here is to remove the customers in the front of the queue, i. The generator matrices enjoy the same near-Toeplitz structure. Toeplitz-circulant preconditioners can be constructed similarly and the preconditioned systems are proved to have singular values clustered around one, Ching [54].

### Samenvatting

Interested readers may consult the following references Bramble [32], Chan et al. Unfortunately, very often there can be thousands of webpages which are relevant to the queries. Therefore a proper list of the webpages in certain order of importance is necessary. The list should also be updated regularly and frequently. Thus it is important to seek for fast algorithm for the computing the PageRank so as to reduce the time lag of updating.

### Introduction

The reason is not just because of the huge size of the webpages in the Internet but also the size keeps on growing rapidly. PageRank has been proposed by Page et al. Larry Page and Sergey Brin are the founder of Google. And while we have dozens of engineers working to improve every aspect of Google on a daily basis, PageRank continues to provide the basis for all of our web search tools.