DNNE MATLAB Package

Fast Decorrelated Neural Network Ensembles

 

Overview

The assurance of well-balanced trade-off between the bias and variance of single RVFL learner is a difficult task due to the uncertainties in the learning process that are caused by the random initializations of basis functions. Different initial settings for the RVFL networks lead to different solutions. In order to reduce the impact of these factors on the generalization error of a learning system, a cluster of RVFL networks are combined together to produce efficient predictions. DNNE is a new ensemble learning approach that uses RVFL networks as ensemble components and it is fitted in negative correlation learning framework. Since RVFL networks do not require learning all their parameters (basis functions are set randomly), we seek a simple and fast solution to calculate the output weights of the base RVFL networks. This solution should take into account the correlation among the base RVFL networks in the output space and make sure it is at minimum. Although DNNE aims at encouraging the diversity among ensemble components by reducing the correlation among their outputs, it still maintains an overall good ensemble accuracy. The following pseudocode illustrates the DNNE learning algorithm. For more information about DNNE, see the original paper.

Installation and Usage

Download the DNNE MATLAB package from here. To use DNNE MATLAB code, you can copy the m-files to your working directory or add the m-files folder to your MATLAB search path as it is shown in Mathworks website. The DNNE package is composed of four m-files: newdnne.m, traindnne.m, simdnne.m and dnne.m. Each m-file stores one function that creates, trains, stimulates and evaluates DNNE model. Next, we explain the usage of these functions
  • newdnne is used to create a DNNE model.
    dnne = newdnne(ensSize, baseSize, X, T, lambda) RETURNS an empty DNNE model. It creates a RVFL neural network ensemble of size ensSize and all its parameters are set to zero.
    The number of hidden neurons in each base RFVL network is baseSize.
    X and T are the input and targets training vectors, respectively. Each row in these two matrices represent one input-target instance. The number of columns in these two matrices determine the number of inputs and outputs for the DNNE model.
    lambda is a regularization factor that is used in the DNNE learning algorithm.

  • traindnne runs the DNNE learning algorithm to optimize the ensemble model parameters.
    [dnne, rmse] = traindnne(dnneInit, X, T) RETURNS a trained DNNE model dnne and the root mean square error rmse as an estimate to the learning error.
    It takes an empty DNNE model dnneInit, input feature vectors X that is N x n matrix and target vectors T that is N x t matrix

  • simdnne simulates the DNNE model on a given input data.
    T = simdnne(dnne, X, method) RETURNS the prediction values in case of regression problems or class labels in case of classification T. It takes a trained DNNE model dnne as an argument along with input feature vectors X. Each row in X represents one input instance.
    method determines whether a regressor or classifier predictions are computed. It takes one of these two values 'reg' or 'class'.

  • dnne is the main function to create a DNNE model using a training data and then evaluate is performance using a testing data. It returns the training and testing accuracies as the RMSE for regresion problems and classification accuracy for classification datasets.
    It can be invoked in several ways as follows
    [dnneModel, TrainingAccuracy, TestingAccuracy, TrainingRMSE] = dnne(ensSize, baseSize, lambda, trainData) if you do not want to use testing dataset.

    [dnneModel, TrainingAccuracy, TestingAccuracy, TrainingRMSE] = dnne(ensSize, baseSize, lambda, trainData, testData) if it is a regression problem and you do not want to specify a random stream.

    [dnneModel, TrainingAccuracy, TestingAccuracy, TrainingRMSE] = dnne(ensSize, baseSize, lambda, trainData, testData, method, randStream) ALLOWS users to use specific Random Stream to initialize the random numbers generator. This allows users to have a control on the randomness of the generated DNNE models.

Practical Examples

Computer Activity Dataset (Regression)

Computer Activity dataset describes the portion of time that CPUs run in user-mode, based on 8192 computer system activities collected from a Sun SPARCstation 20/712 with 2 CPUs and 128 MB of memory running in a multi-user university department, and each system activity is evaluated using 12 system measures (number of reads and number of writes between system memory and user memory, number of system calls of all types, number of system read calls, number of system write calls, number of system fork calls, number of system exec calls, number of characters transferred by read calls, number of characters transferred by write calls, process run queue size, number of memory pages available to user processes, and number of disk blocks available for page swapping).
Number of Attributes: 12
Number of Instances: 8,192

The approximated function

clear all;

data = csvread('data/computer_activity.data');

X = data(:, 2:end);

T = data(:, 1);

 

dnne = newdnne(5, 70, X, T, 0.5);

 

[dnne, rmse] = traindnne(dnne, X, T);

 

netOut = simdnne(dnne, X);

 

rmse1 = sqrt(sum((T - netOut).^2) / size(T,1));

Calfornia Housing Dataset (Regression)

California Housing dataset contains 20,640 observations for estimating the median house prices in California, U.S. Each characterized by eight continuous features (median income, housing median age, total rooms, total bedrooms, population, households, latitude, and longitude).
Number of Attributes: 8
Number of Instances: 20,640

The approximated function

clear all;

data = csvread('data/calhousing.data');

X = data(:, 2:end);

T = data(:, 1);

 

dnne = newdnne(5, 50, X, T, 0.55);

 

[dnne, rmse] = traindnne(dnne, X, T);

 

netOut = simdnne(dnne, X);

 

rmse1 = sqrt(sum((T - netOut).^2) / size(T,1));

German Credit Dataset (Binary Classification)

Number of Attributes: 24
Number of Instances: 1,000
Number of Classes: 2

clear all;

data = csvread('data/credit_german.data');

X = data(:, 2:end);

TOrig = data(:, 1);

noClasses = max(TOrig);

T = ones(size(TOrig,1), noClasses) * -1;

for i=1:noClasses

    T(TOrig == i, i) = 1;

end

clear noClasses data i;

   

dnne = newdnne(5, 100, X, T, 0.55);

 

[dnne, rmse] = traindnne(dnne, X, T);

 

predLabels = simdnne(dnne, X, 'class');

 

acc = sum(TOrig == predLabels) / size(TOrig,1) * 100;

Led-7 Dataset (Multi-class Classification)

This simple domain contains 7 Boolean attributes and 10 concepts, the set of decimal digits. Recall that LED displays contain 7 light-emitting diodes and hence the reason for 7 attributes. The problem would be easy if not for the introduction of noise. In this case, each attribute value has the 10% probability of having its value inverted. All attribute values are either 0 or 1, according to whether the corresponding light is on or not for the decimal digit. Each attribute (excluding the class attribute, which is an integer ranging between 0 and 9 inclusive) has a 10% percent chance of being inverted.
Number of Attributes: 7
Number of Instances: 2,000
Number of Classes: 10

clear all;

data = csvread('data/led_7.data');

X = data(:, 2:end);

TOrig = data(:, 1);

if min(TOrig) == 1

    noClasses = max(TOrig);

    T = ones(size(TOrig,1), noClasses) * -1;

    for i=1:noClasses

        T(TOrig == i, i) = 1;

    end

elseif min(TOrig) == 0

    noClasses = max(TOrig) + 1;

    T = ones(size(TOrig,1), noClasses) * -1;

    for i=1:noClasses

        T(TOrig == i - 1, i) = 1;

    end

end

 

clear noClasses data i;

  

dnne = newdnne(5, 50, X, T, 0.55);

 

[dnne, rmse] = traindnne(dnne, X, T);

 

predLabels = simdnne(dnne, X, 'class');

if min(TOrig) == 1

    predLabels = simdnne(dnne, X, 'class');

elseif min(TOrig) == 0

    predLabels = simdnne(dnne, X, 'class') - 1;

end

 

acc = sum(TOrig == predLabels) / size(TOrig,1) * 100;

Evaluation Example

In this example, we show how to evaluate the performance of a DNNE model on testing dataset that is extracted from the original dataset. This example mainly shows how to use the dnne m-function. We show this on two datasets that model regression and classification problems.

clear all;

s = RandStream('mt19937ar','Seed',1986);

for i=1:10

    % California Housing Dataset (REGRESSION)

    data = csvread('data/calhousing.data');

    indexes = randperm(s, size(data,1));

    t = ceil(0.60 * size(data,1));

    trainData = data(indexes(1:t), :);

    testData =  data(indexes(t+1:end), :);

    [housing_dnneModel{i}, housing_trnAcc(i), housing_tstAcc(i), housing_rmse(i)] = dnne(5, 50, 0.55, trainData, testData, 'reg', s);

 

    % German Credit Card Dataset (CLASSIFICATION)

    data = csvread('data/credit_german.data');

    indexes = randperm(s, size(data,1));

    t = ceil(0.60 * size(data,1));

    trainData = data(indexes(1:t), :);

    testData =  data(indexes(t+1:end), :);

    [credit_dnneModel{i}, credit_trnAcc(i), credit_tstAcc(i), credit_rmse(i)] = dnne(5, 100, 0.55, trainData, testData, 'class', s);

end

clear data trainData testData s indexes t i


Development Notice

This software package has been developed based on the paper
M. Alhamdoosh and D. H. Wang, Fast decorrelated neural network ensembles with random weights, Information Sciences, Vol. 264, pp. 104-117, 2014.

For technical support and/or help, please contact m.hamdoosh [at] gmail [dot] com