Product Classification

The train data set has 61878 rows and 95 columns. Here is a sample of a few rows and columns. The target column has 9 classifiers:

The number of elements in each class is shown below.

To get some data for inspection first create a random sample of 4000 rows. This speeds up calculations.

sample_rows <- sample(1:dim(train_data)[1], 4000)

The sampled train_data has 4000 rows and 95 columns.

and then for plotting etc. convert it to a “long” format with each observation in one unique row.

## Get packages
require(plyr); require(ggplot2); require(tidyr)
## munge data into long format 
long_train<-gather(train_data, feature, data, feat_1:feat_93)

Let's look at the means and std deviations of the features by class….

## use ddply to get means and standard deviations
train_morph<-ddply(long_train, c("target", "feature"), summarize, mean_data = mean(data), sdev_data = sqrt(var(data)))


## calculate inverse coeff of variation 
## (which I will label as z_stat for later use)
## add small value to prevent overflow errors
train_morph$z_stat<-train_morph$mean_data/(train_morph$sdev_data+0.00001)

This is beginning to look promising. We can see at least some variation between classes & features (though others are weak)

Here the mean and 1/CV within a product class is plotted with color coded for the target class.

If we require that 1/CV be above 0.5 (arbitrarily)

This starts to look selective

Decision trees also appear to offer a good way to distinguish.

This shows just a few features (e.g. 17, 78, and 84) can do a pretty good job discriminating between Class_1 and Class_2.

Here's a look at a different pairing (Class_1 and CLass_6)

This shows many more features (e.g. 8, 78, 6, &c.) are needed to discriminate between Class_1 and Class_6. Though feat_78 is common between the two analyses.

Examine more pairwise trees.
Understand if the variable values contribute information.
Run generic random forest as benchmark.

Cheers!