Linear SVMs

The internal methods of these classifiers are beyond the scope of this class, although we will discuss some of aspects. At a practical level, we can think of these as advanced classifiers that in fact are rooted in statistical decision theory from which LDA methods arose. Let’s first consider ‘linear’ SVMs. These are in fact nearly identical to LDA, but instead of computing the discriminant and finding the best decision criterion, it uses other optimization methods to determine a hyperplane that best discriminates the groups, minimizing errors. But usually there are many possible hypeplanes that are equally good–ranges of slopes in each direction. The approach of an SVM–that gives the SVM its name–is that it tries to find a set of points on the boundary between the two classes. This set of points are usually a small subset of the total number of points available, so they are more efficient to work with. This set provides the support to model a boundary (the support vector). Linear SVMs attempt to find a plane that optimally separates these points.

Let’s look at the engineering data set first. We will run a classifier on the engineering data first. Because the predicted variable is numeric, we will make it a factor so the svm knows to predict the class. The svm function needs to be told to use a linear svm, which seems reasonable in this case.

library(e1071)
library(DAAG)
joint <- read.csv("eng-joint.csv")[, -1]
joint$eng <- as.factor(joint$eng)
s1 <- svm(y = joint$eng, x = joint[, -1], scale = T, kernel = "linear")

coef(s1)

(Intercept)         eer         eeu          ep         ppr         ppu 
 0.05154417 -0.15567504  0.16576103  0.36205481  0.25004473  0.57847773 
      eer.1       eeu.1        ep.1       ppr.1       ppu.1 
 0.10664820 -0.31357906  0.60989730 -0.43790979  0.40541747 

summary(s1)

Call:
svm.default(x = joint[, -1], y = joint$eng, scale = T, kernel = "linear")


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  linear 
       cost:  1 

Number of Support Vectors:  67

 ( 33 34 )


Number of Classes:  2 

Levels: 
 0 1

confusion(joint$eng, predict(s1, data = joint))

Overall accuracy = 0.697 

Confusion matrix 
      Predicted (cv)
Actual     0     1
     0 0.684 0.316
     1 0.289 0.711

First, consider the arguments we gave. We specified that the variables should be rescaled–this makes sense here because the variables are on substantially different scales. Next, the svm used C-classification. This is the default for predicting factor values–if you give a numeric value to predict, it will implement a regression. You can also specify nu-classification which is essentially the same but uses a different parameter. Next, we used a linear kernel Different types of data (networks, sparse text-document matrices, images, etc.) are usually best fit by one or another of the available kernels. Next, the model specifies its ‘support vectors’. This is in a sense a number of parameters used to make the decisions. For linear classifiers, we can extract the coefficients–the actual plane used to discriminate between

The kernel trick for non-linear classifiers

If you cannot separate cleanly with a linear classifier, svm allows use of kernels (called the kernel trick). A kernel is just a way of transforming the features into a higher-dimensional space. Different kernels provide different higher-dimensional spaces, but essentially you can create new dimensions by combining existing features, enabling interactions and other complex classification patterns. A linear boundary on those new dimensions may produce a better classification. So the same SVM algorithm can apply after we generate a higher-dimensional representation. So rather than, for example, trying to find a curve that separates two classes of data, you transform the data into a higher-dimensional space that curves the data, and fit a linear separation in that space.

Polynomial kernel

A polynomial kernel provides something like polynomial regression. It might be worthwhile to examine a polynomial kernel, which is akin to QDA. We can see the fit improves to 79%

s1.pol <- svm(y = joint$eng, x = joint[, -1], kernel = "polynomial", scale = T)
s1.pol

Call:
svm.default(x = joint[, -1], y = joint$eng, scale = T, kernel = "polynomial")


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  polynomial 
       cost:  1 
     degree:  3 
     coef.0:  0 

Number of Support Vectors:  74

confusion(joint$eng, (predict(s1.pol)))

Overall accuracy = 0.789 

Confusion matrix 
      Predicted (cv)
Actual     0     1
     0 1.000 0.000
     1 0.421 0.579

The linear SVM does not do as well, while the polynomial performs better. But it gets 100% accuracy on the 0-0 cases, and does much more poorly on the 1-1 case. But these are as good or better than any of the other methods. The C-classification has two parameters that control the solution, the cost (C) of mis-classification and \(\gamma\). By increasing the cost function, we can punish errors more and force the model to fit the data better:

s1.pol <- svm(y = joint$eng, x = joint[, -1], kernel = "polynomial", scale = T, cost = 100)
s1.pol

Call:
svm.default(x = joint[, -1], y = joint$eng, scale = T, kernel = "polynomial", 
    cost = 100)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  polynomial 
       cost:  100 
     degree:  3 
     coef.0:  0 

Number of Support Vectors:  60

confusion(joint$eng, (predict(s1.pol)))

Overall accuracy = 1 

Confusion matrix 
      Predicted (cv)
Actual 0 1
     0 1 0
     1 0 1

Cross-validation

The high error cost is likely leading to overfitting. It is also typical to use cross-validation to help determine reasonable values for the decision bounds. This svm library will do n-fold crossvalidation by specifying a cross function. Maybe that will help with overfitting.

s1.pol <- svm(y = joint$eng, x = joint[, -1], kernel = "polynomial", scale = T, cross = 10,
    cost = 100)
s1.pol$tot.accuracy

[1] 43.42105

confusion(joint$eng, (predict(s1.pol)))

Overall accuracy = 1 

Confusion matrix 
      Predicted (cv)
Actual 0 1
     0 1 0
     1 0 1

It does not.
##

RBF and sigmoid kernels

The radial basis function kernel is useful for especially for images, as it is sort of a guassian transformation of each pair of features. Similarly, a sigmoid kernel enables other

Exercise: iPhone data

phone <- read.csv("data_study1.csv")
phone$Smartphone <- factor(phone$Smartphone)
phone$Gender <- as.numeric(as.factor(phone$Gender))
phone.svm <- svm(y = phone$Smartphone, x = phone[, -1], scale = T, cross = 50)
phone.svm

Call:
svm.default(x = phone[, -1], y = phone$Smartphone, scale = T, cross = 50)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  radial 
       cost:  1 

Number of Support Vectors:  429

summary(phone.svm)

Call:
svm.default(x = phone[, -1], y = phone$Smartphone, scale = T, cross = 50)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  radial 
       cost:  1 

Number of Support Vectors:  429

 ( 227 202 )


Number of Classes:  2 

Levels: 
 Android iPhone

50-fold cross-validation on training data:

Total Accuracy: 63.89414 
Single Accuracies:
 90 72.72727 50 72.72727 90 45.45455 54.54545 60 72.72727 80 54.54545 50 45.45455 81.81818 90 36.36364 70 81.81818 45.45455 60 81.81818 70 36.36364 60 81.81818 100 70 36.36364 70 54.54545 80 36.36364 54.54545 80 81.81818 40 63.63636 72.72727 60 45.45455 80 45.45455 70 72.72727 54.54545 70 36.36364 60 63.63636 72.72727 

Here, we are about as good as other methods using cross-validation. It again would be worthwhile to select the C and gamma parameters via a grid search.

Exercise: MNist data

SVMs and the RBF kernel are especially well-suited for image data. We can use the image data, but this svm does not like features that have 0 variance, so let’s remove them.

train <- (read.csv("trainmnist.csv"))
var <- apply(train, 2, sd)
train2 <- train[, var > 0]
test <- as.matrix(read.csv("testmnist.csv")[, var > 0])
class <- as.factor(rep(c("0", "1"), each = 250))
mn.svm <- svm(y = class, x = train2, kernel = "radial", scale = F, cross = 10)
table(class, predict(mn.svm, test))

     
class   0   1
    0 249   1
    1   2 248