Differences

This shows you the differences between two versions of the page.

--- ece4580:module_recognition [2017/01/29 16:12] – pvela
+++ ece4580:module_recognition [2024/08/20 21:38] (current) – external edit 127.0.0.1
@@ Line 59: / Line 59: @@
 -------------------------
 ===== Module #2: Alternative Classifiers =====
 The vanilla Bag-of-Words algorithm utilizes the //k nearest neighbors// (kNN) algorithm to output the final decision.  As the size of the training set increases, so does the kNN algorithm.  Certainly, we humans don't have that problem.  We don't take longer to classify things even though our corpus of known objects increase from age 1 to age 10 and beyond. kNN doesn't try to distill any inherent structure in the data to simplify the decision process.
 Here we will explore support vector machines (SVM) as a means to perform the final decision.
----------------
-;#;
-[[ECE4580:Modules|ECE4580 Learning Modules]]
-;#;
+__**Week #1: understanding SVM**__ \\
+  - Go to [[https://www.csie.ntu.edu.tw/~cjlin/libsvm/#download|libSVM]] and scroll down to Graphic Interface.
+  - See square in the middle? Select a color by clicking on the "change" button and draw some dots within the square. Then, change another color and draw some dots again. Finally, click on the "run" button, you will find SVM found a boundary to separate these two groups!
+  - Here we would like to use SVM classifier to help us to train a model to predict if a coming image is a car or face by drawing a hyper-plane in feature space, so we no longer have to compare each new image to all the training image(k-nearest neighbor approach in module #1)
+  - Go to "Download LIBSVM" section and download the libraries(select appropriate one based on your OS)
+  - Extract the compressed file, open your Matlab, and browse into the folder, say C:\libsvm
+  - Now type command >> mex -setup
+  - After you select your compiler for MEX-files, get into /matlab folder >> cd('C:\libsvm\matlab')
+  - Compile it >> make
+  - Okay, now your should have SVM libraries in your computer! you can add path by >> addpath('C:\libsvm\matlab')
+  - Now, we'd like to run some toy example from [[http://stackoverflow.com/questions/15557908/how-to-run-libsvm-in-matlab|here]]
+  - >> labels = double(rand(10,1)>0.5);
+  - >> data = rand(10,5);
+  - >> model = svmtrain(labels, data, '-s 0 -t 2 -c 1 -g 0.1')
+  - Now you should be able to understand the basic usages. Please carefully read the README file in the folder and run the example. Output the result of accuracy_l and accuracy_p to demonstrate you've run the example yourself. It is very important your have correctly installed libSVM. We will use libSVM for the next task!
+  - (optional) anyone who is interested in SVM, I highly recommend [[https://www.youtube.com/watch?v=_PwhiWxHK8o|Prof. Winston's online video course]].
+__**Week #2: apply SVM to car and face dataset**__ \\
+  - If you did the toy example in libSVM and understood commands correctly, now you are ready to apply this powerful library on our previous dataset - car and face!.
+  - In the toy example, data is actually a list of feature and it's true label. Now, we would like to use our bag-of-words features here.
+  - Generate bag-of-words feature for car and face as you did in your previous tasks (collect sift features, kmeans, and compute the histogram for each image. We use histogram as the feature for each image).
+  - Mimic the toy example, for each of the image, we set an label for car (and different label for face). Also for each of the image we have the histogram as its feature like you see in toy example.
+  - You have to generate two files, one for training and one for testing.
+  - Use the commands you learned last week, report your SVM accuracy.
+__**Week #3: Kernel Trick**__ \\
+  - So far we've applied SVM on our dataset to distinguish car and face. We are going to learn more about SVM to see how it works.
+  - If you played with the nice SVM GUI [[https://www.csie.ntu.edu.tw/~cjlin/libsvm/|here]] by generating some points in different colors, you can observe that the separating hyperplanes are not always a straight line. Why is it?
+  - Actually, one of the good properties of SVM is that you can apply different "kernels" to project your data to higher dimension in order to keep them apart and separate them.
+  - Please read carefully about [[http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html|Kernel trick]]. Figure.1 is an example that you can use a straight line to separate two groups. Figure 3. is an example that a straight line is not able to separate them. Can you plot another example that you need a kernel to help you?
+  - Now, figure out how you can change different kernel in libSVM. Write down your answer.
+  - Here we provide an easier dataset [[https://www.dropbox.com/s/740kqcyc6q3cu4u/mnist_49_3000.mat?dl=0|"mnist_4_9_3000"]]. It contains hand-written digit "4" and "9" for you to do binary classification as you did on car and face. Use first 2000 for training and 1000 for testing.
+  - You can use the following command to visualize your data
+  - >> load mnist_49_3000;
+  - >> [d,n] = size(x);
+  - >> i = 1; % index of image to be visualized
+  - >> imagesc(reshape(x(:,i),[sqrt(d),sqrt(d)])’) % notice the transpose
+  - Train SVM as you did before on this 4_9_3000 dataset with linear kernel. Compare the results with different kernels.
+  - Train SVM on your car and face dataset with linear kernel. Compare the results with different kernels. Which dataset has more improvements with different kernels. Why?
+__**Week #4: Cross Validation**__ \\
+  - Besides kernel, another important thing to apply SVM correctly is to select good hyper-parameters.
+  - If you checked SVM GUI [[https://www.csie.ntu.edu.tw/~cjlin/libsvm/|here]], you might notice that different C might lead to different performance.
+  - Please quickly review the SVM material again, and explain what is C parameter here?
+  - One way to select good hyper-parameter is to apply "cross validation".
+  - Please read carefully about [[https://www.cs.cmu.edu/~schneide/tut5/node42.html|Cross Validation]]. The idea is mainly to leave some data untouched, and use it to test your selected parameter. And repeat this step using different untouched portion in your training data.
+  - Now, apply cross validation by using 10% of your data for cross validation. Try it on mnist_49_3000. Test C = 10^-2, 10^-1, 1, 10^1, 10^2, which one gives you best performance?
+  - What is the difference between K-fold and LOO(leave one out)?
+---------------
+;#;
+[[ECE4580:Modules|ECE4580 Learning Modules]]
+;#;