VB.NET :: Multiple face detection and recognition in real time

Introduction

The facial recognition has been a problem worked on around the world for many persons; this problem has emerged in multiple fields and sciences, especially in computer science, others fields that are very interested In this technology are: Mechatronic, Robotic, criminalistics, etc. In this article I work in this interesting topic using EmguCV cross platform .Net wrapper to the Intel OpenCV image processing library and C# .Net, these library’s allow me capture and process image of a capture device in real time. The main goal of this article is show and explains the easiest way how implement a face detector and recognizer in real time for multiple persons using Principal Component Analysis (PCA) with eigenface for implement it in multiple fields.

Background

Facial recognition is a computer application composes for complex algorithms that use mathematical and matricial techniques, these get the image in raster mode(digital format) and then process and compare pixel by pixel using different methods for obtain a faster and reliable results, obviously these results depend of the machine use to process this due to the huge computational power that these algorithms, functions and routines requires, these are the most popular techniques used for solve this modern problem:

Techniques:

Traditional

Some facial recognition algorithms identify faces by extracting landmarks, or features, from an image of the subject's face. For example, an algorithm may analyze the relative position, size, and/or shape of the eyes, nose, cheekbones, and jaw. These features are then used to search for other images with matching features. Other algorithms normalize a gallery of face images and then compress the face data, only saving the data in the image that is useful for face detection. A probe image is then compared with the face data. One of the earliest successful systems is based on template matching techniques applied to a set of salient facial features, providing a sort of compressed face representation. Recognition algorithms can be divided into two main approaches, geometric, which looks at distinguishing features, or photometric, which is a statistical approach that distill an image into values and comparing the values with templates to eliminate variances. Popular recognition algorithms include Principal Component Analysis with eigenface, Linear Discriminate Analysis, Elastic Bunch Graph Matching fisherface, the Hidden Markov model, and the neuronal motivated dynamic link matching.
text taken from [1]
An example of EigenFaces:

image taken from [4]

3-D

A newly emerging trend, claimed to achieve previously unseen accuracies, is three-dimensional face recognition. This technique uses 3-D sensors to capture information about the shape of a face. This information is then used to identify distinctive features on the surface of a face, such as the contour of the eye sockets, nose, and chin. One advantage of 3-D facial recognition is that it is not affected by changes in lighting like other techniques. It can also identify a face from a range of viewing angles, including a profile view. Even a perfect 3D matching technique could be sensitive to expressions. For that goal a group at the Technion applied tools from metric geometry to treat expressions as isometries.

image taken from [2]

Skin texture analysis

Another emerging trend uses the visual details of the skin, as captured in standard digital or scanned images. This technique, called skin texture analysis, turns the unique lines, patterns, and spots apparent in a person’s skin into a mathematical space Tests have shown that with the addition of skin texture analysis, performance in recognizing faces can increase 20 to 25 percent. It is typically used in security systems and can be compared to other biometrics such as fingerprint or eye iris recognition systems.
text taken from [1]

EmguCV

Emgu CV is a cross platform .Net wrapper to the Intel OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc. The wrapper can be compiled in Mono and run on Linux / Mac OS X.
text taken from [3]
In my own words EmguCV is an awesome Wrapper, this let make very interesting things and tasks of computer vision, this library set let do an unlimited amount of wonderful projects in this field, EmguCV have many functions that let us work with CPU and GPU increases the performance dramatically with the latest mentioned.
This Awesome SW project let work and do:

• Optical Character Recognition(OCR)
• Face Detection
• Pedestrian Detection
• Kinect projects
• 3D reconstruction
• SURF feature detector ...between many others interesting tasks.

EmguCV basics: How i start to work?

If you don't had work never with this wrapper, you want see how add refereces to project or solve troubles look this god article/tutorial by C_Johnson:
Creating Your First EMGU Image Processing Project
Other interesting Web/blog whit multiple tutorials to star with emguCV, image processing, and face recognition bymehwish87 is :
EmguCV and basic image processing tutorials

Using the code

First declare all variables an important objects to use:

Hide   Copy Code

//Declararation of all variables, vectors and haarcascades
        Image<bgr,> currentFrame;
        Capture grabber;
        HaarCascade face;
        HaarCascade eye;
        MCvFont font = new MCvFont(FONT.CV_FONT_HERSHEY_TRIPLEX, 0.5d, 0.5d);
        Image<gray,> result, TrainedFace = null;
        Image<gray,> gray = null;
        List<image<gray,>> trainingImages = new List<image<gray,>>();
        List<string> labels= new List<string>();
        List<string> NamePersons = new List<string>();
        int ContTrain, NumLabels, t;
        string name, names = null;

Then load the haarcascades for face detection, then I do a little “procedure” to load of previous trained faces and labels for each image stored previously:

Hide   Copy Code

//Load haarcascades for face detection
face = new HaarCascade("haarcascade_frontalface_alt_tree.xml");
eye = new HaarCascade("haarcascade_eye.xml");
try
{
    //Load of previus trainned faces and labels for each image
    string Labelsinfo = File.ReadAllText(Application.StartupPath + "/TrainedFaces/TrainedLabels.txt");
    string[] Labels = Labelsinfo.Split('%');
    NumLabels = Convert.ToInt16(Labels[0]);
    ContTrain = NumLabels;
    string LoadFaces;

    for (int tf = 1; tf < NumLabels+1; tf++)
    {
        LoadFaces = "face" + tf + ".bmp";
        trainingImages.Add(new Image<gray,>(Application.StartupPath + "/TrainedFaces/" + LoadFaces));
        labels.Add(Labels[tf]);
    }
}
catch(Exception e)
{
    //MessageBox.Show(e.ToString());
    MessageBox.Show("Nothing in binary database, please add at least a face", "Triained faces load", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}

Initialize the capture device, and FrameGrabber event that performs the detection and procces of images for each frame captured:

Hide   Copy Code

grabber = new Capture();
grabber.QueryFrame();
//Initialize the FrameGraber event
Application.Idle += new EventHandler(FrameGrabber);
button1.Enabled = false;

Passing to FrameGrabber event (main part of prototype) we use the most important methods and objects: DetectHaarCascade And EigenObjectRecognizer and perform operations For each face detected in one frame:

Hide   Shrink    Copy Code

MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
                  face,
                  1.2,
                  10,
                  Emgu.CV.CvEnum.HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,
                  new Size(20, 20));

                    //Action for each element detected
                    foreach (MCvAvgComp f in facesDetected[0])
                    {
                        t = t + 1;
                        result = currentFrame.Copy(f.rect).Convert<gray,>().Resize(100, 100, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);
                        //draw the face detected in the 0th (gray) channel with blue color
                        currentFrame.Draw(f.rect, new Bgr(Color.Red), 2);


                        if (trainingImages.ToArray().Length != 0)
                        {
                            //TermCriteria for face recognition with numbers of trained images like maxIteration
                        MCvTermCriteria termCrit = new MCvTermCriteria(ContTrain, 0.001);

                        //Eigen face recognizer
                        EigenObjectRecognizer recognizer = new EigenObjectRecognizer(
                           trainingImages.ToArray(),
                           labels.ToArray(),
                           5000,
                           ref termCrit);

                        name = recognizer.Recognize(result);

                            //Draw the label for each face detected and recognized
                        currentFrame.Draw(name, ref font, new Point(f.rect.X - 2, f.rect.Y - 2), new Bgr(Color.LightGreen));

                        }
}        

Parameters:
haarObj: Haar classifier cascade in internal representation scaleFactor: The factor by which the search window is scaled between the subsequent scans, for example, 1.1 means increasing window by 10%
minNeighbors: Minimum number (minus 1) of neighbor rectangles that makes up an object. All the groups of a smaller number of rectangles than min_neighbors-1 are rejected. If min_neighbors is 0, the function does not any grouping at all and returns all the detected candidate rectangles, which may be useful if the user wants to apply a customized grouping procedure
flag: Mode of operation. Currently the only flag that may be specified is CV_HAAR_DO_CANNY_PRUNING. If it is set, the function uses Canny edge detector to reject some image regions that contain too few or too much edges and thus cannot contain the searched object. The particular threshold values are tuned for face detection and in this case the pruning speeds up the processing.
minSize: Minimum window size. By default, it is set to the size of samples the classifier has been trained on (~20x20 for face detection)

How to Train the Prototype?

I do this part the most easy possible, the prototype detect faces constantly (Each frame) and you can add this detected face in the image database with one label respectably, the face trained image will show in the imageBoxFrameGrabber and the process will be finished!!
Keep in mind: The face recognition algorithms based in PCA (Principal Component Analysis) do multiple comparisons and matches between a face detected and the trained images stored in binary database for this reason And for improve the accurate of recognition you should add several images of the same person in different angles, positions and luminance conditions, this training do this prototype solid and very accurate.
Example:

Code of training button (This perform the adding of training faces and labels for each):

Hide   Shrink    Copy Code

try
           {
               //Trained face counter
               ContTrain = ContTrain + 1;

               //Get a gray frame from capture device
               gray = grabber.QueryGrayFrame().Resize(320, 240, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

               //Face Detector
               MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
               face,
               1.2,
               10,
               Emgu.CV.CvEnum.HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,
               new Size(20, 20));

               //Action for each element detected
               foreach (MCvAvgComp f in facesDetected[0])
               {
                   TrainedFace = currentFrame.Copy(f.rect).Convert<gray,>();
                   break;
               }

               //resize face detected image for force to compare the same size with the
               //test image with cubic interpolation type method
               TrainedFace = result.Resize(100, 100, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);
               trainingImages.Add(TrainedFace);
               labels.Add(textBox1.Text);

               //Show face added in gray scale
               imageBox1.Image = TrainedFace;

               //Write the number of triained faces in a file text for further load
               File.WriteAllText(Application.StartupPath + "/TrainedFaces/TrainedLabels.txt", trainingImages.ToArray().Length.ToString() + "%");

               //Write the labels of triained faces in a file text for further load
               for (int i = 1; i < trainingImages.ToArray().Length + 1; i++)
               {
                   trainingImages.ToArray()[i - 1].Save(Application.StartupPath + "/TrainedFaces/face" + i + ".bmp");
                   File.AppendAllText(Application.StartupPath + "/TrainedFaces/TrainedLabels.txt", labels.ToArray()[i - 1] + "%");
               }

               MessageBox.Show(textBox1.Text + "&acute;s face detected and added :)", "Training OK", MessageBoxButtons.OK, MessageBoxIcon.Information);
           }
           catch
           {
               MessageBox.Show("Enable the face detection first", "Training Fail", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
           }
       }

How to improve the recognition?

The default parameters (scale_factor=1.1, min_neighbors=3, flags=0) are tuned for accurate yet slow object detection.
Also you may modify the size for a big value, Modify this in the code:

Hide   Copy Code

MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
face,
1.1,
3,
0,
new Size(20, 20));

Aditionally modify the 3rd param to 2500 or 3000 instead 5000, this modification do that the EigenObjectRecognizer be more strict/acurate.

Hide   Copy Code

//Eigen face recognizer
                       EigenObjectRecognizer recognizer = new EigenObjectRecognizer(
                          trainingImages.ToArray(),
                          labels.ToArray(),
                          5000,
                          ref termCrit);

How to improve the performance for slower CPUs?

All image processing algorithms demand many computational power, in this case the internals process carried on for the CPU with this sw prototype are so hard for slower o monocore CPUS, the Easy way for improve the performance of this Demo is modify the parameters that use the DetectHaarCascade method, these allow decrement the number of iteration, critic sections and Comparisons of the real time image captured for the Webcam improving notoriously the application performance.
Keep in mind: reduce the values of these parameters will affect the efficiency of recognition Algorithms.

First option:

For a faster operation on real video images the settings are: scale_factor=1.2, min_neighbors=2, flags=CV_HAAR_DO_CANNY_PRUNING, min_size= (for example, ~1/4 to 1/16 of the image area in case of video conferencing).
Also you may modify the Minsize parameter for a big value.

Hide   Copy Code

// DetectHaarCascade Config for optimal performance

MCvAvgComp[][] facesDetected = gray.DetectHaarCascade(
                  face,
                  1.2,
                  2,
                  Emgu.CV.CvEnum.HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,
                  new Size(20, 20));

Second option:

Get a “thumbnail”or resize the original image capture for reduce the time of processing In the FrameGrabber method modify the size values for a minor size (originally is 320x240)
Example:

Hide   Copy Code

//Get the current frame form capture device
                currentFrame = grabber.QueryFrame().Resize(260, 200, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

Remember do the same in the Training button:

Hide   Copy Code

gray = grabber.QueryGrayFrame().Resize(260, 200, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

The prototype does not show the strangers as unknown, Why?

Is a limitation of Eigenface approach, isn't error, use LBPH algorithm instead(have pros and shortcomings) available on EmguCV 2.4.2 and above.

What do you need to run/use this project without errors?

For avoid errors like this:

1. First download and decompress FaceRecPro_Demo.zip it have “external” OS libraries(DLLs) used for some OpenCV functions, after copy these DLLs in C:/Windows/System32 or in bin folder of this Project
2. Then download EmguCV(Wrapper of OpenCV for C#) here: http://sourceforge.net/projects/emgucv/files/, install it and after go to C:\Emgu\emgucv-windows-x86 2.2.1.1150\bin folder and copy: opencv_calib3d220, opencv_contrib220, opencv_core220, opencv_features2d220, opencv_ffmpeg220, opencv_flann220, opencv_gpu220, opencv_highgui220, opencv_imgproc220, opencv_legacy220, opencv_ml220, opencv_objdetect220 and opencv_video220 in C:/Windows/System32 or in bin folder of this Project
remember this DLLs are OpenCV Libraries and are necessary to run anything project that use EgmuCV
this is the easiest way for run the demo or project, found and Copy These files to folder demo or bin folder(if you want "run" the source code):
opencv_calib3d220.dll, opencv_contrib220.dll, opencv_core220.dll, opencv_features2d220.dll, opencv_ffmpeg220.dll, opencv_flann220.dll, opencv_gpu220.dll, opencv_highgui220.dll, opencv_imgproc220.dll, opencv_legacy220.dll, opencv_ml220.dll, opencv_objdetect220.dll and opencv_video220.dll(Dlls from OpenCV Content in EmguCV Download) and: Emgu.CV.dll Emgu.CV.UI.dll Emgu.Util.dll cv110.dll cvaux110.dll cvextern.dll cxcore110.dll highgui110.dll(Content into Demo Download zip)
Keep in mind: you can copy these files in Windows/System32/ folder and forget your problems of dependencies for this and other project that use Emgu or openCV(Emgu.CV.dll, Emgu.CV.UI.dll, Emgu.Util.dll SHOULD go always in the bin or .exe folder) or Download all ready project(Optimized version) and files here:

• Download FaceRecPro Optimized Version: Source code, Demo: All binaries references, algorithm enhanced - 6.5 MB

Points of Interest

I had many problems with vectors use with EmguCV, for this reason i had learn how use list of components like vectors and it really works for my project.
I learn many of image processing, PCA and EigenFaces and to optimize the code, due to the huge demand of resources for part of artificial vision algorithms.
This project idea emerge after see an Iron man scene…XD

What can you do whit this article and OpenCV?

Something like this....
http://youtu.be/T4z11lIKnDU

Books about it

The official reference book for OpenCV is: "Learning OpenCV: Computer Vision with the OpenCV Library", from O'Reilly (2008)
There are hundreds of other good books for computer vision that are not specific to OpenCV:

• "Computer Vision: A Modern Approach" by Forsyth and Ponce (2002).
• "Computer Vision: Algorithms and Applications" by Szeliski (2011).
• "Digital Image Processing" by Gonzalez and Woods (2001).
• "The Essential Guide to Image Processing", by Bovik (2009).
• "Computer Vision and Applications: A Guide for Students and Practitioners", by Jähne and Haußecker (2000).
• HIPR2 Image Processing Worksheets (simple explanations of many computer vision topics)

References

[1] http://en.wikipedia.org/wiki/Facial_recognition_system
[2] "Firms point to biometric future" By Dominic Bailey, BBC News, at Biometrics 2006
[3] http://www.emgu.com/wiki/index.php/Main_Page
[4] http://www.shervinemami.info/faceRecognition.html

License

This article and prototype is only for academic purposes, not use for commercial or public releases without author permissions.

History

11-07-2011: Initial release
28-01-2012: Final release(Optimized version)

Tweet