Biometrics 2
Face recognition
PV181 Laboratory of security and applied cryptography
Seminar 8. 12. 2021
Agáta Kružíková, kruzikova@mail.muni.cz
Katarína Galanská, xgalansk@fi.muni.cz
Lecture structure
Seminar 1
1. Introduction
2. Fingerprints
3. Seminar activity
– Fake fingerprints
4. Homework
– Report on selected
biometric system
Seminar 2
1. Fake fingerprints
– Completion
2. Face recognition
3. Seminar activity
– Hypotheses
4. Homework
– Face detection
2
Fingerprint experiment
Completion
3
Fingerprints Completion
4
• Scan your genuine and fake fingerprints
– In case of the bonus task do not forget to scan the
fake without Gabor filtering it too!
• You will receive the scans in your Depository in
IS
• Process the scans with NBIS software
• Try to “hack” your own smartphone
Preparation for post processing
• You will receive scans of your fingers
• Prepare the image for post processing
– 8-bit grayscale raster
– width at least 512 pixels
– height at least 480 pixels
– .png format
5
Fingerprints post processing
● Use NIST Biometric Image Software (NBIS)
○ It is preinstalled on lab PC in the within Ubuntu VM
● Create a minutia map in .xyt format (Mindtct)
○ /opt/nbis/bin>./mindtct -m1 <fingerprint_scan_from_reader.png>
<new_name_of_fingerprint_in_xyt_format>
● Check quality of the fingerprint (Nfig)
○ /opt/nbis/bin>./nfiq -d <fingerprint_scan_from_reader.png>
○ Output: 1-5 where 1-best, 5-worst
● Check the number of identified minituas in .min
file
6
Fingerprints post processing
● Repeat it for all fingerprint inputs which you
scanned via reader
● Compare fake and real fingerprints (Bozoroth3)
○ Score above 40 means true match (“rule of thumb”)
○ /opt/nbis/bin>./bozorth3 <fake.xyt> <original.xyt>
● Bonus task: compare the real fingerprint to fake
without Gabor filters
● Open the questionnaire and fill in the data from
your observations
○ Mandatory for all, only the research questions are not
7
Help with future fingerprint image processing
8
1. Open your image in
ImageJ.
2. Select your fingerprint
with polygon selection.
3. Process → Noice → Add
Noice
4. Save as → .png
5. Upload to:
https://is.muni.cz/auth/el/fi/p
odzim2021/PV181/ode/121
202412/
Face recognition
Theory and examples
9
Real-life example
10
Face recognition – Input
• Single picture
• Video sequence
• 3D image
• Facial thermograms
11
Face recognition: The manual way
12
Face recognition: The automatic way
• Statistical
– Eigenface, PCA, LDA, ...
• Neural networks
– Microsoft: Face API
– Facebook: DeepFace
– VK: FindFace (“best results” in MegaFace comp.)
– Google: FaceNet
13
FindFace – example
Subway photo (left), social network photo (right)
14
Challenges in face recognition
• Illumination
• Pose
• Environment
– Noisy background
• Aging
• Feature occlusion
– Hats, glasses, hair, ...
• Image quality
– colour, resolution, ...
15
New challenge in face recognition...
• NIST study on the effects of face masks
– Error rates 5–50% on face masks
– Nose and mask color matter
• NtechLab: “Even balaclava is OK.”
– Focus (even more) on eyes
16
Face recognition overview (OpenFace)
Photo © The OpenFace project, cmusatyalab.github.io/openface
17
Face detection: Haar cascades
• Machine learning based approach based on comparing
pixel intensities in adjacent regions
18
Face detection: Haar cascades
Face Detection: Visualized https://vimeo.com/12774628
19
CV Dazzle: Anti face-detection
20
Photo © 2010-2016 Adam Harvey, CV Dazzle
CV Dazzle: Anti face-detection
21
Photo © 2010-2016 Adam Harvey, CV Dazzle
Microsoft: Face API
22
Automatic passport control
23
Biometric passports
• “Smart card”, contain NFC chip
• Two security levels:
- BAC: Reading your photo+personal information
(Try Android app Passport reader)
- EAC: Reading your biometrics
- Fingerprint, Face and Iris support
24
KFC AliPay
- Introduced 2015
- Only one KFC in China
- Liveness detection
- 3D camera
- 2017: login in Alibaba
services
- See AliPay promo video at
https://www.theverge.com/2017/9/4/16251304/kfc
-china-alipay-ant-financial-smile-to-pay
25
KFC AliPay
26
Face impersonation
27
Photo © 2016 Carnegie Mellon University, Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Face impersonation
• Fooling deep-neural-networks-based face
recognition systems (e.g. Face++)
– Over 90% success rate
– The principle is more general
• "physically realizable and inconspicuous"
Sharif, Mahmood, et al. "Accessorize to a crime: Real and stealthy
attacks on state-of-the-art face recognition." Proceedings of the 2016
ACM SIGSAC Conference on Computer and Communications Security.
ACM, 2016.
28
Apple FaceID hacked
- Liveness detection feature hacked in 2019
- Researchers used a pair of modified glasses
- A victim has to sleep :-)
Soucce: https://threatpost.com/researchers-bypass-apple-faceid-using-biometrics-achilles-heel/147109/
29
Detecting sexual orientation from faces
30
Photo © 2017 Deep neural networks are more accurate than
humans at detecting sexual orientation from facial images.
Journal of Personality and Social Psychology
Detecting sexual orientation from faces
• Classifying sexual orientation (straight vs. gay)
on men/women photos
– Human success: 61% / 54%
– Neural networks: 81% / 71%
– Neural networks (5 images): 91% / 83%
• May be a privacy issue!
Wang, Y., & Kosinski, M. (in press). Deep neural networks are more
accurate than humans at detecting sexual orientation from facial
images. Journal of Personality and Social Psychology, 2017.
31
Mugshots
32
Face recognition ban in San Francisco
- “Threat to civil liberties”
- Ban for government agencies (city police and
sheriff)
- Federal agencies not affected
- Reason: discrimination, privacy issues
- Less accurate at people of colour!
- Suppliers see it as a step back
- See more at www.banfacialrecognition.com
Gregory Barber, San Francisco Bans Agency Use of Facial-Recognition Tech. 2019, Wired.
33
Ethical use of technology?
Code of Ethics (ACM)
1. Society and human well-being
2. No harm for participants & risk analysis
3. Honesty (transparency)
4. No plagiarism
5. Respect privacy
6. Confidentiality
7. High quality & standards (competence)
8. Professional review
9. Inform society
ACM Code of Ethics and Professional Conduct., Online [2019]: acm.org/code-of-ethics
34
Homework
Exploring automatic face detection
35
Homework: Overview
• Explore what influences face detection
– Use deep learning modules from OpenCV
github.com/crocs-muni/biometrics-utils
– Use a webcam or your own picture(s)
•Your pictures will not be shared
– Test real-live modifications or digital touch-up
• Submit to IS MUNI a single ZIP file with
– Report (PDF) with proper methodology (see next slide)
– Used adjusted images
• Deadline: 15. 12. 2020 8:00
36
Homework: Overview
Step 1: State the hypotheses.
E.g., obstructing eyes decreases face detection accuracy
significantly more than obstructing other face parts.
Step 2: Set the criteria for a decision.
Set baseline (no obstructions) and test different settings, do
multiple small changes (progressively obstructing eyes, mouth, ...).
Step 3: Interpret the results.
Summarize the results, reject the hypothesis if appropriate.
37
Homework: Hypotheses
● Measurable (we can make observations)
○ NOT: “There are invisible creatures all around us.”
● Falsifiable (if it’s false, we can show it)
○ NOT: “There are other planets in the universe where life exists.”
● Precise (can be made into experiment)
○ NOT: “Candles repel mosquitoes.”
● Reproducible (others can verify it)
○ NOT: “Putting an African bush elephant on the top of the Leaning tower of Pisa
will crash it.“
● Useful enough (predictive, not too general, ...)
○ NOT: “A Škoda Superb car with (...specification...) will drive more than 2 km
with 20 l of petrol.”
Note: Hypothesis is always a statement, not a question
38
Task: Formulating Hypotheses
Formulate possible good hypotheses based on these
sentences:
1. Do people like iris eye readers?
2. 256b AES keys are secure.
3. PV080 is the best course at FI MU.
4. You can make a lock that opens with three
different keys.
5. Closing the browser deletes the cookies.
39
Task: Formulating Hypotheses
Possible nice hypotheses:
1. Non-IT university students consider using fingerprint
readers more usable than iris eye readers for day-to-day
authentication.
2. You cannot successfully break 256b AES encryption in CBC
mode in one hour on machine XYZ.
3. Among all bachelor students at FI MU, the average selfreported
satisfaction with PV080 is significantly higher than
for IB000.
4. You cannot make a lock that opens with three different
keys.
5. All non-permanent cookies are removed after closing
40
Homework: Report
• Write a summarizing report
– Your hypotheses and how you tested them
– Test at least 5 distinct features
• Concentrate on:
– Having a formulated hypotheses for each feature
– Having several images supporting/falsifying your idea
• Avoid:
– Many changes in the face at once
– Radical changes (deleting half the face)
– Overgeneralization
41
Homework: Scoring
• Up to 10 points awarded
– Scoring rubric available in the Information system
– The rubric can help you understand what is important
in the task!
• Have a look at old homework submission with
good methodology in the Study Materials.
– Special thanks to Vladimír Bouček for providing it.
42