Biometrics
Fingerprints & Face recognition
PV181 Laboratory of security and applied cryptography
Seminar 05. 12. 2024
Katarína Galanská, galanska@mail.muni.cz
Lecture structure
1. Intro to biometrics
2. Demo of fingerprint processing
3. Face recognition
– Face recognition theory
– Selected attacks
4. Seminar activities
– Building biometric authentication
5. Homework
– Face detection
3
“What uses of biometrics have you seen
in your life?”
4
Biometrics – introduction
• Authentication based on:
– something I know (e.g. password)
– something I have (e.g. access card)
– something I am (e.g. fingerprint)
5
• Never 100% match
– FAR (false acceptance rate)
– FRR (false rejection rate)
Basic criteria for biometrics
• Uniqueness (sufficiently different across population)
• Universality (everybody has it)
• Permanence (invariant in the period of time)
• Collectability (possible to measure and digitize it)
• Performance (recognition accuracy should good)
• Acceptability (individuals should be OK to present it)
• Circumvention (hard to fake)
6
Attack vectors
7
Fingerprints
Theory and examples
8
Fingerprint
features
9
Fingerprint minutiae
10
Other Resources: Fingerprint recognition colab - BioLab Bologna
Fingerprint enrollment
11
Training AI for reversing fingerprint
minutiae (inputs)
12
Bouzaglo, Rafael and Yosi Keller. “Synthesis and Reconstruction of Fingerprints using
Generative Adversarial Networks.” ArXiv abs/2201.06164 (2022): n. pag.
Training AI for reversing fingerprint
minutiae (step 2000)
13
Training AI for reversing fingerprint
minutiae (step 13000)
14
Face recognition
Use cases, examples and attacks
15
Usage: Smartphone unlocking
16
Usage: Bank registration
17
Source: https://developers-old.innovatrics.com/digital-onboarding/docs/use-cases/onboarding/
Project Citizen K.
• 2010 “art” project by
• 12 people had their Czech ID cards reissued
with morphed photographs
– Similar haircut & style
– Photos 50% themselves,
50% someone else
– In 2010 you brought
your own photo for
the ID card
18
• Using someone else’s identity for several months
– Wedding, gun licence, pilot licence, bank operations,
out-of-Schengen travel, elections, ...
PS: Czech documentary can be legally streamed for 60 Kč
19
KFC AliPay
- Introduced 2015
- Only one KFC in China
- Liveness detection
- 3D camera
- 2017: login in Alibaba
services
- See AliPay promo video at
https://www.theverge.com/2017/9/4/16251304/kfc
-china-alipay-ant-financial-smile-to-pay
20
Biometric passports
• “Smart card”, contains NFC chip
• Two security levels:
- BAC: Reading your photo+personal information
(Try Android app Passport reader)
- EAC: Reading your biometrics
- Fingerprint, Face and Iris support
21
Passport control?
22
Automatic passport control
24
The
future
of
travel
25
Fly
to
Gate
26
Face recognition – Input
• Single picture
• Video sequence
• 3D image
• Facial thermograms
27
Face recognition: The manual way
28
Face recognition: The automatic way
• Statistical
– Eigenface, PCA, LDA, ...
• Neural networks
– Microsoft: Face API
– Facebook: DeepFace
– VK: FindFace (“best results” in MegaFace comp.)
– Google: FaceNet
29
FindFace example
Subway photo (left), social network photo (right)
30
Face recognition overview (OpenFace)
Photo © The OpenFace project, cmusatyalab.github.io/openface
31
Microsoft: Face API
32
Face detection: Haar cascades
• Machine learning based approach based on comparing
pixel intensities in adjacent regions
• Face Detection: Visualized
https://vimeo.com/12774628
33
34
Haar Cascade Visualization
I take no credit for this video.
This is a video created by the absolutely amazing Adam Harvey. I have only uploaded it on my channel because I couldn't find it on YouTube, and some of my students could not access vimeo.
Here's the original description with the video at vimeo:
This video visualizes the detection process of OpenCV's face detector. The algorithm uses the Viola Jones method of calculating the integral image and then performing some calculations on all the areas defined by the black and white rectangles to analyze the differences between the dark and light regions of a face. The sub-window (in red) is scanned across the image at various scales to detect if there is a potential face within the window. If not, it continues scanning. If it passes all stages in the cascade file, it is marked with a red rectangle. But this does not yet confirm a face. In the post-processing stage all the potential faces are checked for overlaps. Typically, 2 or 3 overlapping rectangles are required to confirm a face. Loner rectangles are rejected as false-positives.
This visualization was done as part of the documentation for CV Dazzle, camouflage from face detection.
For more information, visit cvdazzle.com
Challenges in face recognition
• Illumination
• Pose
• Environment
– Noisy background
• Aging
• Feature occlusion
– Hats, glasses, hair, ...
• Image quality
– colour, resolution, ...
35
Newer challenge in face recognition...
• NIST study on the effects of face masks
– Error rates 5–50% on face masks
– Nose and mask color matter
• NtechLab: “Even balaclava is OK.”
– Focus (even more) on eyes
36
CV Dazzle:
Anti face-
detection
37
Photo © 2010-2016 Adam Harvey, CV Dazzle
CV Dazzle: Anti face-detection
38
Photo © 2010-2016 Adam Harvey, CV Dazzle
Face impersonation
39
Photo © 2016 Carnegie Mellon University, Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Face impersonation
• Fooling deep-neural-networks-based face
recognition (e.g. Face++)
– Over 90% success rate
– The principle is more general
• "physically realizable and inconspicuous"
Sharif, Mahmood, et al. "Accessorize to a crime: Real and stealthy
attacks on state-of-the-art face recognition." Proceedings of the 2016
ACM SIGSAC Conference on Computer and Communications Security.
ACM, 2016.
40
Makeup attacks
C. Rathgeb, P. Drozdowski and C. Busch, "Detection of Makeup Presentation Attacks based
on Deep Face Representations," 2020 25th International Conference on Pattern Recognition
(ICPR), Milan, Italy, 2021, pp. 3443-3450, doi: 10.1109/ICPR48806.2021.9413347.
41
Makeup attacks
42
Liveness detection
- Protection against: spoofs (digital replicas),
photos, videos, masks (physical replicas)
- Liveness detection:
- AI in real time
- Passive
- E.g., eye movements, skin structure
- Active
- User asked to do something
- Combination of liveness checks
Liveness Detection in Face Recognition: Evolution of Biometrics
44
Liveness detection: other resources
Facia Unveils How It Will Defend Against $24B Identity Thefts: From
Paper Masks to Deepfakes
Can I unlock it with my photo? Face ID vs Windows Hello vs Samsung
Facial Recognition
iPhone X Review: Testing (and Tricking) FaceID
iPhone X Face ID fooled with mask, claims hackers
https://support.apple.com/en-us/102381
E. Lavens, D. Preuveneers, and W. Joosen. 2023. Mitigating undesired
interactions between liveness detection components in biometric
authentication. In Proceedings of the 18th International Conference on
Availability, Reliability and Security (ARES '23). ACM, New York, NY,
USA, Article 141, 1–8. https://doi.org/10.1145/3600160.3604992
45
Apple FaceID hacked
- Liveness detection feature hacked in 2019
- Researchers used a pair of modified glasses
- A victim has to sleep :-)
Source: https://threatpost.com/researchers-bypass-apple-faceid-using-biometrics-achilles-heel/147109/
46
Face recognition
Privacy issues
47
Detecting sexual orientation from faces
48
Photo © 2017 Deep neural networks are more accurate than
humans at detecting sexual orientation from facial images.
Journal of Personality and Social Psychology
Detecting sexual orientation from faces
• Classifying sexual orientation (straight vs. gay)
on men/women photos
– Human success: 61% / 54%
– Neural networks: 81% / 71%
– Neural networks (5 images): 91% / 83%
• May be a privacy issue!
Wang, Y., & Kosinski, M. (in press). Deep neural networks are more
accurate than humans at detecting sexual orientation from facial
images. Journal of Personality and Social Psychology, 2017.
49
Mugshots (publicly published)
50
Face recognition ban in San Francisco
- “Threat to civil liberties”
- Ban for government agencies (city police and
sheriff)
- Federal agencies not affected
- Reason: discrimination, privacy issues
- Less accurate at people of colour!
- Suppliers see it as a step back
- See more at www.banfacialrecognition.com
Gregory Barber, San Francisco Bans Agency Use of Facial-Recognition Tech. 2019, Wired.
51
Ethical use of technology?
Code of Ethics (ACM)
1. Society and human well-being
2. No harm for participants & risk analysis
3. Honesty (transparency)
4. No plagiarism
5. Respect privacy
6. Confidentiality
7. High quality & standards (competence)
8. Professional review
9. Inform society
ACM Code of Ethics and Professional Conduct., Online [2019]: acm.org/code-of-ethics
52
Seminar task
• Design a biometric authentication system for one
of the scenarios:
a. Secure access to a personal device (e.g., smartphone, laptop)
b. Access to a secure building (e.g., office or research lab)
c. Student identification for exams
d. Airport security/boarding
e. Healthcare patient identification
53
Building Biometric Authentication
System (Online Banking Authentication)
Modality: Voice Recognition + OTP (One-Time Password)
User Experience:
• Voice Enrollment: The user records their voice by speaking a set of words or numbers
into their phone or computer microphone.
• Secure Login: During the authentication process, the user speaks their passphrase into
the microphone. The system compares the voice sample to the stored one and verifies
the user’s identity.
• OTP for Additional Security: Once voice authentication is successful, the system sends
an OTP to the user’s phone, which they must enter to complete the login process.
54
Building Biometric Authentication
System (Online Banking Authentication)
Security Features:
• Two-Factor Authentication: The combination of voice recognition and OTP ensures that
even if someone gains access to the user’s phone or computer, they would still need the
OTP to complete the authentication process.
• Anti-Spoofing: The system uses liveness detection to confirm the authenticity of the
voice sample (i.e., preventing playback from recorded audio or synthetic voices).
• Voiceprint Encryption: Voiceprint data is encrypted and stored in a secure,
decentralized manner to prevent unauthorized access or data breaches.
Challenges Addressed:
• Phishing and Fraud Prevention: OTP adds an additional layer to prevent unauthorized
access even if a voiceprint is spoofed.
• User Convenience: The voice recognition system allows for hands-free authentication,
making it convenient for users to access their accounts on the go.
55
Seminar task (what to consider)
• Which biometric modality to use (e.g., fingerprint, facial recognition,
iri scan, voice recognition, etc.)
• User experience: How easy is it for a user to enroll and use the
system? What about people with disabilities?
• Security features: How can they make the system
more secure? Should there be multi-factor
authentication (e.g., combining fingerprint + PIN)?
• Context of use: Consider factors like lighting
conditions for facial recognition or the need for
speed in an airport scenario.
56
Seminar task (questions)
• Why you chose this biometric modality?
• How the system works?
• How you addressed usability
and security concerns?
• How the system solves
challenges in the specific
scenario?
57
Homework
Exploring automatic face detection
58
Homework: Overview
• Explore what influences face detection
– Use deep learning modules from OpenCV
github.com/crocs-muni/biometrics-utils
– Use a webcam or your own picture(s)
•Your pictures will not be shared
– Test real-live modifications or digital touch-up
• Submit to IS MUNI a single ZIP file with
– Report (PDF) with proper methodology (see next slide)
– Used adjusted images
• Deadline: 12. 12. 2024 8:00
59
Homework: Overview
Step 1: State the hypotheses.
E.g., obstructing eyes decreases face detection accuracy
significantly more than obstructing other face parts.
Step 2: Set the criteria for a decision.
Set baseline (no obstructions) and test different settings, do
multiple small changes (progressively obstructing eyes, mouth, ...).
Step 3: Interpret the results.
Summarize the results, reject the hypothesis if appropriate.
60
Homework: Hypotheses
● Measurable (we can make observations)
○ NOT: “There are invisible creatures all around us.”
● Falsifiable (if it’s false, we can show it)
○ NOT: “There are other planets in the universe where life exists.”
● Precise (can be made into experiment)
○ NOT: “Candles repel mosquitoes.”
● Reproducible (others can verify it)
○ NOT: “Putting an African bush elephant on the top of the Leaning tower of Pisa
will crash it.“
● Useful enough (predictive, not too general, ...)
○ NOT: “A Škoda Superb car with (...specification...) will drive more than 2 km
with 20 l of petrol.”
Note: Hypothesis is always a statement, not a question
61
Task: Formulating Hypotheses
Formulate possible good hypotheses based on these
sentences:
1. Do people like iris eye readers?
2. 256b AES keys are secure.
3. PV181 is the best course at FI MU.
4. You can make a lock that opens with three
different keys.
5. Closing the browser deletes the cookies.
63
Task: Formulating Hypotheses
Possible nice hypotheses:
1. Non-IT university students consider using fingerprint
readers more usable than iris eye readers for day-to-day
authentication.
2. You cannot successfully break 256b AES encryption in CBC
mode in one hour on machine XYZ.
3. Among all bachelor students at FI MU, the average selfreported
satisfaction with PV181 is significantly higher than
for IB000.
4. You cannot make a lock that opens with three different
keys.
5. All non-permanent cookies are removed after closing
64
Homework: Report
• Write a summarizing report
– Your hypotheses and how you tested them
– Test at least 5 distinct features
• Concentrate on:
– Having a formulated hypotheses for each feature
– Having several images supporting/falsifying your idea
• Avoid:
– Many changes in the face at once
– Radical changes (deleting half the face)
– Overgeneralization
65
Homework: Scoring
• Up to 10 points awarded
– Scoring rubric available in the Information system
– The rubric can help you understand what is important
in the task!
66