PV226 – Lasaris seminar / 2021 / R. Ošlejšek 1
Visual Analytics in the KYPO Cyber Range
– Principles and Challenges
Radek Ošlejšek
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 2
KYPO Cyber Range Platform
Remote connection
Tasks (an example):
●
Find an unusual service running on
a server
●
Exploit its vulnerability to access
the server
●
Steal SSH credentials
●
Crack them to see the passwords
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 3
Problem statement
No tangible output (like a code in programming courses)
●
Tutors have no idea, what trainees do, whether they are stacked in some task, etc.
●
Trainees don’t know whether what they did wrong, or whether there was a faster
solution to the tasks.
●
Training designers don’t know whether the game was too easy or difficult.
●
Research Goal: To research and develop data analysis tools providing insight into
educational aspects of cybersecurity training and enable comparison, assessment, and
continuous improvement.
Avoiding confusion… What is [not] visual analysis
IT IS NOT about the design of GUI, e.g., where to place info window, what color to choose
(although these UX aspects are part of any good graphical tool).
IT IS about finding ways to provide insight into complex data and their hidden relationships
by means of “smart” interactive visualizations.
[Furmanová, K., et al. "Multiscale Visual Drilldown for the Analysis of Large Ensembles of Multi-Body Protein Complexes.", TVCG, 2019]
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 5
VA examples: Forensic investigation
BERAN, Martin, František HRDINA, Dan KOUŘIL, Radek OŠLEJŠEK, Kristína ZÁKOPČANOVÁ.
Exploratory Analysis of File System Metadata for Rapid Investigation of Security Incidents.
In IEEE Symposium on Visualization for Cyber Security (VizSec’20).
FIMETIS – A tool for forensic investigation of disk images
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 6
VA methodology
●
The development of a really useful VA tool is challenging. It is necessary to follow
many rules and best practices to achieve good results and to prove usability
– Tight cooperation with domain experts for both requirements analysis and usability
evaluation
– Using iterative design methodologies, e.g., user-centered design (it isn’t an ad-hoc
process)
– Formal evaluation of results, e.g., quantitative and qualitative methods of measuring
user experience
●
The development process can be considered a special discipline of software
engineering
UNDERSTAND EXPLORE MATERIALIZE
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 7
VA high-level concepts
●
Regardless of the methodology and application domain, it is always
necessary to
– clarify users roles, actors, personas in given application domain;
– identify their analytical goals and data processes;
– propose visualization techniques that reflect available data and address
analytical goals of user roles.
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 8
VA for Hands-on Cybersecurity Training
OŠLEJŠEK, Radek, Vít RUSŇÁK, Karolína DOČKALOVÁ BURSKÁ, Valdemar ŠVÁBENSKÝ, Jan VYKOPAL and Jakub ČEGAN.
Conceptual Model of Visual Analytics for Hands-on Cybersecurity Training.
In IEEE Transactions on Visualization and Computer Graphics, 2021.
Personalized feedback to trainees
Goal: Learning from own mistakes
●
What did I do wrong in selected tasks?
●
Where I lost most points and why?
●
...
OŠLEJŠEK, Radek, Vít RUSŇÁK, Karolína BURSKÁ, Valdemar ŠVÁBENSKÝ a Jan VYKOPAL.
Visual Feedback for Players of Multi-Level Capture the Flag Games: Field Usability Study.
In IEEE Symposium on Visualization for Cyber Security (VizSec‘19)
VYKOPAL, Jan, Radek OŠLEJŠEK, Karolína BURSKÁ and Kristína ZÁKOPČANOVÁ.
Timely Feedback in Unstructured Cybersecurity Exercises.
In ACM Technical Symposium on Computer Science Education (SIGCSE’18)
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 11
Insight for organizing participants
Goal: Situational awareness and timely intervention
●
Which trainees are in trouble? Why?
●
Is the training session on schedule, or is there some delay?
●
...
DOČKALOVÁ BURSKÁ Karolína, Vít RUSŇÁK and Radek OŠLEJŠEK.
Enhancing Situational Awareness for Tutors of Cybersecurity Capture the Flag Games.
In International Conference Information Visualization (iV’21).
Post-training analysis
Goal: Improve the impact of learning
●
Was training too easy or difficult?
●
What are the sources of losing motivation and giving up the training?
●
Are there some flows in the scenario, requirements, etc.?
●
...
DOČKALOVÁ BURSKÁ Karolína, Vít RUSŇÁK and Radek OŠLEJŠEK.
Data-driven insight into the puzzle-based cybersecurity training.
In Computers & Graphics, 2021.
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 13
Bottom-up approach to VA
●
Our recent approach to VA reflects a standard domain-specific paradigm
●
Game data and events
– Estimated time of tasks
– Start/end of the exercise
– Submission of a correct flag, i.e., successful solution of a task
– Submission of an incorrect flag, i.e., wrong attempts to solve the task
– Taking a hint
●
Assessment data
●
Bash history
analytical goalsdomain-specific
data
visualizations
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 14
Tailored domain-specific approach
●
Precise support of users and their analytical requirements
●
The introduction of new data types usually requires adaptation or extension of
existing visualizations
●
Application to other learning domains that follow puzzle-based gamification
principles is also limited
– Puzzles are used as a metaphor for getting students to think about how to frame
and solve unstructured problems.
– Division of learning tasks into smaller connected parts (puzzles)
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 15
Tailored vs. unified approach to design VA
Is there some more general conceptual approach
to design exploratory visualizations
for cybersecurity exercises?
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 16
Process mining
●
Cybersecurity learning is process-oriented
●
There exist a process mining research area
– A bridge between traditional data analysis techniques, like data mining, and
business process management analysis
– Provides algorithms that take event logs as input and produces process graphs
reconstructed from the logs (it is called process discovery)
– Process graphs provide better cognitive features than row event logs and then
simplify comprehension
Process mining for cybersecurity training
●
The idea of using process graphs is not new, even in the subdomain of
cybersecurity training
– Weiss, R. et al.: A reflective approach to assessing student performance in cybersecurity
exercises. ACM SIGCSE’16
– Mirkovic, J. et al.: Using terminal histories to monitor student progress on hands-on exercises.
ACM SIGCSE’20
●
But they utilize tailored process graphs (i.e., domain-specific approach) while
omitting generic process mining approaches
●
Using process mining approaches brings many open problems
– Data pre-processing and mapping affect obtained graphs
– The selection of process discovery algorithm affects obtained graphs
– Problem with the scalability of obtained graphs
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 18
Open problems – current research
●
Tackling comprehensibility:
– We defined necessary pre-processing tasks and formulated data abstraction that
enables us to get reasonable process graphs from cybersecurity exercises
– We conducted initial experiments that proved its usability for learning analytics.
However, a more robust evaluation with more participants is necessary.
●
Tackling scalability:
– Data aggregation and filtering at the input side of the process mining algorithms
– Structural properties of puzzle-based games
– Providing complementary views to process graphs
●
What is the same or sufficiently similar commands?
– User 1: ssh root@147.251.8.28
– User 2: ssh 147.251.8.28
– User 3: ssh -4 root@147.251.8.28
– User 4: ssh 127.1.5.8
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 19
Data aggregation and filtering
LevelStarted (4)
ssh (4)
4
... (4)
4
LevelStarted (4)
ssh 147.251.8.28 (3)
3
... (3)
3
ssh 127.1.5.8 (1)
... (3)
1
1
LevelStarted (4)
ssh root@147.251.8.28 (2)
2
... (3)
3
ssh 127.1.5.8 (1)
... (3)
1
1
ssh 147.251.8.28 (1)
1
... (3)
1
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 20
Structural properties of puzzle-based games
●
High cohesion inside puzzles (tasks)
●
Low decoupling between puzzles (tasks)
●
Schneiderman’s visual information-seeking mantra: Overview first, zoom and
filter, then details-on-demand
“Weakly connected islands of complexity“
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 21
Complementary views to process graphs
●
Idea: Provide alternate view to a traditional graph representation
●
From the VA perspective, process graphs are so-called multivariate networks
– Nobre, C. et al. The state of the art in visualizing multivariate networks. In
Computer Graphics Forum, Vol. 38, No. 3. 2019
●
But still, the design of a concrete tool is challenging
PV226 – Lasaris seminar / 2021 / R. Ošlejšek 22
Infrastructure analysis
Thank you for your attention!