Development of information security
management standard and evaluation
instrument, Estonian case
2023/04/27
Mari Seeba
NCSC-EE leading expert of cybersecurity, UT PhD student
mari.seeba@{ria|ut}.ee
Agenda
• How to choose standard?
• Seeba, Mari, Raimundas Matulevičius, and Ilmar Toom. "Development of the
Information Security Management System Standard for Public Sector Organisations
in Estonia." Business Information Systems. 2021.
• How to evaluate the standard compliance level?
• Seeba, Mari, Sten Mäses, and Raimundas Matulevičius. "Method for Evaluating
Information Security Level in Organisations." Research Challenges in Information
Science: 16th International Conference, RCIS 2022, Barcelona, Spain, May 17–20,
2022, Proceedings. Cham: Springer International Publishing, 2022.
• MUSE – why we need method for updating security evaluation tool?
• MASS – tool and benchmarks to evaluate security (work in progress)
• Security level evaluation intermediate results
3
Motivation
• Estonia has 1.33 million
inhabitants
• Digital services via data exchange
layer X-tee (Estonian instance of
X-Road)
• 3000 digital services
• 225 million request per
month via X-tee
• From 2004 since now Estonian
public sector organisations use
security framework ISKE
• based on previous approach
of BSI IT Grundscutz
• BSI ITG changed their
approach at 2017
Fig. source: https://abi.ria.ee/xtee/et/x-tee-juhend/x-tee-kasutusstatistika/x-tee-visualiseerija
What should be the criterias or requirements of
the information security management standard
for public sector organisations?
• RQ1: How to find the national states requirements to the ISMS
standard?
• RQ2: How to use these requirements when developing the
national ISMS standard?
Requirements elicitation
• NCSI database
• Cybersecurity strategies and
implementation plans of EU countries
• GR, CZ, LT, ES, BE, FI, SK, HR, FR, LV, PL,
NL
• Requirements for security standard or
guidelines
• Similar requirements aggregation (15
requirements)
• Requirements grouping into modules:
• National Security module
• Content Module
• Assessment ModuleSource: https://ncsi.ega.ee/
Requirements elicitation results
National Security Module
N1 Developer and
Jurisdiction
N2 Development financing
N3 Licence conditions
N4 Language
N5 Update Cycle
Content Module
C1 Scope
C2 ISMS Compliance
C3 Basic Controls
C4 Leveled Controls
C5 Risk Management
Approach
C6 Technology Dependence
C7 Integrability of local
needs
C8 Controls Approach
Assessment Module
A1 Auditability
A2 Certification Schema
Requirements
ISMS standards
comparison example
Estonian
Case
Most suitable
Suitable
Suitable with some exceptions
Not suitable
• Limitations
• Different detail and maturity level
• Differences in requirements importance
• Conclusion
• Reusable requirements to compare standards or guidelines
• Each country has to do its decision by itself
• Suggestion to ENISA to develop EU based security standard
Building blocks of
security level
evaluation •framework and its principles
•Seeba, M., Mäses, S., Matulevičius, R. (2022). Method for Evaluating Information Security
Level in Organisations. In: RCIS 2022. Lecture Notes in Business Information Processing,
vol 446. Springer, Cham. https://doi.org/10.1007/978-3-031-05760-1_39
•Content versions http://dx.doi.org/10.23673/re-298; http://dx.doi.org/10.23673/re-372
F4SLE- Framework for Security level Evaluation
•How to update the F4SLE
•process, principles, inputs
•submitted manuscript: Seeba,M., Affia, A.-a.,O., Mäses, S., Matulevičius, R. Create Your
Own MUSE: a Method for Updating Security Level Evaluation Instruments
MUSE - Method for Updating Security Level Evaluation Instruments
•tool to use F4SLE
•2023Master thesis project of Maria Pibilota Murumaa “Designing a tool for security level
evaluation framework”
•CHESS mini project
•immidiate results to respondents and sending the aggregated results to central server
MASS
• INITIAL
• The need to deal with
information security has been
acknowledged and addressed
• DEFINED
• Formal processes have been
agreed, and the necessary
information security supporting
documents have been
prepared
• BASIC
• Practical basic activities have
been implemented to manage
information security
• STANDARD
• There are clear organisational
policies and principles.
Activities are standardised,
documented, regular and
monitored. There is ongoing
monitoring and improvement.
Use case 0
• The organisations used the coloured table to interpret
organisation security using traffic light colours and the
dominant visual colour to indicate the current
security status before calculations.
Use case 1
• Then average result of the organisation by each
dimension and maturity level the Fig 1. Colours
transferred into quantifiable form:
• Three level scale: red= 1, yellow =2, green=3;
• Four-level scale: red= 0, orange= 1, yellow=2, green=
3.
Use case 2
• The sum each level's average value by dimension to
get the information security level of organisation by
dimensions (Fig 2, blue line).
Use case 3
• The average value of information security level by
dimension based on all organisations for the
benchmark (Fig 2: red line).
Use case 4
• The benchmark usage as an input for state-level
political and strategical decisions.
Standard levelBasic levelDefined levelInitial level
Dimension, its
description and
comments
Attributes
Req. 1: Framework should
cover a wide area of
security-related topics
• Procedural and technical measures.
• Comprehensive categories should still allow
minor modifications or additions to the more
specific topics as the technology evolves.
• Technology independent
• It should be possible to categorise any upcoming
security control to an already existing category.
Req. 2: Framework should
produce quantifiable and
comparable
• Organisation security dynamics observation
• Evaluation should be based on evidence
• To compare different organisations between
each other or against a security benchmark.
Req. 3: Framework should
be quick and easy to
implement and understand
• While the actual implementation of the security
controls might take a long time, the evaluation
should be intuitive to follow and take less than 1
hour.
Req. 4: Framework should
be aligned with a security
standard
• Following the standard structure helps to give
the measurements a more coherent structure
and avoids extra effort done to comply with the
standard.
Baseline standard
We used the Estonian information security standard (E-ITS) [1]
Baseline Catalogue (compliant with ISO27001)
Dimensions of the framework
Ten module groups of E-ITS:
• ISMS, ORP, CON, OPS, DER are procedural,
• INF, NET, SYS, APP, IND are system based technical modules.
Framework levels
E-ITS measures are ordered Basic, Standard and High.
Exclusion of High to include only mandatory part.
E-ITS Basic divided into three levels:
1. Initial Level - organisation solves its security ad hoc and on a
need-based
2. Defined Level - formal compliance documentation
requirements
3. Basic level - processes taking place
Standard level - equals with E-ITS Standard security measures.
Allows the organisation to deal with unknown risks by
significantly reducing their potential impact and loss.
Figure 1. An organisation’s security levels Figure 2. Comparison with bencmark
Example fragment of framework content
Requirements for security evaluation framework
Information Security Evaluation Framework Design
Attributes of the Framework
Respondent could find evidence for each attribute
implementation status.
Evaluation scale for attributes
Four-level scale
• quantifies the dynamics of organisation security even in the
case of minor changes
• forces the respondent to decide whether the situation is
somewhat positive or rather negative.
Interpretation Use Cases
Demonstration and evaluation Limitations
Framework with its full content is available at [2].
Method for Evaluating Information Security
Level in Organisations
Mari Seeba1, Sten Mäses2 and Raimundas Matulevičius1
1 University of Tartu, Institute of Computer Science, Tartu, Estonia
2 Tallinn University of Technology, Department of Software Science, Tallinn, Estonia
Use case 0
• The organisations used the coloured table to interpret
organisation security using traffic light colours and the
dominant visual colour to indicate the current
security status before calculations.
Use case 1
• Then average result of the organisation by each
dimension and maturity level the Fig 1. Colours
transferred into quantifiable form:
• Three level scale: red= 1, yellow =2, green=3;
• Four-level scale: red= 0, orange= 1, yellow=2, green=
3.
Use case 2
• The sum each level's average value by dimension to
get the information security level of organisation by
dimensions (Fig 2, blue line).
Use case 3
• The average value of information security level by
dimension based on all organisations for the
benchmark (Fig 2: red line).
Use case 4
• The benchmark usage as an input for state-level
political and strategical decisions.
Standard levelBasic levelDefined levelInitial level
Dimension, its
description and
comments
Attributes
1. Problem
identification
2. Solution
objectives definition
3. Design and
development
4. Demonstration 5. Evaluation
Process iteration
6. Communication
RQ: How to
evaluate the level
of information
security?
Reqirements of
security evaluation
framework
Design decisions
for framework
development
Testing the framework
content on real
organisastion.
Interpretation of
results
Validation of framework,
its alignment with
requirements, limitations
of the framework
This paper itself
Research questions:
How to evaluate the level of information security in the
organisation?
RQ1: What are the requirements of the security evaluation?
RQ2: How to conduct the evaluation of security level?
RQ3: How to use and interpret the results of information security evaluation?
Design science research method
Req. 1: Framework should
cover a wide area of
security-related topics
• Procedural and technical measures.
• Comprehensive categories should still allow
minor modifications or additions to the more
specific topics as the technology evolves.
• Technology independent
• It should be possible to categorise any upcoming
security control to an already existing category.
Req. 2: Framework should
produce quantifiable and
comparable
• Organisation security dynamics observation
• Evaluation should be based on evidence
• To compare different organisations between
each other or against a security benchmark.
Req. 3: Framework should
be quick and easy to
implement and understand
• While the actual implementation of the security
controls might take a long time, the evaluation
should be intuitive to follow and take less than 1
hour.
Req. 4: Framework should
be aligned with a security
standard
• Following the standard structure helps to give
the measurements a more coherent structure
and avoids extra effort done to comply with the
standard.
• For benchmark validation bigger reference group is
needed
• Self-assessment or third party assessment or partly
automated?
• Benchmark tool and falsification threat
• Updating responsibility – clear criterias
• Difficulties with interpretation – need to know the
dimensions content
• Generalisation difficulties
Baseline standard
We used the Estonian information security standard (E-ITS) [1]
Baseline Catalogue (compliant with ISO27001)
Dimensions of the framework
Ten module groups of E-ITS:
• ISMS, ORP, CON, OPS, DER are procedural,
• INF, NET, SYS, APP, IND are system based technical modules.
Framework levels
E-ITS measures are ordered Basic, Standard and High.
Exclusion of High to include only mandatory part.
E-ITS Basic divided into three levels:
1. Initial Level - organisation solves its security ad hoc and on a
need-based
2. Defined Level - formal compliance documentation
requirements
3. Basic level - processes taking place
Standard level - equals with E-ITS Standard security measures.
Allows the organisation to deal with unknown risks by
significantly reducing their potential impact and loss.
References:
[1] RIA (Estonian Information System Authority): E-ITS. https://eits.ria.ee/
[2] Seeba, M.: Estonian Information Security Standard (E-ITS) Based Security Level Evaluation Instrument (2021). https://doi.org/10.23673/re-298
Figure 1. An organisation’s security levels Figure 2. Comparison with bencmark
Acknowledgements: This paper is supported in part by European Union's Horizon 2020 research and innovation programme under grant agreement No 830892, project SPARTA.
Example fragment of framework content
Requirements for security evaluation framework
Information Security Evaluation Framework Design
Attributes of the Framework
Respondent could find evidence for each attribute
implementation status.
Evaluation scale for attributes
Four-level scale
• quantifies the dynamics of organisation security even in the
case of minor changes
• forces the respondent to decide whether the situation is
somewhat positive or rather negative.
Interpretation Use Cases
Demonstration and evaluation Limitations
Experiment
First iteration:
10 organisations
self-assesment, DOCX coloring
Second iteration:
1 organisation
self-assesment, DOCX coloring
Third iteration:
security expert independent review
Framework with its full content is available at [2].
Seeba, M., Mäses, S., Matulevičius, R.
(2022). Method for Evaluating Information
Security Level in Organisations. In: RCIS
2022. Lecture Notes in Business Information
Processing, vol 446. Springer, Cham.
https://doi.org/10.1007/978-3-031-05760-
1_39
14
How to update security level evaluation instrument attributes in a way that
• results are comparable
• in long-term for all use cases
• organisation level,
• benchmark providing,
• central view?
Method to
update security
evaluation
instrument
MUSE
• Baseline
• Source of attributes security
controls,
principles, regular
updateing
• E-ITS 2022
• Threat landscape report
(attributes relevance):
• ENISA Threat
Landscape Report
2022,
• RIA annyal
cybersecurity book
(2023 predictions)
• Reference standard
• fixed scope:
• ISO27002:2022
MS Word MS Excel MASS Tool
MASS – web based tool to
simplify F4SLE usage
• Privacy principle – raw data does not leave from the
respondent
• Only aggregated data will be sent to the server
• Immediate results to respondent
• Benchmark creation based on aggregated data
Test environment: https://mass.cloud.ut.ee/test-massui/#/
Production environment: https://mass.cloud.ut.ee/massui/#/
Organisation result
What to do with the results?
• Preparing for audit
• Input to security implementation plan, priorities
• Management review input
• Security dynamics monitoring
• Understanding the standard
• Partners assessment (sh X-tee teenused)
• Partner self-assessment / auditor tool
• Central analysis
• Industry based benchmark
• Input to plan supporting activities
• Monitor the changes
PILOOT 2/2 2023 PILOOT 2/1 2022
Conclusion
• Requirements of choosing standard
• Implementing requires evaluation
• Evaluation instrument needs updating
• Estonian case
Thank you!
Mari.Seeba@{ria|ut}.ee