prof. Barbara Russo
Free University of Bozen-Bolzano, Italy
SwSE - Software and Systems Engineering
Masaryk University, November 10th 2016
Mining logs to predict
system errors
2
•Computational power increases
•(2015) Tianhe-2, China - 3,120,000 cores
•The larger the system, the more frequent
critical events
•Lower overall system utilisation
•Hardware failure, software failure, and
user errors
Today systems
3
•Crashes
•immediately stop the system
•easily identifiable (e.g., disk failure)
•but can originate a large number of events
spread across components
•Deviations from the expected output
•let the system run
•reveal only at completion of system tasks
Manage failures …
4
… we need information on system behaviour
and make failure predictions
To better manage failures …
5
• Part of such data traces the change in
behaviour of the system and its sub-
components
• Logging services store state changes of a
system in archives, logs
Systems generate big data
6
•How can we exploit log data to model and
predict system behaviour?
Mining logs
7
•A log event represents a change in a
system state
Log events
8
xml log event
<Info
TimeStamp=“2015-04-08T07:32:37.345”
File=“XXX.Control.ObservingModes.ObservingModeBaseImpl"
Line="231" Routine=“beginSubscan"
Host="XXX01"
Process=“XXX/javaContainer"
SourceObject=“XXX/Array005"
Thread="RequestProcessor-35023"
LogId=“343355"
Audience=“Operator">
<![CDATA[Text message … here]]>
</Info>
9
•Some events can tell about undesirable
system behaviour
System misbehaviour
10
•Events in error state (error events) act as
alerts of system failures:
• Interpretation of event data might be hard
• Originated from a series of preceding events
Error events act as alerts
11
Logs can be cryptic
SAP
12
•Failure, but the program exited cleanly
Interpretation
YY-MM-DD-HH:MM:SS NULL ZZZ MYHOST FAILURE xxx exited
normally with exit code 0
13
•If the system administrator was doing
maintenance on the machine, this
message is a harmless artefact of his
actions
•If it was generated during normal
machine operation, this message
indicates that all running jobs on the
computer were undesirably killed
Interpretation
14
•We need to understand the operational
context
Operational context
15
•System changes can have introduced errors
much earlier than an error manifests in logs
Originated from a series of
preceding events
16
The classification problem
Data Sets Classifier
G2 =Non-Faulty
G1= Faulty
Features
17
•Event sequence: set of events ordered by
their timestamp occurring within a given
time window
•A sequence abstraction is a
representation of such sequence (e.g.,
vector) that can be used to feed classifiers
(features)
Sequences
18
A. Isolating sequences
•Identify sequence length
•Characterise sequence information
B. Build sequence abstraction (e.g., a vector)
C. Build features
Building features
19
Isolating sequences
Different length, different types
20
•μi – number of the events of type i in a
sequence (multiplicity)
•sv=[μ1, …,μn] – vector of event
multiplicities
Sequence abstraction
21
{General, Log In, Performance, Systems}
sv1=[0,1,0,1]
sv2=[2,1,1,0]
Example – sequence
abstraction
22
Multiple sequences and users
µ1 … µn
s7
s30
s2
s14
s10
Same length, same types
23
•v= [sv, μ(sv), ν(sv)] – feature
•μ(sv) = # sequences mapping onto sv
•ν(sv) = average # of users in sequences mapping
onto sv
•ρ(sv) = number of errors in sequences
mapping onto sv
•v is an faulty feature if at least one event
in one sequence is in error state, ρ(sv)>0.
Features
24
v1= [0,1,0,1;1,1], sv1=[0,1,0,1]
μ(sv1) =1, ν(sv1)=1, ρ(sv1)=0
v2 = [2,1,1,0;1,2], sv2=[2,1,1,0]
μ(sv2) =1, ν(sv2)=2, ρ(sv2)=2
Example - features
Which models do we use to
predict system behaviour?
25
26
The classification problem
Data Sets Classifier
Different ex-ante
distributions:
(faulty, non-faulty)
G2 =Non-Faulty
G1= Faulty
Ex-post classification differs
on different classifier’s
thresholds
Features
27
•The problem varies depending on how
many errors we allow in the system
•c – cut-off value, i.e., number of errors in a
feature
•Categories:
•G1(c)={v = [sv, μ(sv),ν(sv)] | ρ(sv)≥c} - faulty
•G2(c)={v = [sv, μ(sv),ν(sv)] | ρ(sv)<c} - non-faulty
Parametric classification
28
Build classifiers on historical
data
Classifier
Training Set
Test Set
1. To tune classifier’s
parameters
2. To compute classifier’s
fitting performance
29
Compare prediction
performance
Classifier1
Validation
Set
Classifier2
Classifiern
…
30
•Did we put too much information in our
features?
•Information Gain selects feature
attributes that most contribute to the
information of a given classification
category
Information Gain
The case study - a telemetry
system
32
System applications
33
•Different kernels
•Multilayer perceptron
•Linear
•Radial Basis Function
Support Vector Machines
34
•Best performance at individual application
(MP, c=3):
•1% false positive rate, 94% true positive rate,
and 95% precision
•Best performance across applications
averaged over models for c=2,
•9% false positive rate, 78% true positive rate,
and 95% precision,
Findings
What can predictions tell
administrators?
35
36
FPr=11%, TPr=82%
37
Example
38
•Application that manages software tools of
cars
•Pervasive in the telemetry system
•106 distinct sequences of 10 different
event types, 18% multiple sequences, and
89% with more than one user	
Example - ST6
39
•c=1
•G1(1)={v = [sv, μ(sv),ν(sv)] | ρ(sv)≥1}
•G2(1)={v = [sv, μ(sv),ν(sv)] | ρ(sv)<1}
•IG reduction from 12 to 7 still including μ
and ν
ST6 - Analysis
40
Confusion matrix - MP pred.
TPr
FPr
41
•Behaviour is the same in next three
months
•1000 sequences
•Category balance in future sets is the one
of the test set (39%)
•390 faulty sequences and 610 non- faulty
sequences
Prediction - assumptions
42
•450 (45%*1000) predicted faulty
sequences
•Predicted faulty sequences that have no
errors:
•67 = 11%*610
•Predicted non-faulty sequences that have
an error
• 70 =18%*390
In numbers
Pred pos Pred neg Total
Pos 82% 18% 100%
Neg 11% 89% 100%
Total 45% 54% 100%
43
• Inspection cost.
•Wasting time ≥ 67 * average cost to fix
one error
• Cost for undiscovered errors.
•Defect slippage ≥ 70
Cost of prediction
Recapitulation
Accuracy to measure
costs in prediction
Sequences to model
system changes
Classifiers to model and
predict system behaviour
Thank you