Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis PA197 Secure Network Design 8. Network Monitoring Eva Hladká, Luděk Matýska Faculty of Informatics March 26, 2019 Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Q Traffic monitoring (principles and tools) Q Traffic analysis • Tools 0 Netflow • Principles • IPFIX • Advantages and usability 0 Network Behavior Analysis • DDoS vs flash crowds Eva Hladká', Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis 9 Continuously monitor the computer network o Collect information • Perform analysis o Send alerts • Part of network management • Wider scope than IDS • "natural" causes of network problems • Traffic monitoring versus service monitoring • e.g. status of a particular web server Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis • Information about flows in the network • to improve Quality of Service to get global view on flows • flow between different networks o bandwidth optimization for content providers o Information about applications and frequency of their use to tune network parameters to get better performance • To group users sharing the same network • To allow smart logging • conforms to the law • optimize log files • To have sufficient data for experiments • traffic generators • To detect malicious traffic Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Tools Network Behavior Analysis • By port • applications operating on fixed port numbers • simple • unreliable • Deep packet inspection (DPI) • QoS based • rather unreliable • Statistical classification • remember IDS? Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis • Diagnostic tools • usually active • connectivity and reachability tests • Monitoring tools active or passive • run "on background" • collect events (passive) • initiate own probes (active) 9 Performance tools • flow monitoring Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis -,- , Netflow Tods Network Behavior Analysis • M RTG • Wireshark • ntopng • SolarWinds Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools M RTG • Multi Router Traffic Grapher • Free software to monitor and measure traffic load on network links written in Perl • available on Linux, OS X. MS Widows, UNIX, ... o Uses SNMP calls to send requests • only SNMP-enabled devices could be monitored • Creates an HTML document with the list of graphs to display traffic from selected devices Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis -,- , Netflow Tools Network Behavior Analysis Rfŕ íflr. Í/Hr S* JxfPOrti T.vfc Hŕfc H.. - f|ÍÍ --j'- * lillť-.'.ljŇhh iiiVlihi íjiiViiiHli "3 isa±Jn^™^r M RTG Index Page n i! ía » a t e i e s » u m m m » a .la*.* t. 500.0 ř 4-f- 44. i..j..ĺ. 4-i.. ..j..:. 4.j.. 4-f- -Y-i- .. 't' - "it I--J-4-- .44- .44. .44- .44- .44- "1" ..U. ■H- ■ - .j.j. -f\j- ■\-\~ ■■ H H li » e l ť 6 K li U H H » CPU Lpail Syďhľn. Use: -und Mice P; -+-řŤ+-!-Ť-tlÍ-Ť+-!-Ť-t:-!-Ť + -!-Ť+-!-Ť + -| 1 -h- -U J.4. U H LJ Jv Ü (■ 1 4 t t 10 U U U U A ^ 7».« H ■. '. . ■ e».0 H 0.0 H —i-"— ■—!-.- i—.j— ŕ-'-ŕ"-!— i— — ■— I— í— ■—1-—í— ■— ■— E— -Í--Ž--Š--Í--Í--Š--Í--Í--Š--Í-- U US LI 20 22 O Z * E I ld U U U II H Z Nüiv ~.\:\> Cnnnnr.f r r r. l-sl.i blishň r. ' i: P tľnnnc ntipn r, MO.* em,* .. . 4,4 -í- t"! "ľ"f *4- 14 K 19 X 21 -y- .-i-t-i- .....■■ i ■■ ŕ ' 6 S 1( 11 U K Ľ) ;i i " t] e.4 -f-ř- - 1 tllllllilllllHTklttU, HtW -h-ř' 14 li U K e ť ř S 11 li 14 U 14 » i ^Ihlx ■ ■ I ■ ear mrtg.gif Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis -,- , Netflow Tools Network Behavior Analysis • An open source packet analyzer • free software (under GNU GPL) • available for Linux, OS X, MS Windows • Similar to tcpdump, with with extensive GUI <* Understands the structure of many network protocols • protocol field parsing • Uses promiscuous mode on the monitored interface Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools Jitopng • Next generation of ntop • see http://www.ntop.org • Network traffic probe • shows network usage in a similar way to the UNIX top command 9 Use of web browser interface • ntop servers as a web server o Features (selected) • sort network traffic according to the protocols used • show IP traffic distribution among various protocols • analyze IP traffic and sort it 9 display IP traffic subnet matrix (who is talking with who?) • geolocate hosts • store traffic statistics in RRD format Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis -,- , Netflow Tools Network Behavior Analysis • A commercial product • http://www.solarwinds.com 9 Extensive suite of monitoring tools • multi-vendor fault, performance, and availability monitoring • dynamic network maps • customizable topology and dependency-aware intelligent alerts • automated capacity forecasting, alerting, and reporting • deep packet inspection and analysis • Also other products • applications and system optimization • database performance acceleration • security and compliance enhancement Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis -,- , Netflow Tools Network Behavior Analysis Router- i Al I Details nvtrigt mipotitl Tl m* 8. PJekil Leu Aic, R.-1J1 ľ in- ■■ Puck* Dm Avtngt CPU Lc-Jd « Mamory Utlllutlon IM I 5 1.1 Memory Used Kir* rjttAlll uonur^eturcr Service Too l j 11 Poll Ttma .... Currant Hardware Haallh Mam vkvt 39 Jf Fan !+! f Power Supply + Temporalum Tflp 10 Flapping Reulei IDC * S l»«J0 ■: :': It «10 TS! 6: cien v Kl 1* i* a i* « B i' 4 0 ť 3 0 •■: .■ 1 M ■.->;:":." - SWOiPW) «MI Pu. -wm é M>< Ajo 1TJ1MQ1 + 247:14 All] 4*flMI ? Ů*y* 400 (M *» 14 E M 47PU.i H4VI2 4*yt ijrt (17143014 3 0«t*ult Kout* Chang*! (1 -k ordsl u** * Last 7 Akr* * n c 0.1 *mm it>cs.í if '*«Ki»poif-iWŮ«4M PU'i H*iwork >»ifr-f, & P«fc»1 Lfltl Core ftouler ŕ> H»". UM**. Ji* 17»14. 1? Jfl tv? ,'. ■, lt. I2h24h I ? 100 mi IWwJiy, Juy ir Km 1214.» MiruM«JAv*riä* BEipsn« Hrň* Con Mi* »u. iz»o«-M «»K íl Mem-limn Iti I2h2*h =j 200ms J 100 *J Í 0 mi i:ooam iooam e00am $00am IJ0OPM □ < ■ >*c«.w Hit Cen Routif JuH72flU I7»»ps-Jul Iľjalf 12»pm 2oom )l> I2l>21h 1ÍO0MI j «au 6«ca« □ Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) p . . , Traffic analysis IPFIX * i i o i ■ Netf'ow Advantages and usability Network Behavior Analysis 9 Introduced by CISCO • Available at CISCO routers to collect IP traffic at interfaces • Analysis of netflow traffic can help • to determine source and destination of traffic • class of service • congestion • Components • flow exporter • router: aggregates packets into flow • sends them to collector • flow collector • reception, storage and preprocessing • analysis application Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis • NetFlow version 5 9 Flow is a unidirectional sequence of packets that all share the following 7 values: O ingress interface Q source IP address Q destination IP address O IP protocol Q source port for UDP or TCP; 0 for other protocols O destination port for UDP or TCP; type and code for ICMP; 0 for other protocols O IP type of service Routing information is not included as it may change during flow lifetime (e.g. due load balancing) • Also user defined key are allowed in advanced implementations Principles IPFIX Advantages and usability Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) p . . , Traffic analysis IPFIX * i i o i ■ Netf'ow Advantages and usability Network Behavior Analysis o NetFlow designed to process all packets • router implementation • Performance implications for high bandwidth links • Sampled NetFlow • only one packet in n is processed • deterministic sampling: exactly each n-th packet • random sampling • more complex patterns per flow sampling • sampling rate per router or per interface Sampling introduces errors • INVEA-TECH probes for wire speed at multigigabit networks Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Principles IPFIX Advantages and usability BF|X • IP Flow Information Export • IETF protocol • Standard of export for IP flow information from routers, probes, ... • Based on NetFlow version 9 • Defined in the following RFCs: 5103, 7011-7015 • IPFIX flow • packets that share same properties observed in a specific timeframe • Basic Architecture contains • metering process collects data at an observation point • exporter sends collected flow information to a collector o A many-to-many relationship exists between collectors and exporters • IPFIX is push protocol Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Principles IPFIX Advantages and usability • Unobtrusive • the attackers can't detect flow monitoring 9 can slow down high traffic bandwidth • esp. not sampled monitoring 9 Relatively easy to implement 9 information taken from routers • probes in the network • Substantial processing power required • for real-time monitoring • more extensive analysis possible off-line for limited time periods Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) p . . , Traffic analysis IPFIX * i i o i ■ Netf'ow Advantages and usability Network Behavior Analysis • Observing limits and security policies • users' compliance with network use policy • service use (for network optimization) • QoS monitoring • passive, but potentially biased • Traffic accounting Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) p . . , Traffic analysis IPFIX * i i o i ■ Netf'ow Advantages and usability Network Behavior Analysis P2P network/service detection IP port scanning detection • TCP RESET packets increase for vertical scan • high increase of ICMP Host Unreachable packets for horizontal scan DoS attacks detection • e.g. TCP SYN-flood attack • flash-crowd effect (see later) Worms and viruses spread detection high number of unexpected open connections to other computers Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds ior Analysis o Detection of unusual actions through traffic monitoring • Monitor network inside an organization 9 many monitoring points • aggregation • trends spotting • including e.g. bandwidth fluctuation o Machine learning methods • what is normal behavior? • Complements IDS, firewalls, . .. Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis c „ , , k i ^r, DDob vs flash crowds Netflow Network Behavior Analysis Basic steps • uses history of traffic observation to build a model of selected relevant characteristic of network behavior • predict these characteristics for the future traffic o identify the source of discrepancy between predicted and measured values Adaptable, no limit for the detection strength • artificial intelligence approach Error rate the main potential problem o single NBA methods usually prone to high number of false negatives • multistage collaborative methods, trust modeling etc. used to overcome this shortcoming Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowd DDoS vs flash crowds • Web server example • highly variable usage patterns o unexpected increase in the traffic • attack or information attractivity • DDoS attack • malicious activity • aim to shutdown the web server • distributed access patterns • Flash crowd (Slashdot effect) • massive increase of traffic to a web site • due to sudden interest • often through linking from a popular site 9 Difficult (impossible?) to distinguish Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds Distinguishing Flash Crowds o An example taken from the following article • Ke Li et al (2009): Distinguishing DDoS Attacks from Flash Crowds using probability Metrics. Network and System Security NSS'09, pp. 9-17, DOI 10.1109/NSS.2009.35 • Differences between Flash crowds and DDoS attacks • intent: users want content, DDOS wants the site shut down • users coming from the whole community network or the whole Internet • aggregated source IP addresses resemble flat fractional Gaussian noise distribution • DDOS from attackers/botnet • aggregated source IP addresses follow Poison distribution • difference in traffic increase/decrease • users follow the spread wave (gradually increase the traffic) attackers use rather short time frame during the initial phase of attack Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis c „ , , k i ^r, DDob vs flash crowds Netflow Network Behavior Analysis • Based on a hybrid probabilistic method • using similarity between flows to distinguish normal versus flash crowd versus DDoS flows • similarity measured as n i=l where P = (pi, P2,..., pn) and Q = (qi, q2,..., qn) are two probability distributions p(P, (?) = 1 for P = Q and p(P, (?) = 0 when P and Q are • total variation calculated as T{P, Q) = Y,i = l"\Pi-qi Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds 9 The algorithm (applied at the last router preceding the server) • set grouping thresholds GTs (similarity) and GTj (variance); each threshold has an lower and upper bound • calculate probabilistic distribution for each aggregated flow • calculate total variation 7~(P, Q) and similarity Q) for each two flows • if T > upper(G7~7-) and p < lower(G7~s) the DDoS is detected from Flash crowds • if lower(G7~7-) < T < upper(G7~7-) and lower(G7~s) < p < upper(G7~s) then DDoS is detected from Normal flow 9 if T < upper(G7"7-) and p > lower(G7~s) than Flash crowds is detected form Normal flow • otherwise Normal flow is assumed • The values for upper and lower band of thresholds GTj and GTs was derived from simulations and are (0,5921,1.1045) _and (0.7220,0.8708), resp._ Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds • Based on article o P.R.Reddy et al (2013): Techniques to Differentiate DDoS Attacks from Flash Crowd. Int. J. Adv. Res. Comp. Sci. Soft. Eng., Vol 3(6), pp. 295-299. • Uses flow correlation coefficient • Similar observations as above • individual attack flows show an internal similarity—flow standard deviation is usually smaller than that of genuine flash crowd flows o smaller number of botnet nodes compared to number of genuine flash crowd users • each botnet node must initiate higher number of attack flows to mimics the expected number of users Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis c „ , , k i ^r, DDob vs flash crowds Netflow Network Behavior Analysis • Flow correlation coefficient o Packet arrival patterns • Information distance o In all cases, the differentiation is based on smaller variance in DDoS attack flows • the correlation coefficient use experimentally verified as the most promising metrics Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis c „ , , k i ^r, DDoS vs flash crowds Netflow Network Behavior Analysis • Traffic monitoring as a very strong mechanism • unobtrusive • not detectable by attacker • Usable in a large range of scenarios • performance as well as security related • Support from network elements needed • probes • router implementation • NetFlow and IPFIX • Network behavior analysis • example of DDoS versus Flash crowd detection 9 Next session: Wireless Networks (introduction) Eva Hladká, Luděk Matýska PA197 Secure Network Design 8. Network Monitoring