Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis PA197 Secure Network Design 10. Network Monitoring Eva Hladká, Luděk Matýska Faculty of Informatics April 5th, 2023 Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Q Traffic monitoring (principles and tools) Q Traffic analysis • Tools 0 Netflow • Principles • IPFIX • Advantages and usability Q Network Behavior Analysis • DDoS vs flash crowds Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis 9 Continuously monitor the computer network o Collect information • Perform analysis o Send alerts • Part of network management • Wider scope than IDS • "natural" causes of network problems • Traffic monitoring versus service monitoring • e.g. status of a particular web server Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis • Information about flows in the network • to improve Quality of Service to get global view on flows • flow between different networks o bandwidth optimization for content providers o Information about applications and frequency of their use • to tune network parameters to get better performance • To group users sharing the same network • To allow smart logging • conforms to the law • optimize log files • To have sufficient data for experiments • traffic generators • To detect malicious traffic Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools Traffic Classification • By port • applications operating on fixed port numbers • simple • unreliable • Deep packet inspection (DPI) • QoS based 9 rather unreliable • Statistical classification • remember IDS? Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis • Diagnostic tools • usually active • connectivity and reachability tests • Monitoring tools active or passive • run "on background" • collect events (passive) • initiate own probes (active) • Performance tools • flow monitoring Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools • M RTG • Wireshark • ntopng SolarWinds Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools M RTG B • Multi Router Traffic Grapher • Free software to monitor and measure traffic load on network links written in Perl • available on Linux, OS X. MS Widows, UNIX, ... o Uses SNMP calls to send requests • only SNMP-enabled devices could be monitored • Creates an HTML document with the list of graphs to display traffic from selected devices Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools -Jaj ■! 0 ^] - f§J :'.ŕ: ^ ft.hnp.'.ljFjt.jj iiij-iid L.jiiVii.nM M RTG Index Page Trark Arminia Fťrf 192.186.1.1M> ..bfrJHty n la » a 6 n u u n u » a CPU LoaJ System. Use: and Nfcť Ť+Ť-lt-řŤ+-ř--+-řŤ+ŤŤ-*Ť+-řŤ-l:-řŤ+-řŤ+-řŤ+-: -h- -!-|- H Li L9 JO 22 0 1 H ŕ [.L0L2L-ILSL120 22 ■ .. . 4,4 -í- h- -r-ř- ■ —i —■ —: —í ■ť- U 1! li » a t E 1 ť ->-f M U M M u » a n^^ui^H írmi iLiui hl» . Ľ.+J.0 t. SSO.« h k Kfl.0 t. E. j BD.D L B 4.4 t. '-[.- 4-í- -i-!-- ■4-1. 4-í. 4-L- 4-í- 4-[- 4-i- .. Ý ■it' 'Í"Š" - j-i-j.. ■it' "Í" ■f {- -fi" - -j.j. ■f|" ■.vi"' -■ H lij u » a t E t ť 8 M li M li M » a H LS Ll 20 22 10 U LI LS LI 20 22 -s:.i r uhň r. ™ i i ~" Cnnnn:.ľ r: r ä í E.4 S J.o 4.4 -l-ř-h-K---K-l-ř-h -T-f- - -h-r-r H H li » E t ť ŕ M li M M M » a| mrtg.gif Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools • An open source packet analyzer • free software (under GNU GPL) • available for Linux, OS X, MS Windows • Similar to tcpdump, with with extensive GUI <* Understands the structure of many network protocols • protocol field parsing • Uses promiscuous mode on the monitored interface Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools • Next generation of ntop • see http://www.ntop.org • Network traffic probe • shows network usage in a similar way to the UNIX top command 9 Use of web browser interface • ntop servers as a web server a Features (selected) • sort network traffic according to the protocols used • show IP traffic distribution among various protocols • analyze IP traffic and sort it • display IP traffic subnet matrix (who is talking with who?) • geolocate hosts • store traffic statistics in RRD format Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Tools 1 • A commercial product • http://www.solarwinds.com • Extensive suite of monitoring tools • multi-vendor fault, performance, and availability monitoring • dynamic network maps • customizable topology and dependency-aware intelligent alerts • automated capacity forecasting, alerting, and reporting • deep packet inspection and analysis • Also other products • applications and system optimization • database performance acceleration • security and compliance enhancement Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Network Behavior Analysis Traffic analysis Tools Netflow Router -üAII Details Average Ruponie Tlrn* & Packet Lůss A vij Rt*p ľimc Average CPU Load & Memory UWnation Hardware Detain Minu r tourer Uwta-1 L*it Poll Time Curreni Hardware Healih ■■■ ľ i:: T/1 MOU IÍ*?.JÍftí ■mop saví S Jf Fan S • Powof Supply 4 TempoMUif* Tep 10 Flapping Rautet MKT HOP T ^o™0- ja K] M4U11 íij-Hf? (m mo u 55T04PU) »o« »e LeM *t-x/1 ľHfvjo WÚU JJ5?fll PU) 10450 » H # AMtrS H w*CSPf ;::i :a, j;; ■ :: 2S7:14AII) 104 80 M J si c:aÉSs looeo 2* 3 • Rout**? u tciCP —^- Oefautt Route Chjngei (1 records) ^ |Seent) 11 co 1 «Med Heiwork Latency & Packet Lett ■Cor* ftouter M K »14. UH*n-Jrf17 2014. Ii M fr* 2mid IS :ih£4k B tiporttaPW TtvwMy, JuV 17.2614 1246» Min.iMjj;JAv*rig* Re i pen íl- Time Core Router M1 * W14. »2 » «n - Jul lľ S»U12 » Cffl g 20Cms S 0*M UM AM 6«AW í 00 AU UMPH 1200 FM i0 1: c y. ■ lírtVu R«Sfr-H Tŕň* C*f* «Mi* Latency Tí,-.: ! . EWC*T KU> Core Routes Jul 17 HU >ZM i- ■ Jul I72CU 12»» Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) p . . . Traffic analysis idciv N tfl rl/\ i r-, i ■ a i ■ Advantages and usability Network Behavior Analysis 9 Introduced by CISCO • Available at CISCO routers to collect IP traffic at interfaces • Analysis of netflow traffic can help • to determine source and destination of traffic • class of service • congestion • Components • flow exporter • router: aggregates packets into flow • sends them to collector • flow collector • reception, storage and preprocessing • analysis application Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) p . . . Traffic analysis idciv N tfl rl/\ i r-, i ■ a i ■ Advantages and usability Network Behavior Analysis NetFlow version 5 Flow is a unidirectional sequence of packets that all share the following 7 values: O ingress interface Q source IP address O destination IP address O IP protocol O source port for UDP or TCP; 0 for other protocols O destination port for UDP or TCP; type and code for ICMP; 0 for other protocols Q IP type of service Routing information is not included as it may change during flow lifetime (e.g. due load balancing) Also user defined key are allowed in advanced implementations Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) p . . . Traffic analysis idciv N tfl rl/\ i r-, i ■ a i ■ Advantages and usability Network Behavior Analysis o NetFlow designed to process all packets • router implementation • Performance implications for high bandwidth links • Sampled NetFlow • only one packet in n is processed • deterministic sampling: exactly each n-th packet • random sampling • more complex patterns per flow sampling • sampling rate per router or per interface Sampling introduces errors • INVEA-TECH probes for wire speed at multigigabit networks Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) p . . , Traffic analysis idciv N tfl rl/\ i r-, i ■ a i ■ Advantages and usability Network Behavior Analysis IP Flow Information Export IETF protocol Standard of export for IP flow information from routers, probes, ... Based on NetFlow version 9 Defined in the following RFCs: 5103, 7011-7015 IPFIX flow • packets that share same properties observed in a specific timeframe Basic Architecture contains • metering process collects data at an observation point • exporter sends collected flow information to a collector o A many-to-many relationship exists between collectors and exporters • IPFIX is push protocol Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Principles IPFIX Advantages and usability 1 • Unobtrusive • the attackers can't detect flow monitoring o can slow down high traffic bandwidth • esp. not sampled monitoring 9 Relatively easy to implement • information taken from routers • probes in the network • Substantial processing power required • for real-time monitoring • more extensive analysis possible off-line for limited time periods Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Principles IPFIX Advantages and usability • Observing limits and security policies • users' compliance with network use policy • service use (for network optimization) • QoS monitoring • passive, but potentially biased • Traffic accounting Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis Principles IPFIX Advantages and usability P2P network/service detection IP port scanning detection • TCP RESET packets increase for vertical scan • high increase of ICMP Host Unreachable packets for horizontal scan DoS attacks detection • e.g. TCP SYN-flood attack • flash-crowd effect (see later) Worms and viruses spread detection high number of unexpected open connections to other computers Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds rior Analysis 1 o Detection of unusual actions through traffic monitoring • Monitor network inside an organization • many monitoring points • aggregation • trends spotting • including e.g. bandwidth fluctuation o Machine learning methods • what is normal behavior? • Complements IDS, firewalls, . .. Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis «-,-,, , ,, DDob vs flash crowds Netflow Network Behavior Analysis 1 Basic steps • uses history of traffic observation to build a model of selected relevant characteristic of network behavior • predict these characteristics for the future traffic o identify the source of discrepancy between predicted and measured values 9 Adaptable, no limit for the detection strength • artificial intelligence approach • Error rate the main potential problem o single NBA methods usually prone to high number of false negatives • multistage collaborative methods, trust modeling etc. used to overcome this shortcoming Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds DDoS vs flash crowd • Web server example • highly variable usage patterns 9 unexpected increase in the traffic • attack or information attractivity • DDoS attack • malicious activity • aim to shutdown the web server 9 distributed access patterns • Flash crowd (Slashdot effect) • massive increase of traffic to a web site • due to sudden interest • often through linking from a popular site 9 Difficult (impossible?) to distinguish Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds Distinguishing Flash Crowds o An example taken from the following article • Ke Li et al (2009): Distinguishing DDoS Attacks from Flash Crowds using probability Metrics. Network and System Security NSS'09, pp. 9-17, DOI 10.1109/NSS.2009.35 • Differences between Flash crowds and DDoS attacks • intent: users want content, DDOS wants the site shut down • users coming from the whole community network or the whole Internet • aggregated source IP addresses resemble flat fractional Gaussian noise distribution • DDOS from attackers/botnet • aggregated source IP addresses follow Poison distribution • difference in traffic increase/decrease • users follow the spread wave (gradually increase the traffic) attackers use rather short time frame during the initial phase of attack Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis «-,-,, , ,, DDob vs flash crowds Netflow Network Behavior Analysis 1 Based on a hybrid probabilistic method • using similarity between flows to distinguish normal versus flash crowd versus DDoS flows • similarity measured as n i=l where P = (pi, P2,..., pn) and Q = (qi, q2,..., qn) are two probability distributions p(P, Q) = 1 for P = Q and p(P, Q) = 0 when P and (? are • total variation calculated as T{P, Q) = Y,i = l"\Pi-qi Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds The Algorithm • The algorithm (applied at the last router preceding the server) • set grouping thresholds GTs (similarity) and GTj (variance); each threshold has an lower and upper bound • calculate probabilistic distribution for each aggregated flow • calculate total variation 7~(P, Q) and similarity p(P, Q) for each two flows • if T > upper(G7~7-) and p < lower(G7~s) the DDoS is detected from Flash crowds o if lower(G7~7-) < T < upper(G7~7-) and lower(G7~s) < p < upper(G7~s) then DDoS is detected from Normal flow • if T < upper(G7"7-) and p > lower(GTs) than Flash crowds is detected from Normal flow • otherwise Normal flow is assumed • The values for upper and lower band of thresholds GTj and GTs was derived from simulations and are (0,5921,1.1045) and (0.7220,0.8708), resp. Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds • Based on article o P.R.Reddy et al (2013): Techniques to Differentiate DDoS Attacks from Flash Crowd. Int. J. Adv. Res. Comp. Sci. Soft. Eng., Vol 3(6), pp. 295-299. • Uses flow correlation coefficient • Similar observations as above • individual attack flows show an internal similarity—flow standard deviation is usually smaller than that of genuine flash crowd flows o smaller number of botnet nodes compared to number of genuine flash crowd users • each botnet node must initiate higher number of attack flows to mimics the expected number of users Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis Netflow Network Behavior Analysis DDoS vs flash crowds • Flow correlation coefficient o Packet arrival patterns 9 Information distance o In all cases, the differentiation is based on smaller variance in DDoS attack flows • the correlation coefficient use experimentally verified as the most promising metrics Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring Traffic monitoring (principles and tools) Traffic analysis «-,-,, , ,, DDob vs flash crowds Netflow Network Behavior Analysis 1 9 Traffic monitoring as a very strong mechanism • unobtrusive • not detectable by attacker • Usable in a large range of scenarios • performance as well as security related • Support from network elements needed • probes • router implementation 9 NetFlow and IPFIX • Network behavior analysis • example of DDoS versus Flash crowd detection • Next session: Operational Security Management Eva Hladká, Luděk Matýska PA197 Secure Network Design 10. Network Monitoring