Network Services Delivery Michal Mesaros Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE What is Shared Network Infrastructure (SNI)? § Provides secure way how to connect from IBM internal network to customer network § SNI is special network architecture inside IBM Global Services Data Center. § Security requirements are very difficult § Is based on few network segment with different security access levels Tier Definitions for SNI (e.g. eSNI “simplified”) Implementation Example (e.g. eSNI “simplified”) Abbreviations § CML – Central Management LAN § CSL – Central Service LAN § SML – Shared Management LAN § SSL – Shared Service LAN § DML – Dedicated Management LAN § DAL – Dedicated Access LAN § IAL – Infrastructure Access LAN § IAL_IBM – Infrastructure Access LAN IBM What Advantages/Disadvantages are there for SNI? Advantages § Standard solution § Secure solution § Reuse of environment § Cost reduction Disadvantages § Sharing of network environment got much higher security and management requirement as single-customer one. § It’s not always possible standardize all customer specific requests § Possibility of conflicts in private IP address ranges Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE Organization Structure – Network management GNMC model NOC - Level 1 support § Proactive monitoring of different tools. Coordinates problem resolution and communication. § Use simple and clear processes. Require best knowledge of these processes, tools usage and got global overview of systems. § Necessary 24/7 support Examples § Coordinates outages of WAN providers, communicate WAN related problems. § Update problem tickets in ticketing systems and inform other teams in case of problem resolution § Communication point for CSC – provide feedback for customer § Coordinates HW replacement NOC - Level 2 support § Advanced problem resolution of troubles coming from 1^st level. § Processes are not so clear for 2^nd level § Level 2 require skills and experiences Examples § Analyze and correct routing problems § Correct security findings in configuration, patch/upgrade OS on devices § Setting and modifying configuration on devices, activation of new customers or devices § Change of ACLs, cooperation with 3^rd level and vendor support if needed Level 3 support § Level 3 support work with complex problems. 3rd level is involved in problems affecting huge infrastructure. § Solving all not standard solutions § Cooperating and coordinating complex changes in network structure. § Act as Network Architects Examples § Providing prevention in wrong setup of routing protocols § Finding solution for slow application performance § Deploying new customer to SNI Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE Network Management Toolset § Tivoli Netview – Detection of problems with implementation of L3 map § Entuity Eye of the Storm – Perofmance and advanced monitoring / analysis – Monitor device with SNMP - can detect more than 70 type of errors. § Cisco Works (CW) – Provides advanced configuration / problem detection for Cisco Platform § CACTI / Vital suite Statistics – SNMP orientated performance management tool § Other tools – TACACS/RADIUS/LDAP – Authentification services – Evidence databases – CEP+ / MAD / eAMT – Ticket tracking tools Network Management Toolset Fault detection with Netview § Netview is standard tool used by IBM all over the world for most customers. § Monitoring of device status § Clear picture of network infrastructure § Netview support easy implementation of various scripts which can automation work. § With SNMP support of all devices provides advanced monitoring (not based only on UP/DOWN functionality with ICMP) § Can receive/forward SNMP traps from/to other tools (EotS/Cacti…) Fault detection - Netview Entuity Eye of the Storm § Advanced monitoring of devices (LAN, WAN and firewalls) with SNMP § Forward major issues to netview § Provides advanced troubles finding § Feature performance monitoring gives us possibility for prevention in outages based on wrong implementation § Provides statistic for core lines (Trunks, Etherchannels) § Availability management § Keeps historical data Entuity Eye of the Storm – port listing Entuity Eye of the Storm – device report Entuity Eye of the Storm Configuration with Cisco Works § CW support mapping devices in network made by Cisco devices. § CW is able to download configs but it also allow to upload them to device, modify directly on CW which allow to made small common changes by “one click” on many devices § CW give you chance to work with device like with real (show physical surface) § Data colleting from devices / mass changes / security activities § Can create reports for Cisco platform Configuration – Cisco works Performance with Lucent Vital suite / CACTI § One of the most important part of our work is troubleshooting are network performance problems. § Requirements for Performance Tools: § Collect variable information from device and store them for analyze . (historical data) § Fast analyze of network performance situations – On which point is network overload. – And what kind of traffic is overloading it. § Proactive Information to prevent overload of WAN / LAN networks § Lucent vital suite are the standard tool for Performance § Can analyze QoS separately § List of TOP talkers Cacti – graphs Evidence Databases & Other Databases § All databases are bind § Asset Evidence (eAMT) § Central Evidence of all devices – Device type/hardware information – Location information – IP address, hostname, interfaces – Contacts for other support groups / provider / on-site support – Security Evidence with historical data – Etc. § Evidence for Security findings – Keeps OS bugs – With each finding in configuration bug reports to responsible support Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE LAN Management § LAN = Local Area Network § Device’s vendors – Cisco, Nortel, 3com, Alel, Allied Telesyn, Blue Coat, Digital, D-link, Enterasys, HP, IBM, Intel, Intermac, Kingston, KTI Networks, LANart, LinkSys, Netgear, Nokia, Olicom, Planet, Symbol, Synoptics, Xtreme – Migration of all existing platforms to Cisco for providing best centralized support § Device’s categories – Firewalls – Routers – Switches LAN – simple connection LAN – Data Centre Datacentre example Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE WAN Management § WAN = Wide Area Network § Used solutions – Leased line – ATM/Frame Relay – MPLS – DSL/ADSL/ISDN – Internet tunnel (iVPN) § WAN lines are usually provided by external companies (BT, AT&T, HP, DT…) § NOC (1^st level) is contact point between customer and provider WAN Management – providers WAN Specifications and requirements § Setting QoS on WAN lines leads to better performance and usage of line § 80 – 100 % WAN link utilization (“we pay 100, we use 100”) § For monitoring of QoS we need good tools QoS – Basic categorization § Category 1 – interactive applications with non-packet burst traffic (e.g. telnet, VoIP) – Packet loss should be avoided § Category 2 – Interactive applications with packet bursty traffic (e.g. http) – Few packet loss § Category 3 – Non-interactive batch traffic (e.g. replication, UDP packets) – Packet loss possible § Category Default – Non classified traffic – High packet loss on congestions, best effort WAN Problem determination Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE Firewall § Firewall types § Standard used FW § Checkpoint ProviderOne § Usage of FW Types of existing Firewalls § Software – Checkpoint Firewall-1 (diverse versions) – Cisco PIX § Operating Systems – Checkpoint Secure Platform (SPlat) – Sun Solaris – Microsoft Windows – Linux – Nokia IPSO § Hardware – PC Architecture – Sun – Nokia – Cisco PIX Firewall Standard for all replaced and new build firewalls § Software – Checkpoint Firewall-1 Next Generation with Application Intelligence – Cisco PIX § Operating Systems – Checkpoint Secure Platform – Cisco PIX Firewall OS § Hardware – IBM x-Series Servers – Cisco PIX Checkpoint - ProviderOne § Easy centralized management § Saved all FW rule sets § Central Logging § Multi-platform management (Nokia, Splat) Checkpoint - ProviderOne Checkpoint - ProviderOne Usage of Firewalls § All network environments (Internet/DMZ/Corporate networks) § Secure separation of networks § Advanced security (not only ACLs) § Implementation of statefull FW § VPN implementation – VPN concentrators Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE IP Services (IPSE) § DNS/DHCP § NTP § Proxy QIP – central management for DNS/DHCP § One central (with backup feature) QIP management server § Structure-based implementation of QIP provides opportunity to use other QIP servers which are reporting to QIP management server § Location types: – Less than 250 users DHCP – IP helper – Less than 499 users local DHCP server or IP helper – More than 500 users (Super location), local DHCP is provided by redundant servers § Rules – Static Addresses for Servers and active network devices – Dynamic addresses for PCs and Printers NTP § Time synchronization service § NTP is installed on Intranet DNS servers § NTP could be distributed for each domain to different servers (location based) § More NTP for one location provide redundancy. Also internet backup is possible Proxy Solution § In past main scope of proxy servers was to provide better usage of WAN lines (http proxy) § Today’s usage of Proxy servers is to provide secure and balanced connection § We can recognize two types of proxies – Transparent (act as proxy for any traffic – mainly socks proxies) – Passive (use proxy feature only if application provide such functionality – http/ftp) Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE Network Security § Configuration standards § Checking or real configuration § Actualized SW/HW § User revalidation Network Security – Standard configuration § General Rules § Applicable for different HW/OS § Pre-defined standards pro Cisco, Nortel, IPSO and other platforms Network Security – Checking actual configuration § Correct setup for new device in network § Revalidation is made at least each half of year § Documentation of findings § Corrective actions if applicable Network security – Actual versions SW/HW § Monitoring for new information/releases – Patches – New versions § Risk management § Planning upgrade Network Security – User revalidation § Quarterly revalidation if users still exists – User verification § Yearly revalidation if users still needs access – Business need § Storing of evidence Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE Typical problems LAN/WAN § Slow network – LAN – Internet/WAN § Device unreachable - LAN § Location unreachable - WAN Example 1 – Slow LAN network § User reports slow network § It’s needed to identify if problem occurs on local server or remote site/internet § Find port settings (speed/duplex) on switch and settings on user PC and server. § Find statistic data for port errors § Cooperate with Server support group to eliminate possible server problems § Replace cable if port settings are not showing any incorrect settings and errors are shown on port report Example 2 – Slow Internet/WAN network § Up/Down Management – Find if there is no WAN issue (only backup running) § QoS Statistics and Reports – Find if there are peak on network or load near threshold § Netflow Traffic Analysis – Find which traffic cause big network load Example 3 – Device unreachable § Incoming event in NW management tool (Netview, …) § Event verification (Ping, SNMP request) § Try to connect from different location (using different paths) § Contact On-Site Support to eliminate power outage or cabling problems § Manual restart (cold reboot) § Console connect § HW replacement Example 4 – Location unreachable § Incoming event in NW management tool (Netview, …) § Contact On-Site Support to eliminate power outage or cabling problems § Connect via manual backup solution if available (dial up) § Contact WAN provider for check line problems Agenda § Shared Network Infrastructure § Organization structure § Network monitoring tools § LAN Management § WAN Management § Firewall § IP Services § Network Security § Typical problems - LAN/WAN § Typical problems - FW, IPSE Example 1 – User can’t connect to network § Check IP address § If IP don’t correspond to location there could be problem with IPSE § Check DHCP service on server § Check if there are free IPs in pool Example 2 – User can’t connect to service § Check IP addresses and locations (source / destination) § Check if there is no network / server / service outage) § Find route § Check all rules on FW - ProviderOne § Check all ACLs on routers / Switches § Check VLANs Example 3 – New server in location § Get necessary approvals § Find required connections § Find data flow in network § Correct all FW rules and ACLs Questions & Answers