2   CESNET2 Backbone Network Development

Within the development of the CESNET2 backbone network in 2004, we focused on building the Prague-Brno DWDM line, representing a foundation for developing the optical transfer layer and upgrading the transfer capacities of the IP network to 10GE. The optical transfer layer will also allow for providing services on the optical paths level (Lambda services) for research activities and connected participants within the national and international scopes.

By initiating the development of the new generation NREN, CESNET2, we have completed the design of the new backbone network architecture. In the IP network development area, we have accelerated the processes, aiming to simplify the general topology and phase out and replace the GSR12016 network core routers which could no longer support the development of other services and features, such as the IPv6 unicast/multicast and jumbo frame support because of their moral obsoleteness. Concerning the logical architecture and network properties, we have completed the migration to Sup720-3BXL processors in access routers that was indispensable for providing support for the dual-stack IPv4/IPv6 including a connection to the GÉANT network.

Crucial changes were made during the IPv4 multicast implementation (AnyCast RP solution) when we also changed the method for connecting large university MANs and campus networks. Migration to external connections utilizing eBGP, eMBGP and MSDP protocols (these networks have also their own RP) and private AS now excludes any mutual influence with the backbone network. The administration of these connections has been simplified, increasing the reliability and availability of all IP network services, especially the IPv4 multicast.

An inseparable part of the IP network services development is the provision of the Quality of Services (QoS) that includes - in addition to the necessary technical implementation - also the need to deal with many issues concerning the definition of an acceptable general framework of rules for applying the QoS policy to particular user categories.

The development of the CESNET2 NREN also covers the development and operation of the essential supportive infrastructure needed to maintain a continuous and reliable operation of the network and ensure support for projects and participants of NREN on the national and international levels (coordination and cooperation with the GÉANT network and other European NRENs).

The development and changes of the backbone network can be summarized into several points:

2.1   Backbone Network Physical Topology

The basic physical network topology (see Figure) employs leased pairs of optical fibres. Some of the smaller PoPs (e.g., Dìèín, Cheb) are connected via a single fibre only, utilizing the available technology of MRV fast Ethernet converters. The remaining PoPs are connected to the backbone network with 10 Mbps and 34 Mbps radio circuits or leased circuits.

[Figure]

Figure 2.1: Physical optical topology of the CESNET2 backbone network

The basic logical topology of the backbone network comprises ten GigaPoPs interconnected through data circuits with transfer capacities of at least 10 Gbps (see Figure). The backbone circuits utilize 10GE PHY LAN, 1GE (Gigabit Ethernet) and POS STM-16/OC-48 to provide transfer rates of 2.5 Gbps. GigaPoPs are connected to the backbone network always with two data circuits to provide for an access backup.

Optical circuits are fitted with various technologies. Gigabit circuits spanning over shorter distances (up to approx. 120-140 km) need no regeneration or amplification, enabling us to deploy replaceable optical transceivers (Pluggable Optics) in interfaces of routers and CWDM-1550 switches (superseding formerly used GBIC-ZX) and DWDM GBICs with 100 Ghz channel spacing in compliance with ITU-T (e.g., Èeské Budìjovice-Brno, 308 km with the EDFA amplification via Keopsys amplifiers).

In the field of new generation replaceable optical transceivers, we have started using also SFP (Small Form Factor Pluggable) GBICs that offer smaller dimensions and lower power consumption and will support extended optical parameter monitoring (DOM, Digital Optical Monitoring) within a greater range of operating temperatures (-5 to +85°C). However, these transceivers can be installed only in the appropriate new type of interfaces (they are not interchangeable with standard GBICs). For 10GE interfaces, we intend to perform the testing of DWDM Xenpaks for 10GE transfers via a DWDM system. A new generation of 10GE GBICs are the so-called XFP (Xenpak Form Factor Pluggable), however, these will be supported only by newer types of 10GE router interfaces.

Optical circuits with higher attenuations (more than 32 dB) are fitted with Keopsys optical EDFA amplifiers (preamplifiers and boosters). Some lines feature also L2 switches functioning as signal repeaters. Use of switches along an optical line is a cheaper fitting alternative, though its great disadvantage is its need to be placed on the line (installing the switch at a required location is not always possible) and protocol dependency (L2 Ethernet).

Using EDFA amplifiers located only at the ends of lines (NIL, Nothing in Line method) is therefore a preferred alternative. These amplifiers boost only the optical signal and are protocol-independent (Gigabit Ethernet, POS, etc.). In cooperation with the Optical networks activity, we test our own EDFA PC-based amplifiers (PC Light) as an alternative to commercially available EDFA amplifiers in the operating part of CESNET2. These amplifiers are installed in the 1GE circuit Prague-Hradec Králové where we also employ DWDM GBICs for the connected routers.

Migrating the backbone network to leased optical fibres is the basic prerequisite for building an optical transport network. A technology suitable for this is DWDM, allowing multiple optical channels to be operating within a single leased fibre and providing the means for integration with the existing IP network and migration to a backbone network combining IP services and optical services in future stages (using an appropriate control plane, such as GMPLS), including installation of optical switches for dynamically switching the optical transfer lines.

The first DWDM line was put into service by the end of 2004. The public tender winner was a solution provided by Cisco Systems based on a modular DWDM system, ONS 15454 (for the system description see www.cisco.com). When the line sections were measured, the DWDM deployment project had to be modified since the measured chromatic dispersion values did not match the original assumptions (e.g., negative values were measured in one section). The DWDM system comprises two end DWDM terminals (ET, End Terminal) with tuning-enabled 10GE transponders and three active elements needed to amplify and compensate for the dispersion along the optical line (see Figure).

[Figure]

Figure 2.2: Equipment used on the Prague-Brno DWDM line (large image)

The DWDM system installation allowed for increasing the capacity of the connection between the Prague and Brno nodes to 10GE. For the upcoming period, we plan expanding the DWDM transport system with additional sections, aiming to complete the main backbone optical ring (Prague-Brno-Olomouc-Hradec Králové-Prague) using the promising ROADM technology. The optical transport layer development includes also a proposal for establishing a connection with the GÉANT2 network on this level. The operation of the GÉANT2 network should be launched in the third quarter of 2005. However, it has not been made clear yet which solution will be chosen for the optical backbone of this network (DWDM system, SONET/SDH transport, or optical switches).

2.2   Backbone Network Logical Topology

The basic transfer protocol employed within the backbone network is IP/MPLS. OSPFv2 is used locally as the IGP protocol of the MPLS network. The actual network logical topology is divided into two functional levels to which the topology of each GigaPoP is adapted:

[Figure]

Figure 2.3: Logical topology of the CESNET2 backbone network (large image)

Core Backbone Level

The network core comprises the GS22016 and OSR7609 routers (marked red in Fig. 3) where all backbone data circuits are terminated. The new backbone network topology design, in compliance with the planned development of the DWDM network, preserves core routers only in the basic network circuit GigaPoPs (Prague-Brno-Olomouc-Hradec Králové-Prague). Besides, these routers need replacing with newer versions, since the GSR12016 no longer allows further network development (10GE, IPv6 multicast, jumbo frame support) and due to their inconvenient properties, their upgrade would not be financially feasible. We have therefore installed OSR7609 routers in the network core in the Prague and Brno GigaPoPs with sufficient technical/economical parameters (especially concerning the price per single 10GE port). The upcoming period will see the replacement of the remaining GSR12016 routers (in Prague, Olomouc and Hradec Králové GigaPoPs). We will also focus on selecting suitable routers with features that are better suited for the network core router requirements (with respect to the 10GE support, buffer memory capacities on ports, performance and stability).

Network Access Level

We utilize the Cisco OSR7609 (with high-performance SUP720-3BXL processors) as access routers and Cisco 7206-VXR with NPE-G1 as MPLS PE in smaller nodes (Ústí n. Labem, Zlín). These routers provide connected participants with all functions and services of the backbone network (MPLS, MPLS VPN, QoS, IPv4/IPv6 routing, IPv4 multicast, NetFlow statistics export, access filters). Academic metropolitan (Pasnet, BAPS, etc.) and university networks are connected via the gigabit Ethernet, while other participants and detached workplaces use lower capacities (10/100 Ethernet). The interconnection of access routers with core routers is ensured with 10GE LAN PHY, OC-48 POS, or 1GE.

Smaller network nodes (PoP) are connected to these access routers by 100 Mbps optical circuits (single-fibre connection), 34/10 Mbps radio circuits (conversion to 100BASE-T at circuit ends is used in case of 34 Mbps), and leased fixed circuits with smaller capacities. These nodes are fitted with smaller routers with limited functions (MPLS CE) from the C2621/C2651-XM/C2691 family that support only VRF Lite functionality. These routers are now being replaced with new L3 switches, Catalyst 3750, with parameters and features fully in compliance with the requirements of gigabit networks (10/100/1000 ports, 32 Gbps throughput, SFP GBIC support, IP routing, giant frame support). The IPv6 protocol has not yet been implemented in stable IOS versions. However, it is supported in experimental versions in the testing and verification of which we actively participate within the manufacturer's beta test program.

Logical topology of every node is divided into several parts: the service segment, the participants' connection, and the connection of testing segments. The service segment includes OOB access, VoIP gateway, UPS, and other service equipment and servers. Connection of participants is ensured through dedicated 802.1Q VLANs in the access router. VLANs are usually terminated in the participants' access routers; in some cases, VLANs are also used to create EoMPLS L2VPN within the backbone network (mapping of a given participant's VLAN to EoMPLS is employed here).

The MPLS technology enables us to create L3VPN and point-to-point L2VPN (in accordance with draft-martini-l2circuit-trans-mpls-05) within the entire backbone network. EoMPLS (Ethernet over MPLS) is used for L2 connections of detached sites. This connection method has an indisputable advantage if seen from the backbone network perspective, since the participant's traffic does not affect the L3 MPLS backbone (an example is the peering of the SANET and PIONIER networks ensured via the CESNET2 backbone).

The next network development stage assumes testing of features provided by the VPLS technology (Virtual Private LAN Services) in accordance with the PWE3 concept (Pseudo Wire Edge to Edge Emulation), when the MPLS network performs the function of a distributed L2 switch. The VPLS technology will be supported on deployed router types in 2005.

2.3   IPv4 Unicast Routing

As an internal routing protocol within the backbone network, we use the internal BGPv4 (iBGP) among access PE routers (see Figure). External R84, R85, and R98 routers contain RR1, RR2, and R3 route-reflectors needed to provide routing redundancy. Other PE routers are fitted with route-reflector clients. Use of route-reflectors lowers the necessary number of peers in the backbone network. Static aggregated blocks are used in GigaPoP access routers and no redistribution from internal routing protocols is carried out. Large metropolitan and university networks (PASNET, ÈVUT, etc.) use their private autonomous systems and are connected to the backbone via the eBGP protocol. Preferred connections for smaller 3rd party networks are based on static routes.

[Figure]

Figure 2.4: Internal unicast routing (example of the RR1 route-reflector in R84) (large image)

2.4   IPv4 Multicast Routing

To exchange multicast (group-oriented transmission) routing information, we use the internal MBGP protocol. Again, iMGBP uses three route-reflectors in the R84, R85 and R98 routers. Unlike in the case of the unicast routing, route-reflector clients are configured in all P and PE routers, since proper RPF (Reverse Path Forwarding) functions must be ensured also in network core routers. RPF, as a part of the forwarding process, provides protection from loops or duplicated multicast packets. This inspection is provided in the backbone network environment by iMGBP (assuming that all multicast data sources are announced using this protocol).

The entire backbone network uses the PIMv2-SM protocol. We have considerably simplified the general multicast topology to leave only one central RP (IP 195.113.144.2) in use in the R85 and R98 routers where redundant AnyCast RP topology is implemented. Large metropolitan and university networks now operate their own RP (within an independent multicast domain) and active multicast data sources are announced with the MSDP protocol. Other connected participants can make use of the central RP where the dynamic protocol (Cisco Auto-RP or BSR) announcing has been cancelled. These participants must configure the RP address statically in their access routers.

The existing logical multicast topology is incongruent (unicast is transferred via MPLS whereas the multicast transfer lacks the MPLS labels). Filtering of announced active sources and groups is also carried out in all nodes on the MSDP level in compliance with the recommendations of Cisco and the GÉANT2 network (restricting operation of the Novell NDS, ImageCast, and other services utilizing the multicast, which are not intended for announcing to the backbone network and MBone).

The current solution of the multicast distribution in the backbone network supports also SSM/IGMPv3 (Source Specific Multicast) that uses a reserved IP range of 232.0.0.0 0.255.255.255 and does not require RP to operate.

In addition to the traditional allocation of multicast addresses using dynamic protocols such as SDR (SAP), these addresses can be allocated statically in accordance with RFC 2770 based on the AS number. The given mechanism implies that our AS2852 has a corresponding range of public multicast addresses of 233.11.36.0/24 from which addresses are allocated.

2.5   IPv6 Unicast/Multicast Implementation

Deploying new SUP720-3BXL processors that support hardware IPv6 transfers in OSR7609 access routers allowed for converting the temporary IPv6 solution with dedicated 6PE C7500 routers to access routers. The backbone network now supports the dual-stack IPv4/IPv6 unicast routing via MPLS (PE/6PE). The participants using IPv6 are typically connected by a dedicated 802.1Q VLAN since most participants do not support dual-stack in their networks.

The IPv6 multicast is not currently supported. Results of our laboratory testing are used to prepare a workaround solution for which we will utilize the original solution with 6PE C7500 routers. We will use IPv6 multicast tunnels among these routers for the multicast distribution. Our alternative solution will be implemented in the beginning of the second quarter of 2005. IOS versions supporting the IPv6 multicast for OSR7609 access routers will be available by the end of 2005.

2.6   QoS Implementation

Implementing the QoS (Quality of Services) within the NREN environment has the following objectives:

The "point-to-cloud" type of the DiffServ model, using the E-LSP technique over the MPLS infrastructure, was chosen as a basis for the technical architecture needed to implement the QoS. If the capacity of the backbone network is not fully utilized, some QoS classes can make use of the remaining capacity as an addition to the minimum guaranteed bandwidth. We have defined the following service classes for various types of applications, depending on the bandwidth, latency and packet loss requirements:

Proposals for IP packets labeling using DSCP, mapping to the Exp field of the MPLS heading, and the bandwidth reservation for each class within the Exp-LSP MPLS/DiffServ CESNET2 domain are listed in the Table.

Service class PHBDSCPExp MPLS Bandwidth
Premium IP EFEF, CS5525 %
Net. Control AFCS7, CS67, 65 %
Gold IP+ AFAF4x, CS4430 %
Silver IP+ AFAF3x, CS3, AF2x, CS23, 214 %
Best Effort Default0 (ostatní)025 %
LBE AFAF1x, CS111 %

Table 2.1: Proposal of PHB, DSCP/Exp mapping, and bandwidth reservation for individual QoS classes

During our testing operation, we implemented QoS first in Plzeò and then in Liberec and Ostrava. The testing operation has taken place without problems and the QoS implementation in other nodes is scheduled for the beginning of 2005.

2.7   External Connections (Peering)

[Figure]

Figure 2.5: External peering of the CESNET2 network

The CESNET2 uses the following external connections and peering.

Foreign connectivity (commodity traffic)

We are provided with the global foreign connectivity by the Telia International Carrier. The connection capacity is 2.5 Gbps (POS STM-16/OC-48 in R84) restricted to 800 Mbps. A backup connection is set up with another 2.5 Gbps circuit leading to the second router of the Telia node (R85 Internet peering router).

Connection to the GÉANT network

Featuring the capacity of 2.5 Gbps, the connection to the GÉANT network uses a local POS STM-16/OC-48 connection to a node located in the premises of the CESNET Association. The GÉANT node in Prague is connected to Germany (Frankfurt am Main) with a 10 Gbps circuit and to Poland (Poznan) and Slovakia (Bratislava) using two POS STM-16/OC-48 circuits. The GÉANT network provides IPv4 unicast/multicast and IPv6 unicast connections to European, American and Asian scientific & research NRENs and centres.

National Peering in NIX.CZ

Access to the NIX.CZ is provided via two GE circuits terminated in R84 and R85 Internet peering routers (load balancing and backup). Circuits are implemented using leased optical fibres and fitted with appropriate GBICs (GBIC-LX/LH and CWDM-GBIC on the second, longer line). The R85 router provides also a native IPv6 peering. As the total traffic to NIX is now exceeding 1 Gbps, we are planning to upgrade the connection to 10GE.

Peering with the SANET and PIONIER Networks

The interconnection with the Slovak academic network, SANET, uses optical fibres leading from Brno to Bratislava. The line is fitted with CWDM-GBIC-1550, using the Catalyst 3524 switch functioning as a repeater. The connection to the Polish academic network called PIONIER is also based on optical fibres, this time connecting Ostrava and Bielsko-Biala and fitted with CWDM-GBIC-1550. This line is shorter, measuring approx. 120 km, and hence no amplification is needed here.

To connect both networks, we employ the 802.1Q encapsulation. One VLAN is always used for the CESNET peering whereas the second VLAN is mapped to the MPLS L2 tunnel (EoMPLS) that allows direct L2 peering of these networks. The CESNET2 network in this case provides only the L2 transport service for these NRENs.

In addition to the mutual peering of our networks, we provide - within this connection - peering for SANET in NIX.CZ and vice versa, and the SANET network provides CESNET2 with peering in SIX. This saves foreign connectivity resources for both networks. Mutual announcing of networks to peering centers is based on BGP communities (tags).

2.8   Backbone Network Development Plans for the Upcoming Period

We are planning to expand the transport optical DWDM network to other nodes and to complete the basic DWDM ring if possible (see Figure) in 2005. In addition, we will test properties of the new ROADM technology as a means for creating optical channels within the DWDM system.

[Figure]

Figure 2.6: Planned DWDM development in 2005

In connection with the optical layer development, we will continue also building the IP layer and migrating to 10GE. This migration will involve the deployment of higher-performance routers in the network core and the addition of the 10GE interface to access routers. The deployment of new routers is crucial for the future development of network properties, in particular:

We also assume reaching full QoS implementation and QoS compatibility with the GÉANT network, which is needed to provide QoS services in the international scope.

As far as new protocols and features are concerned, we will concentrate on testing and deploying the IPv6 multicast. Initially, a workaround solution will be implemented, with migration to backbone routers to be performed later. We will also continue developing the MPLS technology in the VPLS area (Virtual Private LAN Services) and MPLS QoS.

During the year, we will perform necessary upgrades of connection capacities for the foreign connectivity and the connection to NIX.CZ (to 2×10GE). An important task will be to establish a connection to the newly created pan-European research network GÉANT2 which should be put into service in the second half of 2005, with a capacity of at least 1×10GE, and to ensure the compatibility of this connection (in the area of IPv4/IPv6 unicast/multicast and QoS implementation) as well as the connection on the optical level in future stages.

previous
contents 		next
metacentrum elearning liberouter live shows videoserver eduroam