| 1 | = Authors = |
| 2 | [https://wiki.ittc.ku.edu/gpeni_wiki/index.php/User:Pangu '''Pragatheeswaran Angu'''], University of Nebraska-Lincoln[[BR]] |
| 3 | [https://wiki.ittc.ku.edu/gpeni_wiki/index.php/User:Byrav '''Dr. Byrav Ramamurthy'''], University of Nebraska-Lincoln[[BR]] |
| 4 | |
| 5 | with acknowledgements to Ciena, DRAGON/DCN (USC/ISI) and MAX/MANFRED. |
| 6 | |
| 7 | = !CoreDirector CI System Description = |
| 8 | |
| 9 | == Overview == |
| 10 | |
| 11 | The !CoreDirector Multi-Service Switch and !CoreDirector CI |
| 12 | Multi-Service Switch delivers a wide range of optical capacities and a |
| 13 | variety of protection options. !CoreDirector Switches are intelligent |
| 14 | switches that provide unmatched, managed capacity and bandwidth |
| 15 | density. They provide nonblocking, bidirectional switching capacity |
| 16 | that can be configured to switch and groom traffic from any input port |
| 17 | to any output port down to the STS-1/VC-3 level. These switches |
| 18 | support OC-3/12/STM-1/4, OC-48/STM-16, OC-192/STM-64 optical |
| 19 | interfaces, STM-1e electrical interfaces and Gigabit Ethernet |
| 20 | interfaces. The switch matrix architecture is designed to operate on a |
| 21 | single wavelength or the composite signals of a single wavelength. |
| 22 | |
| 23 | == Switching Capacity == |
| 24 | |
| 25 | For smaller core or larger metropolitan central offices, the |
| 26 | !CoreDirector CI Switch delivers up to 160 Gbps of switching capacity |
| 27 | using a maximum of 16 OC-192/STM-64 optical interfaces, 64 |
| 28 | OC-48/STM-16 optical interfaces, or 128 OC-3/12/STM-1/4 optical |
| 29 | interfaces. The !CoreDirector CI Switch can use a combination of these |
| 30 | interface types to achieve the total capacity. |
| 31 | |
| 32 | The !CoreDirector CI Switch handle both Synchronous Optical Network |
| 33 | (SONET) and Synchronous Digital Hierarchy (SDH) optical |
| 34 | interfaces. The Transport Bandwidth Unit (TBU) is the minimum |
| 35 | granularity of switched bandwidth in the !CoreDirector Switch. Each |
| 36 | TBU corresponds to a clear-channel bandwidth of approximately 52.56 |
| 37 | megabits per second (Mb/s). For SONET interfaces, each TBU supports |
| 38 | one STS-1 signal. In a fully configured !CoreDirector CI Switch with |
| 39 | SONET/SDH interfaces, a total of 3072 STS-1/VC-3 circuits can be |
| 40 | established between ports. |
| 41 | |
| 42 | == Hardware Overview == |
| 43 | |
| 44 | [[Image(cdci.png, 600px)]] |
| 45 | Figure 1: Hardware Overview |
| 46 | |
| 47 | The !CoreDirector CI Switch consists of a chassis containing: |
| 48 | |
| 49 | * A single shelf housing all Line, Control, Switch, and Timing Modules |
| 50 | * Modular display panel |
| 51 | * Two blower modules |
| 52 | * Power distribution unit |
| 53 | * I/O module, either SONET-DS1 (T1) or SDH-E1 |
| 54 | |
| 55 | The !CoreDirector CI Switch is designed to be installed in a standard |
| 56 | Electronic Industries Alliance (EIA) 23-inch equipment rack. |
| 57 | |
| 58 | The display panel is at the top of the chassis, and the blower modules |
| 59 | are below the display panel. The fans provide filtered cooling airflow |
| 60 | from the bottom of the system up through the chassis, then exhaust the |
| 61 | warm air through top front and rear vents. The shelf is below the fans, |
| 62 | and the PDU is at the lower back of the chassis. |
| 63 | |
| 64 | A fully populated !CoreDirector CI Switch contains eight Line Modules, |
| 65 | two Control Modules, two Timing Modules, and four Switch Modules. These |
| 66 | are located on the same shelf and share a common backplane. |
| 67 | |
| 68 | === Modules in the !CoreDirector Switches === |
| 69 | |
| 70 | The !CoreDirector Switches contain the following types of modules: |
| 71 | |
| 72 | * Line Modules |
| 73 | * Optical Modules and Ethernet Interfaces |
| 74 | * Control Modules |
| 75 | * Timing Modules |
| 76 | * Switch Modules |
| 77 | |
| 78 | ==== Line Modules ==== |
| 79 | |
| 80 | !CoreDirector Line Modules provide bidirectional ports, either through |
| 81 | integrated optical ports, installed Optical Modules, or integrated |
| 82 | electrical ports. The Line Modules also contain part of the switching |
| 83 | fabric for the switch. Some Line Modules have integrated |
| 84 | (nonremovable) ports. The LM-16 Line Module has 16 integrated optical |
| 85 | ports; the LM-16e Line Module has 16 ports to support STM-1e |
| 86 | electrical interface; the LM-20-GE Gigabit Ethernet service Line |
| 87 | Module accepts 20 replaceable transceivers, Ethernet Services Line |
| 88 | Module (ESLM) provides a single 10G port group supporting 10 Gigabit |
| 89 | Ethernet ports or a single 10GbE port. The LM-8 Line Module can have |
| 90 | up to eight installed Optical Modules; the LM-2 Line Module |
| 91 | accommodates two Optical Modules. The optical connections are accessed |
| 92 | from the front of the rack. |
| 93 | |
| 94 | ==== Optical Modules and Ethernet Interfaces ==== |
| 95 | |
| 96 | The following types of Optical Modules are available: |
| 97 | * Short Reach (SR), Intermediate/Medium Reach/ (IR/MR), |
| 98 | and Long Reach (LR1 and LR2) OC- 48/STM-16 Optical Modules |
| 99 | * MR OC-3/12 STM-1/4 Optical Modules |
| 100 | * SR1, SR2, IR/MR, and LR2 OC-192/STM-64 Optical Modules |
| 101 | * OC-192/STM-64 Optical Modules with Dense Wavelength Division |
| 102 | Multiplexing (DWDM)capability |
| 103 | * OM1GE-850-SXL-A |
| 104 | * OM1GE-1310-LXL-A |
| 105 | * GbE SX and LX Transceivers |
| 106 | * 10GbE SR, LR and ER Transceivers |
| 107 | |
| 108 | ==== Control Modules ==== |
| 109 | |
| 110 | Two Control Modules are installed in the !CoreDirector Switch to serve |
| 111 | as the central system controllers. At any given time, only one Control |
| 112 | Module is active; the other is in standby mode to be used if a failure |
| 113 | occurs in the active (primary) one. A hard drive on each Control |
| 114 | Module provides persistent (nonvolatile) storage for all configuration |
| 115 | data. The Control Modules contain external user interface ports. |
| 116 | |
| 117 | ==== Timing Modules ==== |
| 118 | |
| 119 | The !CoreDirector Switch is equipped with two Timing Modules that |
| 120 | perform the network timing and switch fabric synchronization functions |
| 121 | for the system. Each Timing Module contains a Stratum 3E clock whose |
| 122 | output is used to time all outgoing optical signals. For SONET |
| 123 | networks, the reference for the Timing Module clock can be any optical |
| 124 | input or a Building Integrated Timing Source (BITS) input. For SDH |
| 125 | networks, the reference for the Timing Module clock can be any optical |
| 126 | input or Synchronization Supply Unit (SSU) input. |
| 127 | |
| 128 | In the absence of a qualified input, the clock can operate in the |
| 129 | holdover mode. One Timing Module is primary, and the other is used in |
| 130 | the event of a failure of the primary Timing Module. |
| 131 | |
| 132 | ==== Switch Modules ==== |
| 133 | |
| 134 | Switch Modules provide the central part of the system switching |
| 135 | fabric. (The other part of the switching fabric is on the Line |
| 136 | Modules.) Each Switch Module has paths to all Line Modules. In the |
| 137 | !CoreDirector Switch, the Switch Modules are installed in a |
| 138 | chassis-wide shelf between the Line Module shelves. In the |
| 139 | !CoreDirector CI system, the Switch Modules occupy part of the middle |
| 140 | of the single system shelf. The exact number of Switch Modules in a |
| 141 | !CoreDirector system depends on the system configuration. The |
| 142 | !CoreDirector Switch accommodates up to 15 installed Switch Modules; |
| 143 | the !CoreDirector CI Switch accommodates up to four Switch Modules. |
| 144 | |
| 145 | == Software Overview == |
| 146 | |
| 147 | !CoreDirector embedded operating software contains all the features |
| 148 | and functionality necessary for the !CoreDirector Switch and |
| 149 | !CoreDirector CI Switch to be used in a variety of network |
| 150 | applications. !CoreDirector provides the flexibility to support a wide |
| 151 | range of network applications by offering multiple protection and |
| 152 | restoration schemes as well as intelligent, end-to-end service |
| 153 | provisioning across multiple bandwidth sizes, ranging from STS-1/VC-3 |
| 154 | to STS-192c/VC-4-64c. |
| 155 | |
| 156 | === Features of the !CoreDirector Software Packages === |
| 157 | |
| 158 | '''!CoreDirector Cross Connect software''' enables basic optical cross |
| 159 | connect capabilities for simple wavelength switching and automated |
| 160 | patch panel applications. This infrastructure package provides the |
| 161 | following basic capabilities: |
| 162 | |
| 163 | * Synchronous Optical Network (SONET) and Synchronous Digital Hierarchy (SDH) |
| 164 | Operations Administration, Maintenance, and Provisioning (OAM&P) |
| 165 | * 1+1; 1:N Automatic Protection Switching/Multiplex Section Protection |
| 166 | (APS/MSP) protectionswitching, Ethernet Port Protection, Equipment Protection |
| 167 | Switching |
| 168 | * Timing and synchronization |
| 169 | * Node Manager Graphical User Interface (GUI) |
| 170 | * Transaction Language One (TL1) Management |
| 171 | * Data Plane Fault Isolation (DPFI) |
| 172 | * Timing Plane Fault Isolation (TPFI) |
| 173 | * Data Communications Network (DCN) access management |
| 174 | * Audit logs |
| 175 | * Circuit test capabilities |
| 176 | |
| 177 | ''!CoreDirector - Ring software'' adds the following ring protection |
| 178 | capabilities to those of the !CoreDirector Cross Connect software |
| 179 | package: |
| 180 | |
| 181 | * Two-Fiber Bi-directional Line Switched Ring/Two-Fiber Multiplex |
| 182 | Section Shared Protection Ring (2F-BLSR/MS-SPRing) |
| 183 | * Four-Fiber Bi-directional Line Switched Ring/Four-Fiber Multiplex |
| 184 | Section Shared Protection Ring (4F-BLSR/MS-SPRing) |
| 185 | * Virtual Line Switched Ring (VLSR) |
| 186 | * Asymmetric Virtual Line Switched Ring (A-VLSR) |
| 187 | * Transoceanic Line Switched Ring (TLSR) (G.841 protection) |
| 188 | * Unidirectional Path Switched Ring/Subnetwork Connection |
| 189 | Protection (UPSR/SNCP) |
| 190 | |
| 191 | ''!CoreDirector - Mesh software'' adds the following optical switching |
| 192 | and mesh protection capabilities to the !CoreDirector Cross Connect |
| 193 | software package: |
| 194 | |
| 195 | * Optical Signal and Routing Protocol (OSRP) functionality |
| 196 | * Point-and click auto-provisioning |
| 197 | * Automatic route computation |
| 198 | * Network (topology) autodiscovery |
| 199 | * Link aggregation for large network scalability |
| 200 | * Mesh protection (!FastMesh™) |
| 201 | * Protection service classes |
| 202 | * Multiple protection bundle IDs |
| 203 | |
| 204 | = Management Interfaces = |
| 205 | |
| 206 | The !CoreDirector CI can be managed using two interfaces: Node Manager and TL1. |
| 207 | |
| 208 | == Node Manager == |
| 209 | |
| 210 | The !CoreDirector Node Manager application provides a GUI that allows |
| 211 | users from any point in a network to manage or configure operation of |
| 212 | the !CoreDirector CI Switches, including connections, provisioning, |
| 213 | protection, and statistical collection at the network element |
| 214 | level. It also permits craft personnel to monitor and verify all |
| 215 | aspects of the connected node element operation and hardware |
| 216 | configuration and to access historical data about alarms, links, |
| 217 | routing, system alarms, and network activity. |
| 218 | |
| 219 | [[Image(nodemanager.png, 800px)]] |
| 220 | Figure 2: Node Manager Software |
| 221 | |
| 222 | The Node Manager screen is divided into the following six sections: |
| 223 | |
| 224 | * Menu Bar |
| 225 | * Toolbar |
| 226 | * Status Bar |
| 227 | * Equipment Tree |
| 228 | * List Frame |
| 229 | * Details Frame |
| 230 | * Database Synchronization Status Bar |
| 231 | |
| 232 | === Menu Bar === |
| 233 | |
| 234 | The menu bar is located at the top of the main window beneath the |
| 235 | title bar. The menu bar consists of six functions File, View, Go, |
| 236 | Action, Tools and Help |
| 237 | |
| 238 | === Toolbar === |
| 239 | |
| 240 | The toolbar provides icons for navigating quickly to the Node Manager |
| 241 | screens. |
| 242 | |
| 243 | === Status Bar === |
| 244 | |
| 245 | The status bar is located below the menu bar. The status bar consists |
| 246 | of Alarm Light Emitting Diodes (LEDs), Alarm Data bar, the Screen Name |
| 247 | box, and Active and Standby LEDs. |
| 248 | |
| 249 | === Equipment Tree === |
| 250 | |
| 251 | The equipment tree is located on the left side panel of the Node |
| 252 | Manager window beneath the status bar. The equipment tree identifies |
| 253 | the !CoreDirector Switch hardware, such as the bay, shelf, and line |
| 254 | modules. When an object is selected on the equipment tree, its |
| 255 | corresponding information is displayed in the List frame. |
| 256 | |
| 257 | === List Frame === |
| 258 | |
| 259 | The List frame provides summary information about the item selected in |
| 260 | the equipment tree. All List frame elements are read-only. To view |
| 261 | detailed information about an object in the List frame, the user |
| 262 | selects the object; information about that object is displayed in the |
| 263 | Details frame (to deselect an item, the user presses CTRL and |
| 264 | left-clicks the mouse). To sort objects in the List frame, the user |
| 265 | clicks the column heading. This sorts the column in ascending order |
| 266 | (indicated by a black up arrow in the column heading). To sort the |
| 267 | column in descending order (indicated by a black down arrow in the |
| 268 | column heading), the user clicks the column heading again. To unsort |
| 269 | objects, the user right-clicks on the column heading. |
| 270 | |
| 271 | === Database Synchronization Status Bar === |
| 272 | |
| 273 | The Database Synchronization status bar is located at the bottom of |
| 274 | the screen. The left side of the status bar lists the last database |
| 275 | synchronization operation. The right side of the status bar displays |
| 276 | the DB status icon when database synchronization is in progress. |
| 277 | |
| 278 | == Transaction Language 1 (TL1) == |
| 279 | |
| 280 | TL1 is a text-based human and machine protocol developed in the 1980s |
| 281 | by Bellcore (now known as Telcordia Technologies, Inc.). TL1 is a |
| 282 | standardized set of American Standard Code for Information Interchange |
| 283 | (ASCII) telecommunications network management commands and |
| 284 | messages. TL1 was developed as a subset adaptation of the |
| 285 | International Telecommunications Union Telecommunications (ITU-T) |
| 286 | recommendation for machine-to-machine environments. TL1 is used by |
| 287 | most manufacturers of network elements. |
| 288 | |
| 289 | === Accessing TL1 on the !CoreDirector Switch === |
| 290 | |
| 291 | Telnet is used to access !CoreDirector TL1 software from a remote |
| 292 | location, or after connecting a laptop to the Ethernet port on the |
| 293 | !CoreDirector Switch. |
| 294 | |
| 295 | STEP 1: Obtain the Internet Protocol (IP) address of the intended |
| 296 | !CoreDirector Switch and verify connectivity to the intended |
| 297 | !CoreDirector system. |
| 298 | |
| 299 | STEP 2: Telnet to the !CoreDirector Switch from the DOS prompt of the |
| 300 | PC. From the prompt of a DOS terminal window, type the Telnet command, |
| 301 | a blank space, the !CoreDirector IP address, a blank space, and the |
| 302 | TL1 port number 10201. |
| 303 | |
| 304 | === TL1 Input Conventions === |
| 305 | |
| 306 | TL1 input commands are composed of the command type (a verb) followed |
| 307 | by command modifiers and blocks. Input commands can contain up to 512 |
| 308 | alphanumeric characters. |
| 309 | |
| 310 | All commands consist of the following: |
| 311 | |
| 312 | * TL1 command name |
| 313 | * !CoreDirector Target Identifier TID |
| 314 | * Correlation Tag (CTAG) |
| 315 | * A semicolon |
| 316 | |
| 317 | The TID is the unique name, limited to 20 alphabetical characters, |
| 318 | that is assigned to the !CoreDirector Switch during !CoreDirector |
| 319 | system setup and configuration. |
| 320 | |
| 321 | As an example of a TL1 command input and response, a user with account |
| 322 | name “BOBSMITH” and password “bobs2E” can start a TL1 session on a |
| 323 | Network Element (NE) or TID named “!CoreDirector777” by entering the |
| 324 | following command (using the CTAG “Myctag”): |
| 325 | |
| 326 | {{{ |
| 327 | ACT-USER:COREDIRECTOR777:BOBSMITH:Myctag::bobs2E; |
| 328 | }}} |
| 329 | |
| 330 | The semicolon at the end of the command line indicates the end of the command input. The |
| 331 | !CoreDirector system sends a command acknowledgement indicating that the |
| 332 | command is in progress (IP): |
| 333 | |
| 334 | {{{ |
| 335 | IP Myctag |
| 336 | }}} |
| 337 | |
| 338 | If the command is entered correctly and no errors are encountered, the !CoreDirector |
| 339 | Switch processes the command and issues a completed (COMPLD) response; |
| 340 | otherwise, the !CoreDirector Switch provides an error code and an explanation. |
| 341 | |
| 342 | === TL1 Output Conventions === |
| 343 | |
| 344 | !CoreDirector Switch outputs messages and command responses in plain text (ASCII). TL1 |
| 345 | returns any of the following four types of responses and messages to |
| 346 | commands: |
| 347 | |
| 348 | * Normal Responses |
| 349 | * Acknowledgement Messages |
| 350 | * Error Messages and Codes |
| 351 | * Autonomous Messages |
| 352 | |
| 353 | ==== Normal Responses ==== |
| 354 | |
| 355 | Normal responses to commands are distinguished by an uppercase letter M as the first |
| 356 | character of the second line of the response. |
| 357 | |
| 358 | A response message may contain anywhere from three to hundreds of lines of text. If non-zero |
| 359 | data is available for requested parameters, a normal response report is |
| 360 | displayed as indicated in the following example: |
| 361 | |
| 362 | 1. SID DATE TIME |
| 363 | 2. M MYCTAG COMPLD |
| 364 | 3. "A-1-1:NEND,S&J&O,15-MIN" |
| 365 | 4. ; |
| 366 | |
| 367 | Line 1 indicates the source, date, and time of the response.[[BR]] |
| 368 | |
| 369 | Line 2 indicates the status of the response with the correlation tag and the completion code as follows: |
| 370 | |
| 371 | * COMPLD - Completed (or normal) |
| 372 | * DENY - Denied (or error) |
| 373 | * PRTL - Partial response |
| 374 | * |
| 375 | |
| 376 | Line 3 displays the specified report data or error codes (for example, |
| 377 | ENEQ, IPNV, and so forth). The TL1 Interface Manual provides additional |
| 378 | detailed information on error codes. |
| 379 | |
| 380 | Line 4 displays a semicolon (;) to indicate the end of the response. |
| 381 | |
| 382 | ==== Acknowledgement Messages ==== |
| 383 | |
| 384 | The !CoreDirector Switch returns an acknowledgment message if the system |
| 385 | is unable to issue a NORMAL or DENY response within 2 seconds of |
| 386 | receiving the input command. An acknowledgment response is repeated |
| 387 | every 20 seconds until concluded with a NORMAL or DENY message. |
| 388 | |
| 389 | The acknowledgment response message structure is as follows: |
| 390 | |
| 391 | {{{<acknowledgmentcode>^<ctag><cr><lf>}}} |
| 392 | |
| 393 | TL1 acknowledgment codes are as follows: |
| 394 | |
| 395 | * IP - In progress |
| 396 | * RL - Repeat later |
| 397 | * NA - No acknowledgment |
| 398 | * PF - Printout follows |
| 399 | |
| 400 | ==== Error Messages and Codes ==== |
| 401 | |
| 402 | TL1 input command error codes are four-character codes generated in |
| 403 | response to the user’s input commands. |
| 404 | |
| 405 | ==== Autonomous Messages ==== |
| 406 | |
| 407 | Autonomous messages are generated by the !CoreDirector Switch to report |
| 408 | alarms or events automatically. |
| 409 | |
| 410 | = Configurations & Provisioning = |
| 411 | |
| 412 | !CoreDirector configuring and provisioning commands are used to |
| 413 | create, configure, modify, and delete physical ports, subnetwork |
| 414 | connections, Open Systems Interconnection (OSI) protocols, Optical |
| 415 | Signaling and Routing Protocol (OSRP) links, cross connects, |
| 416 | facilities, and equipment. It is possible to do the configuration and |
| 417 | provisioning using either Node Manager interface or TL1 commands. In |
| 418 | this section we give a brief description of the basic configurations |
| 419 | and provisioning needed to operate the !CoreDirector CI in the GpENI |
| 420 | network from the view of Node Manager. |
| 421 | |
| 422 | == Configuration == |
| 423 | |
| 424 | === Physical Termination Points (PTPs) === |
| 425 | |
| 426 | The Physical TP screen consists of up to seven tabs (Basic, DCC, |
| 427 | Fault, Performance, Section Trace, LW Peer Comms, and Physical) that |
| 428 | provide PTP information. The number of tabs available is determined by |
| 429 | the type of Optical Module (OM) or port and is driven by the OM or |
| 430 | port capabilities. If the tree selection is not valid (for example |
| 431 | when a LM is selected in the equipment tree), Node Manager displays a |
| 432 | blank screen with a message "Not a valid screen for current |
| 433 | selection". |
| 434 | |
| 435 | '''Port Groups''' (Basic—Port Group) has one tab (Basic) which is |
| 436 | unique to Ethernet Port Groups. All remaining OMs and ports have at |
| 437 | least five tabs (Basic, DCC, Fault, Performance, and Section Trace). |
| 438 | OC-192 OMs have up to two additional tabs (LW Peer Comms and |
| 439 | Physical). |
| 440 | |
| 441 | All OC-192 OMs have the Physical tab. |
| 442 | |
| 443 | OM10G-WDM1 modules also have LW Peer. Basic is the default tab. |
| 444 | |
| 445 | === Group Termination Points (GTPs) === |
| 446 | |
| 447 | A GTP is a defined set of STS-1/AU-3 CTPs or STS3c/AU-4 CTPs that are |
| 448 | configured and treated as a single entity. For example, if multiple |
| 449 | STS-1/AU-3 CTPs are cross-connected in the same manner, creating a GTP |
| 450 | facilitates provisioning by requiring only a single cross connect for |
| 451 | all CTPs in the group. From the Group TPs screen, GTPs are created |
| 452 | from individual CTPs, and existing CTPs can be added or removed from a |
| 453 | GTP. |
| 454 | |
| 455 | === Connection Termination Points (CTPs) === |
| 456 | |
| 457 | CTPs are logical connection points used for manual cross-connecting |
| 458 | and automated provisioning of end-to-end circuits. CTPs are composed |
| 459 | of one or more STS-1or VC-3 time slots. |
| 460 | |
| 461 | === Ethernet Trail Termination Point (ETTP) === |
| 462 | |
| 463 | An ETTP is automatically created for each port when an LM-20-GE or |
| 464 | ESLM is inserted. ETTPs contain the provisioning and monitoring |
| 465 | related to the ethernet protocol and have states, provisioning |
| 466 | attributes, alarms and performance monitoring thresholds and counts |
| 467 | which are retrievable and editable by the user by way of the ETTP |
| 468 | screen. |
| 469 | |
| 470 | === Ethernet Flow (Eflow) === |
| 471 | |
| 472 | An Eflow represents the unidirectional stream of packets that |
| 473 | originate and terminate on given Ethernet Ports or VCGs and is used |
| 474 | for routing and priority assignment for mapping ETTPs within a port |
| 475 | group. An Eflow classifies a group of packets entering a specific |
| 476 | layer 2 port and controls the packets modification and routing. |
| 477 | |
| 478 | === Ethernet Virtual Concatenation Group (VCG) === |
| 479 | |
| 480 | An Ethernet VCG consists of one or more CTPs/GTPs/SNCs. The default |
| 481 | vlan id parameter specifies the vlan id assigned to packets arriving |
| 482 | on this interface. This also maps to the default VLAN tag to be |
| 483 | applied to the packet if a tag is to be added to the packet and an |
| 484 | explicit tag value is not provided. |
| 485 | |
| 486 | == Provisioning == |
| 487 | |
| 488 | === OSRP === |
| 489 | |
| 490 | OSRP allows networked !CoreDirector Switches to communicate, share |
| 491 | topology information, and calculate routes for individual connections. |
| 492 | OSRP is a key technology component behind !FastMesh™ Add-On Software |
| 493 | Package and automatic circuit provisioning capability. |
| 494 | |
| 495 | OSRP key features include: |
| 496 | |
| 497 | * Constraint-based routing |
| 498 | * Automatic network topology discovery (network topology auto-discovery) |
| 499 | * Automatic connection configuration and setup (end-to-end provisioning) |
| 500 | * OSRP link aggregation |
| 501 | * Multiple protection bundle ID |
| 502 | * Interface to Modeling and Planning Suite (MPS) for network optimization, |
| 503 | traffic engineering, and capacity planning |
| 504 | |
| 505 | The primary purpose of OSRP is to facilitate rapid connection |
| 506 | provisioning and mesh restoration. These connections can have a |
| 507 | variety of protection requirements, such as requiring the connection |
| 508 | to traverse only linear 1:N or ring-protected spans or the |
| 509 | establishment of restoration priority levels. OSRP supports two types |
| 510 | of provisioning: explicitly routed connections and automatically |
| 511 | routed connections. |
| 512 | |
| 513 | ==== Explicitly Routed Connection ==== |
| 514 | |
| 515 | Explicitly routed connections contain user-defined routes that OSRP |
| 516 | provisions. In this provisioning mode, the explicit route for the |
| 517 | working path and an optional set of protection routes are defined as |
| 518 | preferred or exclusive routes. (Routes provided by the user must still |
| 519 | satisfy the routing constraints associated with the connection, such |
| 520 | as required protection class and maximum end-to-end delay.) |
| 521 | |
| 522 | ==== Automatically Routed Connection ==== |
| 523 | |
| 524 | Automatically routed connections contain OSRP-defined routes that OSRP |
| 525 | also provisions. Working and protection paths are automatically |
| 526 | computed based on the existing network conditions, user-specified |
| 527 | connection constraints, and class of service. OSRP provisions the |
| 528 | end-to-end connection, creates dynamic cross connects across the |
| 529 | network, and manages the connection as a single end-to-end entity. |
| 530 | After the connections are provisioned, the working route becomes the |
| 531 | connection's home route; protected traffic can revert to the home |
| 532 | route if that option is enabled. The protection path can be shared by |
| 533 | other end-to-end connections, and OSRP regularly recomputes the |
| 534 | protection path (approximately every 30 seconds) to take changes in |
| 535 | network conditions into account. |
| 536 | |
| 537 | Automatic connections begin at the originating node.The above figure |
| 538 | illustrates how a connection is configured and provisioned. This |
| 539 | example shows a request to connect Endpoint X and Endpoint Y. The |
| 540 | connection is configured at !CoreDirector Switch {{{#1}}}. OSRP |
| 541 | computes the route and automatically creates cross connects on |
| 542 | !CoreDirector Switches {{{#1, #2, #5, and #4}}} as it sends traffic to |
| 543 | the destination port. |
| 544 | |
| 545 | [[Image(osrp.png, 600px)]] |
| 546 | Figure 3: OSRP |
| 547 | |
| 548 | Automatically routed connections can be manually regroomed at any |
| 549 | time. The regrooming operation attempts to locate a new optimal path |
| 550 | through the network according to user-specified constraints, such as |
| 551 | least cost. If a more optimal route is found, the regrooming operation |
| 552 | establishes the connection across the new home route and takes the |
| 553 | existing connection out of service. |
| 554 | |
| 555 | === Subnetwork Connections (SNCs) === |
| 556 | |
| 557 | There are three types of SNCs: dynamic, permanent, and SNCP Auto |
| 558 | CrossConnect. |
| 559 | |
| 560 | A dynamic SNC is an end-to-end circuit whose path can traverse any |
| 561 | number of nodes and can change over time. The route for dynamic SNCs |
| 562 | can be automatically computed by OSRP, or it can be a user-provisioned |
| 563 | explicit route. Dynamic SNCs can also be provisioned with revertive |
| 564 | and mesh-restorable attributes. |
| 565 | |
| 566 | A Permanent SNC (or PSNC) is a fixed end-to-end circuit. A PSNC |
| 567 | connection can use an explicit, exclusive user-defined routing |
| 568 | profile. When a line failure occurs, the P-SNC remains connected and |
| 569 | a SNC-Unavailable alarm is generated. An explicit change in the |
| 570 | physical path where one end point of a physical link belonging to the |
| 571 | PSNC’s path changes (such as link misconnection, node removal, or node |
| 572 | insertion) is perceived as a conscious action on the part of the user |
| 573 | and results in the immediate release of the PSNC. PSNCs are not |
| 574 | mesh-restorable, cannot have a routing profile consisting of protected |
| 575 | line routes, and cannot be manually regroomed. PSNCs can participate |
| 576 | in APS/MSP, BLSR, !SmartRing™ VLSR, or MS-SPRing protection. |
| 577 | |
| 578 | A Subnetwork Connection Protection (SNCP) is a type of SNC that is 1+1 |
| 579 | protected across the network using the SNCP protocol. A head-end |
| 580 | bridge function transmits two copies of the path signal across the |
| 581 | network while a tail-end select function selects the better of two |
| 582 | received path signals. |
| 583 | |
| 584 | === Cross Connect Provisioning and Connection Management === |
| 585 | |
| 586 | A cross connect is a port-to-port connection contained within the |
| 587 | !CoreDirector Switch. The two types of cross connects are static and |
| 588 | dynamic. |
| 589 | |
| 590 | Static, or manual, cross connects are user-provisioned, fixed |
| 591 | connections between an originating and a terminating end point on a |
| 592 | single node. |
| 593 | |
| 594 | Dynamic cross connects are established and cleared within each |
| 595 | !CoreDirector Switch in support of an SNC. This type of cross connect |
| 596 | cannot be manually provisioned or deleted. A dynamic cross connect is |
| 597 | automatically created along the path that the SNC takes through the |
| 598 | network and can be mesh-restored. |
| 599 | |
| 600 | |
| 601 | !CoreDirector Cross Connect supports the following cross connect manual |
| 602 | provisioning features: |
| 603 | |
| 604 | * Standards-based static cross connect control |
| 605 | * Path-level cross connects from any SONET |
| 606 | STS-1/STS-3c/STS-12c/STS-48c to any |
| 607 | STS-1/STS-3c/STS-12c/STS-48c or from any |
| 608 | SDH VC3/VC4/VC4-4c/VC4-16c/VC4-64c to any |
| 609 | VC3/VC4/VC4-4c/VC4-16c/VC4-64c (in other words, any time |
| 610 | slot of any port to any time slot of an port) |
| 611 | * Port-to-port cross connects from any SONET |
| 612 | OC-3/OC-12/OC-48 to any OC-3/OC-12/OC-48 or from |
| 613 | any SDH STM1/STM4/STM16/STM64 to any |
| 614 | STM1/STM4/STM16/STM64 (transparent to path level tributaries) |
| 615 | * Linear APS or Multiplex Section Protection (MSP) for all cross |
| 616 | connectsManual cross connect capability of STS-192c/VC4-64c signals |
| 617 | * Support for arbitrary concatenation (STS-21c/VC4-7c for GigE transport) |
| 618 | * SONET/SDH manual cross connects for international gateway applications |
| 619 | (applicable with theSONET/SDH Gateway Utility). The cross connect (path |
| 620 | level) end points must be at compatible rates. For example, a VC-4 |
| 621 | on an STM-1 can be cross connected to an STS-3c on an OC-48. |
| 622 | * Automatic S-bit translation between SONET and SDH interfaces |
| 623 | ==== End Points ==== |
| 624 | |
| 625 | The end point of a cross connect can be either a Connection |
| 626 | Termination Point (CTP) or a Group Termination Point (GTP). |
| 627 | |
| 628 | = DRAGON/DCN interface for !CoreDirector = |
| 629 | |
| 630 | DRAGON software is part of the DCN Software Suite and is utilized as |
| 631 | the Domain Controller (DC) for the Internet2 Dynamic Circuit Network |
| 632 | (DCN). The DRAGON software enables the dynamic control of multiple |
| 633 | dataplane technologies. This includes various Ethernet switches, Ciena |
| 634 | Core Directors, or a topology which includes a mix of both. This |
| 635 | section explains the mechanism of how DRAGON/DCN software controls the |
| 636 | !CoreDirector CI in Internet2/ESnet network and also the the list of |
| 637 | TL1 commands used for provisioning the !CoreDirector switches. This |
| 638 | document assumes basic familiarity with DRAGON/DCN software. |
| 639 | |
| 640 | The network of !CoreDirector's(CD) are controlled under the ''Subnet |
| 641 | Control Model''. Unlike the Ethernet switch case that a VLSR only |
| 642 | controls one switch, VLSR under this model can control multiple |
| 643 | CDs. It uses TL1 to connect to source and/or destination CD to create |
| 644 | a subnet- connection (SNC) and map Ethernet flows into time slots in |
| 645 | the SNC. Depending on the location of source and destination CDs, it |
| 646 | is possible to have one or two VLSRs involved in creation of the |
| 647 | SNC. When the source and destination are on the same CD node, a VLSR |
| 648 | will create cross-connect (CRS) instead of SNC.Currently Ciena TDM |
| 649 | layer subnet topology information is loaded into NARB/RCE via a static |
| 650 | configuration file (/usr/local/etc/ciena_subnet.conf). The Core |
| 651 | Director Ethernet edge ports information is configured in |
| 652 | /usr/local/etc/ospfd.conf on the controlling VLSRs. NARB/RCE will |
| 653 | associate the Ethernet ports with TDM layer topology for cross-layer |
| 654 | path computation. Topology states will be updated based on path |
| 655 | computation and provisioning results. The following is the detailed |
| 656 | explanation of subnet control model. |
| 657 | |
| 658 | == Subnet Control Model == |
| 659 | |
| 660 | [[Image(subnetcontrol.png, 800px)]] |
| 661 | Figure 4: Subnet Control |
| 662 | |
| 663 | This is a special type of networking model that allows a subnet to be |
| 664 | embedded into a domain. The purpose of this subnet control model is to |
| 665 | harness the native control capabilities of vendor-specific |
| 666 | subnets. The advantage in so doing is that we can use the vendor’s |
| 667 | native control for reliability and simplicity while still maintaining |
| 668 | the integrity of the overall GMPLS control plane. For example, the |
| 669 | Internet2 Dynamic Circuit Network (DCN) has a subnet of Ciena |
| 670 | !CoreDirector SONET switches, which provide flexible Ethernet-to-SONET |
| 671 | mapping at every switch. The !CoreDirector switches have robust |
| 672 | Ethernet service provisioning capability between two edge ports via |
| 673 | TL1 or GMPLS based OIF UNI interfaces. The idea is to make use of such |
| 674 | native provisioning capability without going through the hassles to |
| 675 | control the subnet from inside. By overlaying one or more Subnet |
| 676 | Controlling VLSRs on top of the subnet, it is possible to control from |
| 677 | edge to edge via standard interfaces such as OIF UNI or via vendor’s |
| 678 | native control API. |
| 679 | |
| 680 | The implications of this model for NARB/RCE design include the |
| 681 | following. Firstly, RCE needs to have a detailed view of the Subnet |
| 682 | topology so it can compute a deterministic path across the |
| 683 | subnet. Secondly, RCE needs to incorporate the Subnet topology into |
| 684 | the domain topology since the two topologies come from different |
| 685 | sources and bridging links need to be identified or created in order |
| 686 | to carry out crossing-layer path computation. Thirdly, NARB/RCE needs |
| 687 | to map the data-flow path into a signaling-flow path because the |
| 688 | actual signaling flow traverses the VLSR level instead of the Subnet |
| 689 | level. The number of Subnet VLSRs can range from one to the number of |
| 690 | Subnet switches, but usually less than the latter number so one VLSR |
| 691 | may correspond to multiple subnet switches. This separation of |
| 692 | signaling flow from data flow is not only the nature of GMPLS |
| 693 | control-plane separation but also a separation of DRAGON VLSR level |
| 694 | control from vendor-specific Subnet level control, which makes the |
| 695 | Subnet Control Model unique. |
| 696 | |
| 697 | In addition to the VLSR-level ERO, the NARB/RCE path computation also |
| 698 | generates a subnet ERO for a domain with a subnet. The subnet ERO can |
| 699 | be returned to a client upon request. The client can therefore learn |
| 700 | subnet-level path details. Also, a subnet ERO can be used to guarantee |
| 701 | a deterministic, explicit path across the subnet during signaling |
| 702 | time. This is particularly important for book-ahead or scheduled |
| 703 | services. Subnet ERO can be converted into any vendor-specific |
| 704 | explicit route representation, such as the Designated Transit List |
| 705 | (DTL) for Ciena !CoreDirector subnets. |
| 706 | |
| 707 | When combined with vendor-specific subnet signaling features, a subnet |
| 708 | ERO or its converted form can help eliminate inconsistent routing |
| 709 | paths between scheduling and signaling. |
| 710 | |
| 711 | == List of TL1 Commands == |
| 712 | |
| 713 | The DRAGON software uses TL1 commands to provision !CoreDirector |
| 714 | switches to create DCN circuit across Internet2 network. The following |
| 715 | is the list of TL1 commands used by the DRAGON to provision circuit |
| 716 | from source node CHIN to destination node NEWY in the Internet2 DCN |
| 717 | network. The circuit provisioned uses VLAN tag 3060. |
| 718 | |
| 719 | ''' Visit destination node first via TL1 (NEWY) by telnet'ing to port |
| 720 | 10201:''' |
| 721 | |
| 722 | {{{ |
| 723 | #!rst |
| 724 | +--------------------------------+--------------------------------+ |
| 725 | |act-user::dragon:123:: | | |
| 726 | |\*\*PASSWORD\*\*; | | |
| 727 | |inh-msg-all:::123; |Login via TL1 and inhibit all | |
| 728 | | |messages | |
| 729 | +--------------------------------+--------------------------------+ |
| 730 | |rtrv-vcg::all:503061; |Retrieves the configuration | |
| 731 | | |settings for all the Virtual | |
| 732 | | |Concatenation Groups existing | |
| 733 | | |in the NE | |
| 734 | +--------------------------------+--------------------------------+ |
| 735 | |rtrv-ocn::1-A-5-1:503062; |Retrieves the configuration for | |
| 736 | | |the interface 1-A-5-1. The rate | |
| 737 | | |of the interface can be OC3, | |
| 738 | | |OC12, OC48 or OC192 | |
| 739 | +--------------------------------+--------------------------------+ |
| 740 | |ent-vcg::name=vcg_503062: |Creates the Virtual | |
| 741 | |503062::,pst=IS,suppttp=1-A-5-1,|Concatenation Group in the | |
| 742 | |crctype=CRC_32,,, |interface 1-A-5-1 with name | |
| 743 | |framingmode=GFP, |vcg_503062. This command | |
| 744 | |tunnelpeertype=NONE,,,gfpfcs, |specifies that STS slots from 1 | |
| 745 | |enabled=YES,,,groupmem=1&&21,,; |to 21 forms this VCG group. It | |
| 746 | | |means the capacity of the VCG | |
| 747 | | |is nearly 1 Gig | |
| 748 | +--------------------------------+--------------------------------+ |
| 749 | |ent-eflow::eflow_vcg_503062_in: |Creates a new Ethernet Eflow. | |
| 750 | |503063:::ingressporttype=ettp, |An Eflow classifies a group of | |
| 751 | |ingressportname=1-A-5-1-1, |packets entering a specific | |
| 752 | |pkttype=single_vlan_tag, |layer 2 port and controls the | |
| 753 | |outervlanidrange=3060,, |packets modification and | |
| 754 | |priority=1&&8, |routing. This command uses VLAN | |
| 755 | |egressporttype=vcg, |id 3060 for classification of | |
| 756 | |egressportname=vcg_503062, |packets. It means all the | |
| 757 | |cosmapping=cos_port_default; |packets that is coming from the | |
| 758 | | |VCG group vcg_503062 is | |
| 759 | | |attached a VLAN tag of 3060 and | |
| 760 | | |is sent to source via port | |
| 761 | | |1-A-5-1-1 | |
| 762 | +--------------------------------+--------------------------------+ |
| 763 | |ent-eflow::eflow_vcg_503062_out:|This command creates the Eflow | |
| 764 | |503064:::ingressporttype=vcg, |in the reverse direction of the | |
| 765 | |ingressportname=vcg_503062, |previous command. Hence a 2 way | |
| 766 | |pkttype=single_vlan_tag, |Eflow is created between the | |
| 767 | |outervlanidrange=3060,, |VCG group created and the port | |
| 768 | |priority=1&&8, |1-A-5-1-1 | |
| 769 | |egressporttype=ettp, | | |
| 770 | |egressportname=1-A-5-1-1, | | |
| 771 | |cosmapping=cos_port_default; | | |
| 772 | +--------------------------------+--------------------------------+ |
| 773 | }}} |
| 774 | |
| 775 | '''Visit source node via TL1 (CHIN) again by telnet'ing to port |
| 776 | 10201:''' |
| 777 | |
| 778 | {{{ |
| 779 | #!rst |
| 780 | +----------------------------------------------------------------+----------------------------------------------------------------+ |
| 781 | |rtrv-vcg::all:3061; |Retrieves the configuration settings for all the | |
| 782 | | |Virtual Concatenation Groups existing in the NE | |
| 783 | +----------------------------------------------------------------+----------------------------------------------------------------+ |
| 784 | |rtrv-ocn::1-A-6-1:3062; |Retrieves the configuration for the interface 1-A-6-1. The | |
| 785 | | |rate of the interface can be OC3, OC12, OC48 or OC192 | |
| 786 | +----------------------------------------------------------------+----------------------------------------------------------------+ |
| 787 | |ent-vcg::name=vcg_3062:3062::,pst=IS,suppttp=1-A-6-1, |Creates the Virtual Concatenation Group in the interface | |
| 788 | |crctype=CRC_32,,,framingmode=GFP,tunnelpeertype=NONE,,, |1-A-6-1 with name vcg_3062. This command specifies that STS | |
| 789 | |gfpfcsenabled=YES,,,groupmem=1&&21,,; |slots from 1 to 21 forms this VCG group. It means the capacity | |
| 790 | | |of the VCG is nearly 1 Gig | |
| 791 | +----------------------------------------------------------------+----------------------------------------------------------------+ |
| 792 | |ent-eflow::eflow_vcg_3062_in:3063:::ingressporttype=ettp, |Creates a new Ethernet Eflow. This also classifies the packets | |
| 793 | |ingressportname=1-A-6-1-1,pkttype=single_vlan_tag, |based on VLAN id 3060. It means to all the incoming packets | |
| 794 | |outervlanidrange=3060,,priority=1&&8, |associated with the VCG group vcg_3062,is attached VLAN tag of | |
| 795 | |egressporttype=vcg,egressportname=vcg_3062, |3060 and transmitted to the destination via port 1-A-6-1-1 | |
| 796 | |cosmapping=cos_port_default; | | |
| 797 | +----------------------------------------------------------------+----------------------------------------------------------------+ |
| 798 | |ent-gtp::gtp_3065:3065::lbl=gtp-vcg_3062,,ctp=vcg_3062-CTP-1& |Creates a GTP with 21 STS1 ie of capacity 1 Gig | |
| 799 | |vcg_3062-CTP-2&vcg_3062-CTP-3&vcg_3062-CTP-4&vcg_3062-CTP-5& | | |
| 800 | |vcg_3062-CTP-6&vcg_3062-CTP-7&vcg_3062-CTP-8&vcg_3062-CTP-9& | | |
| 801 | |vcg_3062-CTP-10&vcg_3062-CTP-11&vcg_3062-CTP-12&vcg_3062-CTP-13&| | |
| 802 | |vcg_3062-CTP-14&vcg_3062-CTP-15&vcg_3062-CTP-16&vcg_3062-CTP-17&| | |
| 803 | |vcg_3062-CTP-18&[[BR]]vcg_3062-CTP-19&vcg_3062-CTP-20& | | |
| 804 | |vcg_3062-CTP-21; | | |
| 805 | +----------------------------------------------------------------+----------------------------------------------------------------+ |
| 806 | |ent-snc-stspc:CHIC:gtp_3065,1-A-5-1-1&&21:3066::name=snc_3066, |Creates a Subnetwork Connection from source NE CHIC to remote | |
| 807 | |type=dynamic,rmnode=NEWY,lep=gtp_nametype,conndir=bi_direction, |NE NEWY. This is the command that actually creates a circuit of | |
| 808 | |prtt=aps_vlsr_unprotected,pst=is; |capacity 1 Gig across the network between NE's CHIC and NEWY. | |
| 809 | | |This uses the OSRP protocol as described above | |
| 810 | +----------------------------------------------------------------+----------------------------------------------------------------+ |
| 811 | }}} |
| 812 | |
| 813 | The following are the TL1 commands used to teardown the circuit |
| 814 | provisioned above. |
| 815 | |
| 816 | '''Teardown (on CHIN):''' |
| 817 | |
| 818 | ||rtrv-snc-stspc::snc_3066:3067;||Retrieves the Subnet connection with SNC id snc_3066|| |
| 819 | ||ed-snc-stspc::snc_3066:3068::,pst=oos;||This command puts the SNC snc_3066 to state out of service|| |
| 820 | ||dlt-snc-stspc::snc_3066:3069;||Deletes the SNC snc_3066|| |
| 821 | ||rtrv-gtp::gtp_3065:3070;||Retrieves the GTP with gtp name gtp_3065|| |
| 822 | ||dlt-gtp::gtp_3065:3071;||Deletes the GTP with gtp name gtp_3065|| |
| 823 | ||rtrv-vcg::vcg_3062:3072;||Retrieves the parameters of VCG group vcg_3062|| |
| 824 | ||rtrv-eflow::eflow_vcg_3062_in:3073;||Retrieves the parameters of Ethernet Eflow eflow_vcg_3062_in|| |
| 825 | ||dlt-eflow::eflow_vcg_3062_in:3074;||Deletes the Eflow with name eflow_vcg_3062_in|| |
| 826 | ||dlt-eflow::eflow_vcg_3062_out:3075;||Deletes the Eflow with name eflow_vcg_3062_out|| |
| 827 | ||ed-vcg::name=vcg_3062:3076::,pst=OOS;||This command put the VCG group vcg_3062 to out of service|| |
| 828 | ||dlt-vcg::name=vcg_3062:3077;||Deletes the VCG group vcg_3062|| |
| 829 | |
| 830 | '''Teardown (on NEWY):''' |
| 831 | |
| 832 | ||rtrv-vcg::vcg_503062:503065;||Retrieves the parameters of VCG group vcg_503062|| |
| 833 | ||rtrv-snc-stspc::all:503065;||Deletes all the SNC connection in the NE|| |
| 834 | ||rtrv-eflow::eflow_vcg_503062_in:503066;||Retrieves the parameters of Ethernet Eflow eflow_vcg_503062_in|| |
| 835 | ||dlt-eflow::eflow_vcg_503062_in:503067;||Deletes the Eflow with name eflow_vcg_503062_in|| |
| 836 | ||dlt-eflow::eflow_vcg_503062_out:503068;||Deletes the Eflow with name eflow_vcg_503062_out|| |
| 837 | ||ed-vcg::name=vcg_503062:503069::,pst=OOS;||This command put the VCG group vcg_503062 to out of service|| |
| 838 | ||dlt-vcg::name=vcg_503062:503070;||Deletes the VCG group vcg_503062|| |
| 839 | |
| 840 | = Acknowledgements = |
| 841 | |
| 842 | Ciena - Ronald Jones [[BR]] |
| 843 | DRAGON/DCN - Xi Yang [[BR]] |
| 844 | MAX/MANFRED - Chris Tracy [[BR]] |
| 845 | |
| 846 | = References = |
| 847 | |
| 848 | 1. [https://wiki.ittc.ku.edu/gpeni_wiki/images/7/78/CoreDirector_TL1_Document.pdf CoreDirector CI MultiService Switch TL1 Interface Manual (R5.2.6)] |
| 849 | 2. [https://wiki.ittc.ku.edu/gpeni_wiki/images/1/12/NodeManagerUserGuide.pdf CI MultiService Switch Node Manager User Guide (R5.2.6)] |
| 850 | 3. [https://wiki.ittc.ku.edu/gpeni_wiki/images/0/0e/CoreDirector_System_Description.pdf CI MultiService Switch System Description (R5.2.6)] |
| 851 | 4. [https://wiki.internet2.edu/confluence/download/attachments/19074/narb-rce-architecture-v2.1b.pdf?version=1 NetworkAware Resource Broker (NARB) and Resource Computation Element (RCE) Architecture] [[BR]] |
| 852 | 5. https://wiki.internet2.edu/confluence/display/DCNSS/Home [[BR]] |
| 853 | 6. http://dragon.maxgigapop.net/twiki/bin/view/DRAGON/WebHome [[BR]] |
| 854 | 7. http://en.wikipedia.org/wiki/Transaction_Language_1[[BR]] |