PIC Network Information

This page describes PIC network information for WLCG use.

LAST UPDATE: 31/08/2023

Network Overview

PIC is situated within the datacenter from the UAB (Universitat Autònoma de Barcelona). In the same datacenter there is a connection point from CSUC and RedIRIS, our regional network provider and the Spanish NREN. Our connection to CSUC operates at 200Gbps and they provide redundancy from the UAB to Barcelona. At CSUC’s datacenter in Barcelona they interlink us to the project networks (LHCOPN, LHCONE) and also provide the regular traffic and a connection to IAC (Las Palmas de Gran Canaria). Using this 200Gbps link we also provide network connectivity to IFAE (Institut de Física d’Altes Energies).

Network Description

We have a pair of central ARISTA switches which provide external routing to CSUC and also function as the spine switches within a Layer 2 spine-leaf architecture. The ARISTA leafs proffer 10/25Gbps ports to the servers, forming paired connections to ensure redundancy and dual links to both spines. Furthermore, there are two additional leafs switches at the border, where we establish connectivity for a Fortinet Firewall and the DMZ. We also have some 1Gbps port switches although they lack redundancy. We are trying to move all the core servers to be connected in the leaf architecture rather than in single switches. Every rack has also a management switch where the servers’ management interfaces are connected.

The storage pools are usually connected at 2x25Gbps to a leaf pair. The connection for the worker nodes varies based on their server location. In our basement facility, equipped with submersion cooling, these nodes typically interface with 2x1G connections. Just to note that one interface is configured with an IPv6 address while the other IPv4 (we have different routing paths for the worker node external connection). At the main room we have more or less 50% of our computing power capacity, we’re presently constrained by a single 25Gbps due to limited available network ports. We plan to connect them with redundancy in the future.

Peering Description

Please describe how your site connects by responding to the following questions.

How does your site connect for commodity (non research and education) connectivity (www.google.com, www.github.com, etc)? CSUC provides us the commodity network and at this moment we share the same physical network link for the project network and the commodity one. We plan to split this in different physical connections in the future.

If so, who do you peer with and what is the bandwidth available for this type of traffic? We have a 200Gbps physical network and we share it with the projects, the commodity network and also the IFAE network. The commodity network has also a limitation set at our provider site of 30Gbps.
Are there any firewall or security devices in-line with this traffic? If so, please describe. We have a firewall in order to control the traffic from and to the commodity network. We don’t have an IPS or other security devices to control the traffic. We basically apply some policies in order to block and filter the traffic.

If you are connected to LHCONE, who/where do you peer with and at what bandwidth? PIC establishes a 100Gbps connection to LHCONE through RedIRIS. We use BGP and receive prefixes from RedIRIS. While we implement access control lists (ACLs) to filter permissible traffic, we trust in the prefixes provided by RedIRIS and don’t filter on the source network.

Are there any firewall or security devices in-line with this traffic? If so, please describe. No firewalls or security devices for LHCONE

If you are connected to LHCOPN, how to you connect and peer with CERN and at what bandwidth? PIC established a 100Gbps connection to LHCOPN. We establish dual BGP sessions on the ARISTA spine with CERN, segregating for both IPv4 and IPv6 traffic. As with LHCONE, we apply ACLs to filter the traffic in but we rely on the prefix we receive from CERN.

Are there any firewall or security devices in-line with this traffic? If so, please describe. No. Just the ACL in the spines.

Do you have a peering for research and education networks for non-LHCONE sites? Yes, we also have a dedicated circuit to IAC (https://www.iac.es/es/observatorios-de-canarias/centro-de-astrofisica-en-la-palma)

If so, who/where do you peer with and at what bandwidth? We have a 10Gbps circuit from Barcelona to IAC.
Are there any firewall or security devices in-line with this traffic? If so, please describe. Yes. The routing for this circuit is managed by the firewall.

Network Equipment Details

Spines: Arista - 7050CX3-32S-F

Leafs: Arista - 7050SX3-48YC8

Switches 1G : Arista - 7010T-48, Dell - N1148T-ON, Supermicro - SSE-G24-TG4

Management switches: Dell - X1026 1GB, Cisco Business 250

FW: Fortigate 1800F v.7.4.7

Network Monitoring

URL: https://wlcgmon.pic.es/pic-netmon.json

Sample

{
  "Description": "Network statistics for PIC",
  "UpdatedLast": "2023-09-08T10:18:23.812497+00:00",
  "InBytesPerSec": 1428392979.0383334,
  "OutBytesPerSec": 498684098.9637127,
  "UpdateInterval": "60 seconds",
  "MonitoredInterfaces": [
    "swspinea.pic.es_Ethernet1/1",
    "swspineb.pic.es_Ethernet1/1"
  ]
}

The file is updated once per minute.

Additional information: The server is a virtual machine with Alma9 operating system and an Apache web server that serves de JSON file at the previously shown URL.

Network Monitoring Link Into CRIC

See https://wlcg-cric.cern.ch/core/netsite/detail/ES-PIC

Network Diagrams

We are working on suitable diagrams.