ZFS Appliance

Wed, 02/20/2013 - 14:22 — watson

ZFS (see ZFS wiki entry) is a file system with several distinct features and advantages.  Storage is managed as pools, and pools may be hierarchically split into smaller pools.  This gives the effect of hierarchical quotas, useful for our application, and a model for our own in-house storage management software running above Lustre.  ZFS (Z File System) implements RAID-Z, which has a tree of embedded checksums.  These checksums are always checked on read, so data integrity is very high.  Open ZFS is used

Lustre File System

Wed, 02/20/2013 - 13:34 — watson

The Lustre system (a distributed file system) spans multiple file servers, called Object Storage Servers (OSS), while presenting a single flat namespace as if it were a single server.  File system directory information is held in a Meta Data Server (MDS), which at JLab is implemented as a dual headed system with the two servers sharing a SAS disk array. One of the two heads is active and the other inactive, and they implement a hot failover in case the primary server fails.

Xeon Phi Specifications

Thu, 02/14/2013 - 14:58 — watson

Details

 

  • 16p (2016 Phi) -- 264 nodes, Xeon Phi 7230, 16 GB high bandwidth memory, 192 GB main memory, OmniPath fabric (100 Gb/s), 1 TB disk.

Each Knights Landing (KNL) node has 64 cores, hyper-threaded 4 ways (256 virtual cores) running at 1.3 GHz.  The on-package high bandwidth memory has a bandwidth above 450 GB/s, and the main memory has a bandwidth of about 90 GB/s (available concurrently).

Infiniband Fabric

Thu, 02/14/2013 - 14:38 — watson

 

Infiniband Fabric

 

The different clusters have different optimizations of the Infiniband network fabric.  Both have what is referred to as a "fat tree" topology.  Individual nodes are connected to "leaf switches", and the leaf switches are connected to "core" switches.   They differ in the amount of bandwidth between leaf switches.

GPU Specifications

Wed, 02/13/2013 - 15:34 — watson

Host Details

  • 19g (2019 GeForce RTX 2080) -- 32 nodes, octa RTX 2080 GPU, 24 Intel(R) Xeon(R) Gold 5118 cores, 196 GB memory, Omni-Path fabric (100 Gb/s), 1TB disk.
  • 12k (2012 Kepler) -- 46 nodes, 16 Intel 2.0 GHz cores, 128 GB memory, quad GPU (Kepler K20m), FDR (56 Gb/s) Infiniband, 1 TB disk.

GPU cards

Tesla cards (w/ ECC memory)

Scientific Computing at Jefferson Lab

Wed, 02/13/2013 - 11:49 — Anonymous (not verified)

(this will eventually be an outline plus most important news, upcoming events, etc.)


Scientific Computing consists of two main systems, one for Experimental Physics and the other for Lattice QCD (theory) computing. Many resources (file servers, offline storage, wide area networking) are shared between the two, with appropriate allocations. 

See

Wide Area Networking

Tue, 02/12/2013 - 17:14 — watson

Jefferson Lab has a 10g wide area networking connection to a MAN (metropolitan area network) with a 10g connection up to ESnet in Washington D.C. and a redundant 10g connection to ESnet in Atlanta.  JLab can reasonably use 5 Gbps of this, and Scientific Computing can reasonably use 4 Gpbs.  Thus each of CLAS, GlueX, A+C+misc, and LQCD can on average use 1 Gbps, although may on occasion find they can sustain 5-6 Gbps.

Interactive Nodes

Tue, 02/12/2013 - 17:01 — watson
All interactive user systems are available from offsite through the JLab user login gateways:  login.jlab.org.
 
JLab resources are upgrading in 2019, to use the Slurm (SchedMD.com) manager.  The JLab Slurm testbed environment runs CentOS 7, and is available via the systems
  • islurm1201
  • hpci12k01 2012 dual 8-core Sandy Bridge, 128 GB memory, 1 K20m GPU, CUDA 10, CentOS 7; front end to JLab Slurm12k cluster
 

Tape Library (offline storage)

Tue, 02/12/2013 - 16:15 — watson

IBM TS3500 Tape Library

The JLab Mass Storage System (MSS) is an IBM TS3500 tape library with LTO drives,  installed in 2008 to replace JLab's original StorageTek silo with Redwood technology. The TS3500 is a modular system, with an expandable number of frames for tape slots and drives, and an expandable number of tape drives. The lab's JASMine software provides the user interface to the MSS.

Our current configuration consists of

Experimental Physics File System Layout

Mon, 02/11/2013 - 12:59 — watson

Experimental Physics users see a file system layout with many parts:

/home: is a file system accessible from all CUE nodes, and is the user's normal home directory, held on central file servers.

/group: is a file system accessible from all CUE nodes, and is a shared space for a group such as an experiment, held on central file servers.

Pages

Powered by Drupal