Experimental Physics Computing

The batch farm contains ~300 CentOS 7.7 nodes, with 8, 16, 24, 32, or 36 cores. Each core is run with two hardware threads, for two job slots per core, for a total of ~24000 job slots

• 2019: 64 nodes, dual 32 core AMD EPYC 7502 (Rome), 256GB memory, 1TB SSD, FDR IB
• 2018: 88 nodes, dual 20 core 2.4 GHz Xeon (Skylake), 96 GB memory, 480GB SSD, FDR IB
• 2016: 44 nodes, dual 18 core 2.3 GHz Xeon E5-2697V4 (Broadwell), 64 GB memory, 1 TB HDD, FDR IB
• 2014: 104 nodes, dual 12 core 2.3 GHz Xeon (Haswell), 32 GB memory, dual 1 TB HDDs, QDR IB
• 2013: 24 nodes, dual 8 core 2.6 GHz Xeon (Ivy Bridge), 32 GB memory, dual 1 TB HDDs, QDR IB

The batch farm runs mostly serial jobs which spend part of their time in wait states for file I/O, and so the number of cores is oversubscribed by the batch system so that the number of jobslots equals the number of threads.    Memory per core is typically 2GB.  Jobs with larger memory requirements than 2 GB can still run by declaring their memory requirements. The batch system will leave slots on the same node unused so as to effectively allocate that memory to the larger job.

Farm Networking

All farm nodes are connected to both an Ethernet fabric and an Infiniband fabric, where the IB fabric is used for high speed access to the file servers.  The oldest nodes have QDR cards (single data rate IB, 40 Gb/s) The newest nodes use FDR cards.