Using the Jefferson Lab Knights Landing Cluster



Using the Jefferson Lab Knights Landing Cluster

This page describes the Jefferson Lab Knights Landing Cluster

    Tape Scheduling


    Workflow - Swif - will be decommissioned on March 1st

    What is Swif?

    Swif, the" scientific workflow indefatigable factotum", is a system that aims to simplify the use of Jefferson Lab's batch system.  As the name implies, it will work tirelessly on your behalf so that that you need not expend unnecessary effort to make good use of the compute farm.

    The goal of this initial release is to provide some features that are lacking in Auger.  While future versions of Swif may bypass Auger entirely, the current version functions as a middleman between you and Auger, providing the following enhanced capabilities:

    jcache

    Name

    jcache - manage files and directories on the write-through cache area

     

    FAQ

    What changes are needed  to a jsub script when the job stages files to the write-through cache system?

    Write-through Cache

    Migrated to ServiceNow.  Text below may be out of date.


    The write-through cache system is a front end disk cache system to the tape library system. It enables a user with appropriate permission to deposit new files into the cache. The new files will be migrated into the tape library after a short period of time within which the files can be modified or deleted.

    Best Practices

    1. Try to submit multi-threaded jobs when possible. Using a multi-threaded job can save memory foot-print of the job and may increase the performance of the job.
    2. Let Auger manage the input and output files. Do not call jget/jput from the farm job. If jget/jput is called inside one's job, the time period which the job waits for the tape I/O to finish is counted toward the wall time of this job. Note: jget and jput inside Auger will be disabled in the near future.

    DAQ Support: Writing Raw Data to Tape

    All raw data from the experimental physics detectors goes through a number of steps to reach permanent storage on tape:
    1. The DAQ system assembles events and writes them into a raw data file on a local staging area.  This area is large enough to hold a minimum of 24 hours of high rate running.

    jobinfo

    Name
        jobinfo - displays
    detailed information about a job (including completed jobs).
      
    Syntax
        jobinfo auger-job-index

    Description
      
    Display a full status of a farm job. The result is obtained from batch system if job is in pending or active state, or finished within few days. Otherwise display the job information retrieved from auger database.


    Access to Remote Git Repositories

    Access to Remote Git Repositories

    We allow interactive access to off-site Git repositories. Since these can come in as either SSH traffic or HTTPS traffic,
    we allow them only on an as needed basis. We currently allow:

    SSH Based Access

    SSH access to both these repositories should proceed as normal. There is no need for tunneling.

    To enable pushing via https to github you have to modify the remote url with the following after cloning:

    Pages