ePIC Tutorial Series 2: Running Simulations

US/Eastern
    • 1
      Instroducing Tutorial Session 2
    • 2
      Eic-analysis Framework and Physics Variables Reconstruction
      Speaker: Christopher Dilks (Duke University)
    • 3
      Visualization: detector geometry and events
      Speakers: Shyam Kumar (Universita e INFN, Bari (IT)), Shyam Kumar (University and INFN Bari)
    • 15:50
      Coffee Break
    • 4
      Simulation Campaign and HTCondor Example
      Speaker: Wouter Deconinck (University of Manitoba)

      Central campaign scripts (run your simulations like a pro)
      ------------------------

      # Production simulation scripts

      ## Where are they? (github)
      - https://github.com/eic/simulation_campaign_single:
        run single particle simulations like central simulation productions
        Installed in /opt/campaigns/single/
      - https://github.com/eic/simulation_campaign_hepmc3
        run physics event simulations like central simulation productions
        Installed in /opt/campaigns/hepmc3/

      - https://github.com/eic/job_submission_condor
        run simulation productions through HTCondor systems
        Installed in /opt/campaigns/condor/
      - https://github.com/eic/job_submission_slurm
        run simulation productions through Slurm systems
        Installed in /opt/campaigns/slurm/

      ## What do they do?
      Assumptions:
      - input files are local, output files remain local
      - jobs always run single-threaded (predictability)
      - jobs are run in an isolated environment, i.e. a temporary directory
        (modify $TMPDIR if you wish to write elsewhere)

      Example files on S3:
      - S3/eictest/EPIC/EVGEN/SINGLE/e-/1GeV/e-_1GeV_eta0.0.steer
      - S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc
      - S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
      - S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root

      Can also run with s3http:// or root:// URLs as inputs for hepmc3 files:
      - root://dtn-eic.jlab.org//work/eic2/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
      - s3https://eics3.sdcc.bnl.gov:9000/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root

      ## How to run them in eic-shell?
      1. First copy the files from S3 to your local directory
      2. Run the single or hepmc3 script with the file as argument:
         run.sh <input> [n_chunk=10000] [i_chunk=]

      Steering with exported environment variables (in future we may switch to command line arguments):
      - COPYFULL=true # copy into calling directory
      - COPYRECO=true # copy into calling directory
      - COPYLOG=true # copy into calling directory
      - EBEAM=5 # no default, follows geometry definition)
      - PBEAM=41 # no default, follows geometry definition)
      - DETECTOR_VERSION=23.03.0 # default nightly)
      - DETECTOR_CONFIG=epic_brycecanyon # default epic)

      ## How to run them in a compute job (general)?
      - Run these commands inside an explicit eic-shell session
      cat << EOF | eic-shell
      > /opt/campaigns/hepmc3/scripts/run.sh pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
      > EOF

      - Tell your job scheduling system to run inside a container, e.g. with HTCondor, or using `scrun` in Slurm (not at JLab).

      # HTCondor

      ## HTCondor vs slurm
      Advantages of HTCondor:
      - HTCondor can download input files from S3 to the job node, and upload  output files to S3 (Slurm requires pre-staging and post-uploading)
      - HTCondor can run inside a container
      - HTCondor (on OSG) has scaling to O(100k) nodes

      Advantages of Slurm:
      - Targets longer jobs (in typical queues) instead of rapid throughput

      ## HTCondor job definition file
      Important statements:
      ```
      +ProjectName="EIC"
      +SingularityImage="/cvmfs/singularity.opensciencegrid.org/eicweb/jug_xl:nightly"
      Requirements = HAS_SINGULARITY == TRUE && HAS_CVMFS_singularity_opensciencegrid_org == TRUE && OSG_HOST_KERNEL_VERSION >= 31000
      ```
      The templates in templates/osg_csv.submit.in are a good basis.

      # Productions

      ## Automated benchmarking
      - https://github.com/eic/simulation_campaign_datasets and https://eicweb.phy.anl.gov/EIC/campaigns/datasets
        Determines time per event to run datasets, produces csv input files

      ## Scaling up submission
      Usage:
        ./scripts/submit_csv.sh <template> <type> <file or url> [target hours = 2]

      ```
      git clone https://github.com/eic/job_submission_condor condor
      cd condor
      ./scripts/submit_csv.sh
      ```

      Example:
      ```
      export EBEAM=18 PBEAM=275
      export DETECTOR_VERSION=23.03.0 DETECTOR_CONFIG=epic_arches
      ./scripts/submit_csv.sh osg_csv hepmc3 DIS/NC/18x275/minQ2=1/DIS_NC_18x275_minQ2=1.csv 2
      ```

      ## Dealing with holds
      ./scripts/hold_review_and_release.sh

    • 5
      Example of Job submission script with JLab SLURM
      Speaker: Wenliang Li (Stony Brook University CFNS)
    • 6