ePIC Tutorial Series 2: Running Simulations

Name: ePIC Tutorial Series 2: Running Simulations
Start: 2023-03-14T14:00:00-04:00
End: 2023-03-14T17:30:00-04:00
Location: No location set

Tuesday 14 Mar 2023, 14:00 → 17:30 US/Eastern

Description

Zoom Link

One Pager Preparation Note

Recording

- 14:05 → 14:20
  
  Instroducing Tutorial Session 2 15m
  
  Introduction
- 14:20 → 15:05
  
  Eic-analysis Framework and Physics Variables Reconstruction 45m
  
  Speaker: Christopher Dilks (Duke University)
  
  Github Repository
  
  Tutorial (README)
- 15:05 → 15:50
  
  Visualization: detector geometry and events 45m
  
  Speakers: Shyam Kumar (Universita e INFN, Bari (IT)), Shyam Kumar (University and INFN Bari)
  
  Visualization_Tutorial_Shyam.pdf
- 15:50 → 16:05
  
  Coffee Break 15m
- 16:05 → 16:55
  
  Simulation Campaign and HTCondor Example 50m
  
  Speaker: Wouter Deconinck (University of Manitoba)
  
  Central campaign scripts (run your simulations like a pro)
  ------------------------
  
  # Production simulation scripts
  
  ## Where are they? (github)
  - https://github.com/eic/simulation_campaign_single:
  run single particle simulations like central simulation productions
  Installed in /opt/campaigns/single/
  - https://github.com/eic/simulation_campaign_hepmc3
  run physics event simulations like central simulation productions
  Installed in /opt/campaigns/hepmc3/
  
  - https://github.com/eic/job_submission_condor
  run simulation productions through HTCondor systems
  Installed in /opt/campaigns/condor/
  - https://github.com/eic/job_submission_slurm
  run simulation productions through Slurm systems
  Installed in /opt/campaigns/slurm/
  
  ## What do they do?
  Assumptions:
  - input files are local, output files remain local
  - jobs always run single-threaded (predictability)
  - jobs are run in an isolated environment, i.e. a temporary directory
  (modify $TMPDIR if you wish to write elsewhere)
  
  Example files on S3:
  - S3/eictest/EPIC/EVGEN/SINGLE/e-/1GeV/e-_1GeV_eta0.0.steer
  - S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc
  - S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
  - S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
  
  Can also run with s3http:// or root:// URLs as inputs for hepmc3 files:
  - root://dtn-eic.jlab.org//work/eic2/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
  - s3https://eics3.sdcc.bnl.gov:9000/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
  
  ## How to run them in eic-shell?
  1. First copy the files from S3 to your local directory
  2. Run the single or hepmc3 script with the file as argument:
  run.sh <input> [n_chunk=10000] [i_chunk=]
  
  Steering with exported environment variables (in future we may switch to command line arguments):
  - COPYFULL=true # copy into calling directory
  - COPYRECO=true # copy into calling directory
  - COPYLOG=true # copy into calling directory
  - EBEAM=5 # no default, follows geometry definition)
  - PBEAM=41 # no default, follows geometry definition)
  - DETECTOR_VERSION=23.03.0 # default nightly)
  - DETECTOR_CONFIG=epic_brycecanyon # default epic)
  
  ## How to run them in a compute job (general)?
  - Run these commands inside an explicit eic-shell session
  cat << EOF | eic-shell
  > /opt/campaigns/hepmc3/scripts/run.sh pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
  > EOF
  
  - Tell your job scheduling system to run inside a container, e.g. with HTCondor, or using `scrun` in Slurm (not at JLab).
  
  # HTCondor
  
  ## HTCondor vs slurm
  Advantages of HTCondor:
  - HTCondor can download input files from S3 to the job node, and upload output files to S3 (Slurm requires pre-staging and post-uploading)
  - HTCondor can run inside a container
  - HTCondor (on OSG) has scaling to O(100k) nodes
  
  Advantages of Slurm:
  - Targets longer jobs (in typical queues) instead of rapid throughput
  
  ## HTCondor job definition file
  Important statements:
```
+ProjectName="EIC"
+SingularityImage="/cvmfs/singularity.opensciencegrid.org/eicweb/jug_xl:nightly"
Requirements = HAS_SINGULARITY == TRUE && HAS_CVMFS_singularity_opensciencegrid_org == TRUE && OSG_HOST_KERNEL_VERSION >= 31000
```
  The templates in templates/osg_csv.submit.in are a good basis.
  
  # Productions
  
  ## Automated benchmarking
  - https://github.com/eic/simulation_campaign_datasets and https://eicweb.phy.anl.gov/EIC/campaigns/datasets
  Determines time per event to run datasets, produces csv input files
  
  ## Scaling up submission
  Usage:
  ./scripts/submit_csv.sh <template> <type> <file or url> [target hours = 2]
  
```
git clone https://github.com/eic/job_submission_condor condor
cd condor
./scripts/submit_csv.sh
```
  
  Example:
```
export EBEAM=18 PBEAM=275
export DETECTOR_VERSION=23.03.0 DETECTOR_CONFIG=epic_arches
./scripts/submit_csv.sh osg_csv hepmc3 DIS/NC/18x275/minQ2=1/DIS_NC_18x275_minQ2=1.csv 2
```
  
  ## Dealing with holds
  ./scripts/hold_review_and_release.sh
- 16:55 → 17:05
  
  Example of Job submission script with JLab SLURM 10m
  
  Speaker: Wenliang Li (Stony Brook University CFNS)
  
  code_tarball
  
  SLURM Example
- 17:05 → 17:25
  
  Recording 20m
  
  Session_2_1_recording