ePIC Tutorial Series 2: Running Simulations

Tuesday 14 Mar 2023, 14:00 → 17:30 US/Eastern

Zoom Link

One Pager Preparation Note

Recording

14:05 → 14:20 Instroducing Tutorial Session 2

Introduction

14:20 → 15:05 Eic-analysis Framework and Physics Variables Reconstruction

Speaker: Christopher Dilks (Duke University)

Github Repository

Tutorial (README)

15:05 → 15:50 Visualization: detector geometry and events

Speakers: Shyam Kumar (Universita e INFN, Bari (IT)), Shyam Kumar (University and INFN Bari)

Visualization_Tutorial_Shyam.pdf

15:50 → 16:05 Coffee Break

16:05 → 16:55 Simulation Campaign and HTCondor Example

Speaker: Wouter Deconinck (University of Manitoba)

Central campaign scripts (run your simulations like a pro)
------------------------

# Production simulation scripts

## Where are they? (github)
- https://github.com/eic/simulation_campaign_single:
  run single particle simulations like central simulation productions
  Installed in /opt/campaigns/single/
- https://github.com/eic/simulation_campaign_hepmc3
  run physics event simulations like central simulation productions
  Installed in /opt/campaigns/hepmc3/

- https://github.com/eic/job_submission_condor
  run simulation productions through HTCondor systems
  Installed in /opt/campaigns/condor/
- https://github.com/eic/job_submission_slurm
  run simulation productions through Slurm systems
  Installed in /opt/campaigns/slurm/

## What do they do?
Assumptions:
- input files are local, output files remain local
- jobs always run single-threaded (predictability)
- jobs are run in an isolated environment, i.e. a temporary directory
  (modify $TMPDIR if you wish to write elsewhere)

Example files on S3:
- S3/eictest/EPIC/EVGEN/SINGLE/e-/1GeV/e-_1GeV_eta0.0.steer
- S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc
- S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
- S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root

Can also run with s3http:// or root:// URLs as inputs for hepmc3 files:
- root://dtn-eic.jlab.org//work/eic2/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
- s3https://eics3.sdcc.bnl.gov:9000/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root

## How to run them in eic-shell?
1. First copy the files from S3 to your local directory
2. Run the single or hepmc3 script with the file as argument:
   run.sh <input> [n_chunk=10000] [i_chunk=]

Steering with exported environment variables (in future we may switch to command line arguments):
- COPYFULL=true # copy into calling directory
- COPYRECO=true # copy into calling directory
- COPYLOG=true # copy into calling directory
- EBEAM=5 # no default, follows geometry definition)
- PBEAM=41 # no default, follows geometry definition)
- DETECTOR_VERSION=23.03.0 # default nightly)
- DETECTOR_CONFIG=epic_brycecanyon # default epic)

## How to run them in a compute job (general)?
- Run these commands inside an explicit eic-shell session
cat << EOF | eic-shell
> /opt/campaigns/hepmc3/scripts/run.sh pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
> EOF

- Tell your job scheduling system to run inside a container, e.g. with HTCondor, or using `scrun` in Slurm (not at JLab).

# HTCondor

## HTCondor vs slurm
Advantages of HTCondor:
- HTCondor can download input files from S3 to the job node, and upload  output files to S3 (Slurm requires pre-staging and post-uploading)
- HTCondor can run inside a container
- HTCondor (on OSG) has scaling to O(100k) nodes

Advantages of Slurm:
- Targets longer jobs (in typical queues) instead of rapid throughput

## HTCondor job definition file
Important statements:
```
+ProjectName="EIC"
+SingularityImage="/cvmfs/singularity.opensciencegrid.org/eicweb/jug_xl:nightly"
Requirements = HAS_SINGULARITY == TRUE && HAS_CVMFS_singularity_opensciencegrid_org == TRUE && OSG_HOST_KERNEL_VERSION >= 31000
```
The templates in templates/osg_csv.submit.in are a good basis.

# Productions

## Automated benchmarking
- https://github.com/eic/simulation_campaign_datasets and https://eicweb.phy.anl.gov/EIC/campaigns/datasets
  Determines time per event to run datasets, produces csv input files

## Scaling up submission
Usage:
  ./scripts/submit_csv.sh <template> <type> <file or url> [target hours = 2]

```
git clone https://github.com/eic/job_submission_condor condor
cd condor
./scripts/submit_csv.sh
```

Example:
```
export EBEAM=18 PBEAM=275
export DETECTOR_VERSION=23.03.0 DETECTOR_CONFIG=epic_arches
./scripts/submit_csv.sh osg_csv hepmc3 DIS/NC/18x275/minQ2=1/DIS_NC_18x275_minQ2=1.csv 2
```

## Dealing with holds
./scripts/hold_review_and_release.sh

16:55 → 17:05 Example of Job submission script with JLab SLURM

Speaker: Wenliang Li (Stony Brook University CFNS)

code_tarball

SLURM Example

17:05 → 17:25 Recording

Session_2_1_recording