Central campaign scripts (run your simulations like a pro)
------------------------
# Production simulation scripts
## Where are they? (github)
- https://github.com/eic/simulation_campaign_single:
run single particle simulations like central simulation productions
Installed in /opt/campaigns/single/
- https://github.com/eic/simulation_campaign_hepmc3
run physics event simulations like central simulation productions
Installed in /opt/campaigns/hepmc3/
- https://github.com/eic/job_submission_condor
run simulation productions through HTCondor systems
Installed in /opt/campaigns/condor/
- https://github.com/eic/job_submission_slurm
run simulation productions through Slurm systems
Installed in /opt/campaigns/slurm/
## What do they do?
Assumptions:
- input files are local, output files remain local
- jobs always run single-threaded (predictability)
- jobs are run in an isolated environment, i.e. a temporary directory
(modify $TMPDIR if you wish to write elsewhere)
Example files on S3:
- S3/eictest/EPIC/EVGEN/SINGLE/e-/1GeV/e-_1GeV_eta0.0.steer
- S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc
- S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
- S3/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
Can also run with s3http:// or root:// URLs as inputs for hepmc3 files:
- root://dtn-eic.jlab.org//work/eic2/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
- s3https://eics3.sdcc.bnl.gov:9000/eictest/EPIC/EVGEN/CI/pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc3.tree.root
## How to run them in eic-shell?
1. First copy the files from S3 to your local directory
2. Run the single or hepmc3 script with the file as argument:
run.sh <input> [n_chunk=10000] [i_chunk=]
Steering with exported environment variables (in future we may switch to command line arguments):
- COPYFULL=true # copy into calling directory
- COPYRECO=true # copy into calling directory
- COPYLOG=true # copy into calling directory
- EBEAM=5 # no default, follows geometry definition)
- PBEAM=41 # no default, follows geometry definition)
- DETECTOR_VERSION=23.03.0 # default nightly)
- DETECTOR_CONFIG=epic_brycecanyon # default epic)
## How to run them in a compute job (general)?
- Run these commands inside an explicit eic-shell session
cat << EOF | eic-shell
> /opt/campaigns/hepmc3/scripts/run.sh pythia8NCDIS_5x41_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1_20ev.hepmc.gz
> EOF
- Tell your job scheduling system to run inside a container, e.g. with HTCondor, or using `scrun` in Slurm (not at JLab).
# HTCondor
## HTCondor vs slurm
Advantages of HTCondor:
- HTCondor can download input files from S3 to the job node, and upload output files to S3 (Slurm requires pre-staging and post-uploading)
- HTCondor can run inside a container
- HTCondor (on OSG) has scaling to O(100k) nodes
Advantages of Slurm:
- Targets longer jobs (in typical queues) instead of rapid throughput
## HTCondor job definition file
Important statements:
```
+ProjectName="EIC"
+SingularityImage="/cvmfs/singularity.opensciencegrid.org/eicweb/jug_xl:nightly"
Requirements = HAS_SINGULARITY == TRUE && HAS_CVMFS_singularity_opensciencegrid_org == TRUE && OSG_HOST_KERNEL_VERSION >= 31000
```
The templates in templates/osg_csv.submit.in are a good basis.
# Productions
## Automated benchmarking
- https://github.com/eic/simulation_campaign_datasets and https://eicweb.phy.anl.gov/EIC/campaigns/datasets
Determines time per event to run datasets, produces csv input files
## Scaling up submission
Usage:
./scripts/submit_csv.sh <template> <type> <file or url> [target hours = 2]
```
git clone https://github.com/eic/job_submission_condor condor
cd condor
./scripts/submit_csv.sh
```
Example:
```
export EBEAM=18 PBEAM=275
export DETECTOR_VERSION=23.03.0 DETECTOR_CONFIG=epic_arches
./scripts/submit_csv.sh osg_csv hepmc3 DIS/NC/18x275/minQ2=1/DIS_NC_18x275_minQ2=1.csv 2
```
## Dealing with holds
./scripts/hold_review_and_release.sh