@@@@@ @ @ @ @ @ @ @@@ @ @@ @@@@ @ @ @ @@@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @@ @@@@@ @ @ @ @ @ @ @ @ @ @ @ @ @@ @ @ @ @@@@@ @@@ @ @@@@ @ @ @ @@@@ @ @ @ @ @ @ @@@@ @@@@ @ @@ @@@@ @ @ @@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @@ @ @ @ @ @@@@@ @ @@@@ @ @ @ @ @ @ @ @ @ @ @@@@ @ @@@@ @@ @@ @@@ @@ ************************************************************************* * _ _ _____ ____ * * | | | | / ____| |___ \ * * | |__| | | | __) | * * | __ | | | |__ < * * | | | | | |____ ___) | userguide hc3mpi * * |_| |_| \_____| |____/ * * * * Author: juergen.oehlschlaeger@kit.edu 25 July 2016 * ************************************************************************* Instruction, how to run parallel CORSIKA on the HP XC3000 at the KIT with parallelization using the MPI system (MPI after December 2012): path: corsika.trunk/ (i.e. main path of the corsika distribution) (1a) reset all environment optimization flags by typing . unsetoptflags.sh (1b) then run `./coconut` script to select options by ./coconut explanation: reset all environment optimization flags during compilation and linking by typing `. src/parallel/unsetoptflags.sh` ( - use command `env | grep FLAG` to display current environment variables - ); the application of this script avoids higher compiler optimizations of the MPI system for the parallel usage; after that run `./coconut` to create a CORSIKA executable with parallelization by the MPI system by selecting `p - PARALLEL treatment of subshowers` and secondly `2 - Library to be used with MPI system`; the new executable `mpi_corsika75...Linux_QGSII_gheisha_runner` is automatically moved to the subdirectory run/ of the main path of this corsika directory; then switch to `run/` path: corsika.trunk/run/ (or $WORK) (2) rename new executable to `mpi_corsika75098_stnd_QGSII4_gheisha_runner` to distinguish `stnd` or `thin` simulations and more possible options and models like `QGSII4`, `SIBYLL`, `atmext`, `augerhit`, `neutrino`, `stackin` or others by typing mv mpi_corsika75..._runner mpi_corsika75..._stnd_QGSII4_..._runner (3a) run program `pllselectecuts` to get parallelizing ecuts by typing ./pllselectecuts (3b) prepare the corsika steering file by typing vi parallel-001234 explanation: prepare the corsika steering file `parallel-001234` for the parallel run with the keyword PARALLEL by using the fortran utility program `acreinphc3.f`, i.e. `./acreinphc3`, or modify an existing one, i.e. `parallel-000333` which can be found in the run path; the best ratio for the first parameter (named ecutmin) of the keyword PARALLEL is 1/1000 of the second number (named ecutmax), which influences directly the total number of simulated parts of the complete corsika run; each processor should calculate more than 6 parts of this corsika run, use utility code `pllselectecuts.c` for energy-dependent examples (../src/parallel/) to check quantities; see appendix (A-3). (4a) edit mpi-submit-script by typing vi jobhc3-001234 (4b) and submit job to hc3-queue by ./jobhc3-001234 explanation: prepare the submit script `jobhc3-001234` with the organization of the (new) subdirectory csk001234/ by using the fortran utility program `acreinphc3.f`, i.e. `./acreinphc3`, or modify a copy of an existing one, i.e. `jobhc3-000333` which can be found in the run path; the last line of each `jobhc3-00....` is the complete submit command to the hc3 job queueing system, see appendix (A-4); (5a) switch to simulation path csk001234/ by cd csk001234/ (5b) run postprocessing script (for not-coreas simulations) by typing ./postprocessnew.sh (5c) switch back to working path by typing cd .. explanation: after the successfully finished parallel corsika simulation directory csk001234/ will contain a file `time.txt` of about 80 bytes length; next switch to this subdirectory and execute the postprocessing script `./postprocessnew.sh` which writes more infos to the job files `Job001234_*.[err,out]` and to the renamed time file `time.txt001234`; the last line of `Job001234_*.out` now contains the total amount of Gigabytes of all DAT001234-files as formatted number where the next to last line is the keyword `EXIT` from the steering file `parallel-001234`; see appendix (A-5); path: corsika.trunk/run/ (or $WORK) (6) now run script `./showparallel.sh` (no argument necessary) to create the new current tabular of all available parallel simulations in this working path; script has to be copied from src/parallel subdirectory including the fortran program totaltimenew.f; if you need the tabular of coreas simulations type ./showparallel.sh reas ************************************************************************* (A-3) parallel steering file ============================ RUNNR 1234 PARALLEL 40. 40000. 1 F NSHOW 1 EVTNR 1 SEED 2160 0 0 SEED 2161 0 0 SEED 2162 0 0 SEED 2163 0 0 SEED 2164 0 0 SEED 2165 0 0 PRMPAR 14 ERANGE 1.0000E+06 1.0000E+06 THETAP 21.09 21.09 PHIP -26.56 -26.56 OBSLEV 1452.e2 870.000 g/cm^2 MAGNET 19.51 -14.18 Auger MAXPRT 1 ECTMAP 1.E11 ECUTS 0.1000 0.1000 2.5e-4 2.5e-4 RADNKG 200.E2 HADFLG 0 0 0 0 0 2 ELMFLG T T MUADDI T MUMULT T STEPFC 1. HILOW 88. DIRECT csk001234/ HOST hc3.uni USER you EXIT RUNNR 4234 PARALLEL 4000. 4000000. 1 F ...... . . . PRMPAR 402 ERANGE 3.8283E+09 3.8283E+09 ...... . . . HILOW 100. DIRECT csk004234/ HOST hc3.uni USER you EXIT steering file creation ====================== #!/bin/bash # - - - - create steering and submit files: # - - - - compile and link fortran program acreinphc3.f: ifort -C -O0 -check bounds acreinphc3.f -o acreinphc3 # - - - - execute fortran program: ./acreinphc3 # - - - - make jobhc3-001234 file executable: chmod +x jobhc3-* ************************************************************************* (A-4) job submit script incl. path test ======================================= #!/bin/bash # if [ ! -e qgsdat-II-04 ] ; then echo " WARNING: no corsika submit path." exit; fi /bin/rm -rf csk001234/ /bin/mkdir csk001234/ tail -11 jobhc3-001234 /bin/cp jobhc3-001234 csk001234/ /bin/cp parallel-001234 csk001234/ /bin/cp summ* csk001234/ /bin/cp readmthin* csk001234/ /bin/cp readcsk2asci* csk001234/ /bin/cp sortaugerhit* csk001234/ /bin/cp totaltimenew* csk001234/ /bin/cp postprocess* csk001234/ job_submit -cp -p16 -t60 -m2000 -oJob001234_%j.out -eJob001234_%j.err \ mpirun mpi_corsika75098_stnd_QGSII4_gheisha_runner parallel-001234 ************************************************************************* (A-5a) time statistic file csk001234/time.txt001234 =================================================== START TIME STOP TIME TIME (min) 9112626.006491 9114041.460160 23.590894 LONGEST JOB: MPIID = 17 and Time = 707.726820 Total number of jobs = 207 Maximum size of group = 17 TOTAL CPU TIME (days) = 0.438553 time.txt001234 ************************************************************************* (A-5b) analysis script (optional) ================================= #!/bin/bash # # = = = = = job_submit -p1 -cp -t660 -m1000 showanalyhc3.sh001234 # # create file list and run `showanalyhc3` program (printer plots) # --------------------------------------------------------------- # ls -1 DAT001234* | grep t -v | grep n -v > showanalyhc3.i001234 # # gfortran -fbounds-check showanalyhc3.f -o showanalyhc3 # ifort -C -check bounds showanalyhc3.f -o showanalyhc3 # ./showanalyhc3 < showanalyhc3.i001234 > showanalyhc3.out001234 mv fort.9 showanalyhc3.fort001234 ************************************************************************* (A-6) show infos of all parallel simulations ============================================ #!/bin/bash # # create the tabular of available parallel corsika simulations: # ------------------------------------------------------------------------ # Primary lg(E) theta phi runtsk sizeGBy procs # T(days) ecutmax t(min) files RATIO obslev # Xmagn Zmagn _corsika_executable_ # Antns gamma _corsika_executable_ # < ecutha ecutmu ecutel ecutga thilev wmax lg(thirad) > # names of subdirectories csk0[0,7-9]????; # hc3 job protocols Job00????_%jobid.err, Job00????_%jobid.out; # gfortran -fbounds-check showparallel.f -o showparallel # ifort -C -O0 -check bounds showparallel.f -o showparallel # ------------------------------------------------------------------------ # usage: ./showparallel.sh [reas] # ------------------------------------------------------------------------ # juergen.oehlschlaeger@kit.edu # ------------------------------------------------------------------------ plltype="cors" plltext="" # - - - - - check argument of call `showparallel.sh`: if [ $# -eq 1 ] ; then plltype=`echo $1` plltext="-coreas" fi # - - - - - distinguish argument of showparallel.sh: if [ $plltype == "cors" ] ; then # - - - - - - no argument or argument `cors` (check run numbers): ls -1 csk0[0,1]*/Job*.out > showparallel.jobinfos ./showparallel < showparallel.jobinfos > showparallel.hc3-work-jobinfos else # - - - - - - argument `reas` for coreas simulations (check run numbers): if [ $plltype == "reas" ] ; then ls -1 csk0[1,7,8]????/Job*.out > showparallel.jobinfos ./showparallel < showparallel.jobinfos > showparallel.hc3-work-jobinfos$plltext fi fi *************************************************************************