@@@@@ @ @ @ @ @ @ @@@ @ @@ @@@@ @ @ @ @@@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @@ @@@@@ @ @ @ @ @ @ @ @ @ @ @ @ @@ @ @ @ @@@@@ @@@ @ @@@@ @ @ @ @@@@ @ @ @ @ @ @ @@@@ @@@@ @ @@ @@@@ @ @ @@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @@ @ @ @ @ @@@@@ @ @@@@ @ @ @ @ @ @ @ @ @ @ @@@@ @ @@@@ @@ @@ @@@ @@ ************************************************************************* * * * w w w wwwwww wwwww w * * w w w w w w w w * * w w w w w w wwww wwwww w w * * wwwwwww w wwwwww wwwww w w w w * * w w w w w w wwwww w w w * * w w w w w w w w w w w * * w w wwwwwww w w wwwww wwww w www * * * * Author: juergen.oehlschlaeger@kit.edu 12 June 2014 * ************************************************************************* Instruction, how to run parallel CORSIKA on the hermit1.hww.de at HLRS Stuttgart with parallelization using the MPI system: path: corsika.conex/ (i.e. main path of a corsika distribution) (same as on the hc3 parallel processor system). (1) reset all environment optimization flags during compilation and linking by typing `. src/parallel/unsetoptflags.sh` ( - use command `env | grep FLAG` to display current environment variables - ); it avoids higher compiler optimization of the MPI system for the parallel application; then run ./coconut to create a CORSIKA executable with parallelization by the MPI system by selecting `p - PARALLEL treatment of subshowers` and then `2 - Library to be used with MPI system`; `mpi_corsika74000Linux_QGSII_gheisha_runner` is automatically moved to the subdirectory run/ of the main path of the corsika distribution; then switch to path: corsika.trunk/run/ (or also $WORK) (2) rename the executable to `mpi_corsika74056_stnd_QGSII4_gheisha_runner` to distinguish `standard` or `thinning` simulations, and some more possible options like `history`, `augerhit`, `stackin` and others. (3) prepare the corsika steering file `parallel-001234` for the parallel run with the keyword PARALLEL by using the fortran utility program `acreinphc3.f`, i.e. `./acreinphc3`, or modify (a copy of) an existing one for new purposes; the keyword CUTFILE must not be used and is also invalid in a regular parallel simulation; example see appendix (A-3). (4) prepare the submit script `jobhc3-001234` with the organization of the (new) subdirectory csk001234/ by using the fortran utility program `acreinphc3.f`, i.e. `./acreinphc3`, or modify a copy of an existing one to the new run number; the last line of `jobhc3-001234` is the complete submit command to the hc3 job queueing system, see appendix (A-4); to submit it type: ./jobhc3-001234 ; (5) after the successful parallel corsika simulation the subdirectory csk001234/ will contain a file `time.txt` of about 90 bytes length; next switch to this subdirectory and execute the postprocessing script `./postprocessnew.sh` ( - until 2012 `./postprocess.sh` whereby the summation of `.long` files was not automatically performed - ); `postprocessnew.sh` writes additional infos to the job protocol files `Job001234_*.[err,out]` and to the new extended file `time.txt001234`; the last line of `Job001234_*.out` now contains the total amount of Gigabytes of all DAT001234-files as formatted number where the next to last line is the word `EXIT` from the steering file `parallel-001234`; see appendix (A-5); (6) optionally run the script `showanalyhc3.sh` after a complete standard (not thinned) corsika simulation; see appendix (A-6); switch back to path: corsika.trunk/run/ (or $WORK) (7) next run script `showparallel.sh` (no argument necessary) to create the new current tabular of all available parallel simulations in this working pathr, see appendix (A-7). ************************************************************************* (A-3) parallel steering file ============================ See also CORSIKA user guide. RUNNR 1234 PARALLEL 3000. 3000000. 1 F NSHOW 1 EVTNR 1 SEED 2160 0 0 SEED 2161 0 0 SEED 2162 0 0 SEED 2163 0 0 SEED 2164 0 0 SEED 2165 0 0 PRMPAR 14 ERANGE 3.8283E+09 3.8283E+09 THETAP 23.45 23.45 PHIP 26.56 26.56 OBSLEV 1452.e2 870.000 g/cm^2 MAGNET 19.51 -14.18 Auger MAXPRT 1 ECTMAP 1.E11 ECUTS 0.1000 0.1000 2.5e-4 2.5e-4 RADNKG 200.E2 HADFLG 0 0 0 0 0 2 ELMFLG T T QGSJET T 0 QGSSIG T MUADDI T MUMULT T STEPFC 1. * AUGSCT 20 55. 200. ! data reduction to 2% * AUGSCT 20 55. 300. ! data reduction to 1% HILOW 88. DIRECT csk001234/ HOST stuttgart USER you EXIT steering file creation ====================== #!/bin/bash # - - - - compile and link fortran program: ifort -C -check bounds acreinphc3.f -o acreinphc3 # - - - - execute fortran program: ./acreinphc3 # - - - - make jobhc3-001234 files executable: chmod +x jobhc3-* ************************************************************************* (A-4) job submit script at hermit.hww.de ======================================== #!/bin/bash #PBS -l mppwidth=32 #PBS -l mppnppn=32 #PBS -l walltime=00:10:00 # echo $WORK cd $WORK/corsika-74000/run/ if [ ! -e csk001234/ ] ; then /bin/mkdir csk001234/ else /bin/rm -f csk001234/* fi /bin/cp jobpll-001234 csk001234/ /bin/cp parallel-001234 csk001234/ /bin/cp summ* csk001234/ /bin/cp sortaugerhit* csk001234/ /bin/cp showanalyhc3* csk001234/ /bin/cp totaltimenew* csk001234/ /bin/cp postprocessnew* csk001234/ /bin/cp readcsk* csk001234/ # aprun -B mpi_corsika74045_stnd_QGSII4_gheisha_runner parallel-001234 ************************************************************************* (A-5) time statistic file ========================= START TIME STOP TIME TIME (min) 1359112626.006491 1359114041.460160 23.590894 LONGEST JOB: MPIID = 17 and Time = 707.726820 Total number of jobs = 207 Maximum size of group = 17 TOTAL CPU TIME (days) = 0.438553 time.txt001234 ************************************************************************* (A-6) analysis script ===================== #!/bin/bash # # = = = = = job_submit -p1 -cp -t660 -m1000 showanalyhc3.sh001234 # # create file list and run `showanalyhc3` program: # --------------------------------------------------------------- # ls -1 DAT001234* | grep t -v | grep n -v > showanalyhc3.i001234 # # names of sub paths csk00????; # gfortran -fbounds-check showanalyhc3.f -o showanalyhc3 # f77 -fbounds-check showanalyhc3.f -o showanalyhc3 # ifort -C -check bounds showanalyhc3.f -o showanalyhc3 # ./showanalyhc3 < showanalyhc3.i001234 > showanalyhc3.out001234 mv fort.9 showanalyhc3.fort001234 ************************************************************************* (A-7) show infos of all parallel simulations ============================================ #!/bin/bash # # create the tabular of available parallel corsika simulations: # ------------------------------------------------------------------------ # Primary lg(E) theta phi runtsk sizeGBy procs # T(days) ecutmax t(min) files RATIO obslev Xmagn Zmagn # _corsika_executable_ # < ecutha ecutmu ecutel ecutga thilev wmax lg(thirad) > # ------------------------------------------------------------------------ # usage: ./showparallel.sh # ------------------------------------------------------------------------ # juergen.oehlschlaeger@kit.edu # ------------------------------------------------------------------------ # ls -1 csk00*/Job00*.out > showparallel.jobinfos # # names of subdirectories csk00????; # hc3 job protocols Job00????_%jobid.err, Job00????_%jobid.out; # gfortran -fbounds-check showparallel.f -o showparallel # ifort -C -check bounds showparallel.f -o showparallel # ./showparallel < showparallel.jobinfos > showparallel.hc3-work-jobinfos *************************************************************************