Case Timing Data
Overview
CIME uses the General Purpose Timing Library (GPTL) to provide detailed timing information.
Timing files are output to $CASEROOT/timing/$model_timing.$CASE.$datestamp
.
The two areas we plan to focus on to increase the model performance is to
understand the bottlenecks occuring during the Init Time
phase of each job.
Does init time take longer for first job in hybrid run?
No. I suspected that the init time might take longer for the first submission of a given integration since there may be initialization tasks involved for a hybrid run in which continue_run = FALSE but this seems to be fals.
For the first submission of FHIST_BGC.f09_d025.052.e3 Init Time : 539.420
seconds
while in the second submission Init Time: 562.777 seconds
.
Major Issue: Init Time
It’s unclear how to disassemble the Init Time
statistic to determine what
is causing the initialization to be so slow.
$ cd /lustre/project/k1421/cesm2_1_3/cime
$ grep -Rl "Init Time" ./
./doc/source/users_guide/timers.rst
./tools/load_balancing_tool/tests/timing/timing_2
./tools/load_balancing_tool/tests/timing/timing_3
./tools/load_balancing_tool/tests/timing/timing_1
./scripts/lib/CIME/get_timing.py
It looks like get_timing.py
is the best candidate to look through for the
relevant code:
nmax = self.gettime(' CPL:INIT ')[1]
...
self.write(" Init Time : {:10.3f} seconds \n".format(nmax))
Previously Encountered Problem
In
_create_component_modelio_namelists
Note
This is the file to look in to see what is taking so long. /lustre/scratch/x_johnsobk/FHIST_BGC.f09_d025.095.e500/run/cpl_0229.log.17860564.201220-165938.gz