Research in Natural Hazards Engineering > Uncertainty Quantification (quoFEM)

Parallel execution on a Windows HPC

(1/5) > >>

rsam1993:
Dear all,

I am using QuoFEM on my personal computer (20 core and 40 logical processors - Intel Xeon 4210R 2.40GHz) and our server with a Windows HPC (64 core and 128 logical processors - AMD EPYC 7513 2.60GHz). When I use QuoFEM on my PC with 50 samples for forward propagation analysis, 100% of the CPU is active and occupied by QuoFEM, which makes sense. However, when I use the exact same model in QuoFEM on our server, but with 150 samples, I expect to see that 100% of the CPU is working, while the CPU utilization never goes beyond 65%. I was wondering if there is some type of limitation on the setting of our server or if there are any restrictions in QuoFEM parallel execution.

Thank you,

Sang-ri:
Hello!

To help us identify the reason why the CPU is not fully occupied, can you please follow the below steps?

1. In the server, find a file named "dakota.in" created by quoFEM. This should be in the folder where the working directories are located ("C:\Users\SimCenter\Documents\quoFEM\LocalWorkDir\tmp.SimCenter" in my machine)
2. open "dakota.in" using a text editor
3. Please let us know the number written after the keyword "asynchronous evaluation_concurrency"

Thank you,
Sang-ri

rsam1993:
Here is the info written in dakota.in file. It is interesting because the number is 64! Then why the CPU utilization is still around 60%?

interface
  analysis_driver = 'workflow_driver1.bat'
  fork
   parameters_file = 'paramsDakota.in'
   results_file = 'results.out'
   aprepro
   work_directory
     named 'workdir'
     directory_tag
     directory_save
     file_save
     copy_files = 'templatedir/*'
  asynchronous evaluation_concurrency = 64

Sang-ri:
Hi,

Thank you for the info. We think this number should be 128 instead of 64. While we figure out the solution, can you try the following workaround and let us know if this makes CPU occupied 100%?

1. Modify the number after "asynchronous evaluation_concurrency" in dakota.in from 64 to 128
2. Remove all files and folders in the local working directory except for "dakota.in" and "templatedir"
3. Find the path of the Dakota executable from the preference window of quoFEM. Let us denote this {dakota path}
4. Open the command prompt, cd into the folder where dakota.in is located, and type "{dakota path} dakota.in" (without the quotation marks)

It will run the forward propagation analysis, and the results will be shown in dakotaTab.out.

Thank you,
Sang-ri

rsam1993:
The procedure you described seems straightforward and I believe I did it right, but it does not work correctly. It only makes 128 workdir folders while I expect 150 (the analysis was done using 150 samples) and there are other errors regarding some required files which cannot be found when I run dakota.in through the command prompt.  Here is some part of the errors I got:


C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.127>python writeParam.py paramsDakota.in params.in
'python' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.127>call ./driver.bat

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.127>"C:/quoFEM_Windows_Download/applications/performUQ/templateSub/simCenterSub.exe" params.in SimCenterInput.RV SimCenterInput.tcl
ERROR: simCenterDprepro could not open: params.in

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.127>OpenSees SimCenterInput.tcl  1>ops.out 2>&1
nonblocking fork: workflow_driver1.bat paramsDakota.in results.out
Second pass: scheduling 22 remaining local asynchronous jobs
Waiting on completed jobs
Too many processes (128) in wait_setup
Current limit on processes = 64

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.128>python writeParam.py paramsDakota.in params.in
'python' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.128>call ./driver.bat

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.128>"C:/quoFEM_Windows_Download/applications/performUQ/templateSub/simCenterSub.exe" params.in SimCenterInput.RV SimCenterInput.tcl

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter>ERROR: simCenterDprepro could not open: params.in

C:\Users\rsamtaslimi\Documents\quoFEM\LocalWorkDir\tmp.SimCenter\workdir.128>OpenSees SimCenterInput.tcl  1>ops.out 2>&1

Navigation

[0] Message Index

[#] Next page