DAG job
From MediaWiki
(Difference between revisions)
(20 intermediate revisions not shown) | |||
Line 1: | Line 1: | ||
- | ''' | + | '''Submitting DAG job how-to''' |
This is DAG job submiting example. | This is DAG job submiting example. | ||
- | For demonstration use attached file. | + | For demonstration use attached file. In this case job is split in 4 independent jobs. First A job will be executed, then B and C jobs, and finally job D |
+ | |||
+ | http://wiki.ipb.ac.rs/images/4/46/Dag2.png | ||
- | [http://wiki.ipb.ac.rs/images/ | + | [http://wiki.ipb.ac.rs/images/2/21/DAG.tgz DAG.tgz] |
1.Extract file with: | 1.Extract file with: | ||
- | [ngrkic@ui ~]$ | + | [ngrkic@ui ~]$ tar xvzf DAG.tgz |
2.Enter directory single with: | 2.Enter directory single with: | ||
Line 21: | Line 23: | ||
-rw-r--r-- 1 ngrkic ngrkic 1469 Oct 1 15:54 jobDAG.jdl | -rw-r--r-- 1 ngrkic ngrkic 1469 Oct 1 15:54 jobDAG.jdl | ||
-rwxr-xr-x 1 ngrkic ngrkic 92 Oct 1 15:54 job.sh | -rwxr-xr-x 1 ngrkic ngrkic 92 Oct 1 15:54 job.sh | ||
+ | |||
+ | |||
+ | This is content of file jobDAG.jdl | ||
+ | |||
+ | [ | ||
+ | Type = "dag"; | ||
+ | InputSandbox = {"job.sh", "job2.sh"}; | ||
+ | |||
+ | Nodes = [ | ||
+ | nodeA = [ | ||
+ | Description = [ | ||
+ | JobType = "Normal"; | ||
+ | Executable = "job.sh"; | ||
+ | Arguments = "A"; | ||
+ | StdOutput = "std.out"; | ||
+ | StdError = "std.err"; | ||
+ | InputSandbox = {root.InputSandbox[0]}; | ||
+ | OutputSandbox = {"std.out","std.err"}; | ||
+ | ]; | ||
+ | ]; | ||
+ | nodeB = [ | ||
+ | Description = [ | ||
+ | JobType = "Normal"; | ||
+ | Executable = "job2.sh"; | ||
+ | Arguments = "B"; | ||
+ | StdOutput = "std.out"; | ||
+ | StdError = "std.err"; | ||
+ | InputSandbox = {root.InputSandbox[1]}; | ||
+ | OutputSandbox = {"std.out","std.err"}; | ||
+ | ]; | ||
+ | ]; | ||
+ | nodeC = [ | ||
+ | Description = [ | ||
+ | JobType = "Normal"; | ||
+ | Executable = "job3.sh"; | ||
+ | Arguments = "C"; | ||
+ | StdOutput = "std.out"; | ||
+ | StdError = "std.err"; | ||
+ | InputSandbox = {"job3.sh"}; | ||
+ | OutputSandbox = {"std.out","std.err"}; | ||
+ | ]; | ||
+ | ]; | ||
+ | nodeD = [ | ||
+ | Description = [ | ||
+ | JobType = "Normal"; | ||
+ | Executable = "job.sh"; | ||
+ | Arguments = "D"; | ||
+ | StdOutput = "std.out"; | ||
+ | StdError = "std.err"; | ||
+ | InputSandbox = {root.InputSandbox[0]}; | ||
+ | OutputSandbox = {"std.out","std.err"}; | ||
+ | ]; | ||
+ | ]; | ||
+ | ]; | ||
+ | Dependencies = { | ||
+ | {nodeA,nodeB},{nodeA,nodeC},{{nodeB,nodeC},nodeD} | ||
+ | }; | ||
+ | ] | ||
+ | |||
+ | This is content of file job.sh | ||
+ | |||
+ | #!/bin/bash | ||
+ | |||
+ | echo "Job $1 - `date` - BEGIN" | ||
+ | hostname | ||
+ | sleep 100 | ||
+ | echo "Job $1 - `date` - END" | ||
+ | |||
+ | |||
4.Creating VOMS proxy: | 4.Creating VOMS proxy: | ||
Line 34: | Line 105: | ||
Your proxy is valid until Fri Oct 2 01:11:00 2009 | Your proxy is valid until Fri Oct 2 01:11:00 2009 | ||
- | |||
- | |||
Line 42: | Line 111: | ||
[ngrkic@ui DAG]$ glite-wms-job-submit -a jobDAG.jdl | [ngrkic@ui DAG]$ glite-wms-job-submit -a jobDAG.jdl | ||
- | Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server | + | Connecting to the service <nowiki>https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server</nowiki> |
Line 50: | Line 119: | ||
Your job identifier is: | Your job identifier is: | ||
- | https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ | + | <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki> |
========================================================================== | ========================================================================== | ||
Line 60: | Line 129: | ||
6.Requesting Job status: | 6.Requesting Job status: | ||
- | [ngrkic@ui DAG]$ glite-wms-job-status https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ | + | [ngrkic@ui DAG]$ glite-wms-job-status <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki> |
Line 66: | Line 135: | ||
BOOKKEEPING INFORMATION: | BOOKKEEPING INFORMATION: | ||
- | Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ | + | Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki> |
Current Status: Done (Success) | Current Status: Done (Success) | ||
Exit code: 0 | Exit code: 0 | ||
Line 75: | Line 144: | ||
- Nodes information for: | - Nodes information for: | ||
- | Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/4xL7Bu5cI6eaixg9y0oezg | + | Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/4xL7Bu5cI6eaixg9y0oezg</nowiki> |
Current Status: Done (Success) | Current Status: Done (Success) | ||
Logged Reason(s): | Logged Reason(s): | ||
Line 86: | Line 155: | ||
************************************************************* | ************************************************************* | ||
- | Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/LCE9ULBND3jYd2JUJPJ9wQ | + | Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/LCE9ULBND3jYd2JUJPJ9wQ</nowiki> |
Current Status: Done (Success) | Current Status: Done (Success) | ||
Logged Reason(s): | Logged Reason(s): | ||
Line 97: | Line 166: | ||
************************************************************* | ************************************************************* | ||
- | Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/quIESEluISRrAXdcC7ELKQ | + | Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/quIESEluISRrAXdcC7ELKQ</nowiki> |
Current Status: Done (Success) | Current Status: Done (Success) | ||
Exit code: 0 | Exit code: 0 | ||
Line 105: | Line 174: | ||
************************************************************* | ************************************************************* | ||
- | Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/yNu3yqhSwf4sZ25bWuzKDg | + | Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/yNu3yqhSwf4sZ25bWuzKDg</nowiki> |
Current Status: Done (Success) | Current Status: Done (Success) | ||
Exit code: 0 | Exit code: 0 | ||
Line 115: | Line 184: | ||
7.Requesting Job output: | 7.Requesting Job output: | ||
- | [ngrkic@ui DAG]$ glite-wms-job-output --dir /home/ngrkic/test https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ | + | [ngrkic@ui DAG]$ glite-wms-job-output --dir /home/ngrkic/test <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki> |
- | Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server | + | Connecting to the service <nowiki>https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server</nowiki> |
================================================================================ | ================================================================================ | ||
Line 124: | Line 193: | ||
Output sandbox files for the DAG/Collection : | Output sandbox files for the DAG/Collection : | ||
- | https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ | + | <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki> |
have been successfully retrieved and stored in the directory: | have been successfully retrieved and stored in the directory: | ||
/home/ngrkic/test | /home/ngrkic/test | ||
Line 137: | Line 206: | ||
[ngrkic@ui test]$ ll | [ngrkic@ui test]$ ll | ||
- | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17: | + | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeA |
- | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17: | + | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeB |
- | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17: | + | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeC |
- | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17: | + | drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeD |
+ | |||
+ | [ngrkic@ui test]$ cd nodeA | ||
+ | [ngrkic@ui nodeA]$ ll | ||
+ | -rw-rw-r-- 1 ngrkic ngrkic 0 Oct 12 17:31 std.err | ||
+ | -rw-rw-r-- 1 ngrkic ngrkic 112 Oct 12 17:31 std.out | ||
+ | |||
+ | [ngrkic@ui nodeA]$ cat std.out | ||
+ | Job A - Thu Oct 1 17:14:14 CEST 2009 - BEGIN | ||
+ | grid10.elfak.ni.ac.rs | ||
+ | Job A - Thu Oct 1 17:14:54 CEST 2009 - END | ||
- | + | For each node there is similar output. |
Latest revision as of 15:34, 27 May 2012
Submitting DAG job how-to
This is DAG job submiting example.
For demonstration use attached file. In this case job is split in 4 independent jobs. First A job will be executed, then B and C jobs, and finally job D
1.Extract file with:
[ngrkic@ui ~]$ tar xvzf DAG.tgz
2.Enter directory single with:
[ngrkic@ui ~]$ cd DAG
3.List directory:
[ngrkic@ui DAG]$ ll -rw-r--r-- 1 ngrkic ngrkic 1469 Oct 1 15:54 jobDAG.jdl -rwxr-xr-x 1 ngrkic ngrkic 92 Oct 1 15:54 job.sh
This is content of file jobDAG.jdl
[ Type = "dag"; InputSandbox = {"job.sh", "job2.sh"}; Nodes = [ nodeA = [ Description = [ JobType = "Normal"; Executable = "job.sh"; Arguments = "A"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {root.InputSandbox[0]}; OutputSandbox = {"std.out","std.err"}; ]; ]; nodeB = [ Description = [ JobType = "Normal"; Executable = "job2.sh"; Arguments = "B"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {root.InputSandbox[1]}; OutputSandbox = {"std.out","std.err"}; ]; ]; nodeC = [ Description = [ JobType = "Normal"; Executable = "job3.sh"; Arguments = "C"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"job3.sh"}; OutputSandbox = {"std.out","std.err"}; ]; ]; nodeD = [ Description = [ JobType = "Normal"; Executable = "job.sh"; Arguments = "D"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {root.InputSandbox[0]}; OutputSandbox = {"std.out","std.err"}; ]; ]; ]; Dependencies = { {nodeA,nodeB},{nodeA,nodeC},{{nodeB,nodeC},nodeD} }; ]
This is content of file job.sh
#!/bin/bash echo "Job $1 - `date` - BEGIN" hostname sleep 100 echo "Job $1 - `date` - END"
4.Creating VOMS proxy:
[ngrkic@ui DAG]$ voms-proxy-init -voms aegis
Cannot find file or dir: /home/ngrkic/.glite/vomses Enter GRID pass phrase: Your identity: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Nikola Grkic Creating temporary proxy ......................... Done Contacting voms.ipb.ac.rs:15001 [/C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=host/voms.ipb.ac.rs] "aegis" Done Creating proxy ..................................................... Done Your proxy is valid until Fri Oct 2 01:11:00 2009
5.Submiting DAG job:
[ngrkic@ui DAG]$ glite-wms-job-submit -a jobDAG.jdl
Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server ====================== glite-wms-job-submit Success ====================== The job has been successfully submitted to the WMProxy Your job identifier is: https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ ==========================================================================
Copy the job ID.Job is running now, and it should finish in few moments...
6.Requesting Job status:
[ngrkic@ui DAG]$ glite-wms-job-status https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ ************************************************************* BOOKKEEPING INFORMATION: Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: dagman Submitted: Thu Oct 1 16:44:32 2009 CEST ************************************************************* - Nodes information for: Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/4xL7Bu5cI6eaixg9y0oezg Current Status: Done (Success) Logged Reason(s): - - Job terminated successfully Exit code: 0 Status Reason: Job terminated successfully Destination: grid-ce.etf.bg.ac.rs:2119/jobmanager-pbs-aegis Submitted: Thu Oct 1 16:44:32 2009 CEST ************************************************************* Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/LCE9ULBND3jYd2JUJPJ9wQ Current Status: Done (Success) Logged Reason(s): - - Job terminated successfully Exit code: 0 Status Reason: Job terminated successfully Destination: grid01.rcub.bg.ac.rs:2119/jobmanager-pbs-aegis Submitted: Thu Oct 1 16:44:32 2009 CEST ************************************************************* Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/quIESEluISRrAXdcC7ELKQ Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: grid01.elfak.ni.ac.rs:2119/jobmanager-pbs-aegis Submitted: Thu Oct 1 16:44:32 2009 CEST ************************************************************* Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/yNu3yqhSwf4sZ25bWuzKDg Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully Destination: grid01.elfak.ni.ac.rs:2119/jobmanager-pbs-aegis Submitted: Thu Oct 1 16:44:32 2009 CEST *************************************************************
7.Requesting Job output:
[ngrkic@ui DAG]$ glite-wms-job-output --dir /home/ngrkic/test https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server ================================================================================ JOB GET OUTPUT OUTCOME Output sandbox files for the DAG/Collection : https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ have been successfully retrieved and stored in the directory: /home/ngrkic/test ================================================================================
8.Go to test directory and see output with commands:
[ngrkic@ui ~]$ cd [ngrkic@ui ~]$ cd test [ngrkic@ui test]$ ll drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeA drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeB drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeC drwxr-xr-x 2 ngrkic ngrkic 4096 Oct 1 17:31 nodeD
[ngrkic@ui test]$ cd nodeA [ngrkic@ui nodeA]$ ll -rw-rw-r-- 1 ngrkic ngrkic 0 Oct 12 17:31 std.err -rw-rw-r-- 1 ngrkic ngrkic 112 Oct 12 17:31 std.out [ngrkic@ui nodeA]$ cat std.out Job A - Thu Oct 1 17:14:14 CEST 2009 - BEGIN grid10.elfak.ni.ac.rs Job A - Thu Oct 1 17:14:54 CEST 2009 - END
For each node there is similar output.