DAG job

From MediaWiki

(Difference between revisions)
Jump to: navigation, search
 
(20 intermediate revisions not shown)
Line 1: Line 1:
-
'''Submiting DAG job how to'''
+
'''Submitting DAG job how-to'''
This is DAG job submiting example.
This is DAG job submiting example.
-
For demonstration use attached file.
+
For demonstration use attached file. In this case job is split in 4 independent jobs. First A job will be executed, then B and C jobs, and finally job D
 +
 +
http://wiki.ipb.ac.rs/images/4/46/Dag2.png
-
[http://wiki.ipb.ac.rs/images/7/71/DAG.zip DAG.zip]
+
[http://wiki.ipb.ac.rs/images/2/21/DAG.tgz DAG.tgz]
1.Extract file with:
1.Extract file with:
   
   
-
  [ngrkic@ui ~]$ unzip DAG.zip
+
  [ngrkic@ui ~]$ tar xvzf DAG.tgz
2.Enter directory single with:
2.Enter directory single with:
Line 21: Line 23:
  -rw-r--r--  1 ngrkic ngrkic 1469 Oct  1 15:54 jobDAG.jdl
  -rw-r--r--  1 ngrkic ngrkic 1469 Oct  1 15:54 jobDAG.jdl
  -rwxr-xr-x  1 ngrkic ngrkic  92 Oct  1 15:54 job.sh
  -rwxr-xr-x  1 ngrkic ngrkic  92 Oct  1 15:54 job.sh
 +
 +
 +
This is content of file jobDAG.jdl
 +
 +
[
 +
Type = "dag";
 +
InputSandbox = {"job.sh", "job2.sh"};
 +
 +
Nodes = [
 +
nodeA = [
 +
          Description = [
 +
            JobType = "Normal";
 +
            Executable = "job.sh";
 +
            Arguments = "A";
 +
            StdOutput = "std.out";
 +
            StdError = "std.err";
 +
            InputSandbox = {root.InputSandbox[0]};
 +
            OutputSandbox = {"std.out","std.err"};
 +
          ];
 +
        ];
 +
nodeB = [
 +
          Description = [
 +
            JobType = "Normal";
 +
            Executable = "job2.sh";
 +
            Arguments = "B";
 +
            StdOutput = "std.out";
 +
            StdError = "std.err";
 +
            InputSandbox = {root.InputSandbox[1]};
 +
            OutputSandbox = {"std.out","std.err"};
 +
          ];
 +
        ];
 +
nodeC = [
 +
          Description = [
 +
            JobType = "Normal";
 +
            Executable = "job3.sh";
 +
            Arguments = "C";
 +
            StdOutput = "std.out";
 +
            StdError = "std.err";
 +
            InputSandbox = {"job3.sh"};
 +
            OutputSandbox = {"std.out","std.err"};
 +
          ];
 +
        ];
 +
nodeD = [
 +
          Description = [
 +
            JobType = "Normal";
 +
            Executable = "job.sh";
 +
            Arguments = "D";
 +
            StdOutput = "std.out";
 +
            StdError = "std.err";
 +
            InputSandbox = {root.InputSandbox[0]};
 +
            OutputSandbox = {"std.out","std.err"};
 +
          ];
 +
        ];
 +
];
 +
Dependencies = {
 +
                {nodeA,nodeB},{nodeA,nodeC},{{nodeB,nodeC},nodeD}
 +
};
 +
]
 +
 +
This is content of file job.sh
 +
 +
#!/bin/bash
 +
 +
echo "Job $1 - `date` - BEGIN"
 +
hostname
 +
sleep 100
 +
echo "Job $1 - `date` - END"
 +
 +
4.Creating VOMS proxy:
4.Creating VOMS proxy:
Line 34: Line 105:
  Your proxy is valid until Fri Oct  2 01:11:00 2009
  Your proxy is valid until Fri Oct  2 01:11:00 2009
-
 
-
Now we are ready to submit job.
 
Line 42: Line 111:
  [ngrkic@ui DAG]$ glite-wms-job-submit -a jobDAG.jdl  
  [ngrkic@ui DAG]$ glite-wms-job-submit -a jobDAG.jdl  
-
  Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server
+
  Connecting to the service <nowiki>https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server</nowiki>
   
   
   
   
Line 50: Line 119:
  Your job identifier is:
  Your job identifier is:
   
   
-
  https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
+
  <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki>
   
   
  ==========================================================================
  ==========================================================================
Line 60: Line 129:
6.Requesting Job status:     
6.Requesting Job status:     
-
  [ngrkic@ui DAG]$ glite-wms-job-status https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
+
  [ngrkic@ui DAG]$ glite-wms-job-status <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki>
   
   
   
   
Line 66: Line 135:
  BOOKKEEPING INFORMATION:
  BOOKKEEPING INFORMATION:
   
   
-
  Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
+
  Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki>
  Current Status:    Done (Success)
  Current Status:    Done (Success)
  Exit code:          0
  Exit code:          0
Line 75: Line 144:
   
   
  - Nodes information for:  
  - Nodes information for:  
-
     Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/4xL7Bu5cI6eaixg9y0oezg
+
     Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/4xL7Bu5cI6eaixg9y0oezg</nowiki>
     Current Status:    Done (Success)
     Current Status:    Done (Success)
     Logged Reason(s):
     Logged Reason(s):
Line 86: Line 155:
  *************************************************************
  *************************************************************
      
      
-
     Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/LCE9ULBND3jYd2JUJPJ9wQ
+
     Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/LCE9ULBND3jYd2JUJPJ9wQ</nowiki>
     Current Status:    Done (Success)
     Current Status:    Done (Success)
     Logged Reason(s):
     Logged Reason(s):
Line 97: Line 166:
  *************************************************************
  *************************************************************
      
      
-
     Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/quIESEluISRrAXdcC7ELKQ
+
     Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/quIESEluISRrAXdcC7ELKQ</nowiki>
     Current Status:    Done (Success)
     Current Status:    Done (Success)
     Exit code:          0
     Exit code:          0
Line 105: Line 174:
  *************************************************************
  *************************************************************
      
      
-
     Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/yNu3yqhSwf4sZ25bWuzKDg
+
     Status info for the Job : <nowiki>https://wms-aegis.ipb.ac.rs:9000/yNu3yqhSwf4sZ25bWuzKDg</nowiki>
     Current Status:    Done (Success)
     Current Status:    Done (Success)
     Exit code:          0
     Exit code:          0
Line 115: Line 184:
7.Requesting Job output:
7.Requesting Job output:
-
  [ngrkic@ui DAG]$ glite-wms-job-output --dir /home/ngrkic/test https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
+
  [ngrkic@ui DAG]$ glite-wms-job-output --dir /home/ngrkic/test <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki>
-
  Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server
+
  Connecting to the service <nowiki>https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server</nowiki>
   
   
  ================================================================================
  ================================================================================
Line 124: Line 193:
   
   
  Output sandbox files for the DAG/Collection :
  Output sandbox files for the DAG/Collection :
-
  https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
+
  <nowiki>https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ</nowiki>
  have been successfully retrieved and stored in the directory:
  have been successfully retrieved and stored in the directory:
  /home/ngrkic/test
  /home/ngrkic/test
Line 137: Line 206:
  [ngrkic@ui test]$ ll
  [ngrkic@ui test]$ ll
   
   
-
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:13 nodeA
+
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeA
-
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:13 nodeB
+
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeB
-
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:13 nodeC
+
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeC
-
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:13 nodeD
+
  drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeD
 +
 
 +
[ngrkic@ui test]$ cd nodeA
 +
[ngrkic@ui nodeA]$ ll
 +
-rw-rw-r--  1 ngrkic ngrkic  0 Oct 12 17:31 std.err
 +
-rw-rw-r--  1 ngrkic ngrkic 112 Oct 12 17:31 std.out
 +
 +
[ngrkic@ui nodeA]$ cat std.out
 +
Job A - Thu Oct 1 17:14:14 CEST 2009 - BEGIN
 +
grid10.elfak.ni.ac.rs
 +
Job A - Thu Oct 1 17:14:54 CEST 2009 - END
-
Enter directory and list output file with "cat" command.
+
For each node there is similar output.

Latest revision as of 15:34, 27 May 2012

Submitting DAG job how-to


This is DAG job submiting example. For demonstration use attached file. In this case job is split in 4 independent jobs. First A job will be executed, then B and C jobs, and finally job D

Dag2.png

DAG.tgz

1.Extract file with:

[ngrkic@ui ~]$ tar xvzf DAG.tgz

2.Enter directory single with:

[ngrkic@ui ~]$ cd DAG

3.List directory:

[ngrkic@ui DAG]$ ll

-rw-r--r--  1 ngrkic ngrkic 1469 Oct  1 15:54 jobDAG.jdl
-rwxr-xr-x  1 ngrkic ngrkic   92 Oct  1 15:54 job.sh

This is content of file jobDAG.jdl

[
Type = "dag";
InputSandbox = {"job.sh", "job2.sh"};

Nodes = [
nodeA = [
          Description = [
           JobType = "Normal";
           Executable = "job.sh";
           Arguments = "A";
           StdOutput = "std.out";
           StdError = "std.err";
           InputSandbox = {root.InputSandbox[0]};
           OutputSandbox = {"std.out","std.err"};
          ];
        ];
nodeB = [
          Description = [
           JobType = "Normal";
           Executable = "job2.sh";
           Arguments = "B";
           StdOutput = "std.out";
           StdError = "std.err";
           InputSandbox = {root.InputSandbox[1]};
           OutputSandbox = {"std.out","std.err"};
          ];
        ];
nodeC = [
          Description = [
           JobType = "Normal";
           Executable = "job3.sh";
           Arguments = "C";
           StdOutput = "std.out";
           StdError = "std.err";
           InputSandbox = {"job3.sh"};
           OutputSandbox = {"std.out","std.err"};
          ];
        ];
nodeD = [
          Description = [
           JobType = "Normal";
           Executable = "job.sh";
           Arguments = "D";
           StdOutput = "std.out";
           StdError = "std.err";
           InputSandbox = {root.InputSandbox[0]};
           OutputSandbox = {"std.out","std.err"};
          ];
        ];
];
Dependencies = {
               {nodeA,nodeB},{nodeA,nodeC},{{nodeB,nodeC},nodeD}
};
]

This is content of file job.sh

#!/bin/bash 

echo "Job $1 - `date` - BEGIN"
hostname
sleep 100
echo "Job $1 - `date` - END"


4.Creating VOMS proxy:

[ngrkic@ui DAG]$ voms-proxy-init -voms aegis
Cannot find file or dir: /home/ngrkic/.glite/vomses
Enter GRID pass phrase:
Your identity: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Nikola Grkic
Creating temporary proxy ......................... Done
Contacting  voms.ipb.ac.rs:15001 [/C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=host/voms.ipb.ac.rs] "aegis" Done
Creating proxy ..................................................... Done
Your proxy is valid until Fri Oct  2 01:11:00 2009


5.Submiting DAG job:

[ngrkic@ui DAG]$ glite-wms-job-submit -a jobDAG.jdl 
Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ

==========================================================================


Copy the job ID.Job is running now, and it should finish in few moments...


6.Requesting Job status:

[ngrkic@ui DAG]$ glite-wms-job-status https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ


*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
Current Status:     Done (Success)
Exit code:          0
Status Reason:      Job terminated successfully
Destination:        dagman
Submitted:          Thu Oct  1 16:44:32 2009 CEST
*************************************************************

- Nodes information for: 
    Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/4xL7Bu5cI6eaixg9y0oezg
    Current Status:     Done (Success)
    Logged Reason(s):
        - 
        - Job terminated successfully
    Exit code:          0
    Status Reason:      Job terminated successfully
    Destination:        grid-ce.etf.bg.ac.rs:2119/jobmanager-pbs-aegis
    Submitted:          Thu Oct  1 16:44:32 2009 CEST
*************************************************************
   
    Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/LCE9ULBND3jYd2JUJPJ9wQ
    Current Status:     Done (Success)
    Logged Reason(s):
        - 
        - Job terminated successfully
    Exit code:          0
    Status Reason:      Job terminated successfully
    Destination:        grid01.rcub.bg.ac.rs:2119/jobmanager-pbs-aegis
    Submitted:          Thu Oct  1 16:44:32 2009 CEST
*************************************************************
    
    Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/quIESEluISRrAXdcC7ELKQ
    Current Status:     Done (Success)
    Exit code:          0
    Status Reason:      Job terminated successfully
    Destination:        grid01.elfak.ni.ac.rs:2119/jobmanager-pbs-aegis
    Submitted:          Thu Oct  1 16:44:32 2009 CEST
*************************************************************
    
    Status info for the Job : https://wms-aegis.ipb.ac.rs:9000/yNu3yqhSwf4sZ25bWuzKDg
    Current Status:     Done (Success)
    Exit code:          0
    Status Reason:      Job terminated successfully
    Destination:        grid01.elfak.ni.ac.rs:2119/jobmanager-pbs-aegis
    Submitted:          Thu Oct  1 16:44:32 2009 CEST
*************************************************************

7.Requesting Job output:

[ngrkic@ui DAG]$ glite-wms-job-output --dir /home/ngrkic/test https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
Connecting to the service https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server

================================================================================

			JOB GET OUTPUT OUTCOME

Output sandbox files for the DAG/Collection :
https://wms-aegis.ipb.ac.rs:9000/O3Bxi2I9DoE2Lltmcoe0eQ
have been successfully retrieved and stored in the directory:
/home/ngrkic/test

================================================================================


8.Go to test directory and see output with commands:

[ngrkic@ui ~]$ cd 
[ngrkic@ui ~]$ cd test
[ngrkic@ui test]$ ll

drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeA
drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeB
drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeC
drwxr-xr-x  2 ngrkic ngrkic 4096 Oct  1 17:31 nodeD
[ngrkic@ui test]$ cd nodeA
[ngrkic@ui nodeA]$ ll
-rw-rw-r--  1 ngrkic ngrkic   0 Oct 12 17:31 std.err
-rw-rw-r--  1 ngrkic ngrkic 112 Oct 12 17:31 std.out

[ngrkic@ui nodeA]$ cat std.out 
Job A - Thu Oct 1 17:14:14 CEST 2009 - BEGIN
grid10.elfak.ni.ac.rs
Job A - Thu Oct 1 17:14:54 CEST 2009 - END

For each node there is similar output.

Personal tools