From c46aaf3bf7eb6d58f53e7ed55804a4fdfb1e0d17 Mon Sep 17 00:00:00 2001 From: =?utf8?q?Ale=C5=A1=20K=C5=99enek?= Date: Mon, 13 Dec 2004 13:58:30 +0000 Subject: [PATCH] Added usage notes --- org.glite.jp.primary/Makefile | 5 ++- org.glite.jp.primary/doc/USAGE | 100 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+), 2 deletions(-) create mode 100644 org.glite.jp.primary/doc/USAGE diff --git a/org.glite.jp.primary/Makefile b/org.glite.jp.primary/Makefile index 2e9b3c4..688afe0 100644 --- a/org.glite.jp.primary/Makefile +++ b/org.glite.jp.primary/Makefile @@ -129,10 +129,11 @@ distbin: rm -rf tmpbuilddir install: - -mkdir -p ${PREFIX}/bin ${PREFIX}/etc ${PREFIX}/examples ${PREFIX}/etc/init.d + -mkdir -p ${PREFIX}/bin ${PREFIX}/etc ${PREFIX}/examples ${PREFIX}/etc/init.d ${PREFIX}/share/doc/${package}-${version} + ${INSTALL} -m 755 ${daemon} ${PREFIX}/bin ${INSTALL} -m 755 jpps-test ${PREFIX}/examples/glite-jp-primary-test - + ${INSTALL} ${top_srcdir}/doc/USAGE ${PREFIX}/share/doc/${package}-${version} clean: diff --git a/org.glite.jp.primary/doc/USAGE b/org.glite.jp.primary/doc/USAGE new file mode 100644 index 0000000..6d89729 --- /dev/null +++ b/org.glite.jp.primary/doc/USAGE @@ -0,0 +1,100 @@ +PURPOSE, EXPECTED USAGE, AND LIMITATIONS + +The purpose of the Job Provenance service (JP) is keeping track of the +definition of submitted jobs, execution conditions and environment, and +important points of the job life cycle for a long period (months to years). +Those data can be used for debugging, post-mortem analysis, comparison of job +execution within an evolving environment, as well as assisted re-execution of +jobs. Only data of completed (either successful or failed) jobs are handled; +tracking jobs during their active life is the task of L&B and Job Monitoring +services. + +In general, gathered data are stored (i.e. copied) within the JP storage in +order to really conserve a partial snapshot of the Grid environment when the +job was executed, independently of changes of other Grid services. Obviously +there are practical limitations of the extent to which it is feasible to record +the entire job execution environment. (In the ideal case this would encompass a +snapshot of the entire Grid!) We restrict the recorded data to those that are +processed or somehow affect processing of the Workload Management and Computing +Element services. On the other hand, snapshots of the state of other Grid +services are not done, namely queries to the Data Catalog and their results are +not stored, as well as contents of data files downloaded from and uploaded to +Storage Elementsonly references to those are recorded if required. + +The JP Primary Storage, as the name suggests, is the primary interface for +raw data storage. It is complemented with JP Index service, designed to +provide indexing and execute various queries on these data. + + +ENCOMPASSED DATA + +Currently, JP is designed to gather the following data: + +- JOB LIFE LOG is taken over from the L&B database (see Sect. 8.3.5) after job + completion. Among other information useful mainly for debugging and detailed + analysis of job execution it contains the complete definition of the job (in + terms of the submitted JDL), various timestamps (e.g. when the job was + submitted, matched for a particular CE, started and finished execution), + information on the chosen CE (or more of them, if the job was resubmitted), + various IDs assigned to the job (Condor, Globus, LRMS, ...), and the + result of execution including the return code. Another important information + available in L&B are the user-defined annotations (tags) which can be + specified either statically upon job submission, during its execution, or even + overridden after the job terminates. + +- INPUT/OUTPUT SANDBOX. The input sandbox of a job contains miscellaneous files + required for execution, and the output sandbox may contain various output + files, e.g. debug logs, or even a core file in the case of crash. + +- JP TAGS. JP tags are arbitrary "name=value" pairs attached to a singl job. + They are either standalone or override values of their LB counterparts. + However, JP tag values are still distinguishable those inherited from LB. + JP tags may be either strings or blobs. + + +JP PRIMARY STORAGE INTERFACES AND OPERATIONS + +JP primary storage service exposes two interfaces: +- WS control and query interface, +- bulk file transfer interface (currently gsiftp). + +WS interface operations (see JobProvenancePS.wsdl for details): + +- RegisterJob - create the basic job record with JP primary. + + Normally called by L&B upon job registration wigh L&B. + +- StartUpload - initiates upload of input/output sandbox or job log. + JP responds with upload destination URL, which points to its + bulk file transfer interface, and a time limit for the upload. + + Normally called by L&B (job log) and WMS NS (sandboxes). + +- CommitUpload - confirm upload operation. This must be called after the + upload is done, and within the given time limit. If not done so, + the destination URL becomes invalid and any (partially) uploaded + file is discarded. + + Normally called by L&B (job log) and WMS NS (sandboxes). + +- RecordTag - Record a value of JP user tag. + + Called by any component which needs it. + +- FeedIndex - start feeding data into JP index server. + + Called internally by JP index. + +- GetJob - Based on known JobId retrieve URL's of the job record (i.e. job log, + input/output sandboxes, and JP tags). + + Mainly called by the user, either standalone, or after querying + index server. + + + +EXAMPLE PROGRAM + +The installed example program glite-jp-primary-test is a simple client +calling all the service WS operations. Run "glite-jp-primary-test -?" +for its current usage help. -- 1.8.2.3