clean:
rm -rvf JPUG* JPAG* JPDG*
+ rm -rvf log.xml project/ rpmbuild/ RPMS/ tgz/
-.PHONY: all clean purge ps pdf
+
+# dependencies:
+
+JPUG.pdf: JPUG.tex \
+ JPUG-Introduction.tex \
+ JPUG-Tools.tex glite-jpis-client.tex jpimporter.tex gui.tex \
+ JPUG-UseCases.tex
+
+JPAG.pdf: JPAG.tex \
+ JPAG-Introduction.tex LB-JP-interaction.tex \
+ JPAG-Installation.tex \
+ JPAG-Configuration.tex \
+ JPAG-Running.tex \
+ JPAG-Testing.tex \
+ JPAG-Troubleshooting.tex
+
+JPDG.pdf: JPDG.tex \
+ JPDG-Introduction.tex \
+ JPDG-WS.tex
+
+.PHONY: all clean
\section{Configuration}
\TODO{}
-\subsection{Setting up the MySQL database}
+\subsection{JPPS}
+\subsubsection{Setting up the MySQL database}
+
+\subsection{JPIS}
+
+\begin{alltt}
+The JP-IS server daemon assume prior creation of its database. Simple tool
+ for database creation is org.glite.jp.index/config/dbsetup.sh
+
+customize startup script /etc/init.d/glite-jp-indexd (see below)
+ and set up service startup using this script
+
+
+Currently, configuration is done by command line options, and
+some hard-coded options.
+
+
+The index server takes the following options:
+
+./glite-jp-indexd [option]
+ -d, --debug don't run as daemon, additional diagnostics
+ -q, --query-type hist/cont/both (default history)
+ -n, --noauth don't check user identity with result owner
+ -m, --mysql database connect string
+ -p, --port port to listen
+ -i, --pidfile file to store master pid
+ -o, --logfile file to store logs
+ -x, --config file with server configuration
+
+The config file parameter is required. There is the example configuration in
+$GLITE_LOCATION/etc/glite-jpis-config.xml.
+\end{alltt}
\section{Installation}
-\TODO{}
+\TODO{Import the content of \texttt{glite\_installation\_guide\_\[LB\|JP\].doc}.}
\subsection{Complete RPMs description}
+
\subsection{Daemons description}
-\subsection{CLI tools description}
+\subsection{CLI tools description}
+% admin (sbin) tools
\section{Introduction}
-\TODO{This document should contain:}
+\TODO{Do a reasonable merge with the JPUG-Introduction, do not duplicate text...}
+
+\subsection{Job Provenance service overview}
+The information about jobs submitted to gLite Workload Management
+System is collected by the Logging and Bookkeeping (LB) service.
+LB tracks jobs in terms of events and processes them in
+a~real time to give overall view on the actual job state. The user may
+query the bookkeeping server to obtain either the raw events or the
+computed job state, she may also register for receiving notifications
+on particular job state changes.
+
+While the LB is intended to keep track of jobs during its lifetime, it
+is not supposed to be used for long term archival of such data. The
+Job Provenance (JP) service is designed to provide long-term storage
+of all data related to job life and allow the end user to perform
+data-mining in this data.
+The JP is supposed to provide the permanent storage of
+the job related information as stored within the \LB, to couple it with
+the input sandboxes and other system oriented information necessary to
+reproduce the environment where a~particular job run.
+
+\subsubsection{Gathering data into Job Provenance}
+Fig.~\ref{fig:psinter} depicts basic gLite middleware components and
+their interaction with the Job Provenance.
+
+\begin{figure}[htpb]
+ \centering
+ \includegraphics[scale=0.7]{JP-interactions}
+ \caption{Data flow into gLite Job Provenance}
+ \label{fig:psinter}
+\end{figure}
+
+JP is formed of two classes of services: permanent \emph{Primary
+Storage} (JPPS) accepts and stores job data while possibly volatile
+and configurable \emph{Index Servers} (JPIS) provide an optimized
+querying and data-mining interface to the end-users. The only direct
+data retrieval scenario supported by JPPS is the case when user know exact ID
+of jobs in the interest.
+
+\subsubsection{Getting data from Job Provenance}
+
+The role of \emph{Index Servers} (JPIS) is processing and re-arranging the data
+from Primary Storage(s) into a~form suitable for frequent and complex user
+queries. A user query part of JP is shown in Fig.~\ref{fig:query}.
+
+\begin{figure}[htpb]
+ \centering
+ \includegraphics[scale=0.8]{JP-query}
+ \caption{Index Server interactions}
+ \label{fig:query}
+\end{figure}
+
+Index Servers are created, configured, and populated semi-dynamically
+according to particular user community needs. It is responsibility of
+its administrator to setup the JPIS with appropriate configuration. There
+is no prescribed relationship between Primary Storage and Index Server
+installations. An Index Server may retrieve data from multiple
+Primary Storages and vice versa.
+
+% TODO: update
+% The interface exposed by JPIS to the end user is described in the
+% chapter~\ref{reference}. Command line interface tool for end-user
+% interface to the JPIS is described in the chapter~\ref{CLI}. See the
+% next chapter (use cases) for futher description of JP to user
+% interactions.
+
+
+% LB-JP-interaction
+\input{LB-JP-interaction}
+
+%
\subsection{Deployment scenarios}
\section{Running and stopping the services}
\TODO{}
-\subsection{Tests if everything works properly}
+\subsection{JPPS}
+
+
+\subsection{JPIS}
+
+\begin{alltt}
+Preferred way of starting the daemon is using start-up script
+(config/startup). It loads glite.conf file (personal version may be stored
+in ~/.glite.conf) where many variables may be set to configure the daemon.
+The script takes following variables:
+
+GLITE_JPIS_CONFIG - server config file specification
+ (default is $GLITE_LOCATION//etc/glite-jpis-config.xml)
+GLITE_JPIS_DEBUG - setting to '-d' forces the daemon not to daemonize
+GLITE_JPIS_QT - defines query type
+ 'hist' ... history query
+ 'cont' ... continuous query
+ 'both' ... combination of previous types
+GLITE_JPIS_AUTH - setting to '-n' forces the daemon not to check
+ authorisation
+GLITE_JPIS_PORT - used port (default 8902)
+GLITE_JPIS_DB - database connection string
+ (default jpis/@localhost:jpis)
+GLITE_JPIS_LOGFILE - log file
+ (default is $GLITE_LOCATION_VAR/log/glite-jp-indexd.log)
+GLITE_JPIS_PIDFILE - pid file
+ (default is $GLITE_LOCATION_VAR/run/glite-jp-indexd.log)
+
+\end{alltt}
--- /dev/null
+\section{Testing JP functionality}
+% AKA Testplan
+
+\def\req{\noindent\textbf{Prerequisities: }}
+\def\how{\noindent\textbf{How to run: }}
+\def\result{\noindent\textbf{Expected result: }}
+\def\jpps{\noindent\textbf{JP PS log should contain: }}
+\def\jpis{\noindent\textbf{JP IS log should contain: }}
+
+
+\subsection{JPPS standalone tests}
+
+\subsubsection{Job registration}
+
+\paragraph{Basic functionality}
+\label{regjob}
+\req\ Running JPPS
+
+\how
+\begin{itemize}
+\item call RegisterJob operation:
+\begin{verbatim}
+$ jpps-test RegisterJob JOBID OWNER
+\end{verbatim}
+where JOBID and OWNER should be replaced with real values, JOBID should not have
+been registered with JP before.
+
+\item call GetJobAttributes to verify:
+\begin{verbatim}
+$ jpps-test GetJobAttr JOBID http://egee.cesnet.cz/en/Schema/JP/System:owner
+\end{verbatim}
+\end{itemize}
+\result Should print the OWNER value supplied.
+
+\paragraph{AuthZ check}
+\req\ JPPS running, a~job registered with the procedure in~\ref{regjob}
+
+\how\
+Call GetJobAttributes using different user credentials
+
+\result\
+Should fail with ``Permission denied'' error
+
+\subsubsection{Tag recording}
+\label{tagreg}
+
+\paragraph{Basic functionality}
+\req\
+JPPS running, a~job registered with the procedure in~\ref{regjob}
+
+\how
+\begin{itemize}
+\item Call RecordTag operation:
+\begin{verbatim}
+$ jpps-test RecordTag JOBID TAGNAME STRINGVALUE
+\end{verbatim}
+\item Call GetJobAttributes to verify
+\begin{verbatim}
+$ jpps-test GetJobAttr JOBID TAGNAME
+\end{verbatim}
+\end{itemize}
+\result
+The recorded value should be returned.
+
+\how\ Record another values(s) of the same tag by repeating the RecordTag call
+
+\result\ GetJobAttr should return all the recorded values
+
+
+\paragraph{AuthZ check}
+\req\ JPPS running, a~job registered with the procedure in~\ref{tagreg} \\
+\how\ Call RecordTag using different user credentials \\
+\result\ Should fail with ``Permission denied'' error \\
+
+
+\subsubsection{File upload}
+
+
+\paragraph{Basic functionality}
+\req\ JPPS running, my certificate subject amont JPPS trusted peers,
+\verb'globus-url-copy' in PATH
+
+\how
+Run the aggregate test script from \verb'org.glite.jp.primary/build'
+
+\begin{verbatim}
+$ ../examples/jpps_store_test -o 'OWNER' -d ../examples/job_template
+\end{verbatim}
+(substitute real cert.\ subject for OWNER)
+
+\result\
+The script calls JPPS operations RegisterJob and StartUpload,
+uploads an \LB\ job log generated from the template file,
+calls CommitUpload.
+
+Finally, the upload is checked by retrieving two attribute
+values: LB/Attributes:user and LB/Attributes:finalStatusk via the GetJobAttr
+call.
+Both calls should return OK and print reasonable values.
+
+%- call StartUpload, LB dump file type
+%* check with GetJobFiles -- shoud return nothing
+%- upload via ftp
+%- call CommitUpload
+%* check with GetJobFiles -- should return URL
+%- retrieve and check the file
+
+\paragraph{Phase checks}
+\TODO{salvet}
+% soubor nelze zapsat pred otevrenim operaci StartUpload
+% nelze cist pred Commitem
+% nelze zapsat po Commitu
+
+\paragraph{AuthZ checks}
+(should fail)
+
+\TODO{salvet}
+%* call GetJobFiles with different credentials
+%
+%* StartUpload with different credentials
+%
+%- StartUpload
+%* ftp upload with different credentials
+%
+%* ftp GET with different credentials
+
+
+\paragraph{Cleanup}
+(Foreseen test for feature which is not implemented yet)
+%- call StartUpload, short timeout
+%- upload via ftp
+%(don't call CommitUpload)
+%* uploaded file should be purged after timeout
+
+\subsection{\LB\ plugin}
+%\TODO{honik}
+\LB\ plugin is a component integrating the \LB\ functionality into JP.
+
+
+\subsubsection{Standalone tests}
+\LB\ plugin as a standalone component is used for example in the \texttt{glite-lb-statistics}
+program (part of org.glite.lb.utils). This program reads a dump file of events related to
+one particular job and using the \LB\ plugin it computes the job state and many other job
+statistics. See the \LB\ testplan for more details.
+
+\subsubsection{Integrated tests}
+\req JPPS running with the \texttt{-P/path/to/the/glite\_lb\_plugin.so}
+
+\how
+\begin{itemize}
+\item call GetJobAttributes to get the LB attributes
+\begin{verbatim}
+$ jpps-test GetJobAttr JOBID ATTRIBUTE
+
+where ATTRIBUTE is one of the
+http://egee.cesnet.cz/en/Schema/LB/Attributes:jobId
+http://egee.cesnet.cz/en/Schema/LB/Attributes:user
+http://egee.cesnet.cz/en/Schema/LB/Attributes:VO
+http://egee.cesnet.cz/en/Schema/LB/Attributes:eNodes
+http://egee.cesnet.cz/en/Schema/LB/Attributes:eProc
+http://egee.cesnet.cz/en/Schema/LB/Attributes:RB
+http://egee.cesnet.cz/en/Schema/LB/Attributes:CE
+http://egee.cesnet.cz/en/Schema/LB/Attributes:host
+http://egee.cesnet.cz/en/Schema/LB/Attributes:UIHost
+http://egee.cesnet.cz/en/Schema/LB/Attributes:CPUTime
+http://egee.cesnet.cz/en/Schema/LB/Attributes:finalStatus
+http://egee.cesnet.cz/en/Schema/LB/Attributes:finalStatusDate
+http://egee.cesnet.cz/en/Schema/LB/Attributes:finalStatusReason
+http://egee.cesnet.cz/en/Schema/LB/Attributes:LRMSDoneStatus
+http://egee.cesnet.cz/en/Schema/LB/Attributes:LRMSStatusReason
+http://egee.cesnet.cz/en/Schema/LB/Attributes:retryCount
+http://egee.cesnet.cz/en/Schema/LB/Attributes:additionalReason
+http://egee.cesnet.cz/en/Schema/LB/Attributes:jobType
+http://egee.cesnet.cz/en/Schema/LB/Attributes:nsubjobs
+http://egee.cesnet.cz/en/Schema/LB/Attributes:lastStatusHistory
+http://egee.cesnet.cz/en/Schema/LB/Attributes:fullStatusHistory
+\end{verbatim}
+\end{itemize}
+
+\result Should print the corresponding LB attributes
+
+\subsection{JPPS-JPIS interaction (feeds)}
+
+
+%set of queries (how many?) with different "triggering conditions":
+%- on job registration
+%- on LB file upload
+%- on RecordTag
+
+%corresponding sets of jobs to each query, each containing jobs which match
+%and which don't
+
+%- initial IS release -- single query, so just one set of jobs
+%- due to 3.2 no point in pre-loading PS database, use 1.3.1
+
+\subsubsection{Batch feed}
+%- upload jobs to PS
+%- start feed
+%* check IS contents (jobs and expected attr values)
+
+\req\ Clean JP-PS and JP-IS database.
+
+\how\
+\begin{enumerate}
+ \item \emph{Start JP primary server}
+ \item \emph{Register job to PS}
+ \begin{alltt}
+ for j in `seq 1 10`;
+ do
+ for i in glite-jp-primary-sample_job*.lb;
+ do
+ ./glite-jp-primary-store-test -o \emph{CERT_DN}
+ -t "my_tag=car" -s https://localhost:8901 -d $i;
+ done;
+ done
+ \end{alltt}
+ You should see something like:
+ \begin{alltt}
+ ** ./glite-jp-primary-test -s https://localhost:8901 RegisterJob
+ https://nonexistent.test.server/jpps_store_test_7199
+ /O=CESNET/O=Masaryk University/CN=Milos Mulac
+ OK
+ ** ./glite-jp-primary-test -s https://localhost:8901 GetJobAttr
+ https://nonexistent.test.server/jpps_store_test_7199
+ http://egee.cesnet.cz/en/Schema/JP/System:owner
+ OK
+ Attribute values: /O=CESNET/O=Masaryk University/CN=Milos Mulac
+ SYSTEM Thu Feb 16 14:40:02 2006
+ ....
+ Attribute values:
+ car FILE Thu Feb 16 14:40:02 2006
+ \end{alltt}
+ \item \emph{Start JP index server, using history query}\\
+ \item \emph{Check content of IS database}\\
+ \begin{alltt}
+ mysql -u jpis -e "select * from jobs;" jpis
+ \end{alltt}
+ You should get 50 results, similar to:
+ \begin{alltt}
+| jobid | dg_jobid
+| ownerid | aclid | ps
++----------------------------------+---------------------------------+
+| 7bd73b18b33410ba605fba99dbdd803f | https://nonexistent.test.server/jpps_store_test_5993
+| 5864429d57da18e4ecf9ea366c6b2c9c | NULL | https://localhost:18950 |
+...
+50 rows in set (0.00 sec)
+ \end{alltt}
+\end{enumerate}
+\result{} Expected results in the IS database content check (last step).
+
+
+\subsubsection{Incremental feed - simple tests}
+%- register feed
+%- upload job to PS
+%* check PS and IS output
+
+\req\ Clean JP-PS and JP-IS database.
+
+\how\
+\begin{enumerate}
+ \item \emph{Start JP primary server}
+ \item \emph{Start JP index server, using continuous query}
+ \item \emph{Registerjob}
+ \begin{alltt}
+ ./jpps-test -s https://localhost:18950 RegisterJob
+ https://nonexistent.test.server/jpps_store_test_6880 "/O=CESNET/O=Masaryk
+ University/CN=Milos Mulac"
+ OK
+ \end{alltt}
+ \jpps\
+ \begin{alltt}
+ [22004] client DN: /O=CESNET/O=Masaryk University/CN=Milos Mulac
+ __jpsrv__RegisterJob https://nonexistent.test.server/jpps_store_test_6881
+ /O=CESNET/O=Masaryk University/CN=Milos Mulac
+ feed to https://scientific.civ.zcu.cz:8902, job https://nonexistent.test.server/
+ jpps_store_test_6881
+ \end{alltt}
+ \jpis\
+ \begin{alltt}
+ ...
+ [21984] incoming request
+ __jpsrv__UpdateJobs
+ ...
+ glite_jpis_lazyInsertJob: owner '/O=CESNET/O=Masaryk University/CN=Milos Mulac'
+ found
+ glite_jpis_insertAttrVal: (http://egee.cesnet.cz/en/Schema/JP/System:owner)
+ sql=INSERT INTO attr_52942b8c70bab8491ab5d3b9713d79f5 (jobid, value, full_value,
+ origin) VALUES (
+ '6e436919404778b75cd27eef266190bb',
+ 'S:/O=CESNET/O=Masaryk University/CN=Milos Mulac',
+ 'S:/O=CESNET/O=Masaryk University/CN=Milos Mulac',
+ '1'
+)
+ glite_jpis_insertAttrVal: (http://egee.cesnet.cz/en/Schema/JP/System:regtime) ...
+ ...
+ \end{alltt}
+
+ \item \emph{Start upload}
+ \begin{alltt}
+ ./jpps-test -s https://localhost:18950 StartUpload
+ https://nonexistent.test.server/jpps_store_test_6880
+ urn:org.glite.jp.primary:lb 1234 text/plain
+ OK
+ Destination: gsiftp://scientific.civ.zcu.cz:8960//home/mulac/jp/internal/
+ data/5864429d57da18e4ecf9ea366c6b2c9c/1889/jpps_store_test_6880/lb
+ Commit before: Sat Mar 17 10:12:48 2007
+ \end{alltt}
+ \jpps\
+ \begin{alltt}
+ [22004] client DN: /O=CESNET/O=Masaryk University/CN=Milos Mulac
+ data_basename: (null)
+ \end{alltt}
+ \jpis\
+ nothing
+ \item \emph{globus-url-copy}
+ \begin{alltt}
+ globus-url-copy file:/home/mulac/src/ORG/org.glite.jp.primary/build/job.6880
+ gsiftp://scientific.civ.zcu.cz:8960//home/mulac/jp/internal/
+ data/5864429d57da18e4ecf9ea366c6b2c9c/1889/jpps_store_test_6880/lb
+ \end{alltt}
+ \jpps\
+ nothing \\
+ \jpis\
+ nothing \\
+ \noindent\textbf{Other:}
+ File specified in gsiftp URL should be created.
+ \item \emph{Commit upload}
+ \begin{alltt}
+ ./jpps-test -s https://localhost:18950 CommitUpload
+ gsiftp://scientific.civ.zcu.cz:8960//home/mulac/jp/internal/data/
+ 5864429d57da18e4ecf9ea366c6b2c9c/1889/jpps_store_test_6880/lb
+ OK
+ \end{alltt}
+ \jpps\
+ \begin{alltt}
+ [22004] client DN: /O=CESNET/O=Masaryk University/CN=Milos Mulac
+ glite_jpps_match_file: https://nonexistent.test.server/jpps_store_test_6880 lb (null)
+ lb_plugin: opened 8 events
+ lb_plugin: close OK
+ feed to https://scientific.civ.zcu.cz:8902, job https://nonexistent.test.server/
+ jpps_store_test_6880
+ \end{alltt}
+ \jpis\
+ \begin{alltt}
+ ...
+ __jpsrv__UpdateJobs
+ ...
+ glite_jpis_insertAttrVal: (http://egee.cesnet.cz/en/Schema/LB/Attributes:CE)
+ sql=INSERT INTO attr_c47f78255056386d2b3da6d506d1f244 (jobid, value,
+ full_value, origin) VALUES (
+ '39a0a14f4fc084fbb466728986e5ea2f',
+ 'S:destination CE/queue',
+ 'S:destination CE/queue',
+ '3'
+ )
+ ...
+ \end{alltt}
+ \end{enumerate}
+
+\result{} Expected results in logs.
+
+
+\subsubsection{Incremental feed}
+%- register feed
+%- upload jobs to PS one by one
+%* check IS contents (matching jobs should turn up, others not)
+
+\req\ Clean JP-PS and JP-IS database.
+
+\how\
+\begin{enumerate}
+ \item \emph{Start JP primary server}
+ \item \emph{Start JP index server, using continuous query}
+ \item \emph{Register job to PS}
+ The same as in previous test case.
+
+ \item \emph{Check output of IS}\\
+ You should see incomming connection logs, and among them
+ several times something like:
+ \begin{alltt}
+
+ INSERT INTO attr_52942b8c70bab8491ab5d3b9713d79f5 (jobid, value,
+ full_value, origin) VALUES (
+ '6f4866f3e4f8204c269449e6924d73c0',
+ 'S:/O=CESNET/O=Masaryk University/CN=Milos Mulac',
+ 'S:/O=CESNET/O=Masaryk University/CN=Milos Mulac',
+ '1')
+ ....
+ \end{alltt}
+ \item \emph{Check content of IS database}\\
+ Do the same test as in previous test case. It must give you the same
+ result. You can also look whether the insert from previous step was
+ successful:
+ \begin{alltt}
+ mysql -u jpis -e "select * from
+ attr_52942b8c70bab8491ab5d3b9713d79f5;" jpis
+ \end{alltt}
+ should return:
+ \begin{alltt}
+| jobid | value
+| full_value | origin |
++----------------------------------+-----------------------------------+
+| 76698aabbf5d60dfa5b42c279e1f0e8c | S:/O=CESNET/O=Masaryk University/CN=Milos
+Mulac
+| S:/O=CESNET/O=Masaryk University/CN=Milos Mulac | 1 |
+ \end{alltt}
+\end{enumerate}
+\result{} Expected database INSERTs in the JP-IS (last two steps).
+
+\subsubsection{Multiple feeds at time}
+\TODO{TBD}
+
+\subsubsection{Advanced feed features (to be implemented)}
+- remove (not implemented in PS yet)
+- splitted info about one job (check that the PS doesn't duplicate
+ attribute values) - probably covered in 3.2
+
+
+\subsubsection{PS-IS AuthZ}
+\TODO{Not implemented yet}
+
+\subsection{IS queries}
+
+
+%TBD: insert job sets via JP-IS interaction or directly?
+% - better to populate database directly, independent on previous chain
+%
+%All basic tests:
+%- clear IS database
+%- insert prepared job set
+%- ask queries and check answers
+%- clear database
+%
+%TBD: Is one job set enough?
+% - better to have one complete set
+%
+
+A majority of test from this chapter is automated by shell
+script. The script is located in \texttt{org.glite.jp.index} module
+under \texttt{example/query-tests} directory and called \texttt{run-test.sh}.
+It is available as a part of JP index server RPM package.
+
+\begin{hints}
+The testing shell script is highly configurable via
+environmental varibles. Please, run the script (run-test.sh) with
+'-?' option to get list of all variables and their meaning, if you are
+not satisfied with default setting.
+\end{hints}
+
+\subsubsection{Simple query}
+This test starts new index server instance, creates testing DB
+and populate it with prepared data sample. Then simple query is given
+to server, answer is checked with supposed return output and
+cleanup is done.
+
+
+\how\ Run \texttt{run-test.sh}
+
+\begin{hints}
+The query is in file test/simple\_query.in and has following
+ form: (status=Ready)
+\end{hints}
+
+\subsubsection{Complex query test}
+This is similar to simple query test, only tested query is more complicated.
+
+\how\ Run \texttt{run-test.sh}
+
+\begin{hints}
+The query is in file test/complex\_query.in and has followhing
+ form: (status=Done OR status=READY) AND (user!=God)
+\end{hints}
+
+\subsubsection{Feed \& query test}
+This test starts testing index server, feeds it by
+mimicing bahaviour of primary storage server by sending data
+via soap call, and then asks the index server using a complex
+query. After that it checks the responce and does cleanup.
+
+More precise description of steps:
+\begin{enumerate}
+ \item Simulation of response from a primary storage, making appropriate
+ changes in JP-IS database (inserts feedid).
+ \item Invocation of updateJobs wsdl call, normally invoked by JP-PS, and
+ sending this way some data to the JP-IS which stores them in its database.
+ \item Invocation of queryJobs wsdl call, normally called by user
+ program, obtaining previously inserted data. Test query used here has form
+ (status=Done OR status=Ready) AND (user!=God).
+\end{enumerate}
+
+\how\ Run \texttt{run-test.sh}
+
+\subsubsection{AuthZ checks}
+This test verifies that qeury responses are properly restricted by
+authorization checks. Currently only implicit ACLs are implemented
+inside JP-IS server, so explicit ACLs and its evaluation is to be implemented.
+
+There are 3 scenarios to be verified:
+\begin{itemize}
+ \item Authorization (checking ownership) is swithed off (IS with -n
+ option). This scenario is tested by simple query test described above.
+ \item Only user jobs are returned and jobs not owned by the user posing
+ the query are not covered by the query response. This scenario is
+ covered by Feed \& query test described above.
+ \item Check that queries to jobs not owned by the IS user are
+ returning empty response. The same behaviour as simple query test
+ described above but with user credential not matching job
+ owner. This test is implemented by \texttt{run-test.sh} under AuthZ
+ check part.
+\end{itemize}
+
+\subsubsection{Another supposed tests not implemented yet}
+
+\begin{itemize}
+ \item Check "origin" behaviour -- queries with origin tag
+ \item IS CLI tests -- use prepared config files and command line parameters
+ and check expected QueryJobs contents
+\end{itemize}
+
+
+\subsection{IS standalone advanced features}
+\TODO{Not implemented yet}
+
+\subsubsection{Server startup}
+
+\paragraph{Reboot persistency / configuration vs. database content}
+ situations handling
+- prepared config files
+- checking behaviour (how?) after reboot with different config file
+
+\paragraph{Registration of PS feeds}
+! already covered by 3
+- prepared config files
+- checking appropriate FeedIndex calls
+
+\subsubsection{Admin interface}
+\TODO{Admin interface not implemented yet}
+
+\subsubsection{Type plugin}
+\TODO{type plugin tests -- to be designed, future type plugin implementation}
+
+\subsection{Deployment}
+\TODO{tests on JP deployment process}
+\TODO{TBD}
+
\input{JPAG-Running}
\newpage
+\input{JPAG-Testing}
+
+\newpage
\input{JPAG-Troubleshooting}
\nocite{jgc}
\TODO{Best practices :)}
-
-\section{Web Services Interface}
--- /dev/null
+\section{Web Service Interface}
+
+In source code tree the WSDLs are located in these files:\\
+\texttt{org.glite.jp.ws-interface/src/JobProvenanceIS.xml,\\
+org.glite.jp.ws-interface/src/JobProvenancePS.xml,\\
+org.glite.jp.ws-interface/src/JobProvenanceTypes.xml
+}
+
+\TODO{Add more info about the Web Service Interface...}
+
+% JPWS reference:
+{
+\parindent0pt
+\def\chapter#1{}
+\def\section#1{\subsection{#1}}
+\def\subsection#1{\par\medskip\textbf{#1}\par}
+
+\let\odesc=\description
+\let\oedesc=\enddescription
+\renewenvironment{description}{\odesc\itemindent=1em
+\listparindent=2em
+}{\oedesc}
+%\renewenvironment{description}{\list{}{\labelwidth 5cm\leftmargin 5cm}}
+%{\endlist}
+
+\let\null=\relax
+
+% this file was manually generated using db2latex (http://db2latex.sourceforge.net)
+% from org.glite.jp.ws-interface/build/doc-html.xml
+% TODO: generate it automatically here in Makefile
+
+\input{jpws}
+}
+
\newpage
\input{JPDG-Introduction}
-%\newpage
-%\input{web_services}
+\newpage
+\input{JPDG-WS}
\newpage
\nocite{jgc}
together with the \verb'glite-jp-client'. These are the only \JP\ tools that a
regular grid user may need to use.
+
+% This file was manually generated using db2latex (http://db2latex.sourceforge.net)
+% from org.glite.jp.index/doc/glite-jpis-client.sgml
+% TODO: update the original file
+% TODO: generate it automatically here in Makefile
+{
+\parindent0pt
+\def\section#1{\subsection{#1}}
+\newcommand{\dbz}{}
+\newcommand{\docbooktolatexpipe}{\ensuremath{|}}
+\newskip\docbooktolatexoldparskip
+\input{glite-jpis-client}
+}
+
+
\input{jpimporter}
+
+
\input{gui}
-\section{Use Cases}
+\section{Job Provenance use cases}
+
+\subsection{Prerequisities}
+
+\subsubsection{LB/JP relationship}
+When JP deployed, any job in a terminal state will disappear from LB
+after preconfigured timeout (one week for example). If a user wants
+any information about such a job before this timeout (or before it
+reach a terminal state) he must use the LB service (please refer to LB
+user's guide). After that timeout he must use the JP service.
+
+% TODO: update
+% For LB configuration please see gLite installation guide. For a
+% technical description of LB-JP interactions please see
+% \texttt{http://egee.cesnet.cz/en/JRA1/LB-JP-interaction-guide.pdf}.
+
+\subsubsection{JP service location}
+To call JP you need to know JP services address. There are two services:
+\begin{itemize}
+\item JP primary storage (JPPS)\\
+ From JP design point of view there are only few PS in the
+ grid. Expected implementation is that these JPPS locations
+ are preconfigured in a UI instance while one of them is configured as
+ default JPPS.
+\item JP index server (JPIS)\\
+ Each index server is build (configured and started) by site/VO/user
+ group administrator (or even "senior user") based on given
+ community needs (expected queries and its optimization). So in
+ principle the index server location for a given query is to be
+ provided by the user. We expect that the UI instance will provide
+ mechanism allowing selection from preconfigured JPIS servers list.
+\end{itemize}
+
+\subsection{JP use case 1 -- get job info}
+
+The scenario:
+\begin{itemize}
+\item The user wants information about a particular job. He knows a
+ job id. Job isn't longer in the LB. Procedure: Ask the JPPS to get all
+ or selected attributes of job.
+\end{itemize}
+
+The implementation:
+\begin{itemize}
+ \item Let a user to specify attributes to be returned. See section
+ \ref{attributes}.
+ \item Call GetJobAttributes operation of a JPPS and display the values
+ returned.
+\end{itemize}
+
+Examples and hints:
+\begin{itemize}
+ \item \texttt{org.glite.jp.primary/examples/jpps-test.c}\\
+ This utility is used for all JPPS operations. Some hints how to use it
+ can be find in the test plan document.
+\end{itemize}
+
+\subsection{JP use case 2 -- get job files}
+
+The scenario:
+\begin{itemize}
+ \item The user knows a job id, job is in a terminal state. The user wants
+ all files (LB event dump, sandbox) stored by JP for futher processing.
+\end{itemize}
+
+The implementation:
+\begin{itemize}
+ \item Call GetJobFiles operation of a JPPS. You will get a list of URLs
+ which can be used to download the files.
+\end{itemize}
+
+Examples and hints:
+\begin{itemize}
+ \item The same as use case 1.
+\end{itemize}
+
+\subsection{JP use case 3 -- job lookup}
+
+The scenario:
+\begin{itemize}
+ \item The user is looking for jobs with specific properties. In this case
+ (no job id known) a JPIS must be used. There are the same query interface
+ provided by any JPIS but if a particular query can be answered by
+ the given JPIS depends on its configuration (configuration
+ determines which attributes are uploaded by PS to IS, and which of
+ them are indexed).
+ \item The user should know the proper JPIS to use for its particular
+ needs.
+ \item The scenario can continue by the JP use cases number 1 and 2 described
+ above (JPIS answer will contain job ids and identification of JPPSs
+ to ask for all available JP data about the jobs).
+\end{itemize}
+
+The implementation:
+\begin{itemize}
+ \item The user will select a JPIS and provide query. The JPIS operation
+ QueryJobs is called and list of jobs matching the query is returned.
+\end{itemize}
+
+Examples and hints:
+\begin{itemize}
+ \item JPIS CLI tool\\
+ org.glite.jp.index/examples/jpis-client.c
+
+ \item example in org.glite.jp.index/examples/jpis-test.c (starting
+ from line 161)
+\end{itemize}
+
+\subsection{JP use case 4 -- job annotation}
+
+The scenario:
+\begin{itemize}
+ \item The user wants to add a user tag (annotation) to a job. He must know
+ the job id(s) (or use the JP use case number 3 to find it).
+\end{itemize}
+
+The implementation:
+\begin{itemize}
+ \item Call RecordTag operation of JPPS for the job(s) to add requested
+ user tag.
+\end{itemize}
+
+Examples and hints:
+\begin{itemize}
+ \item The same as use case 1.
+\end{itemize}
+
+
+\subsection{Job attributes}
+\label{attributes}
+Job attributes are referenced by its names. Each attribute belongs to
+one namespace (represented by a prefix in the attribute name).
+
+A namespace is defined by a service (currently we have one for LB and
+one for JP) providing its data to the JP or a user group/experiment
+who wants to attach its own data to the job.
+
+It is expected that UI have preconfigured list of available namespaces
+and XML schema for each namespace (the schema can be automatically
+retrieved based on the namespace name). A list of available attributes
+is generated from these schemas when user have to select attributes to
+be retrieved from JP.
+
+\begin{itemize}
+ \item The namespaces (schema is available at the URL representing namespace):\\
+ http://egee.cesnet.cz/en/Schema/LB/Attributes\\
+ http://egee.cesnet.cz/en/Schema/JP/System <<<<<<<(NOT YET)\\
+ \item There are header files with known names of attributes generated from
+ these schema files in our build procedure:\\
+ org.glite.lb.server/build/jp\_job\_attrs.h\\
+ org.glite.jp.common/interface/known\_attr.h <<<<<<<\\
+\end{itemize}
+
+
+\subsection{Authentication and authorization}
+All the calls must be authenticated by user credentials. In the
+current JP release only implicit ACLs are available -- the job
+information is available for job owner only.
-This section describes usage of \JP\ ...
--- /dev/null
+\subsection{Interaction with Logging and Bookeeping (\LB)}
+
+In this section we describe the interaction of JP with Logging and Bookkeeping
+(\LB) service. The data flows between LB and JP services are displayed in
+Figure~\ref{fig:LB-JP-interactions}. These flows are numbered and one can use
+this numbers to find additional information about each flow in
+table~\ref{tab:LB-JP-interactions}.
+
+\begin{figure}[htpb]
+ \centering
+ \includegraphics[width=0.9\hsize]{LB-JP-interaction-details}
+ \caption{LB to JP interactions detail overview}
+ \label{fig:LB-JP-interactions}
+\end{figure}
+
+\begin{table}[htpb]
+ \centering
+ \begin{tabular}{|c|p{3cm}|l|p{9cm}|}
+ \hline
+ &spool directory&initiated by&description\\
+ \hline
+ \hline
+ 1&lb.export.dump,
+ lb.export.dump.keep&lb-exporter&
+ Export of LB job records into spool directory. It uses glite-lb-purge utility. LB-exporter reads this spool directory in a regular manner and implement next processing of LB dumps. Optionally it can keep handled dumps in lb.export.dump.keep.\\
+ \hline
+ 2&lb.export.jpreg&LB server&When new job come to the LB server
+ it stores its
+ registration into the spool directory. It is responsibility of
+ JP-importer process to handle such registrations.\\
+ \hline
+ 3&lb.export.jpdump,
+ lb.export.jobs,
+ lb.export.jobs.keep&lb-exporter&
+ LB-exporter do its processing of LB dumps (they are in per job form) and passes on it to the JP-importer using the spool directory lb.export.jpdump and temporary storage lb.export.jobs. It can keep the job files for futher usage.\\
+ \hline
+ 4&none&jp-importer&JP importer handles registrations received from LB
+ server and sends it to the JP primary server front-end (using its WS
+ interface).\\
+ \hline
+ 5&none&jp-importer&JP importer handles LB dumps received from LB
+ exporter and sends it to the JP primary server back-end using its
+ gridftp interface.\\
+ \hline
+ \end{tabular}
+ \caption{LB to JP data flows description}
+ \label{tab:LB-JP-interactions}
+\end{table}
+
+
+Notes:
+\begin{itemize}
+ \item Only JP Primary Storage (JPPS) server is involved in described
+ data flows. JP Index Servers are not part of this picture (they are
+ feeded via corresponding JPPS).
+ \item Only flows number 4 and 5 are designed to be inter-host. All
+ the other interactions assume the components are on the same host and
+ do use access to a shared filesystem.
+ \item Data flow number 1 use glite-lb-purge utility (see its
+ documentation) and passes to it argument from lb.export.purgeargs
+ clause of the deployment configuration file. This argument contain
+ the timeouts controlling after how long period of time a job
+ staying in a terminal state is to be purged from the LB server.
+ \item The LB exporter have a feature to store LB job event dumps in a
+ directory for further handling (e.g. for job statistic tool). This behaviour
+ is controled by lb.export.jobs.keep deployment config file clause (leave
+ this clause empty if you don't use dumps for futher handling).
+ \item The LB exporter also have a feature to keep all handled LB
+ dumps (in glite-lb-purge format) in filesystem. This feature is
+ controlled by lb.export.dump.keep.
+ \item LB exporter is not a deamon, it's periodic invocation is
+ provided by cron deamon.
+\end{itemize}
+
\usepackage{xspace}
%\usepackage{doxygen}
+\usepackage{alltt}
+\usepackage{comment}
\def\LB{L\&B\xspace}
\def\JP{JP\xspace}
\long\def\TODO#1{\par\noindent\textbf{TODO:} {\sl#1}\par}
\long\def\ludek#1{}
+\def\path#1{{\normalfont\textsf{#1}}}
+\def\code#1{\texttt{#1}}
+
+\specialcomment{hints}{\par\noindent\textbf{Hints: }\begingroup\slshape}{\endgroup}
+
\hyphenation{plug-in}
--- /dev/null
+%
+% -------------------------------------------------------------
+% Refentry
+% -------------------------------------------------------------
+\section{glite-jpis-client}
+\label{glitejpisclient}\hypertarget{glitejpisclient}{}%
+\label{name}
+
+%\section*{Nom}
+glite-jpis-client --- client interface for JP IS\label{synopsis}
+\subsection*{Synopsis}
+\label{id2455104}
+\begin{list}{}{\setlength{\itemindent}{-\leftmargin}\setlength{\parsep}{0mm}}
+\item\raggedright\texttt{glite-jpis-client [ -h | --help ] [ -i | --index-server \textit{JPIS:PORT}] [ -q | --query-file \textit{IN\_FILE.XML}] [ -t | --test-file \textit{IN\_FILE.XML}] [ -e | --example-file \textit{OUT\_FILE.XML}] [ -f | --format {xml | human}]}
+\end{list}
+
+\subsection*{DESCRIPTION}
+\label{id2417458}
+
+{\bfseries{glite-jpis-client}} is command line interface for querying the Job Provenance Index Server. It takes the XML input, process the QueryJobs operation and returns the result in specified format.
+
+\subsection*{OPTIONS}
+\label{id2417597}
+
+With no options you get simple usage message as with {\texttt{{-h}}}.
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\texttt{{-h}}}\docbooktolatexpipe{}{\texttt{{--help}}}}]\null{}
+Displays usage message.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\texttt{{-i}}}\docbooktolatexpipe{}{\texttt{{--index-server}}}}]\null{}
+Specifies Job Provenance Index Server as {\ttfamily\itshape{{HOST:PORT}}}.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\texttt{{-q}}}\docbooktolatexpipe{}{\texttt{{--query-file}}}}]\null{}
+Process the QueryJobs operation. Requires input data in file {\ttfamily\itshape{{IN\_FILE.XML}}}, for using stdin specify {\texttt{{-}}}.
+
+The input and output data are in XML format with XSD schema, which can be found in {\texttt{{JobProvenanceISClient.\dbz{}xsd}}} (element QueryJobs for input and QueryJobsResponse for output).
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\texttt{{-t}}}\docbooktolatexpipe{}{\texttt{{--test-file}}}}]\null{}
+Test the input data from {\ttfamily\itshape{{IN\_FILE.XML}}} (or from stdin if {\texttt{{-}}} is specified) and prints the found content.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\texttt{{-e}}}\docbooktolatexpipe{}{\texttt{{--example-file}}}}]\null{}
+Write the example input data to file {\ttfamily\itshape{{OUT\_FILE.XML}}} (or to stdout if {\texttt{{-}}} is specified). The XML is valid against XSD schema in {\texttt{{JobProvananceISClient.\dbz{}xsd}}}, formating may vary according to used gsoap version.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\texttt{{-f}}}\docbooktolatexpipe{}{\texttt{{--format}}}}]\null{}
+Use {\ttfamily\itshape{{FORMAT}}} as output format type. You can specify {\texttt{{xml}}} for interchangeable XML output or {\texttt{{human}}} for nice looking human readable output.
+\end{description}
+\noindent
+\subsection*{RETURN VALUE}
+\label{id2417767}
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{0}]\null{}
+Success.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{-1}]\null{}
+Communication error or error from the remote server.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{EINVAL}]\null{}
+In most cases XML parsing error.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{other error}]\null{}
+Other error from errno.
+\end{description}
+\noindent
+\subsection*{EXAMPLES}
+\label{id2417822}
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\bfseries{glite-jpis-client --example-file query.xml}}}]\null{}
+Save the example query parameters to file {\texttt{{query.\dbz{}xml}}}.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\bfseries{glite-jpis-client --query-file query.xml}}}]\null{}
+Queries the local index server running on default port with the query parameters specified in the file {\texttt{{query.\dbz{}xml}}}.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\bfseries{glite-jpis-client -i localhost:8902 -q - -f human}}}]\null{}
+Queries the index server running on local host on the port 8902 with the query parameters from stdin and show results in non-XML form.
+\end{description}
+\noindent
+
+
+\subsection*{SEE ALSO}
+\label{id2418102}
+
+glite-jp-indexd(8)
+
+\subsection*{AUTHOR}
+\label{id2418112}
+
+EU DataGrid Work Package 1, CESNET group.
\subsection{glite-jp-importer}
+\TODO{glite-jp-importer description}
+
--- /dev/null
+
+% -------------------------------------------------------------
+% Chapter Job Provenance
+% -------------------------------------------------------------
+\chapter{Job Provenance}
+\label{id271738}\hypertarget{id271738}{}%
+
+% ------------------------
+% Section
+\section{Primary Storage -- Overview}
+\label{id271094}\hypertarget{id271094}{}%
+
+The Job Provenance (JP) Primary Storage Service is responsible to keep the JP data (definition of submitted jobs, execution conditions and environment, and important points of the job life cycle) in a compact and economic form.
+
+The JP Primary storage, as described in section 8.4 of the Architecture deliverable DJRA1.1 {\textless}\url{https://edms.cern.ch/document/594698/}{\textgreater} provides public interfaces for data storing, retrieval based on basic metadata, and registration of Index servers for incremental feed.
+
+Command interface to JP is completely covered by the WS interface covered here. Bulk file transfers are done via specialised protocols, currently gsiftp only.
+
+% ------------------------
+% Section
+\section{Primary Storage -- Operations}
+\label{id271756}\hypertarget{id271756}{}%
+
+{\em{CVS revision: \$Header$}}
+\subsection{CommitUpload}
+\label{op:CommitUpload}\hypertarget{op:CommitUpload}{}%
+
+Confirm a successfully finished file apload.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{destination}}}}]\null{}
+Destination URL returned by StartUpload before.
+\end{description}
+\noindent
+
+Outputs: N/A
+\subsection{FeedIndex}
+\label{op:FeedIndex}\hypertarget{op:FeedIndex}{}%
+
+Request for feeding a JP Index server (issued by this server).
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{destination}}}}]\null{}
+Endpoint of the listening index server.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of xsd:string{\ttfamily\itshape{{attributes}}}}]\null{}
+Which attributes of jobs is the index server interested in.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:primaryQuery}{primaryQuery}{\ttfamily\itshape{{conditions}}}}]\null{}
+Which jobs is the server interested in.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:boolean{\ttfamily\itshape{{history}}}}]\null{}
+Data on jobs stored at PS in the past are required.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:boolean{\ttfamily\itshape{{continuous}}}}]\null{}
+Data on jobs that will arrive in future are required.
+\end{description}
+\noindent
+
+Outputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{feedId}}}}]\null{}
+Unique ID of the created feed session.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:dateTime{\ttfamily\itshape{{feedExpires}}}}]\null{}
+When the session expires.
+\end{description}
+\noindent
+\subsection{FeedIndexRefresh}
+\label{op:FeedIndexRefresh}\hypertarget{op:FeedIndexRefresh}{}%
+
+Refresh an existing feed session.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{feedId}}}}]\null{}
+Existing feed session ID to be refreshed.
+\end{description}
+\noindent
+
+Outputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:dateTime{\ttfamily\itshape{{feedExpires}}}}]\null{}
+New session expiration time.
+\end{description}
+\noindent
+\subsection{GetJobAttributes}
+\label{op:GetJobAttributes}\hypertarget{op:GetJobAttributes}{}%
+
+Query concrete attributes of a given job.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{jobid}}}}]\null{}
+The job.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of xsd:string{\ttfamily\itshape{{attributes}}}}]\null{}
+Which attributes should be retrieved.
+\end{description}
+\noindent
+
+Outputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:attrValue}{attrValue}{\ttfamily\itshape{{attrValues}}}}]\null{}
+Values of the queried attributes.
+\end{description}
+\noindent
+\subsection{GetJobFiles}
+\label{op:GetJobFiles}\hypertarget{op:GetJobFiles}{}%
+
+Return URL's of files for a given single job.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{jobid}}}}]\null{}
+The job.
+\end{description}
+\noindent
+
+Outputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:jppsFile}{jppsFile}{\ttfamily\itshape{{files}}}}]\null{}
+List of the stored files.
+\end{description}
+\noindent
+\subsection{RecordTag}
+\label{op:RecordTag}\hypertarget{op:RecordTag}{}%
+
+Record an additional user tag.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{jobid}}}}]\null{}
+Job to which the tag is added.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:tagValue}{tagValue}{\ttfamily\itshape{{tag}}}}]\null{}
+Name and value of the tag.
+\end{description}
+\noindent
+
+Outputs: N/A
+\subsection{RegisterJob}
+\label{op:RegisterJob}\hypertarget{op:RegisterJob}{}%
+
+Register job with the JP primary storage.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{job}}}}]\null{}
+Jobid of the registered job.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{owner}}}}]\null{}
+Owner of the job (DN of X509 certificate).
+\end{description}
+\noindent
+
+Outputs: N/A
+\subsection{StartUpload}
+\label{op:StartUpload}\hypertarget{op:StartUpload}{}%
+
+Start uploading a file.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{job}}}}]\null{}
+Jobid to which this file is related.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{class}}}}]\null{}
+Type of the file (URI). The server must have a plugin handing this type.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{name}}}}]\null{}
+Name of the file (used to distinguish among more files of the same type).
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:dateTime{\ttfamily\itshape{{commitBefore}}}}]\null{}
+The client promisses to finish the upload before this time.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{contentType}}}}]\null{}
+MIME type of the file.
+\end{description}
+\noindent
+
+Outputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{destination}}}}]\null{}
+URL where the client should upload the file.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:dateTime{\ttfamily\itshape{{commitBefore}}}}]\null{}
+Server's view on when the upload must be finished.
+\end{description}
+\noindent
+
+% ------------------------
+% Section
+\section{Index Server -- Overview}
+\label{id214261}\hypertarget{id214261}{}%
+
+The Job Provenance (JP) Index Server is a volatile counterpart to the permanent JP Primary Storage. Index servers are populated with subsets of data from Primary storage(s) and indexed according to particular user needs.
+
+The interface to Index server contains three logical parts: administraive (control), system and user. The administrative part is used by run-time index server configuration tool, the system one allows Primary storage(s) to feed data into the Index server, and the user one is available to users for queries.
+
+% ------------------------
+% Section
+\section{Index Server -- Operations}
+\label{id214279}\hypertarget{id214279}{}%
+
+{\em{CVS revision: \$Header$}}
+\subsection{AddFeed}
+\label{op:AddFeed}\hypertarget{op:AddFeed}{}%
+
+Called by JP index serve admin tool to ask new primary storage server to feed it. Updates information on PS in index server, according to what JPPS currently knows.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:feedSession}{feedSession}{\ttfamily\itshape{{feed}}}}]\null{}
+New feed IS URL, filter and query type.
+\end{description}
+\noindent
+
+Outputs: N/A
+\subsection{DeleteFeed}
+\label{op:DeleteFeed}\hypertarget{op:DeleteFeed}{}%
+
+Called by JP index serve admin tool to remove one feed session.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{feedId}}}}]\null{}
+ID of feed to be removed.
+\end{description}
+\noindent
+
+Outputs: N/A
+\subsection{GetFeedIDs}
+\label{op:GetFeedIDs}\hypertarget{op:GetFeedIDs}{}%
+
+Called by JP index serve admin tool to find out IS open feeds
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:feedSession}{feedSession}{\ttfamily\itshape{{feeds}}}}]\null{}
+List of active feeds on IS.
+\end{description}
+\noindent
+
+Outputs: N/A
+\subsection{QueryJobs}
+\label{op:QueryJobs}\hypertarget{op:QueryJobs}{}%
+
+User query to index server.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:indexQuery}{indexQuery}{\ttfamily\itshape{{conditions}}}}]\null{}
+Query conditions, similar to LB.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of xsd:string{\ttfamily\itshape{{attributes}}}}]\null{}
+Set of attributes to be retrieved directly from index server (if any).
+\end{description}
+\noindent
+
+Outputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:jobRecord}{jobRecord}{\ttfamily\itshape{{jobs}}}}]\null{}
+List of jobs matching the query.
+\end{description}
+\noindent
+\subsection{UpdateJobs}
+\label{op:UpdateJobs}\hypertarget{op:UpdateJobs}{}%
+
+Called by JP primary storage as a response to FeedIndex request. Updates information on jobs in index server, according to what JPPS currently knows.
+
+Inputs:
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string{\ttfamily\itshape{{feedId}}}}]\null{}
+Id of the feed, as returned by JPPS FeedIndex operation.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:boolean{\ttfamily\itshape{{feedDone}}}}]\null{}
+Flag of completed batch feed.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:jobRecord}{jobRecord}{\ttfamily\itshape{{jobAttributes}}}}]\null{}
+Attributes per job.
+\end{description}
+\noindent
+
+Outputs: N/A
+
+% ------------------------
+% Section
+\section{JP Common Types}
+\label{id214548}\hypertarget{id214548}{}%
+
+{\em{CVS revision: \$Header$}}
+\subsection{attrOrig}
+\label{type:attrOrig}\hypertarget{type:attrOrig}{}%
+
+Specification of attribute origin.
+
+{\em{Enumeration}} (restriction of xsd:string in WSDL), exactly one of the values must be specified.
+
+Values:
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{SYSTEM}}}}]\null{}
+JP system value, e.g. job owner.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{USER}}}}]\null{}
+Explicitely stored by the user via RecordTag operation.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{FILE}}}}]\null{}
+Coming from uploaded file.
+\end{description}
+\noindent \subsection{attrValue}
+\label{type:attrValue}\hypertarget{type:attrValue}{}%
+
+Single value of an attribute.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{name}}}}]\null{}
+Name of the attribute, including namespace.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:stringOrBlob}{stringOrBlob} {\ttfamily\itshape{{value}}}}]\null{}
+(optional) String value.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:dateTime {\ttfamily\itshape{{timestamp}}}}]\null{}
+When this value was recorded.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:attrOrig}{attrOrig} {\ttfamily\itshape{{origin}}}}]\null{}
+Where this value came from.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{originDetail}}}}]\null{}
+(optional)
+\end{description}
+\noindent \subsection{feedSession}
+\label{type:feedSession}\hypertarget{type:feedSession}{}%
+
+One session between IS and PS (aka feed) charactetristics.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{primaryServer}}}}]\null{}
+URL of primary server.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:primaryQuery}{primaryQuery} {\ttfamily\itshape{{condition}}}}]\null{}
+Filter conditions.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:int {\ttfamily\itshape{{history}}}}]\null{}
+Query type.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:int {\ttfamily\itshape{{continuous}}}}]\null{}
+Query type
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{feedId}}}}]\null{}
+(optional) Unique ID of the feed session.
+\end{description}
+\noindent \subsection{genericFault}
+\label{type:genericFault}\hypertarget{type:genericFault}{}%
+
+
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{source}}}}]\null{}
+
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:int {\ttfamily\itshape{{code}}}}]\null{}
+
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{text}}}}]\null{}
+
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{description}}}}]\null{}
+(optional)
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:genericFault}{genericFault} {\ttfamily\itshape{{reason}}}}]\null{}
+(optional)
+\end{description}
+\noindent \subsection{indexQuery}
+\label{type:indexQuery}\hypertarget{type:indexQuery}{}%
+
+Single query condition on a job. Similarly to LB, these outer conditions are logically ANDed.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{attr}}}}]\null{}
+Which attribute the condition refers to.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:attrOrig}{attrOrig} {\ttfamily\itshape{{origin}}}}]\null{}
+(optional) Specific attribute origin (if we do care).
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:indexQueryRecord}{indexQueryRecord} {\ttfamily\itshape{{record}}}}]\null{}
+List of conditions on attribute attr. These conditions are logically ORed.
+\end{description}
+\noindent \subsection{indexQueryRecord}
+\label{type:indexQueryRecord}\hypertarget{type:indexQueryRecord}{}%
+
+Single condition on an attribute.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:queryOp}{queryOp} {\ttfamily\itshape{{op}}}}]\null{}
+Query operation.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:stringOrBlob}{stringOrBlob} {\ttfamily\itshape{{value}}}}]\null{}
+(optional) Value to compare attribute with.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:stringOrBlob}{stringOrBlob} {\ttfamily\itshape{{value2}}}}]\null{}
+(optional) Value to compare attribute with.
+\end{description}
+\noindent \subsection{jobRecord}
+\label{type:jobRecord}\hypertarget{type:jobRecord}{}%
+
+Information on a single job. Used for both feeding JP index server from primary storage and to answer user queries on index server.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{jobid}}}}]\null{}
+ID of the job.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{owner}}}}]\null{}
+Job owner.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of \hyperlink{type:attrValue}{attrValue} {\ttfamily\itshape{{attributes}}}}]\null{}
+(optional) Attribute values, required by query/feed and available right now.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{list of xsd:string {\ttfamily\itshape{{primaryStorage}}}}]\null{}
+(optional) User query only: which primary storage(s) have data on this job.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:boolean {\ttfamily\itshape{{remove}}}}]\null{}
+(optional) UpdateJobs only: this job no longer belongs to the feed. Attribute values are those which caused the change.
+\end{description}
+\noindent \subsection{jppsFile}
+\label{type:jppsFile}\hypertarget{type:jppsFile}{}%
+
+JP primary storage file identification.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{class}}}}]\null{}
+Type of the file (as set on StartUpload).
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{name}}}}]\null{}
+Name of the file (if there are more of the same type per job).
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{url}}}}]\null{}
+Where the file is stored on JP primary storage.
+\end{description}
+\noindent \subsection{primaryQuery}
+\label{type:primaryQuery}\hypertarget{type:primaryQuery}{}%
+
+A single condition on job.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{attr}}}}]\null{}
+Attribute name to query.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:queryOp}{queryOp} {\ttfamily\itshape{{op}}}}]\null{}
+Operation.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:attrOrig}{attrOrig} {\ttfamily\itshape{{origin}}}}]\null{}
+(optional) Where the attribute value came from.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:stringOrBlob}{stringOrBlob} {\ttfamily\itshape{{value}}}}]\null{}
+Value to compare the job attribute with.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:stringOrBlob}{stringOrBlob} {\ttfamily\itshape{{value2}}}}]\null{}
+(optional) Another value (for op = WITHIN).
+\end{description}
+\noindent \subsection{queryOp}
+\label{type:queryOp}\hypertarget{type:queryOp}{}%
+
+Operators used in queries. Most are self-explanatory.
+
+{\em{Enumeration}} (restriction of xsd:string in WSDL), exactly one of the values must be specified.
+
+Values:
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{EQUAL}}}}]\null{}
+
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{UNEQUAL}}}}]\null{}
+
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{LESS}}}}]\null{}
+
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{GREATER}}}}]\null{}
+
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{WITHIN}}}}]\null{}
+The attribute is between two specified values.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{{\frenchspacing\texttt{{EXISTS}}}}]\null{}
+The attribute exists (even having a NULL value).
+\end{description}
+\noindent \subsection{stringOrBlob}
+\label{type:stringOrBlob}\hypertarget{type:stringOrBlob}{}%
+
+
+
+{\em{Union}} (choice complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{string}}}}]\null{}
+String value.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:base64Binary {\ttfamily\itshape{{blob}}}}]\null{}
+Binary value.
+\end{description}
+\noindent \subsection{tagValue}
+\label{type:tagValue}\hypertarget{type:tagValue}{}%
+
+A single user-recorded value for a job attribute.
+
+{\em{Structure}} (sequence complex type in WSDL)
+
+Fields: ( type{\ttfamily\itshape{{name}}} description )
+
+\begin{description}
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{xsd:string {\ttfamily\itshape{{name}}}}]\null{}
+Name of the attribute, including namespace.
+% \null and \mbox are tricks to induce different typesetting decisions
+\item[{\hyperlink{type:stringOrBlob}{stringOrBlob} {\ttfamily\itshape{{value}}}}]\null{}
+(optional) Value.
+\end{description}
+\noindent
git://scientific.zcu.cz / jra1mw.git / commitdiff