LB notifications
----------------

LB infrastructure enables its users to be notified when something interesting happens on a LB server (typically job status change). It enables the user not to poll LB server periodically to find out whatever he/she is interesting in but confortable wait and let the server itself inform you. If user wants to receive the notifications, he/she must first register to the LB notification infarastructure.

User registers via some notification client. (program that uses LB client API to handle notifications). He/she  must specify a condition under which the notification is sent. For example - job XY reaches status DONE. Currently, one or more JOBID's are required in the condition and only a single occurence of a specific attribute is allowed among ANDed conditions. The condition is then delivered to LB server where it is stored. At the same time, server generates an unique notification ID for such registration and returns it to the user.

The registration exists only for limited amount of time. The validity is returned by LB server together with notification ID when registering. During this period user can attach to server and receive notifications, change conditions which triger notification, prolong validity of the registration, or remove the registration from LB server. For all such operation you have to supply the notification ID.


While the registration is valid, user is able repeatable connect to LB server from different places in the net and continue receiving notifications associated with given notification ID. Notifications generated during the time you are not receiving them are stored and waiting when you reconnect. When you reconnect from another place (another IP or PORT), you will receive all not delivered notifications and continue receiving new ones.


How does it work?
-----------------

The request for notification is delivered to LB server. The notification condition specified in the request is stored in LB database and whenever a new event arrives to the LB server, all notification conditions are tested. Always when some of them is true, corresponding notification is sent.

For a notification delivery, a special deamon, a notification interlogger is used. It stores notifications in files and periodically tries to deliver them to the adress of the last connection of notification client. If the user changes the place (IP or PORT) where the client listens, LB server instuct the notification deamon to change delivery destination of all pending notifications. 



Notification test
-----------------

Steps bellow describe basic testing procedure of the notification
system by creating a fake job record in the LB server, registering
a notification on any state change of this job, and forcing the job state
change.

Our example program 'glite-lb-notify' is used as a notification
client here. It uses mentioned client API calls to manipulate with
registrations. 

LB server supporting notifications is running on skurut68-2.cesnet.cz:9100,
local-logger used for sending of events (there's no need to run it on the same 
host) on skurut68-2.cesnet.cz:9002.

1) Register a job
        examples/glite-lb-job_reg -m skurut68-2.cesnet.cz:9100 -s UserInterface

      -> returns jobid and sequence code used in next steps

EDG_JOBID="https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q"
EDG_WL_SEQUENCE="UI=000002:NS=0000000000:WM=000000:BH=0000000000:JSS=000000:LM=000000:LRMS=000000:APP=000000:LBS=000000"


2) Create notification for this jobid

       GLITE_WMS_NOTIF_SERVER=skurut68-2.cesnet.cz:9100 bin/glite-lb-notify
new -j https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q

     -> returns notification id
     notification ID: https://skurut68-2.cesnet.cz:9100/NOTIF:tOsgB19Wz-M884anZufyUw

3) Start process waiting for notifications on job state change

    GLITE_WMS_NOTIF_SERVER=skurut68-2.cesnet.cz:9100 bin/glite-lb-notify
receive -i 120 https://skurut68-2.cesnet.cz:9100/NOTIF:tOsgB19Wz-M884anZufyUw

4) Change status of your job

4a)
    - using glite-lb-logevent (sequence code passed from glite-lb-job_reg
      and between glite-lb-logevent calls

  GLITE_WMS_LOG_DESTINATION=skurut68-2.cesnet.cz:9002 bin/glite-lb-logevent \
    -j https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q \
    -e Running  -s LogMonitor --node hokus \
    -c "UI=000002:NS=0000000000:WM=000000:BH=0000000000:JSS=000000:LM=000000:LRMS=000000:APP=000000:LBS=000000"
   
     -> returns seq. code for later usage:
     UI=000002:NS=0000000000:WM=000000:BH=0000000000:JSS=000000:LM=000002:LRMS=000000:APP=000000:LBS=000000

      
  GLITE_WMS_LOG_DESTINATION=skurut68-2.cesnet.cz:9002 bin/glite-lb-logevent \
    -j https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q \
    -e Done -s LogMonitor --status_code=OK  --reason="hotovo"  --exit_code=0 \
    -c "UI=000002:NS=0000000000:WM=000000:BH=0000000000:JSS=000000:LM=000002:LRMS=000000:APP=000000:LBS=000000"

   -> returns 
 UI=000002:NS=0000000000:WM=000000:BH=0000000000:JSS=000000:LM=000004:LRMS=000000:APP=000000:LBS=000000
  
4b)
    - using examples/glite-lb-running.sh (sends corresponding set of events):

   GLITE_WMS_LOG_DESTINATION=skurut68-2.cesnet.cz:9002 \
     examples/glite-lb-running.sh \
     -j https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q

5) Watch incomming notifications. 

	You should receive several notification, each correspondig 
	with job status change:

https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q        Running /DC=cz/DC=cesnet-ca/O=Masaryk University/CN=Miroslav Ruda

https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q        Done /DC=cz/DC=cesnet-ca/O=Masaryk University/CN=Miroslav Ruda


When you let notification client running several minutes without any incomming notification, it will finish and remove your registration from the server automatically. 

6) Destroy notification
  
    GLITE_WMS_NOTIF_SERVER=skurut68-2.cesnet.cz:9100 bin/glite-lb-notify \
       drop https://skurut68-2.cesnet.cz:9100/D1qbFGwvXLnd927JOcja1Q
