pick_events

top

09.24.02 The first problem is the fraction of pick_events that are successfully returned by the utility. We submit pick_events jobs that attempt to access 99 or 100 events at a time. The first time we submit a list the event return rate varies from 2% to 100% with a peak at about 85%. Subsequent submissions of the list usually return an increasing fraction of the events. For instance, in one attempt to access 99 events, I go 43 the first time, 91 the second attempt, and 99 on the third attempt. Those results aren't untypical. when the raw-data file was already on SAM cache. I am concerned that my measurement for the success rate of pick_events is typical of other kinds of SAM access.

non-raw data, from any tier.

full design of pickevents involving pooling of common requests. caching all for some period of time - recently used files.
10.29.02 There are three issues related to pick events 1) users have reported unreliable results which needs to be traced and fixed, 2) picking all tiers of data in addition to raw needs to be be enabled, and 3) the general pick events design and implementation to include request pooling and event caching and archiving needs to be completed. Guidance from d0 on the urgency for 3) is needed.

batchdesign

top

minusbug

top

import classes

top

The WZ group wants to form several datasets containing different kinds of events. For instance, events with a W to muon, Z to muons, W to electron, Z to electrons, and another, say, overlap. We want to collect the events, perhaps thumbnails or DSTs and return them to SAM for group storage and access by type.

I think the metadata may contain the provision for this but that, perhaps, SAM can't handle the information yet. Here is an example metadata I used to return some picked-events, which is probably not the best example possible, but is the one I have in hand. The file is called mrg_pick_dimuon_081302_001.raw.meta.py

from import_classes import *
TheFile = ProcessedFile( name = 'mrg_pick_dimuon_081302_001.raw',
sizeK = 226622,
events = Events(13961065, 15926, 2311),
stream = 'pick-event-p10dimcand',
tier = 'raw-bygroup',
start_time = '08/13/2002 13:03:05',
end_time = '08/13/2002 13:39:55',
pid = 831675,
parents = ['pick_d51.dat', ... ,'pick_d38.dat'])

We see the datatier "raw-bygroup" and stream
"pick-event-p10dimcand". I believe this field stream could be used to identify the type of event, e.g., W to muons, etc ..., so that one could access events only of that type.
10.29.02 2. import classes: General evaluation of the needs of the experiment have been underway. This includes use cases that have been recently presented and also using the framework of what is now mcrunjob for doing workflow control and metadata generation for applications beyond MC. This includes meeting with d0 users and developers outside of sam before enough is understood to proceed. Estimate at lease 2 weeks to have requirements written down and design started. Probably a month of coding and testing after that to complete the project.

batch

top

dcache

top

10.04.02 This is necessary in order to rate-adapt the 9940b tape drives to disk and also important for issues involving remote sites. Done when in production.
10.29.02 This project has two parts, 1) testing the server that has been provided for d0 and 2) turning dcache on for general d0 use on central analysis and other stations. We are still waiting for the ISD group to update the existing server to the latest software and turn it over to us. We estimate one week to setup the test on d0test and see how the system works under load. Assuming everything looks good, we will discuss the next steps, and begin to use this server for Central-analysis and/or farm. We will then probably generally phase it in for remote station access to enstore data.