CALICE DAQ/online software system instructions
The DAQ/online software can be used for acquiring data
through a VME system or a USB system. It can also produce
simulation data. The same software can then be used
for analysing data files
created previously with the system.
In the following, the
VME use is indicated by "online", the USB use is "onlilc",
and simulation use is "onlsim", while analysis use is "daquser".
- External packages installation
- SBS PCI-VME driver (online use only)
-
The driver software
needs to be installed into the Linux kernel. The driver straight out of
the box is unfortunately not usable and so a version should be downloaded
from the HAL web site (see below). The AHCAL and ECAL are currently (end 2005)
using V2.3.A (aka v2p3pA). The driver can be installed in any directory.
However, note that directly under this directory, it needs to have a
directory structure "installdir"/1003/v2.3.a/ for the make files to work.
This driver version also requires a modified version of the
routine btpapi.h to compile in C++;
the modifed version can be found
here and this should be copied to the
directory "installdir"/1003/v2.3.a/include/btpapi.h.
-
There are several jumpers which need to be set correctly on the VME
card before use; check the SBS manual for these. In addition to the
default settings, jumpers needed to be added to the ARB,
SYS (positions 4, 5 and 6), and T-INT (position 6) arrays.
-
The software comes with various utilities. To use these, go to the
directory "installdir"/1003/v2.3.a/src/. With the SBS cards installed
and powered up, then run bt_revs. This should communicate with the
VME card and report back the version numbers, etc. If everything is
working, it should give something like this.
-
The default SBS settings result in a large number of informational messages
being written to the kernel log file when doing VME block transfers. If
this becomes an issue, then they can be turned off using the bt_info
utility in the same directory as above. To do this, first check which
bits are set in the relevant memory area using
bt_info -p BT_INFO_TRACE
which should give something like the first part of
this. If bit 12 = 0x1000 = 4096 is set (as shown
by the 0x7... in the example shown), then the informational messages
are enabled. To turn them off, simply reset the value to the same
thing with this bit set to zero, using
bt_info -p BT_INFO_TRACE -v
This should give something like the middle part of
this. The last part is simply running the
original command again to check the value is as expected.
-
If you want to use another VME controller card which is supported by
HAL (see below) then this is possible but a little fiddly; contact Paul
directly for information on this.
- HAL VME interface (online use only)
-
The Hardware Access Library is a CERN interface
to various VME controllers (including the SBS 620). The software is documented
at the following
web page. The exact version should not be critical; the AHCAL and ECAL
are both currently (end 2005) using ver-03-06.
Again, the software can be installed in any directory. The installation
is best checked through use of the CALICE software after it is installed (see
below).
- USB_DAQ interface (onlilc use only)
-
Software to drive the USB_DAQ interface needs to be installed.
The code can be downloaded as a tarball from
here.
This should be untarred and will create a directory called decal_sw.
Go into this directory and source the decal_sw_setup.sh file to set
some environmental variables. Then to build the programs, source
the makes.sh file.
-
Communicating with the USB_DAQ requires a working version of udev to
be installed on the PC; most recent versions of Linux will have this.
Two files will need to be added to the udev rules lists, which requires
root access. The two files are for the standard USB_DAQ firmware
and the master USB_DAQ firmware. The files can be found
here and here,
respectively and these should be added to the directory
/etc/udev/rules.d.
-
If all is well, then with a USB_DAQ plugged in, then the
utilities decal_sw/TPAC1x/Tests/bin/Status and
decal_sw/TPACMasterController/Tests/bin/Status, for
the standard and master firmware respectively,
should return sensible values; see
here
and here
for example.
-
It is useful to add the required variables to LD_LIBRARY_LIB. This can
be done in tcsh in a similar way to shown
here.
- CLHEP and GEANT4 (onlsim use only)
-
These can be downloaded from HERE!!!!! and they need to be
installed according to their standard instructions.
During installation of GEANT4, select the option to put all headers
into one common directory.
-
It is useful to add the required variables to LD_LIBRARY_LIB. This can
be done in tcsh in a similar way to shown
here.
- ROOT graphics (all uses)
-
ROOT is used for histogramming online and is documented
at the following web page. A recent version
needs to be downloaded from this site: go to the "Download Binaries" link
and choose a recent version which is compatible with your Linux compiler.
The exact version should not matter as none of the more recent features are
used in CALICE; currently (end 2005) V4.02 and V5.02 are being used by the
AHCAL and ECAL respectively. The installation can be put anywhere on the
Linux PC.
-
Following installation, then define the environmental variable
ROOTSYS to point to the installation top-level directory, add the ROOT
libraries to the library path and add the ROOT binaries to the command
path. For tcsh this can be done as shown
here. To check the installation, try starting
a root interactive session, which should give something like
this.
- CALICE software installation
- Source installation
-
For online/onlilc/onlsim use,
in all the following, work within account caliceon with the properties:
uid=12841(caliceon) gid=3145(calice) groups=3145(calice). For daquser
use, any account can be used.
-
The central source code should be installed separately from the directories
which will be used to actually build and run the code. Currently, the source
should be downloaded as a tar'ed and gzip'ed
file. (In future, this will migrate into CVS.)
-
The file should unpack to give a
directory called code containing five subdirectories
(packages), namely daquser, onlilc, online, onlsim and records.
Within each of these, there
are bin, inc and src subdirectories plus some others specific to each package.
- Directory structures
-
There are four types of structure which are useful; three for acquiring data
and one for analysing it. The former is called "online", "onlilc" or "onlsim",
the latter
"daquser". The instructions for all are similar. First, create a new
directory in which to work and cd into the directory.
-
Soft-link (with the same names, except for onlilc and onlsim, see below)
the packages from the central source installation into the new
working directory as subdirectories. For a daquser structure, only the
records and daquser packages are needed while the others need one of
the online-like packages.
For an online structure, link the online package.
For onlilc and onlsim, link the relevant package
into the working directoty but call the soft link "online" in either case.
Hence, for all three acquiring data uses, there should be three soft
links called daquser, online and records.
-
Create three directories within the new working directory called bin, inc
and src.
-
If acquiring data (i.e. an online/onlilc/onlsim structure),
create a directory "data"
in the new working directory
and within data, create further subdirectories data/log, data/run
data/slw and data/sum. If onlsim, also create a data/sim directory.
Also create a file data/runNumber.txt which contains the
first run number you will want to use. The data directory can of course
be a soft link to a different directory if more convenient. If analysing
data (i.e. a daquser structure), then soft-link the data area containing
the run files which are to be analysed into the new working area as
a directory called "data", which should have the data/run
subdirectory below it, which is the only one needed for analysis.
-
If an online structure, then link in the SBS and HAL software directories.
For the SBS driver software, the link should be called "sbs" and should
point to "sbsinstalldir"/1003/v2.3.a. The HAL link should be called "hal"
and should point to the hal top level directory.
-
If an onlilc structure, then link in the USB_DAQ code directory
The link should be called "usbdaq" and should
point to the decal_sw directory.
-
If an onlsim structure, then link in the CLHEP and GEANT4 directories.
The former link should be called "clhep" and should
point to the CLHEP installation, while the latter should be called
"geant4" and point to the GEANT4 installation.
-
When everything is completed, then the whole online working directory should
look something like this
and the onlilc should
look something like this.
The daquser
working directory will be very similar but missing clhep, geant4, hal,
online, sbs or usbdaq links.
- Environmental variables and path:
-
No new environmental variables are
needed for the CALICE software. However,
it is convenient to add the executable
directories to the PATH environmental variable.
For tcsh, the way to do this is shown
here.
- Building executables
-
Within the working directory, then typing "build programname" should
make the executable programname in the bin directory. This can be run
by simply typing "programname". A simple example is build printHex, which
makes an executable which writes out its integer argument as both a hex and
binary number.
An example of its output is shown
here. A further example is printTime
which converts an Unix epoch time integer given as an argument
into a human-readable form.
-
For an online/onlilc/onlsim structure,
the hardware locations used for a particular system need to be
specified, as in the file daquser/src/runnerDefine.icc_COPY_THIS.
Copy this to inc/runnerDefine.icc in
working directory and edit it to
correspond to your system.
-
For an online/onlilc/onlsim structure, typing buildAll should
build the essential executables needed for the DAQ.
-
The utility
alive should then be run (simply type alive). For online,
this checks if any CRCs
in the crate can be seen. A typical output of this is shown
here. If you have
only one SBS PCI card, then the
other (PCI card 1) will correctly give an error stating it is not installed;
this is expected. For onlilc, then alive should give something like
this.
-
When editing code, copy it to the local working directory src or inc
subdirectories first. These versions will be taken in preference to the
versions in the central code directories and this allows code updates
without losing your changes.
- Online and onlilc operations
- Background tasks
-
The main program for the online DAQ executes in the background. To
start this from the online working directory, type startUp.
-
The actual
program called runner should appear in the process list (use ps to see
it).
-
A pair of output (i.e. from cout) and error (i.e. from cerr)
files will also be created in
data/log/LogXXXXXXXXXX.out and .err respectively, where XXXXXXXXXX is
the Unix epoch time of the startUp command. (Use printTime to see
what date the XXXXXXXXXX value corresponds to.)
Soft-links called ActiveLog.out and .err are also created to point
to the current log file, so it is more convenient to use these
rather than the explicit data/log file name.
Note, some of the SBS
information messages are sent to cerr so they appear in the .err file;
this is normal.
-
Finally, a file data/slw/SlwYYYYYYYYYY.bin will be created; this
contains the slow controls and readout data and the part of the
program collecting this is always running. The value of YYYYYYYYYY
is again the Unix epoch time of the start of the slow monitoring run
which may differ by a second or so from the log file time.
- Run control
-
To start a data-taking run, then the runStart program is used. This
has several arguments; type runStart -h to see them all. You need to
choose a run type; the list of available run types is
given in a the file RunControlTypes.txt created at startUp.
The current list for onlilc is
here.
A brief explanation of what they do and the meaning of the
version numbers is given
here or by running the program runTypeComment, although this
has few fully commented run types.
For example, a basic online run is crcNoise, which simply reads the boards
as if they have no cables connected. To run this, type
runStart -w -t crcNoise -v 5 for version 5.
-
If the -w option for runStart was not selected, then the run data records
are written to disk as data/run/RunNNNNNN.MMM.bin. Here, NNNNNN is the
run number (as originally specified in the data/runNumber.txt file,
which is incremented for each run written to disk). MMM is the file
number within the run starting from 000.
Files are kept below 2GBytes to prevent large filesize
issues in Linux so a single run, which can have more than 2GBytes of data,
is split between several files which number incrementally.
-
The progress of a run can be checked using the runMonitor program. This
can be run at any time but it is convenient to run it
continuously in a small window
in the corner of the terminal being used. It prints out numbers of
records seen in the whole job, the current run and the current
configuration period within the run. To stop the program, Ctrl-C in
the window it is running in.
-
Some runs have a finite series of configuration periods and events
within those periods and so will complete by themselves. Others
will run (effectively) for ever. To stop any run currently in
progress, type runEnd. The runner program will then default
back to a slow monitoring run.
To stop the current configuration period within
the run, type configurationEnd. The next configuration period will
then start, or if all have then been completed, the run will end.
-
To stop the whole runner background task, then type shutDown. This
should always be done before changing any of the hardware, powering
down the VME crate, etc. If the run crashes completely, then type
panic to clean up lock files, etc, before doning the next startUp.
Usually startUp will not work unless this is done following a crash.
- Run monitoring
-
Histograms of the current run can be seen using ROOT. They are
filled by the runner program and stored in shared memory.
For example, to see the basic mean and rms histograms
per CRC, type "hstDisplay HstCrcNoise"
and one
canvas per CRC will appear. They continually refresh and take up
non-negligible CPU so it is best not to leave them running continuously.
To close them, go to the "File" menu on any of the canvases and select
"Quit ROOT".
For a list of available
displays, type "hstDisplay -h" and for any given display for further
information on options for that display, e.g. type "hstDisplay -h HstCrcNoise"
- Simulation (onlsim) use
- Data files
-
The simulation itself first produces a file in the data/sim directory
called SimXXXXXX.YYY.bin
where XXXXXX is the run number
and YYY is 000, 001, 002, etc, for the
various files in a run. The digitisation then processes these
to files in data/run called RunXXXXXX.YYY.bin. Note the number of
files in a run could increase as digitisation adds more data to the
records. There will be correspondingly fewer records in each file
so there will not be a one-to-one correspondence of records between the Sim
and Run files numbered with YYY.
- Offline (daquser) use
- Data files
-
To run over data, then the raw data "bin" files will need to be copied
to the machine being used. These are all accessible from the DESY
dCache. Contact Paul or Roman for further information.
-
For runs in 2004/5, each run corresponded to one bin file, with a name
RunXXXXXX.bin, where XXXXXX was the run number. Since 2006, a run can
continue for more than one 2GByte file and the bin files are now
named RunXXXXXX.YYY.bin, where YYY is 000, 001, 002, etc, for the
various files in a run.
- Example and utility programs
-
The "count" program does a simple count of the types of records in
a run. The code is in daquser/src/count.cc
and is made using "build count". To run it, type "count XXXXXX"
for run number XXXXXX.
Typical output for a run consisting of two files is shown
here.
-
The "example" program shows a few basic steps for accessing
data from the bin files.
The code is in daquser/src/example.cc
This will print out the DaqRunStart subrecord
at the beginning of the run (which says what sort of run it is),
the DaqRunEnd subrecord (which says how long the run was) and the
same count table as above for all the different types of records in the run.
Typical output for a short crcNoise run is shown
here.
-
An example of code which accesses the ADC data is
daquser/inc/hst/HstCrcNoiseShm.hh which is mainly for producing mean and rms
plots for the online monitoring (see above under Run Monitoring).
This can also be run offline
using the main program "hstGeneric". This puts the data into shared
memory, just as for the online, and the histograms can be
displayed using the same hstDisplay utility as for the
online case. Do not run hstGeneric while taking data or the online
histogram shared memory will be corrupted.
The hstDisplay program needs to be run simultaneously
to the hstGeneric program.
The code in
HstCrcNoise.hh can be studied to see how the ADC data
subrecords are obtained from the record.
Go to the CALICE-UK home page