		Coresight - HW Assisted Tracing on ARM
		======================================

   Author:   Mathieu Poirier <mathieu.poirier@linaro.org>
   Date:     September 11th, 2014

Introduction
------------

Coresight is an umbrella of technologies allowing for the debugging of ARM
based SoC.  It includes solutions for JTAG and HW assisted tracing.  This
document is concerned with the latter.

HW assisted tracing is becoming increasingly useful when dealing with systems
that have many SoCs and other components like GPU and DMA engines.  ARM has
developed a HW assisted tracing solution by means of different components, each
being added to a design at synthesis time to cater to specific tracing needs.
Components are generally categorised as source, link and sinks and are
(usually) discovered using the AMBA bus.

"Sources" generate a compressed stream representing the processor instruction
path based on tracing scenarios as configured by users.  From there the stream
flows through the Coresight system (via ATB bus) using links that are connecting
the emanating source to a sink(s).  Sinks serve as endpoints to the Coresight
implementation, either storing the compressed stream in a memory buffer or
creating an interface to the outside world where data can be transferred to a
host without fear of filling up the onboard Coresight memory buffer.

At typical Coresight system would look like this:

  *****************************************************************
 **************************** AMBA AXI  ****************************===||
  *****************************************************************    ||
        ^                    ^                            |            ||
        |                    |                            *            **
     0000000    :::::     0000000    :::::    :::::    @@@@@@@    ||||||||||||
     0 CPU 0<-->: C :     0 CPU 0<-->: C :    : C :    @ STM @    || System ||
  |->0000000    : T :  |->0000000    : T :    : T :<--->@@@@@     || Memory ||
  |  #######<-->: I :  |  #######<-->: I :    : I :      @@@<-|   ||||||||||||
  |  # ETM #    :::::  |  # PTM #    :::::    :::::       @   |
  |   #####      ^ ^   |   #####      ^ !      ^ !        .   |   |||||||||
  | |->###       | !   | |->###       | !      | !        .   |   || DAP ||
  | |   #        | !   | |   #        | !      | !        .   |   |||||||||
  | |   .        | !   | |   .        | !      | !        .   |      |  |
  | |   .        | !   | |   .        | !      | !        .   |      |  *
  | |   .        | !   | |   .        | !      | !        .   |      | SWD/
  | |   .        | !   | |   .        | !      | !        .   |      | JTAG
  *****************************************************************<-|
 *************************** AMBA Debug APB ************************
  *****************************************************************
   |    .          !         .          !        !        .    |
   |    .          *         .          *        *        .    |
  *****************************************************************
 ******************** Cross Trigger Matrix (CTM) *******************
  *****************************************************************
   |    .     ^              .                            .    |
   |    *     !              *                            *    |
  *****************************************************************
 ****************** AMBA Advanced Trace Bus (ATB) ******************
  *****************************************************************
   |          !                        ===============         |
   |          *                         ===== F =====<---------|
   |   :::::::::                         ==== U ====
   |-->:: CTI ::<!!                       === N ===
   |   :::::::::  !                        == N ==
   |    ^         *                        == E ==
   |    !  &&&&&&&&&       IIIIIII         == L ==
   |------>&& ETB &&<......II     I        =======
   |    !  &&&&&&&&&       II     I           .
   |    !                    I     I          .
   |    !                    I REP I<..........
   |    !                    I     I
   |    !!>&&&&&&&&&       II     I           *Source: ARM ltd.
   |------>& TPIU  &<......II    I            DAP = Debug Access Port
           &&&&&&&&&       IIIIIII            ETM = Embedded Trace Macrocell
               ;                              PTM = Program Trace Macrocell
               ;                              CTI = Cross Trigger Interface
               *                              ETB = Embedded Trace Buffer
          To trace port                       TPIU= Trace Port Interface Unit
                                              SWD = Serial Wire Debug

While on target configuration of the components is done via the APB bus,
all trace data are carried out-of-band on the ATB bus.  The CTM provides
a way to aggregate and distribute signals between CoreSight components.

The Coresight framework provides a central point to represent, configure and
manage Coresight devices on a platform.  This first implementation centers on
the basic tracing functionality, enabling components such ETM/PTM, funnel,
replicator, TMC, TPIU and ETB.  Future work will enable more
intricate IP blocks such as STM and CTI.


Acronyms and Classification
---------------------------

Acronyms:

PTM:     Program Trace Macrocell
ETM:     Embedded Trace Macrocell
STM:     System trace Macrocell
ETB:     Embedded Trace Buffer
ITM:     Instrumentation Trace Macrocell
TPIU:    Trace Port Interface Unit
TMC-ETR: Trace Memory Controller, configured as Embedded Trace Router
TMC-ETF: Trace Memory Controller, configured as Embedded Trace FIFO
CTI:     Cross Trigger Interface

Classification:

Source:
   ETMv3.x ETMv4, PTMv1.0, PTMv1.1, STM, STM500, ITM
Link:
   Funnel, replicator (intelligent or not), TMC-ETR
Sinks:
   ETBv1.0, ETB1.1, TPIU, TMC-ETF
Misc:
   CTI


Device Tree Bindings
----------------------

See Documentation/devicetree/bindings/arm/coresight.txt for details.

As of this writing drivers for ITM, STMs and CTIs are not provided but are
expected to be added as the solution matures.


Framework and implementation
----------------------------

The Coresight framework provides a central point to represent, configure and
manage Coresight devices on a platform.  Any Coresight compliant device can
register with the framework for as long as they use the right APIs:

struct Coresight_device *coresight_register(struct coresight_desc *desc);
void coresight_unregister(struct coresight_device *csdev);

The registering function is taking a "struct coresight_device *csdev" and
register the device with the core framework.  The unregister function takes
a reference to a "struct coresight_device", obtained at registration time.

If everything goes well during the registration process the new devices will
show up under /sys/bus/coresight/devices, as showns here for a TC2 platform:

root:~# ls /sys/bus/coresight/devices/
replicator  20030000.tpiu    2201c000.ptm  2203c000.etm  2203e000.etm
20010000.etb         20040000.funnel  2201d000.ptm  2203d000.etm
root:~#

The functions take a "struct coresight_device", which looks like this:

struct coresight_desc {
        enum coresight_dev_type type;
        struct coresight_dev_subtype subtype;
        const struct coresight_ops *ops;
        struct coresight_platform_data *pdata;
        struct device *dev;
        const struct attribute_group **groups;
};


The "coresight_dev_type" identifies what the device is, i.e, source link or
sink while the "coresight_dev_subtype" will characterise that type further.

The "struct coresight_ops" is mandatory and will tell the framework how to
perform base operations related to the components, each component having
a different set of requirement.  For that "struct coresight_ops_sink",
"struct coresight_ops_link" and "struct coresight_ops_source" have been
provided.

The next field, "struct coresight_platform_data *pdata" is acquired by calling
"of_get_coresight_platform_data()", as part of the driver's _probe routine and
"struct device *dev" gets the device reference embedded in the "amba_device":

static int etm_probe(struct amba_device *adev, const struct amba_id *id)
{
 ...
 ...
 drvdata->dev = &adev->dev;
 ...
}

Specific class of device (source, link, or sink) have generic operations
that can be performed on them (see "struct coresight_ops").  The
"**groups" is a list of sysfs entries pertaining to operations
specific to that component only.  "Implementation defined" customisations are
expected to be accessed and controlled using those entries.

Last but not least, "struct module *owner" is expected to be set to reflect
the information carried in "THIS_MODULE".

How to use the tracer modules
-----------------------------

There is two ways to use the Coresight framework: 1) using the perf cmd line
tool and 2) interacting directly with the Coresight devices using the sysFS
interface.  The latter will slowly be faded out as more functionality become
available from the perf cmd line tool but for the time being both are still
supported.  The following sections provide details on using both methods.

1) Using perf framework:

Coresight tracers like ETM and PTM are represented using the Perf framework's
Performance Monitoring Unit (PMU).  As such the perf framework takes charge of
controlling when tracing happens based on when the process(es) of interest are
scheduled.  When configure in a system, Coresight PMUs will be listed when
queried by the perf command line tool:

linaro@linaro-nano:~$ ./perf list pmu

List of pre-defined events (to be used in -e):

  cs_etm//                                           [Kernel PMU event]

linaro@linaro-nano:~$

Regardless of the amount ETM/PTM IP block in a system (usually equal to the
amount of processor core), the "cs_etm" PMU will be listed only once.

Before a trace can be configured and started a Coresight sink needs to be
selected using the sysFS method (see below).  This is only temporary until
sink selection can be made from the command line tool.

linaro@linaro-nano:~$ ls /sys/bus/coresight/devices
20010000.etb  20030000.tpiu  20040000.funnel  2201c000.ptm
2201d000.ptm  2203c000.etm  2203d000.etm  2203e000.etm  replicator

linaro@linaro-nano:~$ echo 1 > /sys/bus/coresight/devices/20010000.etb/enable_sink

Once a sink has been selected configuring a Coresight PMU works the same way as
any other PMU.  As such tracing can happen for a single CPU, a group of CPU, per
thread or a combination of those:

linaro@linaro-nano:~$ perf record -e cs_etm// --per-thread <command>

linaro@linaro-nano:~$ perf record -C 0,2-3 -e cs_etm// <command>

Tracing limited to user and kernel space can also be used to narrow the amount
of collected traces:

linaro@linaro-nano:~$ perf record -e cs_etm//u --per-thread <command>

linaro@linaro-nano:~$ perf record -C 0,2-3 -e cs_etm//k <command>

As of this writing two ETM/PTM specific options have are available: cycle
accurate and timestamp (please refer to the Embedded Trace Macrocell reference
manual for details on these options).  By default both are disabled but using
the "cycacc" and "timestamp" mnemonic within the double '/' will see those
options configure for the upcoming trace run:

linaro@linaro-nano:~$ perf record -e cs_etm/cycacc/ --per-thread <command>

linaro@linaro-nano:~$ perf record -C 0,2-3 -e cs_etm/cycacc,timestamp/ <command>

The Coresight PMUs can be configured to work in "full trace" or "snapshot" mode.
In full trace mode trace acquisition is enabled from beginning to end with trace
data being recorded continuously:

linaro@linaro-nano:~$ perf record -e cs_etm// dd if=/dev/random of=./test.txt bs=1k count=1000

Since this can lead to a significant amount of data and because some devices are
limited in disk space snapshot mode can be used instead.  In snapshot mode
traces are still collected in the ring buffer but not communicated to user
space.  The ring buffer is allowed to wrap around, providing the latest
information before an event of interest happens.  Significant events are
communicated by sending a USR2 signal to the user space command line tool.
From there the tool will stop trace collection and harvest data from the ring
buffer before re-enabling traces.  Snapshot mode can be invoked using '-S' when
launching a trace collection:

linaro@linaro-nano:~$ perf record -S -e cs_etm// dd if=/dev/random of=./test.txt bs=1k count=1000

Trace data collected during trace runs ends up in the "perf.data" file.  Trace
configuration information necessary for trace decoding is also embedded in the
"perf.data" file.  Two new headers, 'PERF_RECORD_AUXTRACE_INFO' and
'PERF_RECORD_AUXTRACE' have been added to the list of event types in order to
find out where the different sections start.

It is worth noting that a set of metadata information exists for each tracer
that participated in a trace run.  As such if 5 processors have been engaged,
5 sets of metadata will be found in the perf.data file.  This is to ensure that
tracer decompression tools have all the information they need in order to
process the trace data.

Metadata information is collected directly from the ETM/PTM management registers
using the sysFS interface.  Since there is no way for the perf command line
tool to associate a CPU with a tracer, a symbolic link has been created between
the cs_etm sysFS event directory and each Coresight tracer:

linaro@linaro-nano:~$ ls /sys/bus/event_source/devices/cs_etm
cpu0  cpu1  cpu2  cpu3  cpu4  format  perf_event_mux_interval_ms
power  subsystem  type  uevent

linaro@linaro-nano:~$ ls /sys/bus/event_source/devices/cs_etm/cpu0/mgmt/
etmccer  etmccr  etmcr  etmidr  etmscr  etmtecr1  etmtecr2
etmteevr  etmtraceidr  etmtssvr

2) Using the sysFS interface:

Most, if not all, configuration registers are made available to users via the
sysFS interface.  Until all Coresight ETM drivers have been converted to perf,
it will also be possible to start and stop traces from sysFS.

As with the perf method described above, a Coresight sink needs to be identify
before trace collection can commence.  Using the sysFS method _only_, there is
no limit on the amount of sinks (nor sources) that can be enabled at
any given moment.  As a generic operation, all devices pertaining to the sink
class will have an "enable_sink" entry in sysfs:

root:/sys/bus/coresight/devices# ls
replicator  20030000.tpiu    2201c000.ptm  2203c000.etm  2203e000.etm
20010000.etb         20040000.funnel  2201d000.ptm  2203d000.etm
root:/sys/bus/coresight/devices# ls 20010000.etb
enable_sink  status  trigger_cntr
root:/sys/bus/coresight/devices# echo 1 > 20010000.etb/enable_sink
root:/sys/bus/coresight/devices# cat 20010000.etb/enable_sink
1
root:/sys/bus/coresight/devices#

At boot time the current etm3x driver will configure the first address
comparator with "_stext" and "_etext", essentially tracing any instruction
that falls within that range.  As such "enabling" a source will immediately
trigger a trace capture:

root:/sys/bus/coresight/devices# echo 1 > 2201c000.ptm/enable_source
root:/sys/bus/coresight/devices# cat 2201c000.ptm/enable_source
1
root:/sys/bus/coresight/devices# cat 20010000.etb/status
Depth:          0x2000
Status:         0x1
RAM read ptr:   0x0
RAM wrt ptr:    0x19d3   <----- The write pointer is moving
Trigger cnt:    0x0
Control:        0x1
Flush status:   0x0
Flush ctrl:     0x2001
root:/sys/bus/coresight/devices#

Trace collection is stopped the same way:

root:/sys/bus/coresight/devices# echo 0 > 2201c000.ptm/enable_source
root:/sys/bus/coresight/devices#

The content of the ETB buffer can be harvested directly from /dev:

root:/sys/bus/coresight/devices# dd if=/dev/20010000.etb \
of=~/cstrace.bin

64+0 records in
64+0 records out
32768 bytes (33 kB) copied, 0.00125258 s, 26.2 MB/s
root:/sys/bus/coresight/devices#

The file cstrace.bin can be decompressed using "ptm2human", DS-5 or Trace32.

Following is a DS-5 output of an experimental loop that increments a variable up
to a certain value.  The example is simple and yet provides a glimpse of the
wealth of possibilities that Coresight provides.

Info                                    Tracing enabled
Instruction     106378866       0x8026B53C      E52DE004        false   PUSH     {lr}
Instruction     0       0x8026B540      E24DD00C        false   SUB      sp,sp,#0xc
Instruction     0       0x8026B544      E3A03000        false   MOV      r3,#0
Instruction     0       0x8026B548      E58D3004        false   STR      r3,[sp,#4]
Instruction     0       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
Timestamp                                       Timestamp: 17106715833
Instruction     319     0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
Instruction     9       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
Instruction     7       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
Instruction     7       0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
Instruction     10      0x8026B54C      E59D3004        false   LDR      r3,[sp,#4]
Instruction     0       0x8026B550      E3530004        false   CMP      r3,#4
Instruction     0       0x8026B554      E2833001        false   ADD      r3,r3,#1
Instruction     0       0x8026B558      E58D3004        false   STR      r3,[sp,#4]
Instruction     0       0x8026B55C      DAFFFFFA        true    BLE      {pc}-0x10 ; 0x8026b54c
Instruction     6       0x8026B560      EE1D3F30        false   MRC      p15,#0x0,r3,c13,c0,#1
Instruction     0       0x8026B564      E1A0100D        false   MOV      r1,sp
Instruction     0       0x8026B568      E3C12D7F        false   BIC      r2,r1,#0x1fc0
Instruction     0       0x8026B56C      E3C2203F        false   BIC      r2,r2,#0x3f
Instruction     0       0x8026B570      E59D1004        false   LDR      r1,[sp,#4]
Instruction     0       0x8026B574      E59F0010        false   LDR      r0,[pc,#16] ; [0x8026B58C] = 0x80550368
Instruction     0       0x8026B578      E592200C        false   LDR      r2,[r2,#0xc]
Instruction     0       0x8026B57C      E59221D0        false   LDR      r2,[r2,#0x1d0]
Instruction     0       0x8026B580      EB07A4CF        true    BL       {pc}+0x1e9344 ; 0x804548c4
Info                                    Tracing enabled
Instruction     13570831        0x8026B584      E28DD00C        false   ADD      sp,sp,#0xc
Instruction     0       0x8026B588      E8BD8000        true    LDM      sp!,{pc}
Timestamp                                       Timestamp: 17107041535

How to use the STM module
-------------------------

Using the System Trace Macrocell module is the same as the tracers - the only
difference is that clients are driving the trace capture rather
than the program flow through the code.

As with any other CoreSight component, specifics about the STM tracer can be
found in sysfs with more information on each entry being found in [1]:

root@genericarmv8:~# ls /sys/bus/coresight/devices/20100000.stm
enable_source   hwevent_select  port_enable     subsystem       uevent
hwevent_enable  mgmt            port_select     traceid
root@genericarmv8:~#

Like any other source a sink needs to be identified and the STM enabled before
being used:

root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/20010000.etf/enable_sink
root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/20100000.stm/enable_source

From there user space applications can request and use channels using the devfs
interface provided for that purpose by the generic STM API:

root@genericarmv8:~# ls -l /dev/20100000.stm
crw-------    1 root     root       10,  61 Jan  3 18:11 /dev/20100000.stm
root@genericarmv8:~#

Details on how to use the generic STM API can be found here [2].

[1]. Documentation/ABI/testing/sysfs-bus-coresight-devices-stm
[2]. Documentation/trace/stm.txt
