tracing tools for 6.2

- New tool "rv" for starting and stopping runtime verification:
 
    example: ./rv mon wip -r printk -v
 
    Enables the wake-in-preempt monitor and the printk reactor in verbose mode
 
 - Fix exit status of rtla usage() calls
 -----BEGIN PGP SIGNATURE-----
 
 iIoEABYIADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCY5e8IRQccm9zdGVkdEBn
 b29kbWlzLm9yZwAKCRAp5XQQmuv6qpicAQD+Ov5Exy0H19l35oK0ojvG7LUeQpiY
 f50AOTOlA71KqAD/fI9cdCT+BfmD6b2lD5iZc0k+VqU+XtCZj0iHsq2H7go=
 =wxpw
 -----END PGP SIGNATURE-----

Merge tag 'trace-tools-6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace

Pull tracing tools updates from Steven Rostedt:

 - New tool "rv" for starting and stopping runtime verification.
   Example:

      ./rv mon wip -r printk -v

   Enables the wake-in-preempt monitor and the printk reactor in verbose
   mode

 - Fix exit status of rtla usage() calls

* tag 'trace-tools-6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
  Documentation/rv: Add verification/rv man pages
  tools/rv: Add in-kernel monitor interface
  rv: Add rv tool
  rtla: Fix exit status when returning from calls to usage()

Documentation/tools/index.rst

@@ -11,6 +11,7 @@ more additions are needed here:
    :maxdepth: 1
 
    rtla/index
+   rv/index
 
 .. only::  subproject and html
 

Documentation/tools/rv/Makefile

@@ -0,0 +1,52 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+INSTALL		?= install
+RM		?= rm -f
+RMDIR		?= rmdir --ignore-fail-on-non-empty
+
+PREFIX		?= /usr/share
+MANDIR		?= $(PREFIX)/man
+MAN1DIR		= $(MANDIR)/man1
+
+MAN1_RST	= $(wildcard rv*.rst)
+
+_DOC_MAN1	= $(patsubst %.rst,%.1,$(MAN1_RST))
+DOC_MAN1	= $(addprefix $(OUTPUT),$(_DOC_MAN1))
+
+RST2MAN_DEP	:= $(shell command -v rst2man 2>/dev/null)
+RST2MAN_OPTS	+= --verbose
+
+TEST_RST2MAN = $(shell sh -c "rst2man --version > /dev/null 2>&1 || echo n")
+
+$(OUTPUT)%.1: %.rst
+ifndef RST2MAN_DEP
+	$(info ********************************************)
+	$(info ** NOTICE: rst2man not found)
+	$(info **)
+	$(info ** Consider installing the latest rst2man from your)
+	$(info ** distribution, e.g., 'dnf install python3-docutils' on Fedora,)
+	$(info ** or from source:)
+	$(info **)
+	$(info **  https://docutils.sourceforge.io/docs/dev/repository.html )
+	$(info **)
+	$(info ********************************************)
+	$(error NOTICE: rst2man required to generate man pages)
+endif
+	rst2man $(RST2MAN_OPTS) $< > $@
+
+man1: $(DOC_MAN1)
+man: man1
+
+clean:
+	$(RM) $(DOC_MAN1)
+
+install: man
+	$(INSTALL) -d -m 755 $(DESTDIR)$(MAN1DIR)
+	$(INSTALL) -m 644 $(DOC_MAN1) $(DESTDIR)$(MAN1DIR)
+
+uninstall:
+	$(RM) $(addprefix $(DESTDIR)$(MAN1DIR)/,$(_DOC_MAN1))
+	$(RMDIR) $(DESTDIR)$(MAN1DIR)
+
+.PHONY: man man1 clean install uninstall
+.DEFAULT_GOAL := man

Documentation/tools/rv/common_appendix.rst

@@ -0,0 +1,16 @@
+REPORTING BUGS
+==============
+
+Report bugs to <linux-kernel@vger.kernel.org>
+and <linux-trace-devel@vger.kernel.org>
+
+LICENSE
+=======
+
+**rv** is Free Software licensed under the GNU GPLv2
+
+COPYING
+=======
+
+Copyright \(C) 2022 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).

Documentation/tools/rv/common_ikm.rst

@@ -0,0 +1,21 @@
+**-h**, **--help**
+
+        Print the monitor's options and the available reactors list.
+
+**-r**, **--reactor** *reactor*
+
+        Enables the *reactor*. See **-h** for a list of available reactors.
+
+**-s**, **--self**
+
+        When tracing (**-t**), also print the events that happened during the **rv**
+        command itself. If the **rv** command itself generates too many events,
+        the tool might get busy processing its own events only.
+
+**-t**, **--trace**
+
+        Trace monitor's events and error.
+
+**-v**, **--verbose**
+
+        Print debug messages.

Documentation/tools/rv/index.rst

@@ -0,0 +1,24 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============================
+Runtime verification (rv) tool
+==============================
+
+**rv** tool provides the interface for a collection of runtime verification
+(rv) monitors.
+
+.. toctree::
+   :maxdepth: 1
+
+   rv
+   rv-list
+   rv-mon
+   rv-mon-wip
+   rv-mon-wwnr
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`

Documentation/tools/rv/rv-list.rst

@@ -0,0 +1,43 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======
+rv-list
+=======
+-----------------------
+List available monitors
+-----------------------
+
+:Manual section: 1
+
+SYNOPSIS
+========
+
+**rv list** [*OPTIONS*]
+
+DESCRIPTION
+===========
+
+The **rv list** command prints all available monitors. These monitors
+can be enabled using the **rv mon** command.
+
+OPTIONS
+=======
+
+**-h**, **--help**
+
+        Print help menu.
+
+SEE ALSO
+========
+
+**rv**\(1), **rv-mon**\(1)
+
+Linux kernel *RV* documentation:
+<https://www.kernel.org/doc/html/latest/trace/rv/index.html>
+
+AUTHOR
+======
+
+Written by Daniel Bristot de Oliveira <bristot@kernel.org>
+
+.. include:: common_appendix.rst

Documentation/tools/rv/rv-mon-wip.rst

@@ -0,0 +1,44 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========
+rv-mon-wip
+==========
+----------------------------
+Wakeup In Preemptive monitor
+----------------------------
+
+:Manual section: 1
+
+SYNOPSIS
+========
+
+**rv mon wip** [*OPTIONS*]
+
+DESCRIPTION
+===========
+
+The wakeup in preemptive (**wip**) monitor is a sample per-cpu monitor that
+checks if the wakeup events always take place with preemption disabled.
+
+See kernel documentation for further information about this monitor:
+<https://docs.kernel.org/trace/rv/monitor_wip.html>
+
+OPTIONS
+=======
+
+.. include:: common_ikm.rst
+
+SEE ALSO
+========
+
+**rv**\(1), **rv-mon**\(1)
+
+Linux kernel *RV* documentation:
+<https://www.kernel.org/doc/html/latest/trace/rv/index.html>
+
+AUTHOR
+======
+
+Written by Daniel Bristot de Oliveira <bristot@kernel.org>
+
+.. include:: common_appendix.rst

Documentation/tools/rv/rv-mon-wwnr.rst

@@ -0,0 +1,43 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===========
+rv-mon-wwnr
+===========
+--------------------------------
+Wakeup While Not Running monitor
+--------------------------------
+
+:Manual section: 1
+
+SYNOPSIS
+========
+
+**rv mon wip** [*OPTIONS*]
+
+DESCRIPTION
+===========
+
+The wakeup while not running (**wwnr**) is a per-task sample monitor.
+
+See kernel documentation for further information about this monitor:
+<https://docs.kernel.org/trace/rv/monitor_wwnr.html>
+
+OPTIONS
+=======
+
+.. include:: common_ikm.rst
+
+SEE ALSO
+========
+
+**rv**\(1), **rv-mon**\(1)
+
+Linux kernel *RV* documentation:
+<https://www.kernel.org/doc/html/latest/trace/rv/index.html>
+
+AUTHOR
+======
+
+Written by Daniel Bristot de Oliveira <bristot@kernel.org>
+
+.. include:: common_appendix.rst

Documentation/tools/rv/rv-mon.rst

@@ -0,0 +1,55 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======
+rv-list
+=======
+-----------------------
+List available monitors
+-----------------------
+
+:Manual section: 1
+
+SYNOPSIS
+========
+
+**rv mon** [*-h*] **monitor_name** [*-h*] [*MONITOR OPTIONS*]
+
+DESCRIPTION
+===========
+
+The **rv mon** command runs the monitor named *monitor_name*. Each monitor
+has its own set of options. The **rv list** command shows all available
+monitors.
+
+OPTIONS
+=======
+
+**-h**, **--help**
+
+        Print help menu.
+
+AVAILABLE MONITORS
+==================
+
+The **rv** tool provides the interface for a set of monitors. Use the
+**rv list** command to list all available monitors.
+
+Each monitor has its own set of options. See man **rv-mon**-*monitor_name*
+for details about each specific monitor. Also, running **rv mon**
+**monitor_name** **-h** display the help menu with the available
+options.
+
+SEE ALSO
+========
+
+**rv**\(1), **rv-mon**\(1)
+
+Linux kernel *RV* documentation:
+<https://www.kernel.org/doc/html/latest/trace/rv/index.html>
+
+AUTHOR
+======
+
+Written by Daniel Bristot de Oliveira <bristot@kernel.org>
+
+.. include:: common_appendix.rst

Documentation/tools/rv/rv.rst

@@ -0,0 +1,63 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==
+rv
+==
+--------------------
+Runtime Verification
+--------------------
+
+:Manual section: 1
+
+SYNOPSIS
+========
+
+**rv** *COMMAND* [*OPTIONS*]
+
+DESCRIPTION
+===========
+
+Runtime Verification (**RV**) is a lightweight (yet rigorous) method
+for formal verification with a practical approach for complex systems.
+Instead of relying on a fine-grained model of a system (e.g., a
+re-implementation a instruction level), RV works by analyzing the trace
+of the system's actual execution, comparing it against a formal
+specification of the system behavior.
+
+The **rv** tool provides the interface for a collection of runtime
+verification (rv) monitors.
+
+COMMANDS
+========
+
+**list**
+
+        List all available monitors.
+
+**mon**
+
+        Run monitor.
+
+OPTIONS
+=======
+
+**-h**, **--help**
+
+        Display the help text.
+
+For other options, see the man page for the corresponding command.
+
+SEE ALSO
+========
+
+**rv-list**\(1), **rv-mon**\(1)
+
+Linux kernel *RV* documentation:
+<https://www.kernel.org/doc/html/latest/trace/rv/index.html>
+
+AUTHOR
+======
+
+Daniel Bristot de Oliveira <bristot@kernel.org>
+
+.. include:: common_appendix.rst

tools/tracing/rtla/src/osnoise.c

@@ -903,7 +903,7 @@ out_err:
 	return NULL;
 }
 
-static void osnoise_usage(void)
+static void osnoise_usage(int err)
 {
 	int i;
 
@@ -923,7 +923,7 @@ static void osnoise_usage(void)
 
 	for (i = 0; msg[i]; i++)
 		fprintf(stderr, "%s\n", msg[i]);
-	exit(1);
+	exit(err);
 }
 
 int osnoise_main(int argc, char *argv[])
@@ -941,8 +941,7 @@ int osnoise_main(int argc, char *argv[])
 	}
 
 	if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) {
-		osnoise_usage();
-		exit(0);
+		osnoise_usage(0);
 	} else if (strncmp(argv[1], "-", 1) == 0) {
 		/* the user skipped the tool, call the default one */
 		osnoise_top_main(argc, argv);
@@ -956,6 +955,6 @@ int osnoise_main(int argc, char *argv[])
 	}
 
 usage:
-	osnoise_usage();
+	osnoise_usage(1);
 	exit(1);
 }

tools/tracing/rtla/src/rtla.c

@@ -14,7 +14,7 @@
 /*
  * rtla_usage - print rtla usage
  */
-static void rtla_usage(void)
+static void rtla_usage(int err)
 {
 	int i;
 
@@ -33,7 +33,7 @@ static void rtla_usage(void)
 
 	for (i = 0; msg[i]; i++)
 		fprintf(stderr, "%s\n", msg[i]);
-	exit(1);
+	exit(err);
 }
 
 /*
@@ -70,11 +70,9 @@ int main(int argc, char *argv[])
 		goto usage;
 
 	if (strcmp(argv[1], "-h") == 0) {
-		rtla_usage();
-		exit(0);
+		rtla_usage(0);
 	} else if (strcmp(argv[1], "--help") == 0) {
-		rtla_usage();
-		exit(0);
+		rtla_usage(0);
 	}
 
 	retval = run_command(argc, argv, 1);
@@ -82,6 +80,6 @@ int main(int argc, char *argv[])
 		exit(0);
 
 usage:
-	rtla_usage();
+	rtla_usage(1);
 	exit(1);
 }

tools/tracing/rtla/src/timerlat.c

@@ -14,7 +14,7 @@
 
 #include "timerlat.h"
 
-static void timerlat_usage(void)
+static void timerlat_usage(int err)
 {
 	int i;
 
@@ -34,7 +34,7 @@ static void timerlat_usage(void)
 
 	for (i = 0; msg[i]; i++)
 		fprintf(stderr, "%s\n", msg[i]);
-	exit(1);
+	exit(err);
 }
 
 int timerlat_main(int argc, char *argv[])
@@ -52,8 +52,7 @@ int timerlat_main(int argc, char *argv[])
 	}
 
 	if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) {
-		timerlat_usage();
-		exit(0);
+		timerlat_usage(0);
 	} else if (strncmp(argv[1], "-", 1) == 0) {
 		/* the user skipped the tool, call the default one */
 		timerlat_top_main(argc, argv);
@@ -67,6 +66,6 @@ int timerlat_main(int argc, char *argv[])
 	}
 
 usage:
-	timerlat_usage();
+	timerlat_usage(1);
 	exit(1);
 }

tools/verification/rv/Makefile

@@ -0,0 +1,141 @@
+NAME	:=	rv
+# Follow the kernel version
+VERSION :=	$(shell cat VERSION 2> /dev/null || make -sC ../../.. kernelversion | grep -v make)
+
+# From libtracefs:
+# Makefiles suck: This macro sets a default value of $(2) for the
+# variable named by $(1), unless the variable has been set by
+# environment or command line. This is necessary for CC and AR
+# because make sets default values, so the simpler ?= approach
+# won't work as expected.
+define allow-override
+  $(if $(or $(findstring environment,$(origin $(1))),\
+            $(findstring command line,$(origin $(1)))),,\
+    $(eval $(1) = $(2)))
+endef
+
+# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix.
+$(call allow-override,CC,$(CROSS_COMPILE)gcc)
+$(call allow-override,AR,$(CROSS_COMPILE)ar)
+$(call allow-override,STRIP,$(CROSS_COMPILE)strip)
+$(call allow-override,PKG_CONFIG,pkg-config)
+$(call allow-override,LD_SO_CONF_PATH,/etc/ld.so.conf.d/)
+$(call allow-override,LDCONFIG,ldconfig)
+
+INSTALL	=	install
+MKDIR	=	mkdir
+FOPTS	:=	-flto=auto -ffat-lto-objects -fexceptions -fstack-protector-strong \
+		-fasynchronous-unwind-tables -fstack-clash-protection
+WOPTS	:= 	-Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -Wno-maybe-uninitialized
+
+TRACEFS_HEADERS	:= $$($(PKG_CONFIG) --cflags libtracefs)
+
+CFLAGS	:=	-O -g -DVERSION=\"$(VERSION)\" $(FOPTS) $(MOPTS) $(WOPTS) $(TRACEFS_HEADERS) $(EXTRA_CFLAGS) -I include
+LDFLAGS	:=	-ggdb $(EXTRA_LDFLAGS)
+LIBS	:=	$$($(PKG_CONFIG) --libs libtracefs)
+
+SRC	:=	$(wildcard src/*.c)
+HDR	:=	$(wildcard src/*.h)
+OBJ	:=	$(SRC:.c=.o)
+DIRS	:=	src
+FILES	:=	Makefile README.txt
+CEXT	:=	bz2
+TARBALL	:=	$(NAME)-$(VERSION).tar.$(CEXT)
+TAROPTS	:=	-cvjf $(TARBALL)
+BINDIR	:=	/usr/bin
+DATADIR	:=	/usr/share
+DOCDIR	:=	$(DATADIR)/doc
+MANDIR	:=	$(DATADIR)/man
+LICDIR	:=	$(DATADIR)/licenses
+SRCTREE	:=	$(or $(BUILD_SRC),$(CURDIR))
+
+# If running from the tarball, man pages are stored in the Documentation
+# dir. If running from the kernel source, man pages are stored in
+# Documentation/tools/rv/.
+ifneq ($(wildcard Documentation/.*),)
+DOCSRC	=	Documentation/
+else
+DOCSRC	=	$(SRCTREE)/../../../Documentation/tools/rv/
+endif
+
+LIBTRACEEVENT_MIN_VERSION = 1.5
+LIBTRACEFS_MIN_VERSION = 1.3
+
+.PHONY:	all warnings show_warnings
+all:	warnings rv
+
+TEST_LIBTRACEEVENT = $(shell sh -c "$(PKG_CONFIG) --atleast-version $(LIBTRACEEVENT_MIN_VERSION) libtraceevent > /dev/null 2>&1 || echo n")
+ifeq ("$(TEST_LIBTRACEEVENT)", "n")
+WARNINGS = show_warnings
+MISSING_LIBS += echo "**   libtraceevent version $(LIBTRACEEVENT_MIN_VERSION) or higher";
+MISSING_PACKAGES += "libtraceevent-devel"
+MISSING_SOURCE += echo "**  https://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git/ ";
+endif
+
+TEST_LIBTRACEFS = $(shell sh -c "$(PKG_CONFIG) --atleast-version $(LIBTRACEFS_MIN_VERSION) libtracefs > /dev/null 2>&1 || echo n")
+ifeq ("$(TEST_LIBTRACEFS)", "n")
+WARNINGS = show_warnings
+MISSING_LIBS += echo "**   libtracefs version $(LIBTRACEFS_MIN_VERSION) or higher";
+MISSING_PACKAGES += "libtracefs-devel"
+MISSING_SOURCE += echo "**  https://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git/ ";
+endif
+
+define show_dependencies
+	@echo "********************************************";				\
+	echo "** NOTICE: Failed build dependencies";					\
+	echo "**";									\
+	echo "** Required Libraries:";							\
+	$(MISSING_LIBS)									\
+	echo "**";									\
+	echo "** Consider installing the latest libtracefs from your";			\
+	echo "** distribution, e.g., 'dnf install $(MISSING_PACKAGES)' on Fedora,";	\
+	echo "** or from source:";							\
+	echo "**";									\
+	$(MISSING_SOURCE)								\
+	echo "**";									\
+	echo "********************************************"
+endef
+
+show_warnings:
+	$(call show_dependencies);
+
+ifneq ("$(WARNINGS)", "")
+ERROR_OUT = $(error Please add the necessary dependencies)
+
+warnings: $(WARNINGS)
+	$(ERROR_OUT)
+endif
+
+rv: $(OBJ)
+	$(CC) -o rv $(LDFLAGS) $(OBJ) $(LIBS)
+
+.PHONY: install
+install: doc_install
+	$(MKDIR) -p $(DESTDIR)$(BINDIR)
+	$(INSTALL) rv -m 755 $(DESTDIR)$(BINDIR)
+	$(STRIP) $(DESTDIR)$(BINDIR)/rv
+
+.PHONY: clean tarball
+clean: doc_clean
+	@test ! -f rv || rm rv
+	@test ! -f $(TARBALL) || rm -f $(TARBALL)
+	@rm -rf *~ $(OBJ) *.tar.$(CEXT)
+
+tarball: clean
+	rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION)
+	echo $(VERSION) > $(NAME)-$(VERSION)/VERSION
+	cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION)
+	mkdir $(NAME)-$(VERSION)/Documentation/
+	cp -rp $(SRCTREE)/../../../Documentation/tools/rv/* $(NAME)-$(VERSION)/Documentation/
+	tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION)
+	rm -rf $(NAME)-$(VERSION)
+
+.PHONY: doc doc_clean doc_install
+doc:
+	$(MAKE) -C $(DOCSRC)
+
+doc_clean:
+	$(MAKE) -C $(DOCSRC) clean
+
+doc_install:
+	$(MAKE) -C $(DOCSRC) install

tools/verification/rv/README.txt

@@ -0,0 +1,38 @@
+RV: Runtime Verification
+
+Runtime Verification (RV) is a lightweight (yet rigorous) method that
+complements classical exhaustive verification techniques (such as model
+checking and theorem proving) with a more practical approach for
+complex systems.
+
+The rv tool is the interface for a collection of monitors that aim
+analysing the logical and timing behavior of Linux.
+
+Installing RV
+
+RV depends on the following libraries and tools:
+
+ - libtracefs
+ - libtraceevent
+
+It also depends on python3-docutils to compile man pages.
+
+For development, we suggest the following steps for compiling rtla:
+
+  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
+  $ cd libtraceevent/
+  $ make
+  $ sudo make install
+  $ cd ..
+  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
+  $ cd libtracefs/
+  $ make
+  $ sudo make install
+  $ cd ..
+  $ cd $rv_src
+  $ make
+  $ sudo make install
+
+For further information, please see rv manpage and the kernel documentation:
+  Runtime Verification:
+    Documentation/trace/rv/runtime-verification.rst

tools/verification/rv/include/in_kernel.h

@@ -0,0 +1,3 @@
+// SPDX-License-Identifier: GPL-2.0
+int ikm_list_monitors(void);
+int ikm_run_monitor(char *monitor, int argc, char **argv);

tools/verification/rv/include/rv.h

@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define MAX_DESCRIPTION 1024
+#define MAX_DA_NAME_LEN	24
+
+struct monitor {
+	char name[MAX_DA_NAME_LEN];
+	char desc[MAX_DESCRIPTION];
+	int enabled;
+};
+
+int should_stop(void);

tools/verification/rv/include/trace.h

@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <tracefs.h>
+
+struct trace_instance {
+	struct tracefs_instance		*inst;
+	struct tep_handle		*tep;
+	struct trace_seq		*seq;
+};
+
+int trace_instance_init(struct trace_instance *trace, char *name);
+int trace_instance_start(struct trace_instance *trace);
+void trace_instance_destroy(struct trace_instance *trace);
+
+int collect_registered_events(struct tep_event *event, struct tep_record *record,
+			      int cpu, void *context);

tools/verification/rv/include/utils.h

@@ -0,0 +1,8 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define MAX_PATH		1024
+
+void debug_msg(const char *fmt, ...);
+void err_msg(const char *fmt, ...);
+
+extern int config_debug;

tools/verification/rv/src/in_kernel.c

@@ -0,0 +1,698 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * in kernel monitor support: allows rv to control in-kernel monitors.
+ *
+ * Copyright (C) 2022 Red Hat Inc, Daniel Bristot de Oliveira <bristot@kernel.org>
+ */
+#include <getopt.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <unistd.h>
+
+#include <trace.h>
+#include <utils.h>
+#include <rv.h>
+
+static int config_has_id;
+static int config_my_pid;
+static int config_trace;
+
+static char *config_initial_reactor;
+static char *config_reactor;
+
+/*
+ * __ikm_read_enable - reads monitor's enable status
+ *
+ * __does not log errors.
+ *
+ * Returns the current status, or -1 if the monitor does not exist,
+ * __hence not logging errors.
+ */
+static int __ikm_read_enable(char *monitor_name)
+{
+	char path[MAX_PATH];
+	long long enabled;
+	int retval;
+
+	snprintf(path, MAX_PATH, "rv/monitors/%s/enable", monitor_name);
+
+	retval = tracefs_instance_file_read_number(NULL, path, &enabled);
+	if (retval < 0)
+		return -1;
+
+	return enabled;
+}
+
+/*
+ * ikm_read_enable - reads monitor's enable status
+ *
+ * Returns the current status, or -1 on error.
+ */
+static int ikm_read_enable(char *monitor_name)
+{
+	int enabled;
+
+	enabled = __ikm_read_enable(monitor_name);
+	if (enabled < 0) {
+		err_msg("ikm: fail read enabled: %d\n", enabled);
+		return -1;
+	}
+
+	debug_msg("ikm: read enabled: %d\n", enabled);
+
+	return enabled;
+}
+
+/*
+ * ikm_write_enable - write to the monitor's enable file
+ *
+ * Return the number of bytes written, -1 on error.
+ */
+static int ikm_write_enable(char *monitor_name, char *enable_disable)
+{
+	char path[MAX_PATH];
+	int retval;
+
+	debug_msg("ikm: writing enabled: %s\n", enable_disable);
+
+	snprintf(path, MAX_PATH, "rv/monitors/%s/enable", monitor_name);
+	retval = tracefs_instance_file_write(NULL, path, enable_disable);
+	if (retval < strlen(enable_disable)) {
+		err_msg("ikm: writing enabled: %s\n", enable_disable);
+		return -1;
+	}
+
+	return retval;
+}
+
+/*
+ * ikm_enable - enable a monitor
+ *
+ * Returns -1 on failure. Success otherwise.
+ */
+static int ikm_enable(char *monitor_name)
+{
+	return ikm_write_enable(monitor_name, "1");
+}
+
+/*
+ * ikm_disable - disable a monitor
+ *
+ * Returns -1 on failure. Success otherwise.
+ */
+static int ikm_disable(char *monitor_name)
+{
+	return ikm_write_enable(monitor_name, "0");
+}
+
+/*
+ * ikm_read_desc - read monitors' description
+ *
+ * Return a dynamically allocated string with the monitor's
+ * description, NULL otherwise.
+ */
+static char *ikm_read_desc(char *monitor_name)
+{
+	char path[MAX_PATH];
+	char *desc;
+
+	snprintf(path, MAX_PATH, "rv/monitors/%s/desc", monitor_name);
+	desc = tracefs_instance_file_read(NULL, path, NULL);
+	if (!desc) {
+		err_msg("ikm: error reading monitor %s desc\n", monitor_name);
+		return NULL;
+	}
+
+	*strstr(desc, "\n") = '\0';
+
+	return desc;
+}
+
+/*
+ * ikm_fill_monitor_definition - fill monitor's definition
+ *
+ * Returns -1 on error, 0 otherwise.
+ */
+static int ikm_fill_monitor_definition(char *name, struct monitor *ikm)
+{
+	int enabled;
+	char *desc;
+
+	enabled = ikm_read_enable(name);
+	if (enabled < 0) {
+		err_msg("ikm: monitor %s fail to read enable file, bug?\n", name);
+		return -1;
+	}
+
+	desc = ikm_read_desc(name);
+	if (!desc) {
+		err_msg("ikm: monitor %s does not have desc file, bug?\n", name);
+		return -1;
+	}
+
+	strncpy(ikm->name, name, MAX_DA_NAME_LEN);
+	ikm->enabled = enabled;
+	strncpy(ikm->desc, desc, MAX_DESCRIPTION);
+
+	free(desc);
+
+	return 0;
+}
+
+/*
+ * ikm_write_reactor - switch the reactor to *reactor
+ *
+ * Return the number or characters written, -1 on error.
+ */
+static int ikm_write_reactor(char *monitor_name, char *reactor)
+{
+	char path[MAX_PATH];
+	int retval;
+
+	snprintf(path, MAX_PATH, "rv/monitors/%s/reactors", monitor_name);
+	retval = tracefs_instance_file_write(NULL, path, reactor);
+	debug_msg("ikm: write \"%s\" reactors: %d\n", reactor, retval);
+
+	return retval;
+}
+
+/*
+ * ikm_read_reactor - read the reactors file
+ *
+ * Returns a dynamically allocated string with monitor's
+ * available reactors, or NULL on error.
+ */
+static char *ikm_read_reactor(char *monitor_name)
+{
+	char path[MAX_PATH];
+	char *reactors;
+
+	snprintf(path, MAX_PATH, "rv/monitors/%s/reactors", monitor_name);
+	reactors = tracefs_instance_file_read(NULL, path, NULL);
+	if (!reactors) {
+		err_msg("ikm: fail reading monitor's %s reactors file\n", monitor_name);
+		return NULL;
+	}
+
+	return reactors;
+}
+
+/*
+ * ikm_get_current_reactor - get the current enabled reactor
+ *
+ * Reads the reactors file and find the currently enabled
+ * [reactor].
+ *
+ * Returns a dynamically allocated memory with the current
+ * reactor. NULL otherwise.
+ */
+static char *ikm_get_current_reactor(char *monitor_name)
+{
+	char *reactors = ikm_read_reactor(monitor_name);
+	char *start;
+	char *end;
+	char *curr_reactor;
+
+	if (!reactors)
+		return NULL;
+
+	start = strstr(reactors, "[");
+	if (!start)
+		goto out_free;
+
+	start++;
+
+	end = strstr(start, "]");
+	if (!end)
+		goto out_free;
+
+	*end = '\0';
+
+	curr_reactor = calloc(strlen(start) + 1, sizeof(char));
+	if (!curr_reactor)
+		goto out_free;
+
+	strncpy(curr_reactor, start, strlen(start));
+	debug_msg("ikm: read current reactor %s\n", curr_reactor);
+
+out_free:
+	free(reactors);
+
+	return curr_reactor;
+}
+
+static int ikm_has_id(char *monitor_name)
+{
+	char path[MAX_PATH];
+	char *format;
+	int has_id;
+
+	snprintf(path, MAX_PATH, "events/rv/event_%s/format", monitor_name);
+	format = tracefs_instance_file_read(NULL, path, NULL);
+	if (!format) {
+		err_msg("ikm: fail reading monitor's %s format event file\n", monitor_name);
+		return -1;
+	}
+
+	/* print fmt: "%d: %s x %s -> %s %s", REC->id, ... */
+	has_id = !!strstr(format, "REC->id");
+
+	debug_msg("ikm: monitor %s has id: %s\n", monitor_name, has_id ? "yes" : "no");
+
+	free(format);
+
+	return has_id;
+}
+
+/**
+ * ikm_list_monitors - list all available monitors
+ *
+ * Returns 0 on success, -1 otherwise.
+ */
+int ikm_list_monitors(void)
+{
+	char *available_monitors;
+	struct monitor ikm;
+	char *curr, *next;
+	int retval;
+
+	available_monitors = tracefs_instance_file_read(NULL, "rv/available_monitors", NULL);
+
+	if (!available_monitors) {
+		err_msg("ikm: available monitors is not available, is CONFIG_RV enabled?\n");
+		return -1;
+	}
+
+	curr = available_monitors;
+	do {
+		next = strstr(curr, "\n");
+		*next = '\0';
+
+		retval = ikm_fill_monitor_definition(curr, &ikm);
+		if (retval)
+			err_msg("ikm: error reading %d in kernel monitor, skipping\n", curr);
+
+		printf("%-24s %s %s\n", ikm.name, ikm.desc, ikm.enabled ? "[ON]" : "[OFF]");
+		curr = ++next;
+
+	} while (strlen(curr));
+
+	free(available_monitors);
+
+	return 0;
+}
+
+static void ikm_print_header(struct trace_seq *s)
+{
+	trace_seq_printf(s, "%16s-%-8s %5s %5s ", "<TASK>", "PID", "[CPU]", "TYPE");
+	if (config_has_id)
+		trace_seq_printf(s, "%8s ", "ID");
+
+	trace_seq_printf(s, "%24s x %-24s -> %-24s %s\n",
+			 "STATE",
+			 "EVENT",
+			 "NEXT_STATE",
+			 "FINAL");
+
+	trace_seq_printf(s, "%16s %-8s %5s %5s ", " | ", " | ", " | ", " | ");
+
+	if (config_has_id)
+		trace_seq_printf(s, "%8s ", " | ");
+
+	trace_seq_printf(s, "%24s   %-24s    %-24s %s\n",
+			 " | ",
+			 " | ",
+			 " | ",
+			 "|");
+
+}
+
+/*
+ * ikm_event_handler - callback to handle event events
+ *
+ * Called any time a rv:"monitor"_event events is generated.
+ * It parses and print event.
+ */
+static int
+ikm_event_handler(struct trace_seq *s, struct tep_record *record,
+		  struct tep_event *trace_event, void *context)
+{
+	/* if needed: struct trace_instance *inst = context; */
+	char *state, *event, *next_state;
+	unsigned long long final_state;
+	unsigned long long pid;
+	unsigned long long id;
+	int cpu = record->cpu;
+	int val;
+
+	if (config_has_id)
+		tep_get_field_val(s, trace_event, "id", record, &id, 1);
+
+	tep_get_common_field_val(s, trace_event, "common_pid", record, &pid, 1);
+
+	if (config_has_id && (config_my_pid == id))
+		return 0;
+	else if (config_my_pid && (config_my_pid == pid))
+		return 0;
+
+	tep_print_event(trace_event->tep, s, record, "%16s-%-8d ", TEP_PRINT_COMM, TEP_PRINT_PID);
+
+	trace_seq_printf(s, "[%.3d] event ", cpu);
+
+	if (config_has_id)
+		trace_seq_printf(s, "%8llu ", id);
+
+	state = tep_get_field_raw(s, trace_event, "state", record, &val, 0);
+	event = tep_get_field_raw(s, trace_event, "event", record, &val, 0);
+	next_state = tep_get_field_raw(s, trace_event, "next_state", record, &val, 0);
+	tep_get_field_val(s, trace_event, "final_state", record, &final_state, 1);
+
+	trace_seq_printf(s, "%24s x %-24s -> %-24s %s\n",
+			 state,
+			 event,
+			 next_state,
+			 final_state ? "Y" : "N");
+
+	trace_seq_do_printf(s);
+	trace_seq_reset(s);
+
+	return 0;
+}
+
+/*
+ * ikm_error_handler - callback to handle error events
+ *
+ * Called any time a rv:"monitor"_errors events is generated.
+ * It parses and print event.
+ */
+static int
+ikm_error_handler(struct trace_seq *s, struct tep_record *record,
+		  struct tep_event *trace_event, void *context)
+{
+	unsigned long long pid, id;
+	int cpu = record->cpu;
+	char *state, *event;
+	int val;
+
+	if (config_has_id)
+		tep_get_field_val(s, trace_event, "id", record, &id, 1);
+
+	tep_get_common_field_val(s, trace_event, "common_pid", record, &pid, 1);
+
+	if (config_has_id && config_my_pid == id)
+		return 0;
+	else if (config_my_pid == pid)
+		return 0;
+
+	trace_seq_printf(s, "%8lld [%03d] error ", pid, cpu);
+
+	if (config_has_id)
+		trace_seq_printf(s, "%8llu ", id);
+
+	state = tep_get_field_raw(s, trace_event, "state", record, &val, 0);
+	event = tep_get_field_raw(s, trace_event, "event", record, &val, 0);
+
+	trace_seq_printf(s, "%24s x %s\n", state, event);
+
+	trace_seq_do_printf(s);
+	trace_seq_reset(s);
+
+	return 0;
+}
+
+/*
+ * ikm_setup_trace_instance - set up a tracing instance to collect data
+ *
+ * Create a trace instance, enable rv: events and enable the trace.
+ *
+ * Returns the trace_instance * with all set, NULL otherwise.
+ */
+static struct trace_instance *ikm_setup_trace_instance(char *monitor_name)
+{
+	char event[MAX_DA_NAME_LEN + 7]; /* max(error_,event_) + '0' = 7 */
+	struct trace_instance *inst;
+	int retval;
+
+	if (!config_trace)
+		return NULL;
+
+	config_has_id = ikm_has_id(monitor_name);
+	if (config_has_id < 0) {
+		err_msg("ikm: failed to read monitor %s event format\n", monitor_name);
+		goto out_err;
+	}
+
+	/* alloc data */
+	inst = calloc(1, sizeof(*inst));
+	if (!inst) {
+		err_msg("ikm: failed to allocate trace instance");
+		goto out_err;
+	}
+
+	retval = trace_instance_init(inst, monitor_name);
+	if (retval)
+		goto out_free;
+
+	/* enable events */
+	snprintf(event, sizeof(event), "event_%s", monitor_name);
+	retval = tracefs_event_enable(inst->inst, "rv",  event);
+	if (retval)
+		goto out_inst;
+
+	tep_register_event_handler(inst->tep, -1, "rv", event,
+				   ikm_event_handler, NULL);
+
+	snprintf(event, sizeof(event), "error_%s", monitor_name);
+	retval = tracefs_event_enable(inst->inst, "rv", event);
+	if (retval)
+		goto out_inst;
+
+	tep_register_event_handler(inst->tep, -1, "rv", event,
+				   ikm_error_handler, NULL);
+
+	/* ready to enable */
+	tracefs_trace_on(inst->inst);
+
+	return inst;
+
+out_inst:
+	trace_instance_destroy(inst);
+out_free:
+	free(inst);
+out_err:
+	return NULL;
+}
+
+/**
+ * ikm_destroy_trace_instance - destroy a previously created instance
+ */
+static void ikm_destroy_trace_instance(struct trace_instance *inst)
+{
+	if (!inst)
+		return;
+
+	trace_instance_destroy(inst);
+	free(inst);
+}
+
+/*
+ * ikm_usage_print_reactors - print all available reactors, one per line.
+ */
+static void ikm_usage_print_reactors(void)
+{
+	char *reactors = tracefs_instance_file_read(NULL, "rv/available_reactors", NULL);
+	char *start, *end;
+
+	if (!reactors)
+		return;
+
+	fprintf(stderr, "  available reactors:");
+
+	start = reactors;
+	end = strstr(start, "\n");
+
+	while (end) {
+		*end = '\0';
+
+		fprintf(stderr, " %s", start);
+
+		start = ++end;
+		end = strstr(start, "\n");
+	};
+
+	fprintf(stderr, "\n");
+}
+/*
+ * ikm_usage - print usage
+ */
+static void ikm_usage(int exit_val, char *monitor_name, const char *fmt, ...)
+{
+
+	char message[1024];
+	va_list ap;
+	int i;
+
+	static const char *const usage[] = {
+		"",
+		"	-h/--help: print this menu and the reactor list",
+		"	-r/--reactor 'reactor': enables the 'reactor'",
+		"	-s/--self: when tracing (-t), also trace rv command",
+		"	-t/--trace: trace monitor's event",
+		"	-v/--verbose: print debug messages",
+		"",
+		NULL,
+	};
+
+	va_start(ap, fmt);
+	vsnprintf(message, sizeof(message), fmt, ap);
+	va_end(ap);
+
+	fprintf(stderr, "  %s\n", message);
+
+	fprintf(stderr, "\n  usage: rv mon %s [-h] [-q] [-r reactor] [-s] [-v]", monitor_name);
+
+	for (i = 0; usage[i]; i++)
+		fprintf(stderr, "%s\n", usage[i]);
+
+	ikm_usage_print_reactors();
+	exit(exit_val);
+}
+
+/*
+ * parse_arguments - parse arguments and set config
+ */
+static int parse_arguments(char *monitor_name, int argc, char **argv)
+{
+	int c, retval;
+
+	config_my_pid = getpid();
+
+	while (1) {
+		static struct option long_options[] = {
+			{"help",		no_argument,		0, 'h'},
+			{"reactor",		required_argument,	0, 'r'},
+			{"self",		no_argument,		0, 's'},
+			{"trace",		no_argument,		0, 't'},
+			{"verbose",		no_argument,		0, 'v'},
+			{0, 0, 0, 0}
+		};
+
+		/* getopt_long stores the option index here. */
+		int option_index = 0;
+
+		c = getopt_long(argc, argv, "hr:stv", long_options, &option_index);
+
+		/* detect the end of the options. */
+		if (c == -1)
+			break;
+
+		switch (c) {
+		case 'h':
+			ikm_usage(0, monitor_name, "help:");
+			break;
+		case 'r':
+			config_reactor = optarg;
+			break;
+		case 's':
+			config_my_pid = 0;
+			break;
+		case 't':
+			config_trace = 1;
+			break;
+		case 'v':
+			config_debug = 1;
+			break;
+		}
+	}
+
+	if (config_reactor) {
+		config_initial_reactor = ikm_get_current_reactor(monitor_name);
+		if (!config_initial_reactor)
+			ikm_usage(1, monitor_name,
+				  "ikm: failed to read current reactor, are reactors enabled?");
+
+		retval = ikm_write_reactor(monitor_name, config_reactor);
+		if (retval <= 0)
+			ikm_usage(1, monitor_name,
+				  "ikm: failed to set %s reactor, is it available?",
+				  config_reactor);
+	}
+
+	debug_msg("ikm: my pid is %d\n", config_my_pid);
+
+	return 0;
+}
+
+/**
+ * ikm_run_monitor - apply configs and run the monitor
+ *
+ * Returns 1 if a monitor was found an executed, 0 if no
+ * monitors were found, or -1 on error.
+ */
+int ikm_run_monitor(char *monitor_name, int argc, char **argv)
+{
+	struct trace_instance *inst = NULL;
+	int retval;
+
+	/*
+	 * Check if monitor exists by seeing it is enabled.
+	 */
+	retval = __ikm_read_enable(monitor_name);
+	if (retval < 0)
+		return 0;
+
+	if (retval) {
+		err_msg("ikm: monitor %s (in-kernel) is already enabled\n", monitor_name);
+		return -1;
+	}
+
+	/* we should be good to go */
+	retval = parse_arguments(monitor_name, argc, argv);
+	if (retval)
+		ikm_usage(1, monitor_name, "ikm: failed parsing arguments");
+
+	if (config_trace) {
+		inst = ikm_setup_trace_instance(monitor_name);
+		if (!inst)
+			return -1;
+	}
+
+	retval = ikm_enable(monitor_name);
+	if (retval < 0)
+		goto out_free_instance;
+
+	if (config_trace)
+		ikm_print_header(inst->seq);
+
+	while (!should_stop()) {
+		if (config_trace) {
+			retval = tracefs_iterate_raw_events(inst->tep,
+							    inst->inst,
+							    NULL,
+							    0,
+							    collect_registered_events,
+							    inst);
+			if (retval) {
+				err_msg("ikm: error reading trace buffer\n");
+				break;
+			}
+		}
+
+		sleep(1);
+	}
+
+	ikm_disable(monitor_name);
+	ikm_destroy_trace_instance(inst);
+
+	if (config_reactor && config_initial_reactor)
+		ikm_write_reactor(monitor_name, config_initial_reactor);
+
+	return 1;
+
+out_free_instance:
+	ikm_destroy_trace_instance(inst);
+	if (config_reactor && config_initial_reactor)
+		ikm_write_reactor(monitor_name, config_initial_reactor);
+	return -1;
+}

tools/verification/rv/src/rv.c

@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * rv tool, the interface for the Linux kernel RV subsystem and home of
+ * user-space controlled monitors.
+ *
+ * Copyright (C) 2022 Red Hat Inc, Daniel Bristot de Oliveira <bristot@kernel.org>
+ */
+
+#include <stdlib.h>
+#include <signal.h>
+#include <unistd.h>
+
+#include <trace.h>
+#include <utils.h>
+#include <in_kernel.h>
+
+static int stop_session;
+
+/*
+ * stop_rv - tell monitors to stop
+ */
+static void stop_rv(int sig)
+{
+	stop_session = 1;
+}
+
+/**
+ * should_stop - check if the monitor should stop.
+ *
+ * Returns 1 if the monitor should stop, 0 otherwise.
+ */
+int should_stop(void)
+{
+	return stop_session;
+}
+
+/*
+ * rv_list - list all available monitors
+ */
+static void rv_list(int argc, char **argv)
+{
+	static const char *const usage[] = {
+		"",
+		"  usage: rv list [-h]",
+		"",
+		"	list all available monitors",
+		"",
+		"	-h/--help: print this menu",
+		NULL,
+	};
+	int i;
+
+	if (argc > 1) {
+		fprintf(stderr, "rv version %s\n", VERSION);
+
+		/* more than 1 is always usage */
+		for (i = 0; usage[i]; i++)
+			fprintf(stderr, "%s\n", usage[i]);
+
+		/* but only -h is valid */
+		if (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))
+			exit(0);
+		else
+			exit(1);
+	}
+
+	ikm_list_monitors();
+	exit(0);
+}
+
+/*
+ * rv_mon - try to run a monitor passed as argument
+ */
+static void rv_mon(int argc, char **argv)
+{
+	char *monitor_name;
+	int i, run;
+
+	static const char *const usage[] = {
+		"",
+		"  usage: rv mon [-h] monitor [monitor options]",
+		"",
+		"	run a monitor",
+		"",
+		"	-h/--help: print this menu",
+		"",
+		"	monitor [monitor options]: the monitor, passing",
+		"	the arguments to the [monitor options]",
+		NULL,
+	};
+
+	/* requires at least one argument */
+	if (argc == 1) {
+
+		fprintf(stderr, "rv version %s\n", VERSION);
+
+		for (i = 0; usage[i]; i++)
+			fprintf(stderr, "%s\n", usage[i]);
+		exit(1);
+	} else if (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
+
+		fprintf(stderr, "rv version %s\n", VERSION);
+
+		for (i = 0; usage[i]; i++)
+			fprintf(stderr, "%s\n", usage[i]);
+		exit(0);
+	}
+
+	monitor_name = argv[1];
+	/*
+	 * Call all possible monitor implementations, looking
+	 * for the [monitor].
+	 */
+	run += ikm_run_monitor(monitor_name, argc-1, &argv[1]);
+
+	if (!run)
+		err_msg("rv: monitor %s does not exist\n", monitor_name);
+	exit(!run);
+}
+
+static void usage(int exit_val, const char *fmt, ...)
+{
+	char message[1024];
+	va_list ap;
+	int i;
+
+	static const char *const usage[] = {
+		"",
+		"  usage: rv command [-h] [command_options]",
+		"",
+		"	-h/--help: print this menu",
+		"",
+		"	command: run one of the following command:",
+		"	  list: list all available monitors",
+		"	  mon:  run a monitor",
+		"",
+		"	[command options]: each command has its own set of options",
+		"		           run rv command -h for further information",
+		NULL,
+	};
+
+	va_start(ap, fmt);
+	vsnprintf(message, sizeof(message), fmt, ap);
+	va_end(ap);
+
+	fprintf(stderr, "rv version %s: %s\n", VERSION, message);
+
+	for (i = 0; usage[i]; i++)
+		fprintf(stderr, "%s\n", usage[i]);
+
+	exit(exit_val);
+}
+
+/*
+ * main - select which main sending the command
+ *
+ * main itself redirects the arguments to the sub-commands
+ * to handle the options.
+ *
+ * subcommands should exit.
+ */
+int main(int argc, char **argv)
+{
+	if (geteuid())
+		usage(1, "%s needs root permission", argv[0]);
+
+	if (argc <= 1)
+		usage(1, "%s requires a command", argv[0]);
+
+	if (!strcmp(argv[1], "-h") || !strcmp(argv[1], "--help"))
+		usage(0, "help");
+
+	if (!strcmp(argv[1], "list"))
+		rv_list(--argc, &argv[1]);
+
+	if (!strcmp(argv[1], "mon")) {
+		/*
+		 * monitor's main should monitor should_stop() function.
+		 * and exit.
+		 */
+		signal(SIGINT, stop_rv);
+
+		rv_mon(argc - 1, &argv[1]);
+	}
+
+	/* invalid sub-command */
+	usage(1, "%s does not know the %s command, old version?", argv[0], argv[1]);
+}

tools/verification/rv/src/trace.c

@@ -0,0 +1,133 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * trace helpers.
+ *
+ * Copyright (C) 2022 Red Hat Inc, Daniel Bristot de Oliveira <bristot@kernel.org>
+ */
+
+#include <sys/sendfile.h>
+#include <tracefs.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <errno.h>
+
+#include <rv.h>
+#include <trace.h>
+#include <utils.h>
+
+/*
+ * create_instance - create a trace instance with *instance_name
+ */
+static struct tracefs_instance *create_instance(char *instance_name)
+{
+	return tracefs_instance_create(instance_name);
+}
+
+/*
+ * destroy_instance - remove a trace instance and free the data
+ */
+static void destroy_instance(struct tracefs_instance *inst)
+{
+	tracefs_instance_destroy(inst);
+	tracefs_instance_free(inst);
+}
+
+/**
+ * collect_registered_events - call the existing callback function for the event
+ *
+ * If an event has a registered callback function, call it.
+ * Otherwise, ignore the event.
+ *
+ * Returns 0 if the event was collected, 1 if the tool should stop collecting trace.
+ */
+int
+collect_registered_events(struct tep_event *event, struct tep_record *record,
+			  int cpu, void *context)
+{
+	struct trace_instance *trace = context;
+	struct trace_seq *s = trace->seq;
+
+	if (should_stop())
+		return 1;
+
+	if (!event->handler)
+		return 0;
+
+	event->handler(s, record, event, context);
+
+	return 0;
+}
+
+/**
+ * trace_instance_destroy - destroy and free a rv trace instance
+ */
+void trace_instance_destroy(struct trace_instance *trace)
+{
+	if (trace->inst) {
+		destroy_instance(trace->inst);
+		trace->inst = NULL;
+	}
+
+	if (trace->seq) {
+		free(trace->seq);
+		trace->seq = NULL;
+	}
+
+	if (trace->tep) {
+		tep_free(trace->tep);
+		trace->tep = NULL;
+	}
+}
+
+/**
+ * trace_instance_init - create an trace instance
+ *
+ * It is more than the tracefs instance, as it contains other
+ * things required for the tracing, such as the local events and
+ * a seq file.
+ *
+ * Note that the trace instance is returned disabled. This allows
+ * the tool to apply some other configs, like setting priority
+ * to the kernel threads, before starting generating trace entries.
+ *
+ * Returns 0 on success, non-zero otherwise.
+ */
+int trace_instance_init(struct trace_instance *trace, char *name)
+{
+	trace->seq = calloc(1, sizeof(*trace->seq));
+	if (!trace->seq)
+		goto out_err;
+
+	trace_seq_init(trace->seq);
+
+	trace->inst = create_instance(name);
+	if (!trace->inst)
+		goto out_err;
+
+	trace->tep = tracefs_local_events(NULL);
+	if (!trace->tep)
+		goto out_err;
+
+	/*
+	 * Let the main enable the record after setting some other
+	 * things such as the priority of the tracer's threads.
+	 */
+	tracefs_trace_off(trace->inst);
+
+	return 0;
+
+out_err:
+	trace_instance_destroy(trace);
+	return 1;
+}
+
+/**
+ * trace_instance_start - start tracing a given rv instance
+ *
+ * Returns 0 on success, -1 otherwise.
+ */
+int trace_instance_start(struct trace_instance *trace)
+{
+	return tracefs_trace_on(trace->inst);
+}

tools/verification/rv/src/utils.c

@@ -0,0 +1,47 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * util functions.
+ *
+ * Copyright (C) 2022 Red Hat Inc, Daniel Bristot de Oliveira <bristot@kernel.org>
+ */
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <utils.h>
+
+int config_debug;
+
+#define MAX_MSG_LENGTH	1024
+
+/**
+ * err_msg - print an error message to the stderr
+ */
+void err_msg(const char *fmt, ...)
+{
+	char message[MAX_MSG_LENGTH];
+	va_list ap;
+
+	va_start(ap, fmt);
+	vsnprintf(message, sizeof(message), fmt, ap);
+	va_end(ap);
+
+	fprintf(stderr, "%s", message);
+}
+
+/**
+ * debug_msg - print a debug message to stderr if debug is set
+ */
+void debug_msg(const char *fmt, ...)
+{
+	char message[MAX_MSG_LENGTH];
+	va_list ap;
+
+	if (!config_debug)
+		return;
+
+	va_start(ap, fmt);
+	vsnprintf(message, sizeof(message), fmt, ap);
+	va_end(ap);
+
+	fprintf(stderr, "%s", message);
+}