PAPI and native events - c

Is it possible to find the set of native events (CPU counter registers) that maps to a PAPI event ?

If you have papi installed, you can use papi_avail to get the native events. For example,
papi_avail -e PAPI_TOT_INS
Event name: PAPI_TOT_INS
Event Code: 0x80000032
Number of Native Events: 1
Short Description: |Instr completed|
Long Description: |Instructions completed|
Developer's Notes: ||
Derived Type: |NOT_DERIVED|
Postfix Processing String: ||
Native Code[0]: 0x40000002 |INSTRUCTION_RETIRED|
Number of Register Values: 0
Native Event Description: |Number of instructions at retirement|
-Ganesh

I think you mean
papi_native_avail
With PAPI_event_name_to_code(...) you can add your event to your eventset for example.

In order to quickly see and export all available mappings from preset events to native events use papi_decode -a command. The output is in the CSV format. An example from my CPU:
pc#pc ~> papi_decode -a
name,derived,postfix,short_descr,long_descr,note,[native,...]
PAPI_L1_DCM,NOT_DERIVED,,"L1D cache misses","Level 1 data cache misses",,L1D:REPLACEMENT
PAPI_L1_ICM,NOT_DERIVED,,"L1I cache misses","Level 1 instruction cache misses",,L2_RQSTS:ALL_CODE_RD
PAPI_L2_DCM,DERIVED_SUB,,"L2D cache misses","Level 2 data cache misses",,LLC_REFERENCES,L2_RQSTS:CODE_RD_MISS
(...)

Related

how to loop over multiple registers in linux device tree node?

I implemented a generic driver, in the driver I just want to go over each registered device and do something.
my driver is generic so I don't want to know anything in advance about the devices.
I have some device tree nodes:
uart0: uart#1000000 {
compatible = "generic-driver";
reg = <0x1000000 0x1000>,<0x1001000 0x1000>,<0x1002000 0x1000>;
};
uart1: uart#2000000 {
compatible = "generic-driver";
reg = <0x2000000 0x1000>,<0x2001000 0x1000>,<0x2002000 0x1000>;
};
in the driver I know how to get first offset and size with platform_get_resource, but what if I have multiple ranges?
I saw an option to get it by add reg-names:
reg = <0x2000000 0x1000>,<0x2001000 0x1000>,<0x2002000 0x1000>;
reg-names = "uart0_0","urat0_1","uart0_2";
so in the driver I can use platform_get_resource_byname, but there I need to know in the driver the regs names in advance - but it'll be no more generic - unwanted!!
so my question is, is there a way to loop over these regs without early information in the driver?
Thanks in advance!!
If you go through the definition for platform_get_resource you will find that the last parameter is the resource number. So, for the example you gave resource numbers will range from 0 to 2.
In case you don't know the number of resource then you can run an loop till you get a NULL as response to platform_get_resource.

Image-classification-transfer-learning Sagemaker issue

Im getting an error like this while trying image classification with Sagemaker:
ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: 1 validation error detected: Value 'ml.t2.medium' at 'resourceConfig.instanceType' failed to satisfy constraint: Member must satisfy enum value set: [ml.p2.xlarge, ml.m5.4xlarge, ml.m4.16xlarge, ml.p3.16xlarge, ml.m5.large, ml.p2.16xlarge, ml.c4.2xlarge, ml.c5.2xlarge, ml.c4.4xlarge, ml.c5.4xlarge, ml.c4.8xlarge, ml.c5.9xlarge, ml.c5.xlarge, ml.c4.xlarge, ml.c5.18xlarge, ml.p3.2xlarge, ml.m5.xlarge, ml.m4.10xlarge, ml.m5.12xlarge, ml.m4.xlarge, ml.m5.24xlarge, ml.m4.2xlarge, ml.p2.8xlarge, ml.m5.2xlarge, ml.p3.8xlarge, ml.m4.4xlarge]
The ml.t2.medium instance type is not available on SageMaker Training as of this writing.
You can refer to https://aws.amazon.com/sagemaker/pricing/ to see the supported instance types in the component and region you are using.
You should also check if the algorithm you are running has additional hardware constraints. For instance, the Image Classification Algorithm doc calls out that it requires GPU instances for training:
For image classification, we support the following GPU instances for training: ml.p2.xlarge, ml.p2.8xlarge, ml.p2.16xlarge, ml.p3.2xlarge, ml.p3.8xlarge and ml.p3.16xlarge. We recommend using GPU instances with more memory for training with large batch sizes. However, both CPU (such as C4) and GPU (such as P2 and P3) instances can be used for the inference. You can also run the algorithm on multi-GPU and multi-machine settings for distributed training.
Both P2 and P3 instances are supported in the image classification algorithm.

How to sessionize / group the events in Akka Streams?

The requirement is that I want to write an Akka streaming application that listens to continuous events from Kafka, then sessionizes the event data in a time frame, based on some id value embedded inside each event.
For example, let's say that my time frame window is two minutes, and in the first two minutes I get the four events below:
Input:
{"message-domain":"1234","id":1,"aaa":"bbb"}
{"message-domain":"1234","id":2,"aaa":"bbb"}
{"message-domain":"5678","id":4,"aaa":"bbb"}
{"message-domain":"1234","id":3,"aaa":"bbb"}
Then in the output, after grouping/sessionizing these events, I will have only two events based on their message-domain value.
Output:
{"message-domain":"1234",messsages:[{"id":1,"aaa":"bbb"},{"id":2,"aaa":"bbb"},{"id":4,"aaa":"bbb"}]}
{"message-domain":"5678",messsages:[{"id":3,"aaa":"bbb"}]}
And I want this to happen in real time. Any suggestions on how to achieve this?
To group the events within a time window you can use Flow.groupedWithin:
val maxCount : Int = Int.MaxValue
val timeWindow = FiniteDuration(2L, TimeUnit.MINUTES)
val timeWindowFlow : Flow[String, Seq[String]] =
Flow[String] groupedWithin (maxCount, timeWindow)

How to find which device is attached to a USB-serial port in Linux using C?

We are making a device and it has 8 serial ports. It runs on the Monta Vista Pro5 kernel. And we are working in C.
Suppose: A device gets attached to ttyUSB0, ttyUSB1 and ttyUSB2. The next device gets connected to ttyUSB3 and another to ttyUSB4. How can I know which device gets attached to which port ?? ie ttyUSB0 or ttyUSB1 or so on. Is there a way to directly query the device and find which port it is attached to. Or, in C, open ttyUSB0, query it somehow and get some reply as to which device it is ??
A rather complicated way: do a stat of, say /dev/ttyUSB0. Get the device number. And search for this in /proc/bus/usb/devices and find the vendor id or something to identify the device.
Or: Is there some way to reserve ttyUSB0,ttyUSB1 and ttyUSB2 for one device, ttyUSB3 for another and so on when they are plugged in ? This way I will know which device is connected to which port.
Help please..... :)
Thanks in advance.
Nubin Stanley
You can use udev rules to create symbolic links just to your device:
(these rules go in /etc/udev/rules.d/-name.rules -- look at your udev documentation
KERNEL=="ttyUSB*", ATTRS{idVendor}=="<vendorid>", MODE="0666", SYMLINK+="mydev"
You have to specify your vendor id and/or product id for your device. Then those devices will be available at /dev/mydev in the above example.
You can also use various other parameters to create appropriate unique symbolic links for your use. Check udev man page.
Here's my code, based on Alex Robinson's, but without global "except":
import os
from os.path import join
def find_tty_usb(idVendor, idProduct):
"""find_tty_usb('067b', '2302') -> '/dev/ttyUSB0'"""
# Note: if searching for a lot of pairs, it would be much faster to search
# for the enitre lot at once instead of going over all the usb devices
# each time.
for dnbase in os.listdir('/sys/bus/usb/devices'):
dn = join('/sys/bus/usb/devices', dnbase)
if not os.path.exists(join(dn, 'idVendor')):
continue
idv = open(join(dn, 'idVendor')).read().strip()
if idv != idVendor:
continue
idp = open(join(dn, 'idProduct')).read().strip()
if idp != idProduct:
continue
for subdir in os.listdir(dn):
if subdir.startswith(dnbase+':'):
for subsubdir in os.listdir(join(dn, subdir)):
if subsubdir.startswith('ttyUSB'):
return join('/dev', subsubdir)
This Python code seems to find the /dev/ttyUSB number for the given vendor ID and product ID. Not hard to translate it to C. Parsing the output from hwinfo --usb can do the trick, too. The regx is:
"\s\sVendor:\susb\s0x([0-9a-f]{4}).*?\s\sDevice:\susb\s0x([0-9a-f]{4}).*?\s\sDevice\sFile:\s/dev/ttyUSB([0-9]+)"
import glob
import os
import re
def find_usb_tty(vendor_id = None, product_id = None) :
tty_devs = []
for dn in glob.glob('/sys/bus/usb/devices/*') :
try :
vid = int(open(os.path.join(dn, "idVendor" )).read().strip(), 16)
pid = int(open(os.path.join(dn, "idProduct")).read().strip(), 16)
if ((vendor_id is None) or (vid == vendor_id)) and ((product_id is None) or (pid == product_id)) :
dns = glob.glob(os.path.join(dn, os.path.basename(dn) + "*"))
for sdn in dns :
for fn in glob.glob(os.path.join(sdn, "*")) :
if re.search(r"\/ttyUSB[0-9]+$", fn) :
#tty_devs.append("/dev" + os.path.basename(fn))
tty_devs.append(os.path.join("/dev", os.path.basename(fn)))
pass
pass
pass
pass
except ( ValueError, TypeError, AttributeError, OSError, IOError ) :
pass
pass
return tty_devs
print find_usb_tty()
The best way to do this is would be to use libusb, but if that doesn't give you enough information about your devices (which it may not), then you'll have to use the /proc filesystem which the kernel makes available, specifically /proc/bus/usb/.
Have a read of this information on /proc/bus/usb: in particular on /proc/bus/usb/devices. But as you say, this is all a bit hacky!

Increasing camera capture resolution in OpenCV

In my C/C++ program, I'm using OpenCV to capture images from my webcam. The camera (Logitech QuickCam IM) can capture at resolutions 320x240, 640x480 and 1280x960. But, for some strange reason, OpenCV gives me images of resolution 320x240 only. Calls to change the resolution using cvSetCaptureProperty() with other resolution values just don't work. How do I capture images with the other resolutions possible with my webcam?
I'm using openCV 1.1pre1 under Windows (videoinput library is used by default by this version of openCv under windows).
With these instructions I can set camera resolution. Note that I call the old cvCreateCameraCapture instead of cvCaptureFromCam.
capture = cvCreateCameraCapture(cameraIndex);
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_WIDTH, 640 );
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_HEIGHT, 480 );
videoFrame = cvQueryFrame(capture);
I've tested it with Logitech, Trust and Philips webcams
There doesn't seem to be a solution. The resolution can be increased to 640x480 using this hack shared by lifebelt77. Here are the details reproduced:
Add to highgui.h:
#define CV_CAP_PROP_DIALOG_DISPLAY 8
#define CV_CAP_PROP_DIALOG_FORMAT 9
#define CV_CAP_PROP_DIALOG_SOURCE 10
#define CV_CAP_PROP_DIALOG_COMPRESSION 11
#define CV_CAP_PROP_FRAME_WIDTH_HEIGHT 12
Add the function icvSetPropertyCAM_VFW to cvcap.cpp:
static int icvSetPropertyCAM_VFW( CvCaptureCAM_VFW* capture, int property_id, double value )
{
int result = -1;
CAPSTATUS capstat;
CAPTUREPARMS capparam;
BITMAPINFO btmp;
switch( property_id )
{
case CV_CAP_PROP_DIALOG_DISPLAY:
result = capDlgVideoDisplay(capture->capWnd);
//SendMessage(capture->capWnd,WM_CAP_DLG_VIDEODISPLAY,0,0);
break;
case CV_CAP_PROP_DIALOG_FORMAT:
result = capDlgVideoFormat(capture->capWnd);
//SendMessage(capture->capWnd,WM_CAP_DLG_VIDEOFORMAT,0,0);
break;
case CV_CAP_PROP_DIALOG_SOURCE:
result = capDlgVideoSource(capture->capWnd);
//SendMessage(capture->capWnd,WM_CAP_DLG_VIDEOSOURCE,0,0);
break;
case CV_CAP_PROP_DIALOG_COMPRESSION:
result = capDlgVideoCompression(capture->capWnd);
break;
case CV_CAP_PROP_FRAME_WIDTH_HEIGHT:
capGetVideoFormat(capture->capWnd, &btmp, sizeof(BITMAPINFO));
btmp.bmiHeader.biWidth = floor(value/1000);
btmp.bmiHeader.biHeight = value-floor(value/1000)*1000;
btmp.bmiHeader.biSizeImage = btmp.bmiHeader.biHeight *
btmp.bmiHeader.biWidth * btmp.bmiHeader.biPlanes *
btmp.bmiHeader.biBitCount / 8;
capSetVideoFormat(capture->capWnd, &btmp, sizeof(BITMAPINFO));
break;
default:
break;
}
return result;
}
and edit captureCAM_VFW_vtable as following:
static CvCaptureVTable captureCAM_VFW_vtable =
{
6,
(CvCaptureCloseFunc)icvCloseCAM_VFW,
(CvCaptureGrabFrameFunc)icvGrabFrameCAM_VFW,
(CvCaptureRetrieveFrameFunc)icvRetrieveFrameCAM_VFW,
(CvCaptureGetPropertyFunc)icvGetPropertyCAM_VFW,
(CvCaptureSetPropertyFunc)icvSetPropertyCAM_VFW, // was NULL
(CvCaptureGetDescriptionFunc)0
};
Now rebuilt highgui.dll.
I've done image processing in linux before and skipped OpenCV's built in camera functionality because it's (as you've discovered) incomplete.
Depending on your OS you may have more luck going straight to the hardware through normal channels as opposed to through openCV. If you are using Linux, video4linux or video4linux2 should give you relatively trivial access to USB webcams and you can use libavc1394 for firewire. Depending on the device and the quality of the example code you follow, you should be able to get the device running with the parameters you want in an hour or two.
Edited to add: You are on your own if its Windows. I imagine it's not much more difficult but I've never done it.
I strongly suggest using VideoInput lib, it supports any DirectShow device (even multiple devices at the same time) and is more configurable. You'll spend five minutes make it play with OpenCV.
Check this ticket out:
https://code.ros.org/trac/opencv/ticket/376
"The solution is to use the newer libv4l-based wrapper.
install libv4l-dev (this is how it's called in Ubuntu)
rerun cmake, you will see "V4L/V4L2: Using libv4l"
rerun make. now the resolution can be changed. tested with built-in isight on MBP."
This fixed it for me using Ubuntu and might aswell work for you.
Code I finally got working in Python once Aaron Haun pointed out I needed to define the arguments of the set function before using them.
#Camera_Get_Set.py
#By Forrest L. Erickson of VRX Company Inc. 8-31-12.
#Opens the camera and reads and reports the settings.
#Then tries to set for higher resolution.
#Workes with Logitech C525 for resolutions 960 by 720 and 1600 by 896
import cv2.cv as cv
import numpy
CV_CAP_PROP_POS_MSEC = 0
CV_CAP_PROP_POS_FRAMES = 1
CV_CAP_PROP_POS_AVI_RATIO = 2
CV_CAP_PROP_FRAME_WIDTH = 3
CV_CAP_PROP_FRAME_HEIGHT = 4
CV_CAP_PROP_FPS = 5
CV_CAP_PROP_POS_FOURCC = 6
CV_CAP_PROP_POS_FRAME_COUNT = 7
CV_CAP_PROP_BRIGHTNESS = 8
CV_CAP_PROP_CONTRAST = 9
CV_CAP_PROP_SATURATION = 10
CV_CAP_PROP_HUE = 11
CV_CAPTURE_PROPERTIES = tuple({
CV_CAP_PROP_POS_MSEC,
CV_CAP_PROP_POS_FRAMES,
CV_CAP_PROP_POS_AVI_RATIO,
CV_CAP_PROP_FRAME_WIDTH,
CV_CAP_PROP_FRAME_HEIGHT,
CV_CAP_PROP_FPS,
CV_CAP_PROP_POS_FOURCC,
CV_CAP_PROP_POS_FRAME_COUNT,
CV_CAP_PROP_BRIGHTNESS,
CV_CAP_PROP_CONTRAST,
CV_CAP_PROP_SATURATION,
CV_CAP_PROP_HUE})
CV_CAPTURE_PROPERTIES_NAMES = [
"CV_CAP_PROP_POS_MSEC",
"CV_CAP_PROP_POS_FRAMES",
"CV_CAP_PROP_POS_AVI_RATIO",
"CV_CAP_PROP_FRAME_WIDTH",
"CV_CAP_PROP_FRAME_HEIGHT",
"CV_CAP_PROP_FPS",
"CV_CAP_PROP_POS_FOURCC",
"CV_CAP_PROP_POS_FRAME_COUNT",
"CV_CAP_PROP_BRIGHTNESS",
"CV_CAP_PROP_CONTRAST",
"CV_CAP_PROP_SATURATION",
"CV_CAP_PROP_HUE"]
capture = cv.CaptureFromCAM(0)
print ("\nCamera properties before query of frame.")
for i in range(len(CV_CAPTURE_PROPERTIES_NAMES)):
# camera_valeus =[CV_CAPTURE_PROPERTIES_NAMES, foo]
foo = cv.GetCaptureProperty(capture, CV_CAPTURE_PROPERTIES[i])
camera_values =[CV_CAPTURE_PROPERTIES_NAMES[i], foo]
# print str(camera_values)
print str(CV_CAPTURE_PROPERTIES_NAMES[i]) + ": " + str(foo)
print ("\nOpen a window for display of image")
cv.NamedWindow("Camera", 1)
while True:
img = cv.QueryFrame(capture)
cv.ShowImage("Camera", img)
if cv.WaitKey(10) == 27:
break
cv.DestroyWindow("Camera")
#cv.SetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, 1024)
#cv.SetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, 768)
cv.SetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, 1600)
cv.SetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, 896)
print ("\nCamera properties after query and display of frame.")
for i in range(len(CV_CAPTURE_PROPERTIES_NAMES)):
# camera_valeus =[CV_CAPTURE_PROPERTIES_NAMES, foo]
foo = cv.GetCaptureProperty(capture, CV_CAPTURE_PROPERTIES[i])
camera_values =[CV_CAPTURE_PROPERTIES_NAMES[i], foo]
# print str(camera_values)
print str(CV_CAPTURE_PROPERTIES_NAMES[i]) + ": " + str(foo)
print ("/nOpen a window for display of image")
cv.NamedWindow("Camera", 1)
while True:
img = cv.QueryFrame(capture)
cv.ShowImage("Camera", img)
if cv.WaitKey(10) == 27:
break
cv.DestroyWindow("Camera")
I am using debian and ubuntu, i had the same problem, i couldn't change the resolution of video input using CV_CAP_PROP_FRAME_WIDTH and CV_CAP_PROP_FRAME_HEIGHT
I turned out that the reason was a missing library.
I installed lib4l-dev through synaptic, rebuilt OpenCV and the problem is SOLVED!
I am posting this to ensure that no one else wastes time on this setproperty function. I spent 2 days on this to see that nothing seems to be working. So I dug out the code (I had installed the library the first time around). This is what actually happens - cvSetCaptureProperty, calls setProperty inside CvCapture class and lo behold setProperty does nothing. It just returns false.
Instead I'll pick up using another library to feed OpenCV a capture video/images. I am using OpenCV 2.2
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_WIDTH, WIDTH );
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_HEIGHT, HEIGHT);
cvQueryFrame(capture);
That will not work with OpenCV 2.2, but if you use OpenCV 2.1 it will work fine !
If you are on windows platform, try DirectShow (IAMStreamConfig).
http://msdn.microsoft.com/en-us/library/dd319784%28v=vs.85%29.aspx
Under Windows try to use VideoInput library:
http://robocraft.ru/blog/computervision/420.html
I find that in Windows (from Win98 to WinXP SP3), OpenCV will often use Microsoft's VFW library for camera access. The problem with this is that it is often very slow (say a max of 15 FPS frame capture) and buggy (hence why cvSetCaptureProperty often doesn't work). Luckily, you can usually change the resolution in other software (particularly "AMCAP", which is a demo program that is easily available) and it will effect the resolution that OpenCV will use. For example, you can run AMCAP to set the resolution to 640x480, and then OpenCV will use that by default from that point onwards!
But if you can use a different Windows camera access library such as the "videoInput" library http://muonics.net/school/spring05/videoInput/ that accesses the camera using very efficient DirectShow (part of DirectX). Or if you have a professional quality camera, then often it will come with a custom API that lets you access the camera, and you could use that for fast access with the ability to change resolution and many other things.
Just one information that could be valuable for people having difficulties to change the default capture resolution (640 x 480) ! I experimented myself a such problem with opencv 2.4.x and one Logitech camera ... and found one workaround !
The behaviour I detected is that the default format is setup as initial parameters when camera capture is started (cvCreateCameraCapture), and all request to change height or width :
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_WIDTH, ...
or
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_HEIGHT, ...
are not possible afterwards ! Effectively, I discovered with adding return error of ioctl functions that V4l2 driver is returning EBUSY for thet requests !
Therefore, one workaround should be to change the default value directly in highgui/cap_v4l.cpp :
*#define DEFAULT_V4L_WIDTH 1280 // Originally 640*
*#define DEFAULT_V4L_HEIGHT 720 // Originally 480*
After that, I just recompiled opencv ... and arrived to get 1280 x 720 without any problem ! Of course, a better fix should be to stop the acquisition, change the parameters, and restart stream after, but I'm not enough familiar with opencv for doing that !
Hope it will help.
Michel BEGEY
Try this:
capture = cvCreateCameraCapture(-1);
//set resolution
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, frameWidth);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, frameHeight);
cvQueryFrame(capture);
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_WIDTH, any_supported_size );
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_HEIGHT, any_supported_size);
cvQueryFrame(capture);
should be just enough!

Resources