I have a server running HP UX 11 OS and i'm trying to have I/O statistics for File System (not disk).
Fo example , I have 50 disk attached to the server when I type iostat (here under output of iostat for 3 disks) :
disk9 508 31.4 1.0
disk10 53 1.5 1.0
disk11 0 0.0 1.0
And I have File Systems (df output) :
/c101 (/dev/VGAPPLI/c101_lv): 66426400 blocks 1045252 i-nodes
/c102 (/dev/VGAPPLI/c102_lv): 360190864 blocks 5672045 i-nodes
/c103 (/dev/VGAPPLI/c103_lv): 150639024 blocks 2367835 i-nodes
/c104 (/dev/VGAPPLI/c104d_lv): 75852825 blocks 11944597 i-nodes
Is it possible to have I/O statistic specifically for these File System ?
Thanks.
Related
I'm trying to understand where uboot stores the environment variables in emmc. I have the following set in the uboot config file -
CONFIG_ENV_IS_NOWHERE=y
CONFIG_ENV_IS_IN_MMC=y
CONFIG_SYS_MMC_ENV_DEV=2
CONFIG_SYS_MMC_ENV_PART=0
CONFIG_ENV_SIZE=0x4000
CONFIG_ENV_OFFSET=0x400000
CONFIG_ENV_SECT_SIZE=0x10000
uboot is able to save the env variables in emmc and I can read them from Linux. Variables set from Linux are also readable in uboot. I fail to understand which partition has the environment variables?
Output lsblk from Linux -
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mtdblock0 31:0 0 16M 0 disk
mmcblk2 179:0 0 7.3G 0 disk
|-mmcblk2p1 179:1 0 83.2M 0 part /run/media/mmcblk2p1
|-mmcblk2p2 179:2 0 1.7G 0 part /
|-mmcblk2p3 179:3 0 83.2M 0 part /run/media/mmcblk2p3
|-mmcblk2p4 179:4 0 1K 0 part
|-mmcblk2p5 179:5 0 1.7G 0 part /run/media/mmcblk2p5
`-mmcblk2p6 179:6 0 1G 0 part /run/media/mmcblk2p6
mmcblk2boot0 179:32 0 4M 1 disk
mmcblk2boot1 179:64 0 4M 1 disk
mmcblk2boot0 is where uboot is located which is 4MB in size, considering the environment is saved at an offset of 0x400000 in mmcblk2 device, it should be located just after mmcblk2boot0 which points to the parition mmcblk2boot1, but if I dump the strings "strings /dev/mmcblk2boot1", I get nothing.
I have provided a file /etc/fw_config which is used by fw_printenv and fw_saveenv, this file contains "/dev/mmcblk2 0x400000 0x10000". All the settings point to the fact that uboot environment is located at an offset of 0x400000 in mmcblk2 device. Any pointers on which partition listed in output of lsblk holds the uboot environment variables..?
Thanks,
Vinay
The confusing thing here is that CONFIG_SYS_MMC_ENV_PART does not refer to sofware partitions, such as /dev/mmcblk2p1 that you see under Linux, but rather hardware partitions such as /dev/mmcblk2 (partition 0) or /dev/mmcblk2boot0 (partition 1).
I am using mongodb version 2.4 due to 32 bit system limitation (nanopi m1 plus). I have a debian jessie OS image (Debian 8) with 4.2 GB space available (emmc). I have a total of about 2.2 GB available after loading my application files. However, my flash gets filled up quickly to 100% after I start running my application.
Then I get the error "Unable to get database instance and mongodb stopped working " and my application stops working.
Can someone please help me with this problem. Thanks in advance!
Memory status of my device when it stopped working:
df –h:
Filesystem- overlay Size- 4.2G Used- 4.2G Avail- 0 Use%- 100% Mounted on- /
df -h command
du –shx /var/lib/mongodb/ | sort –rh | head –n 20
512M /var/lib/mongodb/xyz.6
512M /var/lib/mongodb/xyz.5
257M /var/lib/mongodb/xyz.4
128M /var/lib/mongodb/xyz.3
64M /var/lib/mongodb/xyz.2
32M /var/lib/mongodb/xyz.1
17M /var/lib/mongodb/xyz.ns
17M /var/lib/mongodb/xyz.0
16M /var/lib/mongodb/local.ns
16M /var/lib/mongodb/local.0
4.0K /var/lib/mongodb/journal
0 /var/lib/mongodb/mongodb.lock
du –shx /var/lib/mongodb/journal/* | sort –rh | head –n 20
257M /var/lib/mongodb/journal/prealloc.2
257M /var/lib/mongodb/journal/prealloc.1
257M /var/lib/mongodb/journal/prealloc.0
du –shx /var/log/mongodb/journal/* | sort –rh | head –n 20
399M /var/lib/mongodb/mongodb.log
353M /var/lib/mongodb/mongodb.log.1
3.3M /var/lib/mongodb/mongodb.log.2.gz
752K /var/lib/mongodb/mongodb.log.1.gz
I'm having some troubles to understand the crash dump and to find what is the root cause of the OutOfMemoryException thrown by the WPF application. The exception is thrown after the application has been run for several hours so this clearly indicates that there is a memory leak.
My first step was to look at !address -summary command :
--- Usage Summary ---------------- RgnCount ------- Total Size -------- %ofBusy %ofTotal
<unknown> 2043 58997000 ( 1.384 Gb) 71.43% 69.22%
Heap 152 fcc3000 ( 252.762 Mb) 12.74% 12.34%
Image 1050 bc77000 ( 188.465 Mb) 9.50% 9.20%
Stack 699 7d00000 ( 125.000 Mb) 6.30% 6.10%
Free 518 3f6b000 ( 63.418 Mb) 3.10%
TEB 125 7d000 ( 500.000 kb) 0.02% 0.02%
Other 12 36000 ( 216.000 kb) 0.01% 0.01%
PEB 1 1000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 2186 685b7000 ( 1.631 Gb) 84.14% 81.53%
MEM_IMAGE 1710 f3f3000 ( 243.949 Mb) 12.29% 11.91%
MEM_MAPPED 186 46db000 ( 70.855 Mb) 3.57% 3.46%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_COMMIT 3366 73fe7000 ( 1.812 Gb) 93.52% 90.62%
MEM_RESERVE 716 809e000 ( 128.617 Mb) 6.48% 6.28%
MEM_FREE 518 3f6b000 ( 63.418 Mb) 3.10%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 1650 5e19e000 ( 1.470 Gb) 75.87% 73.52%
PAGE_EXECUTE_READ 224 bc42000 ( 188.258 Mb) 9.49% 9.19%
PAGE_READWRITE|PAGE_WRITECOMBINE 28 439f000 ( 67.621 Mb) 3.41% 3.30%
PAGE_READONLY 573 3d7b000 ( 61.480 Mb) 3.10% 3.00%
PAGE_WRITECOPY 214 f8f000 ( 15.559 Mb) 0.78% 0.76%
PAGE_EXECUTE_READWRITE 265 d0a000 ( 13.039 Mb) 0.66% 0.64%
PAGE_READWRITE|PAGE_GUARD 357 33b000 ( 3.230 Mb) 0.16% 0.16%
PAGE_EXECUTE_WRITECOPY 55 119000 ( 1.098 Mb) 0.06% 0.05%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unknown> 78d40000 2350000 ( 35.313 Mb)
Heap 36db0000 fd0000 ( 15.813 Mb)
Image 64a8c000 e92000 ( 14.570 Mb)
Stack 4b90000 fd000 (1012.000 kb)
Free 7752f000 1a1000 ( 1.629 Mb)
TEB 7ede3000 1000 ( 4.000 kb)
Other 7efb0000 23000 ( 140.000 kb)
PEB 7efde000 1000 ( 4.000 kb)
This shows that the memory is quite high.
Then I'm looking at GC heap size with eeheap -gc command. It shows that the heap is quite big (1.1GB) which indicates that there is a problem within the managed part of the application.
5fc90000 5fc91000 60c7acd4 0xfe9cd4(16686292)
5a060000 5a061000 5b05e9c0 0xffd9c0(16767424)
56de0000 56de1000 57ddf1c4 0xffe1c4(16769476)
57de0000 57de1000 58ddbbbc 0xffabbc(16755644)
73ff0000 73ff1000 74fe0f5c 0xfeff5c(16711516)
50de0000 50de1000 51dcfa58 0xfeea58(16706136)
5b060000 5b061000 5c05ca54 0xffba54(16759380)
4fde0000 4fde1000 50ddfd8c 0xffed8c(16772492)
Large object heap starts at 0x03921000
segment begin allocated size
03920000 03921000 049013d0 0xfe03d0(16647120)
14850000 14851000 15837380 0xfe6380(16671616)
178d0000 178d1000 1889a3e0 0xfc93e0(16552928)
1a1c0000 1a1c1000 1b1abca8 0xfeaca8(16690344)
40de0000 40de1000 41dc8b48 0xfe7b48(16677704)
42de0000 42de1000 43827170 0xa46170(10772848)
54de0000 54de1000 55dd6d18 0xff5d18(16735512)
Total Size: Size: 0x448fde94 (1150279316) bytes.
------------------------------
GC Heap Size: Size: 0x448fde94 (1150279316) bytes.
Notice that there are 64 segments and each about (16MB). It seems that there is some data held in the memory and never released.
Next I look at !dumpheap -stat:
65c1f26c 207530 19092760 System.Windows.Media.GlyphRun
65c2c434 373991 20943496 System.Windows.Media.RenderData
68482bb0 746446 26872056 MS.Utility.ThreeItemList`1[[System.Double, mscorlib]]
65c285b4 746448 29857920 System.Windows.Media.DoubleCollection
64c25d58 299568 32353344 System.Windows.Data.BindingExpression
6708a1b8 2401099 38417584 System.WeakReference
67082c2c 1288315 41226080 System.EventHandler
67046f80 1729646 42238136 System.Object[]
64c1409c 206969 52156188 System.Windows.Controls.ContentPresenter
67094c9c 382163 64812664 System.Byte[]
004b0890 159 65181140 Free
64c150d0 207806 72316488 System.Windows.Controls.TextBlock
6708fd04 1498498 97863380 System.String
6848038c 847783 128775772 System.Windows.EffectiveValueEntry[]
As I understand it, there is no a signle object that takes all the memory. The biggest one is just about 122MB. Summing up all the sizes (8500 lines of outputed lines) gives the (1.1GB) of occupied memory. It seems that all the object graph is somehow duplicated and added to the memory and never released.
The !gcroot 6848038c or !gcroot 6708fd04 to inspect how EffectiveValueEntry and System.String are reachable, never ends, the stack is soooo big...
dumpheap -mt <address> doesn't show me something that srikes me. !finalizequeue shows that there are many objects (more that 2 millions) registered for finalization :
6708a1b8 2401099 38417584 System.WeakReference
Total 2417538 objects
I suspect that the OutOfMemoryException occures when the application tries to duplicate the object graph and allocate new memory, but I cannot find the root cause of it.
Question How can I drill down to the root of the problem (what other command of windbg can I use to check it). As it seems that not just one object is leaking but the whole object graph.. Am I on the right track or there is something else I'm overlooking ? What are other hypothesis ?
Object graph duplication
Your application uses ~1.1 GB of virtual memory by .NET. You can see that from the output of !eeheap -gc directly
GC Heap Size: Size: 0x448fde94 (1150279316) bytes.
or by summing up the values of !dumpheap -stat.
Summing up all the sizes (8500 lines of output lines) gives the (1.1GB)
This roughly correlates to the value displayed as <unknown> in !address -summary.
--- Usage Summary ---------------- RgnCount ------- Total Size -------- %ofBusy %ofTotal
<unknown> 2043 58997000 ( 1.384 Gb) 71.43% 69.22%
There is no reason to assume that the whole object graph is being duplicated. This is a normal situation.
OutOfMemory
At the moment, there are 65 MB of virtual memory already committed by .NET and marked as free (from !dumpheap -stat):
004b0890 159 65181140 Free
Unfortunately, those 65 MB are split into 159 smaller regions. to get the largest block of those, you need to run a !dumpheap -mt 004b0890.
In addition, .NET could obtain another 63 MB from Windows (from !address -summary):
--- Usage Summary ---------------- RgnCount ------- Total Size --------
Free 518 3f6b000 ( 63.418 Mb)
But the largest block is only 1.6 MB, so that's almost useless:
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 7752f000 1a1000 ( 1.629 Mb)
So, clearly, the application is out of memory.
Where is the memory?
252 MB are in native heaps. It seems you're using some native DLLs. While that does not seem too much at the moment, this fact might could indicate the presence of pinned objects. Pinned objects are not garbage collected. Look at the output of !gchandles to find out whether that could be part of the problem.
188 MB are in DLLs. You can unload native DLLs which are not in use, but for .NET assemblies, you probably can't do much about that.
125 MB are in stacks. With a default size of 1 MB, it seems that there are 125 threads in your application. Use !clrstack to find out what they are doing and why they did not finish yet. Potentially, each thread works on something and has not freed the objects yet. If you have it under your control, do not start so many threads in parallel. E.g. use 8 threads only and wait for threads to finish before doing the next piece of work.
Of course the majority of memory is used by .NET objects. However, you drew a few wrong conclusions.
As I understand it, there is no a single object that takes all the memory. The biggest one is just about 122MB.
Note that there is not a single object EffectiveValueEntry[] eating 122 MB ob memory. There are 847.783 of them. This changes the question from "Why does this object use so much memory?" to "Why are there so many of them?". For example, why does your application need 207.806 text blocks? Is it really displaying so much text?
Using !gcroot is a good idea. However, you used it with the address of a method table instead of an object:
!gcroot 6848038c
!gcroot 6708fd04
These were both numbers from the output of !dumpheap -stat. Using them in !gcroot should have given a warning like
Please note that 6848038c is not a valid object.
Instead, !gcroot works on individual objects only which you get from a !dumpheap without the -stat parameter.
You can use !traveseheap filename.log to dump all objects into a file compatible with CLR profiler [Codeplex]. Note that CLR Profiler cannot read the -xml format. After loading the object information, Heap Graph is probably the most useful button for you.
To find out the trigger of the OutOfMemoryException, you can use the !u command. You'll need to read some MSIL code to understand what it does. See also How to identify array type. However, in your scenario, I guess that's useless, because even small objects could trigger this.
In addition to the other answer, you can try to visualize the memory adresses and the GC heap with the WinDbg extension cosos gcview
If I use
/usr/bin/time -f"%e,%P,%M,%I,%O"
I get (for the last three placeholders) the memory the process used, and if there was some input and output during it.
Obviously, it's easy to get %e or something like it using sys/time.h, but is there a way to get %M, %I and %O programmatically?
You could read and parse the files in the /proc filesystem. /proc/self refers to the process accessing the /proc filesystem.
/proc/self/statm contains information about memory usage, measured in pages. Sample output:
% cat /proc/self/statm
1115 82 63 12 0 79 0
Fields are size resident share text lib data dt; see the proc manual page for some additional details.
/proc/self/io contains the I/O for the current process. Sample output:
% cat /proc/self/io
rchar: 2012
wchar: 0
syscr: 6
syscw: 0
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0
Unfortunately, io isn't documented in the proc manual page (at least on my Debian system). I had too check the iotop source code to see how it obtained the per process I/O information.
I was just going through SO and found out a question Determining CPU utilization
The question is interesting and the one which is more intersting is the answer.
So i thought doing some checks on my solaris SPARC unix system.
i went to /proc as root user and i found out some directories with numbers as their names.
I think these numbers are the process id's.Surprisingly i did not find /stat.(donno why?..)
i took one process id(one directory) and checked whats present inside it.below is the output
root#tiger> cd 11770
root#tiger> pwd
/proc/11770
root#tiger> ls
as contracts ctl fd lstatus lwp object path psinfo root status watch
auxv cred cwd lpsinfo lusage map pagedata priv rmap sigact usage xmap
i did check what are those files :
root#tigris> file *
as: empty file
auxv: data
contracts: directory
cred: data
ctl: cannot read: Invalid argument
cwd: directory
fd: directory
lpsinfo: data
lstatus: data
lusage: data
lwp: directory
map: TrueType font file version 1.0 (TTF)
object: directory
pagedata: cannot read: Arg list too long
path: directory
priv: data
psinfo: data
rmap: TrueType font file version 1.0 (TTF)
root: directory
sigact: ascii text
status: data
usage: data
watch: empty file
xmap: TrueType font file version 1.0 (TTF)
i am not sure ..given this how can i determine the cpu utilization?
for eg: what is the idle time of my process.
can anyone give me the right direction?
probably with an example!
As no one else is taking the bait, I'll add some comments/answers.
1st off, Did you check out the info available for Solaris System tuning? This is for old Solarian, 2.6, v7 & 8. Presumably a little searching at developers.sun.com will find something newer.
You wrote:
I went to /proc as root user and i found out some directories with numbers as their names. I think these numbers are the process id's.Surprisingly i did not find /stat.(donno why?..)
Many non-Linux OS's have their own special conventions on how processes are managed.
For Solaris, the /proc directory is not a directory of disk-based files, but information about all of the active system processes arranged like a directory hierarchy. Cool, right?!
I don't know the exact meaning of stat, status? statistics? something else? but that is just the convention used a different OS's directory structure that is holding the process information.
As you have discovered, below /proc/ are a bunch of numbered entries, these are the active processIDs. When you cd into any one of those, then you're seeing the system information available for that process.
I did check what are those files : ....
I don't have access to Solaris servers any more, so we'll have to guess a little. I recommend 'drilling down' into any file or directory whose name hints at anything related.
Did you try cat psinfo? What did that produce?
If the solaris tuning page didn't help, then is your appropos is working? Do appropos proc and see what man-pages are mentioned. Drill down on those. Else try man proc, andd look near the bottom of the entry for the 'see also' section AND for the Examples section.
(Un)?fortunately, most man pages are not tutorials, so reading through these may only give you an idea on how much more you need to study.
You know about the built-in commands that offer some performance monitoring capabilities, i.e. ps, top, etc?
And the excellent AIX-based nmon has been/is being? ported to Solaris too, see http://sourceforge.net/projects/sarmon/.
There are also expensive monitoring/measuring/utilization tools that network managers like to have, as a mere developer, we never got to use them. Look at the paid ads when you google for 'solaris performance monitoring'.
Finally, keep in mind this excellent observation from the developer of the nmon-AIX system monitor included in the FAQ for nmon :
If you keep using shorter and shorter periods you will eventually see that the CPUs are either 100% busy or 100% idle all the other numbers are just a feature of humans not thinking fast enough and having to average out the CPU use in longer periods.
I hope this helps.
There is no simple and accurate way to get the CPU utilization from Solaris /proc hierarchy.
Unlike Linux which use it to store various system information and statistics, Solaris is only presenting process related data under /proc.
There is also another difference. Linux is usually presenting preprocessed readable data (text) while Solaris is always presenting the actual kernel structures or raw data (binary).
All of this is fully documented in Solaris 46 pages proc manual ( man -s 4 proc )
While it would be possible to get the CPU utilization by summing the usage per process from this hierarchy, i.e. by reading the /proc//xxx file, the usual way is through the Solaris kstat (kernel statistics) interface. Moreover, the former method would be inaccurate by missing CPU usage not accounted to processes but directly to the kernel.
kstat (man -a kstat) is what are using under the hood all the usual commands that report what you are looking for like vmstat, iostat, prstat, sar, top and the likes.
For example, cpu usage is displayed in the last three columns of vmstat output (us, sy and id for time spend in userland, kernel and idling).
$ vmstat 10 8
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr cd s0 -- -- in sy cs us sy id
0 0 0 1346956 359168 34 133 96 0 0 0 53 11 0 0 0 264 842 380 9 7 84
0 0 0 1295084 275292 0 4 4 0 0 0 0 0 0 0 0 248 288 200 2 3 95
0 0 0 1295080 275276 0 0 0 0 0 0 0 3 0 0 0 252 271 189 2 3 95
0 0 0 1295076 275272 0 14 0 0 0 0 0 0 0 0 0 251 282 189 2 3 95
0 0 0 1293840 262364 1137 1369 4727 0 0 0 0 131 0 0 0 605 1123 620 15 19 66
0 0 0 1281588 224588 127 561 750 1 1 0 0 89 0 0 0 438 1840 484 51 15 34
0 0 0 1275392 217824 31 115 233 2 2 0 0 31 0 0 0 377 821 465 20 8 72
0 0 0 1291532 257892 0 0 0 0 0 0 0 8 0 0 0 270 282 219 2 3 95
If for some reason you don't want to use vmstat, you can directly get the kstat counters by using the kstat command but that would be cumbersome and less portable.