Questions about fio output interpretation - benchmarking
I'd like to ask your help to interpret the test result of fit benchmark.
I tested it for AWS EBS volume benchmark.
It writes 16 files, 1G for each, as 16k per request.
I think the total write io size should be 1024MBx16=16384MB, but the output said it is 575184KB.
Could you let me know how this result can come out?
(The target directory /mnt/test is combined by two volumes as RAID 0.)
The output is like below:
$ sudo fio --directory=/mnt/test --name fio_rw_test --direct=1 --rw=randwrite --bs=16k --size=1G --numjobs=16 --time_based --runtime=180 -group_reporting -norandommap
fio_rw_test: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.14
Starting 16 processes
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
fio_rw_test: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 14 (f=14): [w(2),_(1),w(7),_(1),w(5)] [33.5% done] [0KB/3312KB/0KB /s] [0/207/0 iops] [eta 05m:59s]
fio_rw_test: (groupid=0, jobs=16): err= 0: pid=20460: Thu Jan 21 15:41:41 2021
write: io=575184KB, bw=3192.8KB/s, iops=199, runt=180152msec
clat (usec): min=545, max=201975, avg=80143.65, stdev=76991.72
lat (usec): min=546, max=201976, avg=80144.34, stdev=76991.75
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 692], 10.00th=[ 716], 20.00th=[ 764],
| 30.00th=[ 804], 40.00th=[ 860], 50.00th=[80384], 60.00th=[158720],
| 70.00th=[158720], 80.00th=[160768], 90.00th=[160768], 95.00th=[160768],
| 99.00th=[160768], 99.50th=[160768], 99.90th=[168960], 99.95th=[171008],
| 99.99th=[199680]
lat (usec) : 750=18.07%, 1000=25.65%
lat (msec) : 2=1.89%, 4=0.09%, 10=0.40%, 20=0.56%, 50=1.59%
lat (msec) : 100=2.75%, 250=48.99%
cpu : usr=0.01%, sys=0.05%, ctx=36282, majf=0, minf=147
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=35949/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
**WRITE: io=575184KB, aggrb=3192KB/s, minb=3192KB/s, maxb=3192KB/s, mint=180152msec, maxt=180152msec**
Disk stats (read/write):
md1: ios=0/36040, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/18019,
aggrmerge=0/1, aggrticks=0/1442112, aggrin_queue=1350932, aggrutil=91.25%
nvme9n1: ios=0/17898, merge=0/1, ticks=0/368388, in_queue=327596, util=16.08%
nvme8n1: ios=0/18140, merge=0/1, ticks=0/2515836, in_queue=2374268, util=91.25%
Related
Confusion in max connections allowed in AWS RDS types
I know we can find the max connections by {DBInstanceClassMemory/12582880}(12582880 is the magic number for my AWS resources). However, for db.m4.large (8 GB RAM) I checked online at many places that the maxConnections are 648. However, when I made the calculations, I found
8 * 1000000000/12582880 = 635.7844944877
8 * 1024 * 1024 * 1024 / 12582880 = 682.6684027822
Similarly for db.t2.small
2 * 1000000000/12582880 = 158.9461236219
2 * 1024 * 1024 * 1024 / 12582880 = 170.6671006955
acc to the internet: 150
Please help with finding the correct number. I cannot open MySQL console on the AWS instance due to some restrictions.
This AWS page has a table of containers and max_connections limits (default): https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Managing.Performance.html excerpt: db.t2.small 45 db.t2.medium 90 db.t3.small 45 db.t3.medium 90 db.r3.large 1000 Text says: the default connection limit is derived using a formula based on the DBInstanceClassMemory value.
What do the values in 'alsa --dump-hw-params' represent?
I am having problems configuring ALSA on my RHEL 7.5 machine. Part of my solution is to attempt to change settings in /etc/asound.conf. I have tried numerous permutations but I continue to hear "jitter" in my sounds (.raw files). I am using the 'aplay --dump-hw-params to get the params for my sound HW. Using this command: aplay --dump-hw-params Front_Center.wav These are the results I get: Playing WAVE 'Front_Center.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Mono HW Params of device "default": -------------------- ACCESS: MMAP_INTERLEAVED MMAP_NONINTERLEAVED MMAP_COMPLEX RW_INTERLEAVED RW_NONINTERLEAVED FORMAT: S8 U8 S16_LE S16_BE U16_LE U16_BE S24_LE S24_BE U24_LE U24_BE S32_LE S32_BE U32_LE U32_BE FLOAT_LE FLOAT_BE FLOAT64_LE FLOAT64_BE MU_LAW A_LAW IMA_ADPCM S24_3LE S24_3BE U24_3LE U24_3BE S20_3LE S20_3BE U20_3LE U20_3BE S18_3LE S18_3BE U18_3LE U18_3BE SUBFORMAT: STD SAMPLE_BITS: [4 64] FRAME_BITS: [4 640000] CHANNELS: [1 10000] RATE: [4000 4294967295) PERIOD_TIME: (11609 11610) PERIOD_SIZE: (46 49864571) PERIOD_BYTES: (23 4294967295) PERIODS: (0 17344165) BUFFER_TIME: [1 4294967295] BUFFER_SIZE: [92 797831566] BUFFER_BYTES: [46 4294967295] TICK_TIME: ALL -------------------- I'd like to know what the values within parens and braces mean in general. Are they ranges? What is the difference between the use of parens vs. braces? Thanks, Ian
Minimum and maximum values as supported by the specific hardware device you are using.
Numericals on Token Bucket
Question For a host machine that uses the token bucket algorithm for congestion control, the token bucket has a capacity of 1 mega byte and the maximum output rate is 20 mega bytes per second. Tokens arrive at a rate to sustain output at a rate of 10 mega bytes per second. The token bucket is currently full and the machine needs to send 12 mega bytes of data. The minimum time required to transmit the data is _____________ seconds. My Approach Initially token bucket is full. the rate at which it is emptying is (20-10) Mbps. time take to empty token bucket of 1 mb is 1/10 i.e 0.1 sec But answer is given as 1.2sec .
Token bucket has a capacity of 1 mega byte (maximum capacity C ) Here one byte is considered as one token ⇒ C=1 M tokens output rate is 20 mega bytes per second (M=20MBps) Tokens arrive at a rate to sustain output at a rate of 10 mega bytes per second ⇒20-R=10 ⇒ Input Rate R=10MBps Unlike Leaky Bucket , idle hosts can capture and save up c ≤ C tokens in order to send larger bursts later. s When we begin transfer the tokens present in token buckt is transmitted at once to the network ie. if initially capacity of token bucket is 'c' then c tokens will be instantly be present in the network. Time to Empty the token bucket c: is the inital capacity of token bucket R: every sec we are getting R tokens M : evey seconds M tokens are produced INPUT FLOW : Then the number of packets that are ready to enter the network during a time interval 't' is c+Rt OUTPUT FLOW : Then the number of packets that are ready to enter the network during a time interval 't' is Mt INPUT FLOW = OUTPUT FLOW ⇒ c+Rt = Mt t= c/M-R =1/20-10 =0.1sec Given that Token bucket is full (c=C) Now , We have got two cases To transfer 1M tokens , Will it be instantly with t=0 Or to transfer 1M tokens , we take 10/ 20-10 = 0.1sec ? To transfer 1M (inital token) tokens , Will it be instantly with t=0 Consider the equation INPUTFLOW = c+Rt This means that " c tokens (initally contained in token bucket ) are transmitted without any delays " Unlike Leaky bucket , token buckets can keep on reserving token if the sender is idle .Once it is ready to send the packets . Packets will take the token and will be transmitted to the network. ⇒ c And then we are adding the R tokens produced in 't' time to finnaly get the INPUTFLOW ⇒ 1 MB is transmitted instantly . Now we are left with 11 MB to transmit To trnasfer remaining 11 MB at t=0 we begin transmitting 11 MB data. at t=0.1sec : 1MB (1 MB transfered) at t=0.2sec : 1MB (2 MB transfered) .. .. at t=1.1 sec : 1MB (11 MB transfered ) Therefore to transfer 12MB it takes 1.1sec + 0 sec = 1.1 sec Transfer 1M (inital token) tokens , we take = 0.1sec ( if it take 0.1 sec for 1MB i could argue that it will take 1.2ssec for 12MB ) then during 0.1sec . 01 *10MBps = 1M tokens are fulled up . t=0s : begin to transfer 12 MB data. t=0.1s : 1MB t=0.2s : 1MB (2 MB transfered) t=0.3s : 1MB (3 MB transfered) .. .. t=1.2s : 1MB (12 MB transfered) Therefore to transfer 12MB it takes 1.2sec Question does clearly mention about this part . Hence it is common practice to always follw the best case . Therefore the answer would be 1.1 sec More Information : Visit Gate Overflow - Gate 2016 Question on Token Bucket
out of order write a file python
I have multiple segments saved in an input file. The format is; Use case 1: host port start_byte end_byte 127.0.0.1 12345 0 2048 127.0.0.1 12346 0 1024 127.0.0.1 12347 1024 2048 Use case 2: host port start_byte end_byte 127.0.0.1 12345 0 2048 127.0.0.1 12346 1024 2048 127.0.0.1 12347 0 1024 Here, the first line is for the reference to understand what each line has. The host is the localhost but ports are different. Here we have 3 ports. Port #12345 has entire file (say abc.txt). Port #12347 has the second segment whereas the port #12346 has the first segment. Now, I want to read file from the end of file towards the start of file(from line 3 to 1). The code to download each segment and write to a new file is given below. def downloadSegment(threadName, fileNameTemp, server_addr, server_port, segment_beginaddr, segment_endaddr, fileName, maxSegmentSize,ip_address,peer_server_port, relevant_path): downloadSegmentStr = "download," + fileName + ","+segment_beginaddr+"," + segment_endaddr socket1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) socket1.connect((server_addr, int(server_port))) socket1.send(downloadSegmentStr) lock.acquire() with open(fileNameTemp, 'ab') as file_to_write: file_to_write.seek(int(segment_beginaddr),0) while True: data = socket1.recv(maxSegmentSize) #print data if not data: break #print data file_to_write.write(data) file_to_write.close() lock.release(); socket1.close() When I write the segment in increasing order (Use case 1), then, it works perfectly. But, when I try using the out of order like explained in above Use case 2, it doesn't work. Any help is appreciated. Thanks.
You missed these parts of the Python Documentation: open(name[, mode[, buffering]]) … 'a' for appending (which on some Unix systems means that all writes append to the end of the file regardless of the current seek position). file.seek(offset[, whence]) … Note that if the file is opened for appending (mode 'a' or 'a+'), any seek() operations will be undone at the next write. If the file is only opened for writing in append mode (mode 'a'), this method is essentially a no-op… Thus, mode 'a' is unsuitable for the task. Sadly, stdio's fopen() offers no mode without a that creates a file if it doesn't exist and doesn't truncate it if it exists. So I'd use os.open() like in: with os.fdopen(os.open(fileNameTemp, os.O_CREAT|os.O_WRONLY), 'wb') as file_to_write:
capturing webcam stream with V4L2 failed
I am getting started with the V4L2 framework on Ubuntu 10.4. currently I am using an webcam to do some tests. I am following this documentation to start, the installation was worked fine. I downloaded and compiled the application example. The problems is video output,I call the executable using: # modprobe -r pwc # modprobe -v pwc fps=15 compression=3 mbufs=4 fbufs=4 size=vga # ./capturer_mmap -D /dev/video0 -w 640*480 -p 0 | ./viewer -w 640*480 -p 0 given this output: Output on terminal: window size 640*480 Video bytespreline = 1280 Display: Image byte order = LSBFirst Bitmap unit = 32 Bitmap bit order = LSBFirst Bitmap pad = 32 Window: Depth = 24 Red mask = 0x00ff0000 Green mask = 0x0000ff00 Blue mask = 0x000000ff Bits per R/G/B = 8 Image byte order = LSBFirst Bitmap unit = 32 Bitmap bit order = LSBFirst Bitmap pad = 32 Depth = 24 Red mask = 0x00ff0000 Green mask = 0x0000ff00 Blue mask = 0x000000ff Bits per pixel = 32 Bytes per line = 2560 IsShared = True XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":0.0" after 431 requests (19 known processed) with 0 events remaining. root#my-laptop:/home/foo/V4l2_samples-0.4.1# ./capturer_mmap -D /dev/video0 -w 640*480 -p 0 | ./viewer -w 640*480 -p 0 window size 640*480 Video bytespreline = 1280 Display: Image byte order = LSBFirst Bitmap unit = 32 Bitmap bit order = LSBFirst Bitmap pad = 32 Window: Depth = 24 Red mask = 0x00ff0000 Green mask = 0x0000ff00 Blue mask = 0x000000ff Bits per R/G/B = 8 Image byte order = LSBFirst Bitmap unit = 32 Bitmap bit order = LSBFirst Bitmap pad = 32 Depth = 24 Red mask = 0x00ff0000 Green mask = 0x0000ff00 Blue mask = 0x000000ff Bits per pixel = 32 Bytes per line = 2560 IsShared = True XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":0.0" after 101 requests (19 known processed) with 0 events remaining. I have no idea how to fix this. I belive the probrem is in C code because I can to use webcam with Webcam Chesse application. Any help is very appreciated. Thanks a lot!
it looks like you are displaying the image in a completely wrong format. when working with v4l2, you should definitely check out "libv4l" (packaged in debian, so also available in ubuntu). v4l2 allows a device to output it's frames in any of a very large number of video-formats, some of those are compressed (e.g. using jpeg). core v4l2 does not provide any means to convert the image into a given format your application supports, so in theory your application must support all possible formats. in order to avoid code duplication (each v4l2-capable application faces the same problem!), libv4l was created: it allows low-level access to the device, but at the sametime guarantees that the frame can be access using a few standard formats. e.g. if the device only supports jpeg-output and your app requests RGB32 frames, libv4l will transparently convert for you. you can even use libv4l with some LD_PRELOAD tricks, in order to make it work with applications that have been compiled without libv4l-support (just to check whether my suggestion makes sense)