Good afternoon,
I am working on a Matlab project and I have stored some data in an array. I would like to plot a plot like the plot shown below. However, I don't know what plotting function I need to use and how, in order to obtain the image plot (it will be not the same, but this style).
My data is on a 11x16 - matrix.
Thank you guys so much beforehand!
#rayryeng
It was a really useful answer, although I didn't need that exact shape. I need the shape that my data would create, I've been trying to modify the code you wrote in order to obtain what I need but I did not obtained it...
My data is
data = ( 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 ;
8.00 8.02 8.04 8.07 8.12 8.20 8.30 8.42 8.53 8.63 8.72 8.80 8.86 8.91 8.96 9.00;
6.00 6.03 6.07 6.12 6.22 6.37 6.59 6.83 7.07 7.28 7.45 7.60 7.72 7.83 7.92 8.00;
4.00 4.03 4.07 4.14 4.26 4.48 4.85 5.26 5.63 5.95 6.21 6.43 6.61 6.75 6.88 7.00;
2.00 2.02 2.05 2.10 2.20 2.44 3.08 3.70 4.23 4.67 5.01 5.29 5.52 5.70 5.86 6.00;
0 0 0 0 0 0 1.33 2.24 2.93 3.47 3.88 4.21 4.46 4.67 4.84 5.00;
0 0 0 0 0 0 0 1.01 1.78 2.38 2.84 3.19 3.46 3.67 3.84 4.00;
0 0 0 0 0 0 0 0 0.80 1.43 1.91 2.25 2.51 2.70 2.86 3.00;
0 0 0 0 0 0 0 0 0 0.63 1.10 1.41 1.62 1.77 1.89 2.00;
0 0 0 0 0 0 0 0 0 0 0.44 0.66 0.79 0.88 0.94 1.00;
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
This is my matrix of data (sorry I know it's too long), well and when I try to plot writing:
[x,y] = meshgrid(1:16,1:11);
contourf(x,y,data,20,'LineStyle','none');
colorbar
It should have a different shape than what I get. I need to get that the part that are 0 (zeros) are like the white part of the plot I showed before. (Different shape though) I don't really know how to do it (my data should be read properly), if you could help me I would be really thankful.
Thank you so much for last answer.
It depends on your data, I believe you should use contourf.
This is as close as I could get,
[x,y] = meshgrid(1:16,1:11);
data = - y;
data(end,5:10) = NaN;
data(end-1,6:9) = NaN;
data(end-2,7:8) = NaN;
contourf(x,y,data,20,'LineStyle','none');
colorbar
with,
data = - y .* abs(log(sin(.10 * x - 5.5)+.5));
data(data < -4) = NaN;
So I suppose the code is right, it's matter of your data,
with data = max(data(:)) - data;
What you have is almost correct. All you need to do is set any data that is 0 to NaN. That way, when you throw it into contourf, those parts are not visualized. As such:
data = [10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 ;
8.00 8.02 8.04 8.07 8.12 8.20 8.30 8.42 8.53 8.63 8.72 8.80 8.86 8.91 8.96 9.00;
6.00 6.03 6.07 6.12 6.22 6.37 6.59 6.83 7.07 7.28 7.45 7.60 7.72 7.83 7.92 8.00;
4.00 4.03 4.07 4.14 4.26 4.48 4.85 5.26 5.63 5.95 6.21 6.43 6.61 6.75 6.88 7.00;
2.00 2.02 2.05 2.10 2.20 2.44 3.08 3.70 4.23 4.67 5.01 5.29 5.52 5.70 5.86 6.00;
0 0 0 0 0 0 1.33 2.24 2.93 3.47 3.88 4.21 4.46 4.67 4.84 5.00;
0 0 0 0 0 0 0 1.01 1.78 2.38 2.84 3.19 3.46 3.67 3.84 4.00;
0 0 0 0 0 0 0 0 0.80 1.43 1.91 2.25 2.51 2.70 2.86 3.00;
0 0 0 0 0 0 0 0 0 0.63 1.10 1.41 1.62 1.77 1.89 2.00;
0 0 0 0 0 0 0 0 0 0 0.44 0.66 0.79 0.88 0.94 1.00;
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0];
data(data == 0) = NaN;
[x,y] = meshgrid(1:16,1:11);
contourf(x,y,data,20,'LineStyle','none');
colorbar
This is what I get:
Given your comments, you want the y-axis to be reversed. Simply put axis ij; at the end of the code above to flip the y-axis so that y-down is the positive direction. If you do that, we get this figure:
Credit should go to Kamtal as he figured out where you needed to start. I just helped finish off the requirement.
Related
I am trying to add svg in my next project but i keep getting this error 'Error: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.'
code:index.js
import React from 'react'
import Image from 'next/image';
import {ReactComponent as Logo} from '/assets/Logo.svg';
const Header=()=>{
return (
<div className='flex'>
<Logo/>
</div>
)
}
export default Header;
svg file
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 74 20">
<defs>
<style>
.cls-1{fill:#28b16d}
</style>
</defs>
<path d="M51 13.53a.58.58 0 0 0-.11.08 2.77 2.77 0 0 1-2.28 1.1 4.79 4.79 0 0 1-2.32-.45 3.81 3.81 0 0 1-2.07-3 6.57 6.57 0 0 1-.07-1V6.07c0-.56.17-.73.73-.73h1.22c.49 0 .69.21.69.7v4.27a2.45 2.45 0 0 0 .5 1.56A2.25 2.25 0 0 0 51 11.3a1.13 1.13 0 0 0 .06-.41V6.06c0-.59.17-.76.75-.76H53c.49 0 .71.22.71.71v9.07a4.27 4.27 0 0 1-2.38 4 5.79 5.79 0 0 1-3.16.61 5.28 5.28 0 0 1-1.95-.45 1.23 1.23 0 0 1-.86-1.33v-.69a.44.44 0 0 1 .68-.43c.34.15.66.35 1 .48a3.87 3.87 0 0 0 2.42.16A2 2 0 0 0 51 15.46v-1.8a.54.54 0 0 0 0-.13zM24.34 5.55a13 13 0 0 1 1.32-.45 5.6 5.6 0 0 1 3.14.08 4.14 4.14 0 0 1 2.8 3.4 5.52 5.52 0 0 1-.09 2.62 4.48 4.48 0 0 1-2.8 3 6.13 6.13 0 0 1-5.76-.51 4.76 4.76 0 0 1-.73-.61 1.52 1.52 0 0 1-.48-1.08V1.71a.89.89 0 0 1 .48-.85l1.35-.79c.45-.26.77 0 .77.46V5.55zm0 4.3v1.83a.45.45 0 0 0 .22.42 3 3 0 0 0 2.35.47 2.27 2.27 0 0 0 1.85-1.48A3.61 3.61 0 0 0 29 9.44a2.55 2.55 0 0 0-.84-1.82 3.08 3.08 0 0 0-3.63.06.51.51 0 0 0-.13.34c-.07.61-.06 1.22-.06 1.83zM11.26-.07a7.34 7.34 0 0 1 6.91 5.77 7.41 7.41 0 0 1-4.36 8.48 7.16 7.16 0 0 1-8-1.79c-.4-.45-.41-.45-.92-.15l-3.33 2c-.57.33-.81.27-1.14-.3l-.31-.55a.64.64 0 0 1 .27-1l3.43-2c.52-.31.51-.3.35-.86A7.4 7.4 0 0 1 8.87.33a6.52 6.52 0 0 1 1-.26c.47-.07.93-.07 1.39-.14zm5.66 7.43a5.89 5.89 0 0 0-5.75-5.95 5.88 5.88 0 0 0-5.88 5.75A5.88 5.88 0 0 0 11 13.27a5.87 5.87 0 0 0 5.92-5.91zM39.91 13.43a1 1 0 0 0-.15.12 2.61 2.61 0 0 1-1.86 1.1 4.49 4.49 0 0 1-4.17-1.38 4.54 4.54 0 0 1-1.13-2.69 5.23 5.23 0 0 1 .47-3 4.6 4.6 0 0 1 3.13-2.42 7 7 0 0 1 5.28.64l.16.1a1.64 1.64 0 0 1 .9 1.62v6.07a2.13 2.13 0 0 1 0 .42.48.48 0 0 1-.5.42h-1.58c-.33 0-.52-.22-.55-.59v-.41zm0-5.63a.47.47 0 0 0-.25-.45A3.07 3.07 0 0 0 38.28 7a2.55 2.55 0 0 0-2.91 1.9 3.91 3.91 0 0 0 0 1.91A2.43 2.43 0 0 0 39.33 12a1.41 1.41 0 0 0 .57-1.22 59.2 59.2 0 0 1 0-2.98zM62.31 13.43a1.18 1.18 0 0 0-.16.14 2.52 2.52 0 0 1-1.73 1.07 4.54 4.54 0 0 1-3.22-.56 3.82 3.82 0 0 1-1.74-2.87 6.8 6.8 0 0 1-.07-1V6.02c0-.56.16-.73.73-.73h1.25A.59.59 0 0 1 58 6v4.29a2.46 2.46 0 0 0 .54 1.63 2.27 2.27 0 0 0 3.67-.63 1 1 0 0 0 0-.34V6.07c0-.56.18-.74.75-.74h1.19c.48 0 .69.22.69.7v7.69c0 .54-.2.74-.73.74H63c-.48 0-.67-.19-.68-.67l-.01-.36zM66.63 6.9V3.27a.85.85 0 0 1 .45-.81l1.35-.79.12-.06a.46.46 0 0 1 .68.44v2.89c0 .34 0 .35.36.35H73c.67 0 .81.14.82.8v.64c0 .44-.2.63-.63.64h-3.58c-.39 0-.39 0-.39.38v2.39a3.61 3.61 0 0 0 .15 1 2.09 2.09 0 0 0 2.3 1.44 2.52 2.52 0 0 0 1.45-.58l.13-.09a.51.51 0 0 1 .56-.1c.2.11.18.31.18.49v.85a1 1 0 0 1-.47.89 4 4 0 0 1-2.23.65 5 5 0 0 1-2.53-.56 3.91 3.91 0 0 1-2-3.08 16.67 16.67 0 0 1-.08-2c-.06-.72-.05-1.43-.05-2.15z" class="cls-1"/>
<path d="M14.91 8.06v1.88c0 .43-.18.61-.61.61h-1.44c-.34 0-.49-.14-.49-.48V8.56a1.29 1.29 0 0 0-1.09-1.25 1.26 1.26 0 0 0-1.39 1 3.3 3.3 0 0 0 0 .69v1.06c0 .35-.15.52-.51.52h-1.5a.54.54 0 0 1-.6-.58V6.26a.73.73 0 0 1 .4-.71q1.42-.8 2.81-1.65a1.05 1.05 0 0 1 1.21 0c.94.57 1.9 1.13 2.85 1.68a.62.62 0 0 1 .34.6v1.91z" class="cls-1"/>
</svg>
I searched on the error and it says there is a problem with the import/export. So how should i export/import the file??
I tested running bare bones code using ESP IDF on an ESP32 chip using duinotech XC-3800, and obtained the following results in terms of image size.
Analysis Binary Size for ESP32
Folder Structure
temp/
main/
CMakeLists.txt
main.c
CMakeLists.txt
File contents
CMakeLists.txt
# The following lines of boilerplate have to be in your project's
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(temp)
main>CMakeLists.txt
idf_component_register(SRCS "main.c"
INCLUDE_DIRS "")
Test 1 main>main.c
#include <stdio.h>
void app_main(void) {
printf("Hello world!\n");
for (int i = 10; i >= 0; i--) {
printf("Restarting in %d seconds...\n", i);
}
printf("Restarting now.\n");
fflush(stdout);
}
Test 2 main>main.c
#include <stdio.h>
void app_main(void) { printf("Hello world!\n"); }
Test 3 main>main.c
void app_main(void) {}
Size comparison
Obtained by running idf_size.py build/temp.map
Test 1
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75408 bytes
Flash rodata: 23844 bytes
Total image size:~ 146376 bytes (.bin may be padded larger)
Test 2
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75240 bytes
Flash rodata: 23796 bytes
Total image size:~ 146160 bytes (.bin may be padded larger)
Test 3
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75004 bytes
Flash rodata: 23780 bytes
Total image size:~ 145908 bytes (.bin may be padded larger)
Analysis
Size for code obtained by running stat --format="%s" main/main.c
All Sizes are in Bytes
Test No. | Code | Image | Flash Code | Flash rodata
-------- | -----| ------ | ---------- | ------------
1 | 207 | 146376 | 75408 | 23844
2 | 70 | 146160 | 75240 | 23796
3 | 43 | 145908 | 75004 | 23780
At least 145KB of boiler plate code just to get an empty main run.
Speculation
I suspect that the 145KB is made up of a number of libraries that are always loaded onto the chip whether you use them or not. Some of them must be the FreeRTOS, WiFi, HTTP etc.
Can we bring down this size somehow and load only the bare minimum required for operation?
You can get more detailed size information by running idf.py size-components and idf.py size-files. Here are the results for your "Test 3" (i.e. empty function) on my dev environment (note that my Total Image Size is already slightly larger than the one you posted.
Total sizes:
DRAM .data size: 7860 bytes
DRAM .bss size: 4128 bytes
Used static DRAM: 11988 bytes ( 168748 available, 6.6% used)
Used static IRAM: 32706 bytes ( 98366 available, 25.0% used)
Flash code: 76002 bytes
Flash rodata: 32164 bytes
Total image size:~ 148732 bytes (.bin may be padded larger)
Per-archive contributions to ELF file:
Archive File DRAM .data & .bss IRAM Flash code & rodata Total
libc.a 0 0 0 55348 3829 59177
libesp32.a 2132 2351 6767 5976 8076 25302
libfreertos.a 4140 776 12432 0 1653 19001
libdriver.a 76 20 0 4003 7854 11953
libsoc.a 161 4 4953 612 3852 9582
libvfs.a 240 103 0 5464 950 6757
libheap.a 877 4 3095 830 986 5792
libspi_flash.a 24 290 1855 779 890 3838
libefuse.a 16 4 0 1142 2213 3375
libnewlib.a 152 272 766 830 96 2116
libapp_update.a 0 4 109 159 1221 1493
libesp_ringbuf.a 0 0 848 0 209 1057
liblog.a 8 268 429 82 0 787
libhal.a 0 0 515 0 32 547
libpthread.a 8 12 0 256 0 276
libgcc.a 0 0 0 0 160 160
libbootloader_support.a 0 0 0 126 0 126
libm.a 0 0 0 88 0 88
libcxx.a 0 0 0 11 0 11
libxtensa-debug-module.a 0 0 8 0 0 8
libmain.a 0 0 0 5 0 5
(exe) 0 0 0 0 0 0
libmbedcrypto.a 0 0 0 0 0 0
libmbedtls.a 0 0 0 0 0 0
libwpa_supplicant.a 0 0 0 0 0 0
Per-file contributions to ELF file:
Object File DRAM .data & .bss IRAM Flash code & rodata Total
lib_a-vfprintf.o 0 0 0 14193 756 14949
lib_a-svfprintf.o 0 0 0 13838 756 14594
lib_a-svfiprintf.o 0 0 0 9642 1210 10852
lib_a-vfiprintf.o 0 0 0 9945 738 10683
uart.c.obj 44 12 0 2985 7293 10334
tasks.c.obj 12 700 5546 0 531 6789
vfs_uart.c.obj 48 63 0 3680 808 4599
panic.c.obj 2023 5 1989 0 0 4017
esp_err_to_name.c.obj 0 0 0 50 3947 3997
portasm.S.obj 3084 0 484 0 0 3568
lib_a-dtoa.o 0 0 0 3522 13 3535
multi_heap.c.obj 873 0 2277 0 0 3150
intr_alloc.c.obj 8 22 618 1703 722 3073
esp_efuse_utility.c.obj 0 0 0 867 2205 3072
cpu_start.c.obj 0 1 1068 331 1352 2752
queue.c.obj 0 0 2310 0 325 2635
lib_a-mprec.o 0 0 0 2134 296 2430
rtc_clk.c.obj 161 4 2098 0 0 2263
dbg_stubs.c.obj 0 2072 32 108 0 2212
vfs.c.obj 192 40 0 1784 142 2158
rtc_periph.c.obj 0 0 0 0 2080 2080
esp_timer_esp32.c.obj 8 26 1068 262 538 1902
xtensa_vectors.S.obj 8 0 1776 0 36 1820
flash_mmap.c.obj 0 264 1092 114 339 1809
task_wdt.c.obj 53 4 0 1178 548 1783
heap_caps.c.obj 4 0 818 52 617 1491
timers.c.obj 8 56 1006 0 233 1303
cache_utils.c.obj 4 14 749 81 402 1250
soc_memory_layout.c.obj 0 0 0 0 1181 1181
heap_caps_init.c.obj 0 4 0 778 369 1151
port.c.obj 0 16 625 0 493 1134
esp_ota_ops.c.obj 0 4 0 147 965 1116
xtensa_intr_asm.S.obj 1024 0 51 0 0 1075
ringbuf.c.obj 0 0 848 0 209 1057
rtc_time.c.obj 0 0 815 0 198 1013
memory_layout_utils.c.ob 0 0 0 612 393 1005
rtc_init.c.obj 0 0 992 0 0 992
periph_ctrl.c.obj 8 0 0 615 280 903
lib_a-fseeko.o 0 0 0 866 0 866
time.c.obj 0 32 122 703 0 857
clk.c.obj 0 0 64 559 221 844
esp_timer.c.obj 8 16 261 406 134 825
dport_access.c.obj 8 40 444 189 137 818
log.c.obj 8 268 429 82 0 787
rtc_wdt.c.obj 0 0 743 0 0 743
partition.c.obj 0 8 0 522 149 679
ipc.c.obj 0 28 159 283 120 590
locks.c.obj 8 0 485 0 94 587
crosscore_int.c.obj 8 8 193 130 156 495
syscall_table.c.obj 144 240 0 82 0 466
system_api.c.obj 0 0 439 0 0 439
timer.c.obj 16 0 0 112 281 409
int_wdt.c.obj 0 1 91 306 0 398
freertos_hooks.c.obj 8 128 43 216 0 395
esp_app_desc.c.obj 0 0 109 12 256 377
brownout.c.obj 0 0 0 149 201 350
windowspill_asm.o 0 0 311 0 0 311
rtc_module.c.obj 8 8 0 291 0 307
xtensa_context.S.obj 0 0 306 0 0 306
cpu_util.c.obj 0 0 305 0 0 305
dport_panic_highint_hdl. 8 0 242 0 0 250
lib_a-reent.o 0 0 0 232 0 232
lib_a-fopen.o 0 0 0 224 0 224
lib_a-snprintf.o 0 0 0 214 0 214
esp_efuse_api.c.obj 0 4 0 193 0 197
pthread_local_storage.c. 8 4 0 180 0 192
cache_err_int.c.obj 0 0 56 98 0 154
xtensa_intr.c.obj 0 0 108 0 35 143
list.c.obj 0 0 142 0 0 142
syscalls.c.obj 0 0 91 45 0 136
lib_a-assert.o 0 0 0 68 60 128
lib_a-flags.o 0 0 0 127 0 127
bootloader_common.c.obj 0 0 0 126 0 126
lib_a-s_frexp.o 0 0 0 110 0 110
flash_ops.c.obj 20 4 14 62 0 100
lib_a-vprintf.o 0 0 0 94 0 94
lib_a-s_fpclassify.o 0 0 0 88 0 88
lib_a-fiprintf.o 0 0 0 84 0 84
pthread.c.obj 0 8 0 76 0 84
esp_efuse_fields.c.obj 0 0 0 82 0 82
clock.o 0 0 72 0 0 72
reent_init.c.obj 0 0 68 0 2 70
state_asm--restore_extra 0 0 62 0 0 62
state_asm--save_extra_nw 0 0 62 0 0 62
xtensa_vector_defaults.S 0 0 46 0 0 46
lib_a-fseek.o 0 0 0 45 0 45
_divdi3.o 0 0 0 0 40 40
_moddi3.o 0 0 0 0 40 40
_udivdi3.o 0 0 0 0 40 40
_umoddi3.o 0 0 0 0 40 40
xtensa_init.c.obj 0 4 32 0 0 36
interrupts--intlevel.o 0 0 0 0 32 32
esp_efuse_table.c.obj 16 0 0 0 8 24
lib_a-errno.o 0 0 0 10 0 10
pm_esp32.c.obj 0 0 0 8 0 8
int_asm--set_intclear.o 0 0 8 0 0 8
eri.c.obj 0 0 8 0 0 8
cxx_exception_stubs.cpp. 0 0 0 6 0 6
cxx_guards.cpp.obj 0 0 0 5 0 5
hello_world_main.c.obj 0 0 0 5 0 5
FreeRTOS-openocd.c.obj 4 0 0 0 0 4
dummy_main_src.c.obj 0 0 0 0 0 0
bootloader_flash.c.obj 0 0 0 0 0 0
bootloader_random.c.obj 0 0 0 0 0 0
bootloader_sha.c.obj 0 0 0 0 0 0
esp_image_format.c.obj 0 0 0 0 0 0
flash_partitions.c.obj 0 0 0 0 0 0
lib_a-fputs.o 0 0 0 0 0 0
lib_a-printf.o 0 0 0 0 0 0
lib_a-putc.o 0 0 0 0 0 0
lib_a-putchar.o 0 0 0 0 0 0
lib_a-puts.o 0 0 0 0 0 0
lib_a-sprintf.o 0 0 0 0 0 0
lib_a-strerror.o 0 0 0 0 0 0
lib_a-u_strerr.o 0 0 0 0 0 0
lib_a-xpg_strerror_r.o 0 0 0 0 0 0
gpio.c.obj 0 0 0 0 0 0
hw_random.c.obj 0 0 0 0 0 0
pm_locks.c.obj 0 0 0 0 0 0
_addsubdf3.o 0 0 0 0 0 0
_cmpdf2.o 0 0 0 0 0 0
_divdf3.o 0 0 0 0 0 0
_fixdfsi.o 0 0 0 0 0 0
_floatdidf.o 0 0 0 0 0 0
_floatsidf.o 0 0 0 0 0 0
_muldf3.o 0 0 0 0 0 0
_popcountsi2.o 0 0 0 0 0 0
platform.c.obj 0 0 0 0 0 0
platform_util.c.obj 0 0 0 0 0 0
sha256.c.obj 0 0 0 0 0 0
esp_mem.c.obj 0 0 0 0 0 0
gpio_periph.c.obj 0 0 0 0 0 0
spi_flash_rom_patch.c.ob 0 0 0 0 0 0
md5-internal.c.obj 0 0 0 0 0 0
From a library point of view, the major contributor is libc which is the C standard library. While you could probably cherry-pick some functions and drop others from there, I don't think anyone would recommend that.
Next up is libesp32, which provides critical functions such as start_cpu0(). Again, you may be able to cherry-pick only the functions you need, if you really want to.
You can figure out what a library provides by looking up the .a file (e.g. find build -name libesp32.a, and then running nm build/esp-idf/esp32/libesp32.a on the found path.
The second table lists the same size-data, but split per source-file instead of per library.
I want to remove duplicate rows in my table. There is a unique column dataframeid. According to the dataframeid, I want to remove duplicate records
Same records shows up 5 times.
Table - OLTMS_5B0395
Sample data
id DataFrameId DId OT WT AT RC YC BC RV YV BV Oa Aa Gt G M P S Ot O FCNT RSSI SF SNR Rec
2391 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2459 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2282 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2300 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2338 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
Expected output
2282 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
id that can be anything but there shouldn't be any duplicate records.
As per o/p all columns have duplicate values so, you can use row_number() function
delete t from (
select *,
row_number() over (partition by DataFrameId,. . . order by id) Seq
from OLTMS_5B0395
) t
where Seq > 1;
Be aware with order by clause in row_number() function, include column in partition by clause where you got the duplicate values
I suspect this is a straightforward memory leak issue (python27 process has a memory leak with appengine libraries running on the Managed VMs GCE containers), but I'm confused a few things about the data I was collecting during the OOM issues.
After running fine for most of a day, my "vmstat 1" suddenly changed drastically:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 70116 7612 41240 0 0 0 64 231 645 3 2 95 0
0 0 0 70148 7612 41240 0 0 0 0 164 459 2 2 96 0
1 0 0 70200 7612 41240 0 0 0 0 209 712 2 1 97 0
1 0 0 65432 7612 41344 0 0 100 0 602 820 48 5 47 1
1 3 0 69840 5644 29620 0 0 1284 0 812 797 33 6 34 27
0 1 0 69068 5896 30216 0 0 852 68 362 1052 6 1 0 93
0 1 0 68340 6160 30536 0 0 556 0 547 1355 4 2 0 94
0 2 0 67928 6564 30972 0 0 872 0 793 2173 9 5 0 86
0 1 0 63988 6888 34416 0 0 3776 0 716 1940 3 3 0 94
3 0 0 63696 7104 34608 0 0 376 0 353 1006 4 4 34 58
0 0 0 63548 7112 34948 0 0 332 48 379 916 13 1 84 2
0 0 0 63636 7116 34948 0 0 4 0 184 637 0 1 99 0
0 0 0 63660 7116 34948 0 0 0 0 203 556 0 3 97 0
0 1 0 76100 3648 26128 0 0 460 0 409 1142 7 4 85 4
0 3 0 73452 948 15940 0 0 4144 80 1041 1126 53 6 10 31
0 6 0 73828 84 11424 0 0 32924 80 1135 1732 11 4 0 85
0 6 0 72684 64 12324 0 0 52168 4 1519 2397 6 3 0 91
0 11 0 67340 52 12328 0 0 78072 16 1388 2974 2 9 0 89
1 10 0 65992 336 13412 0 0 79796 0 1297 2973 0 9 0 91
0 15 0 69000 48 10396 0 0 78344 0 1203 2739 2 7 0 91
0 15 0 67168 52 11460 0 0 86864 0 1244 3003 0 6 0 94
1 15 0 71268 52 7836 0 0 82552 4 1497 3269 0 7 0 93
In particular, my memory cache and buff dropped, and the io bytes-in surged, and it stayed like this for ~10 minutes afterwards before the machine died and was rebooted by Google Compute Engine. I assume the "bi" represents bytes-in from disk, but I'm curious why swpd showed 0 for this instance, if there was swapping? And why is memory "free" stat still unaffected if things are reaching a swapping point?
Second at the time of the final crash, my top showed:
Top - 15:06:20 up 1 day, 13:23, 2 users, load average: 13.88, 11.22, 9.30
Tasks: 92 total, 3 running, 89 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 8.0 sy, 0.0 ni, 0.0 id, 90.9 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 1745136 total, 1684032 used, 61104 free, 648 buffers
KiB Swap: 0 total, 0 used, 0 free, 12236 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23 root 20 0 0 0 0 R 12.6 0.0 2:11.61 kswapd0
10 root rt 0 0 0 0 S 2.5 0.0 0:52.92 watchdog/0
2315 root 20 0 192m 17m 0 S 2.1 1.0 47:58.74 kubelet
2993 root 20 0 6116m 1.2g 0 S 0.9 70.2 318:41.51 python2.7
6644 root 20 0 55452 12m 0 S 0.9 0.7 0:00.81 python
2011 root 20 0 761m 9924 0 S 0.7 0.6 12:23.44 docker
6624 root 20 0 4176 132 0 D 0.5 0.0 0:00.24 du
140 root 0 -20 0 0 0 S 0.4 0.0 0:08.64 kworker/0:1H
2472 root 20 0 39680 5616 296 D 0.4 0.3 0:27.43 python
1 root 20 0 10656 132 0 S 0.2 0.0 0:02.61 init
3 root 20 0 0 0 0 S 0.2 0.0 2:02.17 ksoftirqd/0
22 root 20 0 0 0 0 R 0.2 0.0 0:24.61 kworker/0:1
1834 root 20 0 53116 756 0 S 0.2 0.0 0:01.79 rsyslogd
1859 root 20 0 52468 9624 0 D 0.2 0.6 0:29.36 supervisord
2559 root 20 0 349m 172m 0 S 0.2 10.1 25:56.31 ruby
Again, I see the python27 process has claimed to 70% (which when combined with the 10% from Ruby, puts me into dangerous territory). Buy why is kswapd going crazy with my 10% CPU when the above vmstat shows 0?
Should I just not trust vmstat's swapd?
After a big break of ~6 months I am back in the world of Perl and Bioinformatics, interning under a different scientist. But the very first assignment is unlike any I had encountered last time, so while I have made some progress, I haven't been able to tackle the problem in its entirety. I am also trying to revise whatever I learnt last time as fast as possible, because I completely lost touch with programming these last 6 months.
The dataset looks like the following:
NR_046018 DDX11L1 , 0 0 1 1 1 1 1 1 1 1 0 0 0 0 1.44 2.72 3.84 4.92 5.6 6.64 7.08 9.12 9.56 8.28 7.16 6.08 5.4 4.36 3.92 1.88 0 0 0.76 1 1 1 1.2 2 2 2 1.72 2 2 2 1.8 1 1.88 2.4 3 3.36 5 6 6 6.72 6.12 5.6 5.44 5.56 5 4.04 5 4.28 4 4 3.08 2.08 1.68 1.96 1.44 3 3.68 4 4.16 5 4.32 4.8 6.16 6 6.28 6.92 7.84 7 7.32 7.2 5.96 5 4.52 4.08 3 3 4.04 4.12 4.44 4 3.52 3.4 4 4 2.64 1.88 1 1 1 0.64 1 1 1.24 2 2.92 3 3 2.96 2 2 2.56 2 1.08 2.12 3 3 3 3 2.6 3 4.64 3.88 3.72 4 4 4.96 4.6 4 2.36 2 1.28 1 1 0.04 0 0.24 1.08 2.68 3.84 4.12 5.72 6 6 5.76 4.92 3.32 3.12 2.88 2.08 2 2 2 2 2 1.44 2.92 3.04 4.28 5.8 7.8 9.48 10.52 13.04 12.08 11.6 11.72 11 9.2 7.52 7.12 7.08 7.08 8.32 7 6.6 7.6 8.04 8.36 6.72 7.88 7.72 8.4 9.24 8.88 8.96 9.88 10.08 9.24 9.28 10.16 11.04 10.52 10 8.56 8 7.8 7.72 6.44 4.32 4 4 3.72 3.68 3.68 3.28 5.56 7.36 9.48 10 10.52 11 12.16 11.96 9.44 8.64 7.52 7 6.48 6 5 5.12 6.28 6 5.52 6 6.68 6.08 7.52 8.16 7.72 8.52 8.56 9.2 9.16 8.92 7.44 6 5 3.48 2.92 2.16 2 2 1.2 1 1 1 1.24 1.64 1 1 1.96 2 2 2 1.76 1 1 1 0.52 1.76 3.64 5.12 6 6 6 6 5.52 4.24 2.36 0.88 0 0 0.68 1 1 1 1 1 1 1 0.32 0 0 1 1 1.44 2.44 3.68 5.4 6.88 7 6 6.52 6.76 6.56 5.32 3.6 2.92 3 3.72 3.96 3.8 3 3 3 2.2 2.4 2.28 1.52 1 1 1 1.72 2 1.6 1 1 1 1 1 0.28 0.92 2 2 2.72 3.64 4 4.84 5 4.08 3 3 2.68 2.36 2 1.16 1 1 2 4.92 4.6 4 4 4 4 4.32 4 1.08 1 1.52 2 2 2 1.68 1 1 1.32 1.48 1 1 1.52 2 2 2 1.68 1 1 1.88 1.48 1 1 1 1 1 1 0.12 0.4 1 1 1.2 3.88 4 5 5 4.6 4 4 3.8 2.08 2 1 1 1.44 2.4 3
NR_047520 LOC643837 , 3 2.2 0.2 0 0 0.28 1 1 1 1 2.2 4.8 5 5.32 5 5 5 5 3.8 1.2 1 0.4 0 0 0 0 0 1 1 1 1 1 1 1 1.56 1 1 1 1 1 1 1 0.44 0.68 1 1.52 3 3.6 4.96 6.8 9 8.32 8.72 8.48 7 7.4 8.8 7.92 7.12 8.84 8.56 9.4 10.2 10 7.24 6.44 6.76 6.16 5.72 4.96 4.8 5.16 6 5.84 4.12 3 3 2.64 2.56 3.08 3 4.16 5 6.72 7 7.16 7.44 5.76 5 4.56 4 3.68 5 5.4 5.52 6 6 5.28 5 3.6 2 2.08 1.48 1 2 2 2 2 2 1.36 1 1 0 0 0.68 1 1 1 1 1 1 1 0.32 0 0 0 1.16 2 2 2 2 2.88 3 3 1.84 1 2 2 2.04 2.12 2 2 2 2 1 1.28 1.96 1.36 2.76 3 3 3 3 2.72 2 1.64 0.76 1 1.36 2 2 2 2 2 1.48 1 0.64 0 0.08 1 1 1.08 2 2 2 2 2.68 2 2 2.16 3.4 4 4 4.2 4.24 4 5.68 6.52 4.6 4 4 3.8 3.8 4 3.12 2.24 2.6 3 4 4 3.2 3 2.2 2 1.4 1.84 1.24 2 2 2 2 2 2 1.16 0.76 0 0 0 0 0 0 0 0.36 1 1.68 2 2 2.92 5.4 6.76 7.64 7 6.88 7 7.36 7.92 6.24 5.92 7.04 9.52 11.52 12.88 14.8 16.36 19.88 22.24 20 19.36 16.92 15.24 13.84 10.88 8.24 5.08 4.96 3.12 3 2.88 2 2.8 2.96 4 4.44 5 6 6 6 5.12 3.28 2 1.56 1 0.08 1.68 2 2 2.84 3 3 3.8 3.92 2.32 2 2.2 2.16 2 2 1.2 1 1 1 0.8 0 0 0 0.72 2.88 3 3 3 3 3 3 2.28 0.12 0 0.52 1 1 1 1 1.44 2 2 1.48 1 1 1 1.56 1.56 1 1 1 1 1 1 0.44 0.8 1.48 3 3 3 3 3 3.56 3.2 2.76 2 2 2 2 2.68 2.44 2 1.76 1 1.4 2 2 1.56 2 2 2 2 2.04 2 2 1.76 1 1 1 1 0.56 0 0 0 0 0 0 0 0 0.72 1.52 2 2 2 2 2 2 1.28 0.48
1. What is needed
For each row in the data file, find the maximum value from the range of numbers.
Once the maximum has been found for all the rows, find average maximum.
2. Strategy I was thinking
Separate the non numerical part from the non-numerical part into "keys" of a hash.
Put the numerical part into the "values" of a hash.
Assign the "values" into array #values
Use module use List::Util qw(max) to find maximum value from the array
Store these maximum values in another array and find average from this array.
3. Code written so far
use warnings;
use List::Util qw(max);
#Input filename
$file = 'test1.data';
#Open file
open I, '<', $file or die;
#Separate data into keys and values, based on ','
chop (%hash = map { split /\s*,\s*/,$_,2 } grep (!/^$/,<I>));
print "$_ => $hash{$_}\n" for keys %hash; #Code is working fine till here
#Create a values array
#values = values %hash;
foreach $value(#values){
print "The values are : ", $value,"\n";
}
4. The Problem
Beyond this, I am not able to figure out how to add each "individual" array
element into a new array so that I may use the max function.
What I mean is that for example, the first array element in #values contains data like 0 0 1 1 3 4.4. The second array element might have data like 3 2.2 0.28 1 1 4.8. So I need to put each of these array elements into a new array, each element going into a different array so that I may be able to use the max function.
5. Points to Note
Most of the rows contain 400 numbers, some have a little less than that, but never more than 400.
There are a total of 23,558 rows.
File is a .txt file and all the numbers in each row are tab delimited.
I would be grateful to anyone who would be kind enough to point me in the right direction, or perhaps provide a better code to tackle the problem as mentioned in 1.
If I understand your problem correctly you're making it overly complicated:
#!/usr/bin/env perl
use strict;
use warnings;
use List::Util qw(max);
#Input filename
my $file = 'test1.data';
#Open file
open my $fh, '<', $file or die "Unable to open $file: $!\n";
my ($total, $num);
while (<$fh>) {
my #values = split;
my $max = max(#values[3 .. $#values]);
$total += $max;
$num++;
}
my $average_max = $total / $num;
Just make one pass over your file, splitting the lines into an array and feeding everything from index 3 to max. Add $max to $total for each line, increment a counter ($num) and calculate average max from that.
You should also always use use strict and lexical filehandles.
Here is a fun solution. If you are using List::Util, you might as well use sum also.
#!usr/bin/perl
use strict;
use warnings;
use List::Util qw/max sum/;
my %line_max = map {
/([\w\s]*?)\s*,\s*(.*)/ or die "bad line";
$1 => max split ' ', $2
} <DATA>;
print "$_: $line_max{$_}\n" foreach (keys %line_max);
my $avg_max = sum (values %line_max) / scalar (values %line_max);
print "average: $avg_max\n";
__DATA__
NR_046018 DDX11L1 , 0 0 1 1 1 1 1 1 1 1 0 0 0 0 1.44 2.72 3.84 4.92 5.6 6.64 7.08 9.12 9.56 8.28 7.16 6.08 5.4 4.36 3.92 1.88 0 0 0.76 1 1 1 1.2 2 2 2 1.72 2 2 2 1.8 1 1.88 2.4 3 3.36 5 6 6 6.72 6.12 5.6 5.44 5.56 5 4.04 5 4.28 4 4 3.08 2.08 1.68 1.96 1.44 3 3.68 4 4.16 5 4.32 4.8 6.16 6 6.28 6.92 7.84 7 7.32 7.2 5.96 5 4.52 4.08 3 3 4.04 4.12 4.44 4 3.52 3.4 4 4 2.64 1.88 1 1 1 0.64 1 1 1.24 2 2.92 3 3 2.96 2 2 2.56 2 1.08 2.12 3 3 3 3 2.6 3 4.64 3.88 3.72 4 4 4.96 4.6 4 2.36 2 1.28 1 1 0.04 0 0.24 1.08 2.68 3.84 4.12 5.72 6 6 5.76 4.92 3.32 3.12 2.88 2.08 2 2 2 2 2 1.44 2.92 3.04 4.28 5.8 7.8 9.48 10.52 13.04 12.08 11.6 11.72 11 9.2 7.52 7.12 7.08 7.08 8.32 7 6.6 7.6 8.04 8.36 6.72 7.88 7.72 8.4 9.24 8.88 8.96 9.88 10.08 9.24 9.28 10.16 11.04 10.52 10 8.56 8 7.8 7.72 6.44 4.32 4 4 3.72 3.68 3.68 3.28 5.56 7.36 9.48 10 10.52 11 12.16 11.96 9.44 8.64 7.52 7 6.48 6 5 5.12 6.28 6 5.52 6 6.68 6.08 7.52 8.16 7.72 8.52 8.56 9.2 9.16 8.92 7.44 6 5 3.48 2.92 2.16 2 2 1.2 1 1 1 1.24 1.64 1 1 1.96 2 2 2 1.76 1 1 1 0.52 1.76 3.64 5.12 6 6 6 6 5.52 4.24 2.36 0.88 0 0 0.68 1 1 1 1 1 1 1 0.32 0 0 1 1 1.44 2.44 3.68 5.4 6.88 7 6 6.52 6.76 6.56 5.32 3.6 2.92 3 3.72 3.96 3.8 3 3 3 2.2 2.4 2.28 1.52 1 1 1 1.72 2 1.6 1 1 1 1 1 0.28 0.92 2 2 2.72 3.64 4 4.84 5 4.08 3 3 2.68 2.36 2 1.16 1 1 2 4.92 4.6 4 4 4 4 4.32 4 1.08 1 1.52 2 2 2 1.68 1 1 1.32 1.48 1 1 1.52 2 2 2 1.68 1 1 1.88 1.48 1 1 1 1 1 1 0.12 0.4 1 1 1.2 3.88 4 5 5 4.6 4 4 3.8 2.08 2 1 1 1.44 2.4 3
NR_047520 LOC643837 , 3 2.2 0.2 0 0 0.28 1 1 1 1 2.2 4.8 5 5.32 5 5 5 5 3.8 1.2 1 0.4 0 0 0 0 0 1 1 1 1 1 1 1 1.56 1 1 1 1 1 1 1 0.44 0.68 1 1.52 3 3.6 4.96 6.8 9 8.32 8.72 8.48 7 7.4 8.8 7.92 7.12 8.84 8.56 9.4 10.2 10 7.24 6.44 6.76 6.16 5.72 4.96 4.8 5.16 6 5.84 4.12 3 3 2.64 2.56 3.08 3 4.16 5 6.72 7 7.16 7.44 5.76 5 4.56 4 3.68 5 5.4 5.52 6 6 5.28 5 3.6 2 2.08 1.48 1 2 2 2 2 2 1.36 1 1 0 0 0.68 1 1 1 1 1 1 1 0.32 0 0 0 1.16 2 2 2 2 2.88 3 3 1.84 1 2 2 2.04 2.12 2 2 2 2 1 1.28 1.96 1.36 2.76 3 3 3 3 2.72 2 1.64 0.76 1 1.36 2 2 2 2 2 1.48 1 0.64 0 0.08 1 1 1.08 2 2 2 2 2.68 2 2 2.16 3.4 4 4 4.2 4.24 4 5.68 6.52 4.6 4 4 3.8 3.8 4 3.12 2.24 2.6 3 4 4 3.2 3 2.2 2 1.4 1.84 1.24 2 2 2 2 2 2 1.16 0.76 0 0 0 0 0 0 0 0.36 1 1.68 2 2 2.92 5.4 6.76 7.64 7 6.88 7 7.36 7.92 6.24 5.92 7.04 9.52 11.52 12.88 14.8 16.36 19.88 22.24 20 19.36 16.92 15.24 13.84 10.88 8.24 5.08 4.96 3.12 3 2.88 2 2.8 2.96 4 4.44 5 6 6 6 5.12 3.28 2 1.56 1 0.08 1.68 2 2 2.84 3 3 3.8 3.92 2.32 2 2.2 2.16 2 2 1.2 1 1 1 0.8 0 0 0 0.72 2.88 3 3 3 3 3 3 2.28 0.12 0 0.52 1 1 1 1 1.44 2 2 1.48 1 1 1 1.56 1.56 1 1 1 1 1 1 0.44 0.8 1.48 3 3 3 3 3 3.56 3.2 2.76 2 2 2 2 2.68 2.44 2 1.76 1 1.4 2 2 1.56 2 2 2 2 2.04 2 2 1.76 1 1 1 1 0.56 0 0 0 0 0 0 0 0 0.72 1.52 2 2 2 2 2 2 1.28 0.48
Note: the map syntax is cute, but if the file is large you should be using a while loop for efficiency. The while loop avoids reading the whole file into memory:
while (<DATA>)
{
if (/^([\w\s]*?)\s*,\s*(.*)/)
{
$line_max{$1} = max split ' ', $2;
}
else
{
print "Line $. is bad.\n";
}
}