I suspect this is a straightforward memory leak issue (python27 process has a memory leak with appengine libraries running on the Managed VMs GCE containers), but I'm confused a few things about the data I was collecting during the OOM issues.
After running fine for most of a day, my "vmstat 1" suddenly changed drastically:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 70116 7612 41240 0 0 0 64 231 645 3 2 95 0
0 0 0 70148 7612 41240 0 0 0 0 164 459 2 2 96 0
1 0 0 70200 7612 41240 0 0 0 0 209 712 2 1 97 0
1 0 0 65432 7612 41344 0 0 100 0 602 820 48 5 47 1
1 3 0 69840 5644 29620 0 0 1284 0 812 797 33 6 34 27
0 1 0 69068 5896 30216 0 0 852 68 362 1052 6 1 0 93
0 1 0 68340 6160 30536 0 0 556 0 547 1355 4 2 0 94
0 2 0 67928 6564 30972 0 0 872 0 793 2173 9 5 0 86
0 1 0 63988 6888 34416 0 0 3776 0 716 1940 3 3 0 94
3 0 0 63696 7104 34608 0 0 376 0 353 1006 4 4 34 58
0 0 0 63548 7112 34948 0 0 332 48 379 916 13 1 84 2
0 0 0 63636 7116 34948 0 0 4 0 184 637 0 1 99 0
0 0 0 63660 7116 34948 0 0 0 0 203 556 0 3 97 0
0 1 0 76100 3648 26128 0 0 460 0 409 1142 7 4 85 4
0 3 0 73452 948 15940 0 0 4144 80 1041 1126 53 6 10 31
0 6 0 73828 84 11424 0 0 32924 80 1135 1732 11 4 0 85
0 6 0 72684 64 12324 0 0 52168 4 1519 2397 6 3 0 91
0 11 0 67340 52 12328 0 0 78072 16 1388 2974 2 9 0 89
1 10 0 65992 336 13412 0 0 79796 0 1297 2973 0 9 0 91
0 15 0 69000 48 10396 0 0 78344 0 1203 2739 2 7 0 91
0 15 0 67168 52 11460 0 0 86864 0 1244 3003 0 6 0 94
1 15 0 71268 52 7836 0 0 82552 4 1497 3269 0 7 0 93
In particular, my memory cache and buff dropped, and the io bytes-in surged, and it stayed like this for ~10 minutes afterwards before the machine died and was rebooted by Google Compute Engine. I assume the "bi" represents bytes-in from disk, but I'm curious why swpd showed 0 for this instance, if there was swapping? And why is memory "free" stat still unaffected if things are reaching a swapping point?
Second at the time of the final crash, my top showed:
Top - 15:06:20 up 1 day, 13:23, 2 users, load average: 13.88, 11.22, 9.30
Tasks: 92 total, 3 running, 89 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 8.0 sy, 0.0 ni, 0.0 id, 90.9 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 1745136 total, 1684032 used, 61104 free, 648 buffers
KiB Swap: 0 total, 0 used, 0 free, 12236 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23 root 20 0 0 0 0 R 12.6 0.0 2:11.61 kswapd0
10 root rt 0 0 0 0 S 2.5 0.0 0:52.92 watchdog/0
2315 root 20 0 192m 17m 0 S 2.1 1.0 47:58.74 kubelet
2993 root 20 0 6116m 1.2g 0 S 0.9 70.2 318:41.51 python2.7
6644 root 20 0 55452 12m 0 S 0.9 0.7 0:00.81 python
2011 root 20 0 761m 9924 0 S 0.7 0.6 12:23.44 docker
6624 root 20 0 4176 132 0 D 0.5 0.0 0:00.24 du
140 root 0 -20 0 0 0 S 0.4 0.0 0:08.64 kworker/0:1H
2472 root 20 0 39680 5616 296 D 0.4 0.3 0:27.43 python
1 root 20 0 10656 132 0 S 0.2 0.0 0:02.61 init
3 root 20 0 0 0 0 S 0.2 0.0 2:02.17 ksoftirqd/0
22 root 20 0 0 0 0 R 0.2 0.0 0:24.61 kworker/0:1
1834 root 20 0 53116 756 0 S 0.2 0.0 0:01.79 rsyslogd
1859 root 20 0 52468 9624 0 D 0.2 0.6 0:29.36 supervisord
2559 root 20 0 349m 172m 0 S 0.2 10.1 25:56.31 ruby
Again, I see the python27 process has claimed to 70% (which when combined with the 10% from Ruby, puts me into dangerous territory). Buy why is kswapd going crazy with my 10% CPU when the above vmstat shows 0?
Should I just not trust vmstat's swapd?
Related
I am trying to add svg in my next project but i keep getting this error 'Error: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.'
code:index.js
import React from 'react'
import Image from 'next/image';
import {ReactComponent as Logo} from '/assets/Logo.svg';
const Header=()=>{
return (
<div className='flex'>
<Logo/>
</div>
)
}
export default Header;
svg file
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 74 20">
<defs>
<style>
.cls-1{fill:#28b16d}
</style>
</defs>
<path d="M51 13.53a.58.58 0 0 0-.11.08 2.77 2.77 0 0 1-2.28 1.1 4.79 4.79 0 0 1-2.32-.45 3.81 3.81 0 0 1-2.07-3 6.57 6.57 0 0 1-.07-1V6.07c0-.56.17-.73.73-.73h1.22c.49 0 .69.21.69.7v4.27a2.45 2.45 0 0 0 .5 1.56A2.25 2.25 0 0 0 51 11.3a1.13 1.13 0 0 0 .06-.41V6.06c0-.59.17-.76.75-.76H53c.49 0 .71.22.71.71v9.07a4.27 4.27 0 0 1-2.38 4 5.79 5.79 0 0 1-3.16.61 5.28 5.28 0 0 1-1.95-.45 1.23 1.23 0 0 1-.86-1.33v-.69a.44.44 0 0 1 .68-.43c.34.15.66.35 1 .48a3.87 3.87 0 0 0 2.42.16A2 2 0 0 0 51 15.46v-1.8a.54.54 0 0 0 0-.13zM24.34 5.55a13 13 0 0 1 1.32-.45 5.6 5.6 0 0 1 3.14.08 4.14 4.14 0 0 1 2.8 3.4 5.52 5.52 0 0 1-.09 2.62 4.48 4.48 0 0 1-2.8 3 6.13 6.13 0 0 1-5.76-.51 4.76 4.76 0 0 1-.73-.61 1.52 1.52 0 0 1-.48-1.08V1.71a.89.89 0 0 1 .48-.85l1.35-.79c.45-.26.77 0 .77.46V5.55zm0 4.3v1.83a.45.45 0 0 0 .22.42 3 3 0 0 0 2.35.47 2.27 2.27 0 0 0 1.85-1.48A3.61 3.61 0 0 0 29 9.44a2.55 2.55 0 0 0-.84-1.82 3.08 3.08 0 0 0-3.63.06.51.51 0 0 0-.13.34c-.07.61-.06 1.22-.06 1.83zM11.26-.07a7.34 7.34 0 0 1 6.91 5.77 7.41 7.41 0 0 1-4.36 8.48 7.16 7.16 0 0 1-8-1.79c-.4-.45-.41-.45-.92-.15l-3.33 2c-.57.33-.81.27-1.14-.3l-.31-.55a.64.64 0 0 1 .27-1l3.43-2c.52-.31.51-.3.35-.86A7.4 7.4 0 0 1 8.87.33a6.52 6.52 0 0 1 1-.26c.47-.07.93-.07 1.39-.14zm5.66 7.43a5.89 5.89 0 0 0-5.75-5.95 5.88 5.88 0 0 0-5.88 5.75A5.88 5.88 0 0 0 11 13.27a5.87 5.87 0 0 0 5.92-5.91zM39.91 13.43a1 1 0 0 0-.15.12 2.61 2.61 0 0 1-1.86 1.1 4.49 4.49 0 0 1-4.17-1.38 4.54 4.54 0 0 1-1.13-2.69 5.23 5.23 0 0 1 .47-3 4.6 4.6 0 0 1 3.13-2.42 7 7 0 0 1 5.28.64l.16.1a1.64 1.64 0 0 1 .9 1.62v6.07a2.13 2.13 0 0 1 0 .42.48.48 0 0 1-.5.42h-1.58c-.33 0-.52-.22-.55-.59v-.41zm0-5.63a.47.47 0 0 0-.25-.45A3.07 3.07 0 0 0 38.28 7a2.55 2.55 0 0 0-2.91 1.9 3.91 3.91 0 0 0 0 1.91A2.43 2.43 0 0 0 39.33 12a1.41 1.41 0 0 0 .57-1.22 59.2 59.2 0 0 1 0-2.98zM62.31 13.43a1.18 1.18 0 0 0-.16.14 2.52 2.52 0 0 1-1.73 1.07 4.54 4.54 0 0 1-3.22-.56 3.82 3.82 0 0 1-1.74-2.87 6.8 6.8 0 0 1-.07-1V6.02c0-.56.16-.73.73-.73h1.25A.59.59 0 0 1 58 6v4.29a2.46 2.46 0 0 0 .54 1.63 2.27 2.27 0 0 0 3.67-.63 1 1 0 0 0 0-.34V6.07c0-.56.18-.74.75-.74h1.19c.48 0 .69.22.69.7v7.69c0 .54-.2.74-.73.74H63c-.48 0-.67-.19-.68-.67l-.01-.36zM66.63 6.9V3.27a.85.85 0 0 1 .45-.81l1.35-.79.12-.06a.46.46 0 0 1 .68.44v2.89c0 .34 0 .35.36.35H73c.67 0 .81.14.82.8v.64c0 .44-.2.63-.63.64h-3.58c-.39 0-.39 0-.39.38v2.39a3.61 3.61 0 0 0 .15 1 2.09 2.09 0 0 0 2.3 1.44 2.52 2.52 0 0 0 1.45-.58l.13-.09a.51.51 0 0 1 .56-.1c.2.11.18.31.18.49v.85a1 1 0 0 1-.47.89 4 4 0 0 1-2.23.65 5 5 0 0 1-2.53-.56 3.91 3.91 0 0 1-2-3.08 16.67 16.67 0 0 1-.08-2c-.06-.72-.05-1.43-.05-2.15z" class="cls-1"/>
<path d="M14.91 8.06v1.88c0 .43-.18.61-.61.61h-1.44c-.34 0-.49-.14-.49-.48V8.56a1.29 1.29 0 0 0-1.09-1.25 1.26 1.26 0 0 0-1.39 1 3.3 3.3 0 0 0 0 .69v1.06c0 .35-.15.52-.51.52h-1.5a.54.54 0 0 1-.6-.58V6.26a.73.73 0 0 1 .4-.71q1.42-.8 2.81-1.65a1.05 1.05 0 0 1 1.21 0c.94.57 1.9 1.13 2.85 1.68a.62.62 0 0 1 .34.6v1.91z" class="cls-1"/>
</svg>
I searched on the error and it says there is a problem with the import/export. So how should i export/import the file??
I tested running bare bones code using ESP IDF on an ESP32 chip using duinotech XC-3800, and obtained the following results in terms of image size.
Analysis Binary Size for ESP32
Folder Structure
temp/
main/
CMakeLists.txt
main.c
CMakeLists.txt
File contents
CMakeLists.txt
# The following lines of boilerplate have to be in your project's
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(temp)
main>CMakeLists.txt
idf_component_register(SRCS "main.c"
INCLUDE_DIRS "")
Test 1 main>main.c
#include <stdio.h>
void app_main(void) {
printf("Hello world!\n");
for (int i = 10; i >= 0; i--) {
printf("Restarting in %d seconds...\n", i);
}
printf("Restarting now.\n");
fflush(stdout);
}
Test 2 main>main.c
#include <stdio.h>
void app_main(void) { printf("Hello world!\n"); }
Test 3 main>main.c
void app_main(void) {}
Size comparison
Obtained by running idf_size.py build/temp.map
Test 1
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75408 bytes
Flash rodata: 23844 bytes
Total image size:~ 146376 bytes (.bin may be padded larger)
Test 2
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75240 bytes
Flash rodata: 23796 bytes
Total image size:~ 146160 bytes (.bin may be padded larger)
Test 3
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75004 bytes
Flash rodata: 23780 bytes
Total image size:~ 145908 bytes (.bin may be padded larger)
Analysis
Size for code obtained by running stat --format="%s" main/main.c
All Sizes are in Bytes
Test No. | Code | Image | Flash Code | Flash rodata
-------- | -----| ------ | ---------- | ------------
1 | 207 | 146376 | 75408 | 23844
2 | 70 | 146160 | 75240 | 23796
3 | 43 | 145908 | 75004 | 23780
At least 145KB of boiler plate code just to get an empty main run.
Speculation
I suspect that the 145KB is made up of a number of libraries that are always loaded onto the chip whether you use them or not. Some of them must be the FreeRTOS, WiFi, HTTP etc.
Can we bring down this size somehow and load only the bare minimum required for operation?
You can get more detailed size information by running idf.py size-components and idf.py size-files. Here are the results for your "Test 3" (i.e. empty function) on my dev environment (note that my Total Image Size is already slightly larger than the one you posted.
Total sizes:
DRAM .data size: 7860 bytes
DRAM .bss size: 4128 bytes
Used static DRAM: 11988 bytes ( 168748 available, 6.6% used)
Used static IRAM: 32706 bytes ( 98366 available, 25.0% used)
Flash code: 76002 bytes
Flash rodata: 32164 bytes
Total image size:~ 148732 bytes (.bin may be padded larger)
Per-archive contributions to ELF file:
Archive File DRAM .data & .bss IRAM Flash code & rodata Total
libc.a 0 0 0 55348 3829 59177
libesp32.a 2132 2351 6767 5976 8076 25302
libfreertos.a 4140 776 12432 0 1653 19001
libdriver.a 76 20 0 4003 7854 11953
libsoc.a 161 4 4953 612 3852 9582
libvfs.a 240 103 0 5464 950 6757
libheap.a 877 4 3095 830 986 5792
libspi_flash.a 24 290 1855 779 890 3838
libefuse.a 16 4 0 1142 2213 3375
libnewlib.a 152 272 766 830 96 2116
libapp_update.a 0 4 109 159 1221 1493
libesp_ringbuf.a 0 0 848 0 209 1057
liblog.a 8 268 429 82 0 787
libhal.a 0 0 515 0 32 547
libpthread.a 8 12 0 256 0 276
libgcc.a 0 0 0 0 160 160
libbootloader_support.a 0 0 0 126 0 126
libm.a 0 0 0 88 0 88
libcxx.a 0 0 0 11 0 11
libxtensa-debug-module.a 0 0 8 0 0 8
libmain.a 0 0 0 5 0 5
(exe) 0 0 0 0 0 0
libmbedcrypto.a 0 0 0 0 0 0
libmbedtls.a 0 0 0 0 0 0
libwpa_supplicant.a 0 0 0 0 0 0
Per-file contributions to ELF file:
Object File DRAM .data & .bss IRAM Flash code & rodata Total
lib_a-vfprintf.o 0 0 0 14193 756 14949
lib_a-svfprintf.o 0 0 0 13838 756 14594
lib_a-svfiprintf.o 0 0 0 9642 1210 10852
lib_a-vfiprintf.o 0 0 0 9945 738 10683
uart.c.obj 44 12 0 2985 7293 10334
tasks.c.obj 12 700 5546 0 531 6789
vfs_uart.c.obj 48 63 0 3680 808 4599
panic.c.obj 2023 5 1989 0 0 4017
esp_err_to_name.c.obj 0 0 0 50 3947 3997
portasm.S.obj 3084 0 484 0 0 3568
lib_a-dtoa.o 0 0 0 3522 13 3535
multi_heap.c.obj 873 0 2277 0 0 3150
intr_alloc.c.obj 8 22 618 1703 722 3073
esp_efuse_utility.c.obj 0 0 0 867 2205 3072
cpu_start.c.obj 0 1 1068 331 1352 2752
queue.c.obj 0 0 2310 0 325 2635
lib_a-mprec.o 0 0 0 2134 296 2430
rtc_clk.c.obj 161 4 2098 0 0 2263
dbg_stubs.c.obj 0 2072 32 108 0 2212
vfs.c.obj 192 40 0 1784 142 2158
rtc_periph.c.obj 0 0 0 0 2080 2080
esp_timer_esp32.c.obj 8 26 1068 262 538 1902
xtensa_vectors.S.obj 8 0 1776 0 36 1820
flash_mmap.c.obj 0 264 1092 114 339 1809
task_wdt.c.obj 53 4 0 1178 548 1783
heap_caps.c.obj 4 0 818 52 617 1491
timers.c.obj 8 56 1006 0 233 1303
cache_utils.c.obj 4 14 749 81 402 1250
soc_memory_layout.c.obj 0 0 0 0 1181 1181
heap_caps_init.c.obj 0 4 0 778 369 1151
port.c.obj 0 16 625 0 493 1134
esp_ota_ops.c.obj 0 4 0 147 965 1116
xtensa_intr_asm.S.obj 1024 0 51 0 0 1075
ringbuf.c.obj 0 0 848 0 209 1057
rtc_time.c.obj 0 0 815 0 198 1013
memory_layout_utils.c.ob 0 0 0 612 393 1005
rtc_init.c.obj 0 0 992 0 0 992
periph_ctrl.c.obj 8 0 0 615 280 903
lib_a-fseeko.o 0 0 0 866 0 866
time.c.obj 0 32 122 703 0 857
clk.c.obj 0 0 64 559 221 844
esp_timer.c.obj 8 16 261 406 134 825
dport_access.c.obj 8 40 444 189 137 818
log.c.obj 8 268 429 82 0 787
rtc_wdt.c.obj 0 0 743 0 0 743
partition.c.obj 0 8 0 522 149 679
ipc.c.obj 0 28 159 283 120 590
locks.c.obj 8 0 485 0 94 587
crosscore_int.c.obj 8 8 193 130 156 495
syscall_table.c.obj 144 240 0 82 0 466
system_api.c.obj 0 0 439 0 0 439
timer.c.obj 16 0 0 112 281 409
int_wdt.c.obj 0 1 91 306 0 398
freertos_hooks.c.obj 8 128 43 216 0 395
esp_app_desc.c.obj 0 0 109 12 256 377
brownout.c.obj 0 0 0 149 201 350
windowspill_asm.o 0 0 311 0 0 311
rtc_module.c.obj 8 8 0 291 0 307
xtensa_context.S.obj 0 0 306 0 0 306
cpu_util.c.obj 0 0 305 0 0 305
dport_panic_highint_hdl. 8 0 242 0 0 250
lib_a-reent.o 0 0 0 232 0 232
lib_a-fopen.o 0 0 0 224 0 224
lib_a-snprintf.o 0 0 0 214 0 214
esp_efuse_api.c.obj 0 4 0 193 0 197
pthread_local_storage.c. 8 4 0 180 0 192
cache_err_int.c.obj 0 0 56 98 0 154
xtensa_intr.c.obj 0 0 108 0 35 143
list.c.obj 0 0 142 0 0 142
syscalls.c.obj 0 0 91 45 0 136
lib_a-assert.o 0 0 0 68 60 128
lib_a-flags.o 0 0 0 127 0 127
bootloader_common.c.obj 0 0 0 126 0 126
lib_a-s_frexp.o 0 0 0 110 0 110
flash_ops.c.obj 20 4 14 62 0 100
lib_a-vprintf.o 0 0 0 94 0 94
lib_a-s_fpclassify.o 0 0 0 88 0 88
lib_a-fiprintf.o 0 0 0 84 0 84
pthread.c.obj 0 8 0 76 0 84
esp_efuse_fields.c.obj 0 0 0 82 0 82
clock.o 0 0 72 0 0 72
reent_init.c.obj 0 0 68 0 2 70
state_asm--restore_extra 0 0 62 0 0 62
state_asm--save_extra_nw 0 0 62 0 0 62
xtensa_vector_defaults.S 0 0 46 0 0 46
lib_a-fseek.o 0 0 0 45 0 45
_divdi3.o 0 0 0 0 40 40
_moddi3.o 0 0 0 0 40 40
_udivdi3.o 0 0 0 0 40 40
_umoddi3.o 0 0 0 0 40 40
xtensa_init.c.obj 0 4 32 0 0 36
interrupts--intlevel.o 0 0 0 0 32 32
esp_efuse_table.c.obj 16 0 0 0 8 24
lib_a-errno.o 0 0 0 10 0 10
pm_esp32.c.obj 0 0 0 8 0 8
int_asm--set_intclear.o 0 0 8 0 0 8
eri.c.obj 0 0 8 0 0 8
cxx_exception_stubs.cpp. 0 0 0 6 0 6
cxx_guards.cpp.obj 0 0 0 5 0 5
hello_world_main.c.obj 0 0 0 5 0 5
FreeRTOS-openocd.c.obj 4 0 0 0 0 4
dummy_main_src.c.obj 0 0 0 0 0 0
bootloader_flash.c.obj 0 0 0 0 0 0
bootloader_random.c.obj 0 0 0 0 0 0
bootloader_sha.c.obj 0 0 0 0 0 0
esp_image_format.c.obj 0 0 0 0 0 0
flash_partitions.c.obj 0 0 0 0 0 0
lib_a-fputs.o 0 0 0 0 0 0
lib_a-printf.o 0 0 0 0 0 0
lib_a-putc.o 0 0 0 0 0 0
lib_a-putchar.o 0 0 0 0 0 0
lib_a-puts.o 0 0 0 0 0 0
lib_a-sprintf.o 0 0 0 0 0 0
lib_a-strerror.o 0 0 0 0 0 0
lib_a-u_strerr.o 0 0 0 0 0 0
lib_a-xpg_strerror_r.o 0 0 0 0 0 0
gpio.c.obj 0 0 0 0 0 0
hw_random.c.obj 0 0 0 0 0 0
pm_locks.c.obj 0 0 0 0 0 0
_addsubdf3.o 0 0 0 0 0 0
_cmpdf2.o 0 0 0 0 0 0
_divdf3.o 0 0 0 0 0 0
_fixdfsi.o 0 0 0 0 0 0
_floatdidf.o 0 0 0 0 0 0
_floatsidf.o 0 0 0 0 0 0
_muldf3.o 0 0 0 0 0 0
_popcountsi2.o 0 0 0 0 0 0
platform.c.obj 0 0 0 0 0 0
platform_util.c.obj 0 0 0 0 0 0
sha256.c.obj 0 0 0 0 0 0
esp_mem.c.obj 0 0 0 0 0 0
gpio_periph.c.obj 0 0 0 0 0 0
spi_flash_rom_patch.c.ob 0 0 0 0 0 0
md5-internal.c.obj 0 0 0 0 0 0
From a library point of view, the major contributor is libc which is the C standard library. While you could probably cherry-pick some functions and drop others from there, I don't think anyone would recommend that.
Next up is libesp32, which provides critical functions such as start_cpu0(). Again, you may be able to cherry-pick only the functions you need, if you really want to.
You can figure out what a library provides by looking up the .a file (e.g. find build -name libesp32.a, and then running nm build/esp-idf/esp32/libesp32.a on the found path.
The second table lists the same size-data, but split per source-file instead of per library.
I want to remove duplicate rows in my table. There is a unique column dataframeid. According to the dataframeid, I want to remove duplicate records
Same records shows up 5 times.
Table - OLTMS_5B0395
Sample data
id DataFrameId DId OT WT AT RC YC BC RV YV BV Oa Aa Gt G M P S Ot O FCNT RSSI SF SNR Rec
2391 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2459 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2282 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2300 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2338 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
Expected output
2282 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
id that can be anything but there shouldn't be any duplicate records.
As per o/p all columns have duplicate values so, you can use row_number() function
delete t from (
select *,
row_number() over (partition by DataFrameId,. . . order by id) Seq
from OLTMS_5B0395
) t
where Seq > 1;
Be aware with order by clause in row_number() function, include column in partition by clause where you got the duplicate values
I'm learning a spread of programming languages in a class, and we're working on an APLX project at the moment. A restriction we have to work around is we cannot use If, For, While, etc. No loops or conditionals. I have to be able to take a plane of numbers, ranging 0-7, and replace each number 2 or greater into the depth of that number, and, ideally, change the 1's to 0's. For example:
0100230 => 0000560
I have no idea how I'm supposed to do the replacement with depth aspect, though the change from ones to zeros is quite simple. I'm able to produce the set of integers in a table and I understand how to replace specific values, but only with other specific values, not values that would have to be determined during the function. The depth should be the row depth, rather than the multi-dimensional depth.
For the record this is not the whole of the program, the program itself is a poker dealing and scoring program. This is a specific aspect of the scoring methodology that my professor recommended I use.
TOTALS„SCORE PHAND;TYPECOUNT;DEPTH;ISCOUNT;TEMPS;REPLACE
:If (½½PHAND) = 0
PHAND„DEAL PHAND
:EndIf
TYPECOUNT„CHARS°.¹PHAND
DEPTH„2Þ(½TYPECOUNT)
REPLACE „ 2 3 4 5 6 7
ISCOUNT „ +/ TYPECOUNT
ISCOUNT „ ³ISCOUNT
((1=,ISCOUNT)/,ISCOUNT)„0
©((2=,ISCOUNT)/,ISCOUNT)„1
©TEMPS „ ISCOUNT
Œ„ISCOUNT
Œ„PHAND
You may have missed the first lessons of your prof and it might help to look at at again to learn about vectors and how easy you can work with them - once you unlearned the ideas of other programming languages ;-)
Assume you have a vector A with numbers from 1 to 7:
A←⍳7
A
1 2 3 4 5 6 7
Now, if you wanted to search for values > 3, you'd do:
A>3
0 0 0 1 1 1 1
The result is a vector, too, and you can easily combine the two in lots of operations:
multiplication to only keep values > 0 and replace others with 0:
A×A>3
0 0 0 4 5 6 7
or add 500 to values >3
A+500×A>3
1 2 3 504 505 506 507
or, find the indices of values > 3:
(A>3)×⍳⍴A
0 0 0 4 5 6 7
Now, looking at your q again, the word 'depth' has a specific meaning in APL and I guess you meant something different. Do I understand correctly that you want to replace values > 2 with the ' indices' of these values?
Well, with what I've shown before, this is easy:
A←0 1 0 0 2 3 0
(A≥2)×⍳⍴A
0 0 0 0 5 6 0
edit: looking at multi-dimensional arrays:
let's look into this example:
A←(⍳5)∘.×⍳10
A
1 2 3 4 5 6 7 8 9 10
2 4 6 8 10 12 14 16 18 20
3 6 9 12 15 18 21 24 27 30
4 8 12 16 20 24 28 32 36 40
5 10 15 20 25 30 35 40 45 50
Now, which numbers are > 20 and < 30?
z←(A>20)∧A<30
z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0 0 0
0 0 0 0 1 0 0 0 0 0
Then, you can multiply the values with that boolean result to filter out only the ones satisfying the condition:
A×z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 21 24 27 0
0 0 0 0 0 24 28 0 0 0
0 0 0 0 25 0 0 0 0 0
Or, perhaps you're interested in the column-index of the values?
z×[2]⍳¯1↑⍴z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 7 8 9 0
0 0 0 0 0 6 7 0 0 0
0 0 0 0 5 0 0 0 0 0
NB: this statement might not work in all APL-dialects. Here's another way to formulate this:
z×((1↑⍴z)⍴0)∘.+⍳¯1↑⍴z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 7 8 9 0
0 0 0 0 0 6 7 0 0 0
0 0 0 0 5 0 0 0 0 0
I hope this gives you some ideas to play with. In general, using booleans to manipulate arrays in mathematical operations is an extremely powerful idea in APL which will take you loooooong ways ;-)
Also, if you'd like to see more of the same, have a look at the FinnAPL Idioms - some useful shorties grown over the years ;-)
edit re. "maintaining untouched values":
going back to example array A:
A←(⍳5)∘.×⍳10
A
1 2 3 4 5 6 7 8 9 10
2 4 6 8 10 12 14 16 18 20
3 6 9 12 15 18 21 24 27 30
4 8 12 16 20 24 28 32 36 40
5 10 15 20 25 30 35 40 45 50
Replacing values between 20 and 30 with the power 2 of these values, keeping all others unchanged:
touch←(A>20)∧A<30
(touch×A*2)+A×~touch
1 2 3 4 5 6 7 8 9 10
2 4 6 8 10 12 14 16 18 20
3 6 9 12 15 18 441 576 729 30
4 8 12 16 20 576 784 32 36 40
5 10 15 20 625 30 35 40 45 50
I hope you get the idea...
Or better: ask a new q, as otherwise this would truly take epic dimensions, whereas the idea of stackoverflow is more like "one issue - one question"...
I'm analysing a dataset with some data-mining tools.The response variable has ten levels and I'm trying to create a classifier.
Here comes the problem.When using nnet and bagging function,the result is not that good and the 5th level is even not in the prediction.
I want to use a confusion matrix to analyse the classifier.but as the 5th level is not shown in the prediction I can't get a well-formed matrix.So how can I get a well-formed matrix?i.e. I want a 10*10 matrix.
The confusion matrix:
library("mda")#This is where **confusion** comes from
> confusion(pre.bag$class,CLASS)#here confusion acts like table
true
predicted 1 2 3 4 6 7 8 9 10 5
1 338 9 6 0 5 12 10 1 15 46
2 9 549 1 59 18 0 3 0 0 6
3 18 1 44 0 0 0 2 0 0 4
4 0 1 0 21 0 0 0 0 0 0
6 2 13 0 1 299 2 9 0 0 0
7 5 2 1 0 10 231 6 0 1 0
8 0 0 0 0 0 5 76 0 0 0
9 5 1 0 0 0 0 0 62 0 0
10 7 3 1 0 0 2 1 6 181 16
attr(,"error")
[1] 0.1231743
attr(,"mismatch")
[1] 0.03386642
Try this:
pred <- factor(pre.bag$class, levels=levels(CLASS) )
confusion(pre.bag$class, CLASS)
(Tested with an fda-object.)