Can we reduce ESP IDF boiler plate size? - c

I tested running bare bones code using ESP IDF on an ESP32 chip using duinotech XC-3800, and obtained the following results in terms of image size.
Analysis Binary Size for ESP32
Folder Structure
temp/
main/
CMakeLists.txt
main.c
CMakeLists.txt
File contents
CMakeLists.txt
# The following lines of boilerplate have to be in your project's
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(temp)
main>CMakeLists.txt
idf_component_register(SRCS "main.c"
INCLUDE_DIRS "")
Test 1 main>main.c
#include <stdio.h>
void app_main(void) {
printf("Hello world!\n");
for (int i = 10; i >= 0; i--) {
printf("Restarting in %d seconds...\n", i);
}
printf("Restarting now.\n");
fflush(stdout);
}
Test 2 main>main.c
#include <stdio.h>
void app_main(void) { printf("Hello world!\n"); }
Test 3 main>main.c
void app_main(void) {}
Size comparison
Obtained by running idf_size.py build/temp.map
Test 1
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75408 bytes
Flash rodata: 23844 bytes
Total image size:~ 146376 bytes (.bin may be padded larger)
Test 2
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75240 bytes
Flash rodata: 23796 bytes
Total image size:~ 146160 bytes (.bin may be padded larger)
Test 3
Total sizes:
DRAM .data size: 8320 bytes
DRAM .bss size: 4072 bytes
Used static DRAM: 12392 bytes ( 168344 available, 6.9% used)
Used static IRAM: 38804 bytes ( 92268 available, 29.6% used)
Flash code: 75004 bytes
Flash rodata: 23780 bytes
Total image size:~ 145908 bytes (.bin may be padded larger)
Analysis
Size for code obtained by running stat --format="%s" main/main.c
All Sizes are in Bytes
Test No. | Code | Image | Flash Code | Flash rodata
-------- | -----| ------ | ---------- | ------------
1 | 207 | 146376 | 75408 | 23844
2 | 70 | 146160 | 75240 | 23796
3 | 43 | 145908 | 75004 | 23780
At least 145KB of boiler plate code just to get an empty main run.
Speculation
I suspect that the 145KB is made up of a number of libraries that are always loaded onto the chip whether you use them or not. Some of them must be the FreeRTOS, WiFi, HTTP etc.
Can we bring down this size somehow and load only the bare minimum required for operation?

You can get more detailed size information by running idf.py size-components and idf.py size-files. Here are the results for your "Test 3" (i.e. empty function) on my dev environment (note that my Total Image Size is already slightly larger than the one you posted.
Total sizes:
DRAM .data size: 7860 bytes
DRAM .bss size: 4128 bytes
Used static DRAM: 11988 bytes ( 168748 available, 6.6% used)
Used static IRAM: 32706 bytes ( 98366 available, 25.0% used)
Flash code: 76002 bytes
Flash rodata: 32164 bytes
Total image size:~ 148732 bytes (.bin may be padded larger)
Per-archive contributions to ELF file:
Archive File DRAM .data & .bss IRAM Flash code & rodata Total
libc.a 0 0 0 55348 3829 59177
libesp32.a 2132 2351 6767 5976 8076 25302
libfreertos.a 4140 776 12432 0 1653 19001
libdriver.a 76 20 0 4003 7854 11953
libsoc.a 161 4 4953 612 3852 9582
libvfs.a 240 103 0 5464 950 6757
libheap.a 877 4 3095 830 986 5792
libspi_flash.a 24 290 1855 779 890 3838
libefuse.a 16 4 0 1142 2213 3375
libnewlib.a 152 272 766 830 96 2116
libapp_update.a 0 4 109 159 1221 1493
libesp_ringbuf.a 0 0 848 0 209 1057
liblog.a 8 268 429 82 0 787
libhal.a 0 0 515 0 32 547
libpthread.a 8 12 0 256 0 276
libgcc.a 0 0 0 0 160 160
libbootloader_support.a 0 0 0 126 0 126
libm.a 0 0 0 88 0 88
libcxx.a 0 0 0 11 0 11
libxtensa-debug-module.a 0 0 8 0 0 8
libmain.a 0 0 0 5 0 5
(exe) 0 0 0 0 0 0
libmbedcrypto.a 0 0 0 0 0 0
libmbedtls.a 0 0 0 0 0 0
libwpa_supplicant.a 0 0 0 0 0 0
Per-file contributions to ELF file:
Object File DRAM .data & .bss IRAM Flash code & rodata Total
lib_a-vfprintf.o 0 0 0 14193 756 14949
lib_a-svfprintf.o 0 0 0 13838 756 14594
lib_a-svfiprintf.o 0 0 0 9642 1210 10852
lib_a-vfiprintf.o 0 0 0 9945 738 10683
uart.c.obj 44 12 0 2985 7293 10334
tasks.c.obj 12 700 5546 0 531 6789
vfs_uart.c.obj 48 63 0 3680 808 4599
panic.c.obj 2023 5 1989 0 0 4017
esp_err_to_name.c.obj 0 0 0 50 3947 3997
portasm.S.obj 3084 0 484 0 0 3568
lib_a-dtoa.o 0 0 0 3522 13 3535
multi_heap.c.obj 873 0 2277 0 0 3150
intr_alloc.c.obj 8 22 618 1703 722 3073
esp_efuse_utility.c.obj 0 0 0 867 2205 3072
cpu_start.c.obj 0 1 1068 331 1352 2752
queue.c.obj 0 0 2310 0 325 2635
lib_a-mprec.o 0 0 0 2134 296 2430
rtc_clk.c.obj 161 4 2098 0 0 2263
dbg_stubs.c.obj 0 2072 32 108 0 2212
vfs.c.obj 192 40 0 1784 142 2158
rtc_periph.c.obj 0 0 0 0 2080 2080
esp_timer_esp32.c.obj 8 26 1068 262 538 1902
xtensa_vectors.S.obj 8 0 1776 0 36 1820
flash_mmap.c.obj 0 264 1092 114 339 1809
task_wdt.c.obj 53 4 0 1178 548 1783
heap_caps.c.obj 4 0 818 52 617 1491
timers.c.obj 8 56 1006 0 233 1303
cache_utils.c.obj 4 14 749 81 402 1250
soc_memory_layout.c.obj 0 0 0 0 1181 1181
heap_caps_init.c.obj 0 4 0 778 369 1151
port.c.obj 0 16 625 0 493 1134
esp_ota_ops.c.obj 0 4 0 147 965 1116
xtensa_intr_asm.S.obj 1024 0 51 0 0 1075
ringbuf.c.obj 0 0 848 0 209 1057
rtc_time.c.obj 0 0 815 0 198 1013
memory_layout_utils.c.ob 0 0 0 612 393 1005
rtc_init.c.obj 0 0 992 0 0 992
periph_ctrl.c.obj 8 0 0 615 280 903
lib_a-fseeko.o 0 0 0 866 0 866
time.c.obj 0 32 122 703 0 857
clk.c.obj 0 0 64 559 221 844
esp_timer.c.obj 8 16 261 406 134 825
dport_access.c.obj 8 40 444 189 137 818
log.c.obj 8 268 429 82 0 787
rtc_wdt.c.obj 0 0 743 0 0 743
partition.c.obj 0 8 0 522 149 679
ipc.c.obj 0 28 159 283 120 590
locks.c.obj 8 0 485 0 94 587
crosscore_int.c.obj 8 8 193 130 156 495
syscall_table.c.obj 144 240 0 82 0 466
system_api.c.obj 0 0 439 0 0 439
timer.c.obj 16 0 0 112 281 409
int_wdt.c.obj 0 1 91 306 0 398
freertos_hooks.c.obj 8 128 43 216 0 395
esp_app_desc.c.obj 0 0 109 12 256 377
brownout.c.obj 0 0 0 149 201 350
windowspill_asm.o 0 0 311 0 0 311
rtc_module.c.obj 8 8 0 291 0 307
xtensa_context.S.obj 0 0 306 0 0 306
cpu_util.c.obj 0 0 305 0 0 305
dport_panic_highint_hdl. 8 0 242 0 0 250
lib_a-reent.o 0 0 0 232 0 232
lib_a-fopen.o 0 0 0 224 0 224
lib_a-snprintf.o 0 0 0 214 0 214
esp_efuse_api.c.obj 0 4 0 193 0 197
pthread_local_storage.c. 8 4 0 180 0 192
cache_err_int.c.obj 0 0 56 98 0 154
xtensa_intr.c.obj 0 0 108 0 35 143
list.c.obj 0 0 142 0 0 142
syscalls.c.obj 0 0 91 45 0 136
lib_a-assert.o 0 0 0 68 60 128
lib_a-flags.o 0 0 0 127 0 127
bootloader_common.c.obj 0 0 0 126 0 126
lib_a-s_frexp.o 0 0 0 110 0 110
flash_ops.c.obj 20 4 14 62 0 100
lib_a-vprintf.o 0 0 0 94 0 94
lib_a-s_fpclassify.o 0 0 0 88 0 88
lib_a-fiprintf.o 0 0 0 84 0 84
pthread.c.obj 0 8 0 76 0 84
esp_efuse_fields.c.obj 0 0 0 82 0 82
clock.o 0 0 72 0 0 72
reent_init.c.obj 0 0 68 0 2 70
state_asm--restore_extra 0 0 62 0 0 62
state_asm--save_extra_nw 0 0 62 0 0 62
xtensa_vector_defaults.S 0 0 46 0 0 46
lib_a-fseek.o 0 0 0 45 0 45
_divdi3.o 0 0 0 0 40 40
_moddi3.o 0 0 0 0 40 40
_udivdi3.o 0 0 0 0 40 40
_umoddi3.o 0 0 0 0 40 40
xtensa_init.c.obj 0 4 32 0 0 36
interrupts--intlevel.o 0 0 0 0 32 32
esp_efuse_table.c.obj 16 0 0 0 8 24
lib_a-errno.o 0 0 0 10 0 10
pm_esp32.c.obj 0 0 0 8 0 8
int_asm--set_intclear.o 0 0 8 0 0 8
eri.c.obj 0 0 8 0 0 8
cxx_exception_stubs.cpp. 0 0 0 6 0 6
cxx_guards.cpp.obj 0 0 0 5 0 5
hello_world_main.c.obj 0 0 0 5 0 5
FreeRTOS-openocd.c.obj 4 0 0 0 0 4
dummy_main_src.c.obj 0 0 0 0 0 0
bootloader_flash.c.obj 0 0 0 0 0 0
bootloader_random.c.obj 0 0 0 0 0 0
bootloader_sha.c.obj 0 0 0 0 0 0
esp_image_format.c.obj 0 0 0 0 0 0
flash_partitions.c.obj 0 0 0 0 0 0
lib_a-fputs.o 0 0 0 0 0 0
lib_a-printf.o 0 0 0 0 0 0
lib_a-putc.o 0 0 0 0 0 0
lib_a-putchar.o 0 0 0 0 0 0
lib_a-puts.o 0 0 0 0 0 0
lib_a-sprintf.o 0 0 0 0 0 0
lib_a-strerror.o 0 0 0 0 0 0
lib_a-u_strerr.o 0 0 0 0 0 0
lib_a-xpg_strerror_r.o 0 0 0 0 0 0
gpio.c.obj 0 0 0 0 0 0
hw_random.c.obj 0 0 0 0 0 0
pm_locks.c.obj 0 0 0 0 0 0
_addsubdf3.o 0 0 0 0 0 0
_cmpdf2.o 0 0 0 0 0 0
_divdf3.o 0 0 0 0 0 0
_fixdfsi.o 0 0 0 0 0 0
_floatdidf.o 0 0 0 0 0 0
_floatsidf.o 0 0 0 0 0 0
_muldf3.o 0 0 0 0 0 0
_popcountsi2.o 0 0 0 0 0 0
platform.c.obj 0 0 0 0 0 0
platform_util.c.obj 0 0 0 0 0 0
sha256.c.obj 0 0 0 0 0 0
esp_mem.c.obj 0 0 0 0 0 0
gpio_periph.c.obj 0 0 0 0 0 0
spi_flash_rom_patch.c.ob 0 0 0 0 0 0
md5-internal.c.obj 0 0 0 0 0 0
From a library point of view, the major contributor is libc which is the C standard library. While you could probably cherry-pick some functions and drop others from there, I don't think anyone would recommend that.
Next up is libesp32, which provides critical functions such as start_cpu0(). Again, you may be able to cherry-pick only the functions you need, if you really want to.
You can figure out what a library provides by looking up the .a file (e.g. find build -name libesp32.a, and then running nm build/esp-idf/esp32/libesp32.a on the found path.
The second table lists the same size-data, but split per source-file instead of per library.

Related

How to correctly export svg in react/next

I am trying to add svg in my next project but i keep getting this error 'Error: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.'
code:index.js
import React from 'react'
import Image from 'next/image';
import {ReactComponent as Logo} from '/assets/Logo.svg';
const Header=()=>{
return (
<div className='flex'>
<Logo/>
</div>
)
}
export default Header;
svg file
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 74 20">
<defs>
<style>
.cls-1{fill:#28b16d}
</style>
</defs>
<path d="M51 13.53a.58.58 0 0 0-.11.08 2.77 2.77 0 0 1-2.28 1.1 4.79 4.79 0 0 1-2.32-.45 3.81 3.81 0 0 1-2.07-3 6.57 6.57 0 0 1-.07-1V6.07c0-.56.17-.73.73-.73h1.22c.49 0 .69.21.69.7v4.27a2.45 2.45 0 0 0 .5 1.56A2.25 2.25 0 0 0 51 11.3a1.13 1.13 0 0 0 .06-.41V6.06c0-.59.17-.76.75-.76H53c.49 0 .71.22.71.71v9.07a4.27 4.27 0 0 1-2.38 4 5.79 5.79 0 0 1-3.16.61 5.28 5.28 0 0 1-1.95-.45 1.23 1.23 0 0 1-.86-1.33v-.69a.44.44 0 0 1 .68-.43c.34.15.66.35 1 .48a3.87 3.87 0 0 0 2.42.16A2 2 0 0 0 51 15.46v-1.8a.54.54 0 0 0 0-.13zM24.34 5.55a13 13 0 0 1 1.32-.45 5.6 5.6 0 0 1 3.14.08 4.14 4.14 0 0 1 2.8 3.4 5.52 5.52 0 0 1-.09 2.62 4.48 4.48 0 0 1-2.8 3 6.13 6.13 0 0 1-5.76-.51 4.76 4.76 0 0 1-.73-.61 1.52 1.52 0 0 1-.48-1.08V1.71a.89.89 0 0 1 .48-.85l1.35-.79c.45-.26.77 0 .77.46V5.55zm0 4.3v1.83a.45.45 0 0 0 .22.42 3 3 0 0 0 2.35.47 2.27 2.27 0 0 0 1.85-1.48A3.61 3.61 0 0 0 29 9.44a2.55 2.55 0 0 0-.84-1.82 3.08 3.08 0 0 0-3.63.06.51.51 0 0 0-.13.34c-.07.61-.06 1.22-.06 1.83zM11.26-.07a7.34 7.34 0 0 1 6.91 5.77 7.41 7.41 0 0 1-4.36 8.48 7.16 7.16 0 0 1-8-1.79c-.4-.45-.41-.45-.92-.15l-3.33 2c-.57.33-.81.27-1.14-.3l-.31-.55a.64.64 0 0 1 .27-1l3.43-2c.52-.31.51-.3.35-.86A7.4 7.4 0 0 1 8.87.33a6.52 6.52 0 0 1 1-.26c.47-.07.93-.07 1.39-.14zm5.66 7.43a5.89 5.89 0 0 0-5.75-5.95 5.88 5.88 0 0 0-5.88 5.75A5.88 5.88 0 0 0 11 13.27a5.87 5.87 0 0 0 5.92-5.91zM39.91 13.43a1 1 0 0 0-.15.12 2.61 2.61 0 0 1-1.86 1.1 4.49 4.49 0 0 1-4.17-1.38 4.54 4.54 0 0 1-1.13-2.69 5.23 5.23 0 0 1 .47-3 4.6 4.6 0 0 1 3.13-2.42 7 7 0 0 1 5.28.64l.16.1a1.64 1.64 0 0 1 .9 1.62v6.07a2.13 2.13 0 0 1 0 .42.48.48 0 0 1-.5.42h-1.58c-.33 0-.52-.22-.55-.59v-.41zm0-5.63a.47.47 0 0 0-.25-.45A3.07 3.07 0 0 0 38.28 7a2.55 2.55 0 0 0-2.91 1.9 3.91 3.91 0 0 0 0 1.91A2.43 2.43 0 0 0 39.33 12a1.41 1.41 0 0 0 .57-1.22 59.2 59.2 0 0 1 0-2.98zM62.31 13.43a1.18 1.18 0 0 0-.16.14 2.52 2.52 0 0 1-1.73 1.07 4.54 4.54 0 0 1-3.22-.56 3.82 3.82 0 0 1-1.74-2.87 6.8 6.8 0 0 1-.07-1V6.02c0-.56.16-.73.73-.73h1.25A.59.59 0 0 1 58 6v4.29a2.46 2.46 0 0 0 .54 1.63 2.27 2.27 0 0 0 3.67-.63 1 1 0 0 0 0-.34V6.07c0-.56.18-.74.75-.74h1.19c.48 0 .69.22.69.7v7.69c0 .54-.2.74-.73.74H63c-.48 0-.67-.19-.68-.67l-.01-.36zM66.63 6.9V3.27a.85.85 0 0 1 .45-.81l1.35-.79.12-.06a.46.46 0 0 1 .68.44v2.89c0 .34 0 .35.36.35H73c.67 0 .81.14.82.8v.64c0 .44-.2.63-.63.64h-3.58c-.39 0-.39 0-.39.38v2.39a3.61 3.61 0 0 0 .15 1 2.09 2.09 0 0 0 2.3 1.44 2.52 2.52 0 0 0 1.45-.58l.13-.09a.51.51 0 0 1 .56-.1c.2.11.18.31.18.49v.85a1 1 0 0 1-.47.89 4 4 0 0 1-2.23.65 5 5 0 0 1-2.53-.56 3.91 3.91 0 0 1-2-3.08 16.67 16.67 0 0 1-.08-2c-.06-.72-.05-1.43-.05-2.15z" class="cls-1"/>
<path d="M14.91 8.06v1.88c0 .43-.18.61-.61.61h-1.44c-.34 0-.49-.14-.49-.48V8.56a1.29 1.29 0 0 0-1.09-1.25 1.26 1.26 0 0 0-1.39 1 3.3 3.3 0 0 0 0 .69v1.06c0 .35-.15.52-.51.52h-1.5a.54.54 0 0 1-.6-.58V6.26a.73.73 0 0 1 .4-.71q1.42-.8 2.81-1.65a1.05 1.05 0 0 1 1.21 0c.94.57 1.9 1.13 2.85 1.68a.62.62 0 0 1 .34.6v1.91z" class="cls-1"/>
</svg>
I searched on the error and it says there is a problem with the import/export. So how should i export/import the file??

how to mathematically have array border without actually increasing the array size?

So I have a game board represented by a 1d array of 64 squares and I wish to know when a piece is trying to escape the borders.
So for example:
Assume a king in the game of chess is on the X:
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 X 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
it is easy to calculate the attacked squares with a direction vector:
-9, -8, -7,
-1, 1,
7, 8, 9
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 1 1 1 0 0
0 0 0 1 X 1 0 0
0 0 0 1 1 1 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
also if he is on the upper or lower edges of the board:
I just add an if that checks if the square + directionvector[i] > 63 or
square + directionvector[i] < 0
so:
0 0 0 1 X 1 0 0
0 0 0 1 1 1 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
The problem then comes when it sits on the side edges:
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 1
1 0 0 0 0 0 1 X
1 0 0 0 0 0 1 1
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
This is what will happen it will "jump" to the wrong place as you see above.
Do you know of any way of checking this?
thanks
At the beginning of your program, you can precompute the legal moves for every piece and every square and store them somewhere.
Assuming that your squares are numbered from 0 to 63, then KING_MOVES[i] will store the legal moves for a king on square i (where i ranges from 0 to 63). For example, KING_MOVES[0] = { 1, 8, 9 } and KING_MOVES[31] = { 22, 23, 30, 38, 39 }. Similarly, for a knight in the corner, KNIGHT_MOVES[0] = { 10, 17 }.
0 1 2 3 4 5 6 7
8 9 10 11 12 13 14 15
16 17 18 19 20 21 22 23
24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39
40 41 42 43 44 45 46 47
48 49 50 51 52 53 54 55
56 57 58 59 60 61 62 63
The number of legal moves is different from every square (a king has 8 moves in the center, but only 3 in the corner). You can store the number of legal moves separately (such as NUM_KING_MOVES[31] = 5) or you can terminate each list with a special value like -1 or 64.
To precompute these lists, you can convert each 1-D number to a pair of (row, column) coordinates and generate moves naively, with boundary checks. For a king on square 31, you convert 31 to (row 3, column 7). When moving to the right, you end up on (row 3, column 8), which is illegal, so you discard this move. When moving to the top-left, you end up on (row 2, column 6), which is legal, so you convert that back to a 1-D representation: 2 * 8 + 6 = 22. You append this move to KING_MOVES[31] and so forth.
This gives you the best of both worlds: faster and shorter code for move generation with no wasted memory on every board due to padding. The downside is a few kilobytes of extra memory.
If you're going down this path, you could also take a look at bitboards for even faster move generation.
You could get the row number from the cell index as i / 8 (floored), and use that to check if e.g. a step to the right or left passed to another row.
E.g. here, on a four-column board, stepping to the "right" from position 7, we go to position 8. The row numbers for these are 7 / 4 = 1 and 8 / 4 = 2, so we changed to another row.
....
...7
8...
....
But that's harder to do for diagonal moves, because you expect them to change the row. In the end, this is pretty much just turning the board into a 2D array, and the step offsets into 2D values, where you can check for overflow in the obvious way.
IMO, in principle what you're asking is impossible for a 1D board. With the board defined as a 1D vector (and similarly, the offsets as 1D offsets), then there are no edges within the board, and nothing to escape. Instead of this:
.ab.
....
...c
d...
you really just have this:
.ab........cd...
And it's obvious that c and d are just as adjacent to each other as a and b are.

How to remove the duplicate rows by using single column in SQL Server

I want to remove duplicate rows in my table. There is a unique column dataframeid. According to the dataframeid, I want to remove duplicate records
Same records shows up 5 times.
Table - OLTMS_5B0395
Sample data
id DataFrameId DId OT WT AT RC YC BC RV YV BV Oa Aa Gt G M P S Ot O FCNT RSSI SF SNR Rec
2391 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2459 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2282 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2300 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
2338 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
Expected output
2282 1525345797494 4 0 0 35 338 333 664 245 244 245 0 0 0 0 0 0 0 0 0 1243 -92 12 -18 2018-03-05 16:39:00
id that can be anything but there shouldn't be any duplicate records.
As per o/p all columns have duplicate values so, you can use row_number() function
delete t from (
select *,
row_number() over (partition by DataFrameId,. . . order by id) Seq
from OLTMS_5B0395
) t
where Seq > 1;
Be aware with order by clause in row_number() function, include column in partition by clause where you got the duplicate values

Appengine/ComputeEngine Memory Issue?

I suspect this is a straightforward memory leak issue (python27 process has a memory leak with appengine libraries running on the Managed VMs GCE containers), but I'm confused a few things about the data I was collecting during the OOM issues.
After running fine for most of a day, my "vmstat 1" suddenly changed drastically:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 70116 7612 41240 0 0 0 64 231 645 3 2 95 0
0 0 0 70148 7612 41240 0 0 0 0 164 459 2 2 96 0
1 0 0 70200 7612 41240 0 0 0 0 209 712 2 1 97 0
1 0 0 65432 7612 41344 0 0 100 0 602 820 48 5 47 1
1 3 0 69840 5644 29620 0 0 1284 0 812 797 33 6 34 27
0 1 0 69068 5896 30216 0 0 852 68 362 1052 6 1 0 93
0 1 0 68340 6160 30536 0 0 556 0 547 1355 4 2 0 94
0 2 0 67928 6564 30972 0 0 872 0 793 2173 9 5 0 86
0 1 0 63988 6888 34416 0 0 3776 0 716 1940 3 3 0 94
3 0 0 63696 7104 34608 0 0 376 0 353 1006 4 4 34 58
0 0 0 63548 7112 34948 0 0 332 48 379 916 13 1 84 2
0 0 0 63636 7116 34948 0 0 4 0 184 637 0 1 99 0
0 0 0 63660 7116 34948 0 0 0 0 203 556 0 3 97 0
0 1 0 76100 3648 26128 0 0 460 0 409 1142 7 4 85 4
0 3 0 73452 948 15940 0 0 4144 80 1041 1126 53 6 10 31
0 6 0 73828 84 11424 0 0 32924 80 1135 1732 11 4 0 85
0 6 0 72684 64 12324 0 0 52168 4 1519 2397 6 3 0 91
0 11 0 67340 52 12328 0 0 78072 16 1388 2974 2 9 0 89
1 10 0 65992 336 13412 0 0 79796 0 1297 2973 0 9 0 91
0 15 0 69000 48 10396 0 0 78344 0 1203 2739 2 7 0 91
0 15 0 67168 52 11460 0 0 86864 0 1244 3003 0 6 0 94
1 15 0 71268 52 7836 0 0 82552 4 1497 3269 0 7 0 93
In particular, my memory cache and buff dropped, and the io bytes-in surged, and it stayed like this for ~10 minutes afterwards before the machine died and was rebooted by Google Compute Engine. I assume the "bi" represents bytes-in from disk, but I'm curious why swpd showed 0 for this instance, if there was swapping? And why is memory "free" stat still unaffected if things are reaching a swapping point?
Second at the time of the final crash, my top showed:
Top - 15:06:20 up 1 day, 13:23, 2 users, load average: 13.88, 11.22, 9.30
Tasks: 92 total, 3 running, 89 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 8.0 sy, 0.0 ni, 0.0 id, 90.9 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 1745136 total, 1684032 used, 61104 free, 648 buffers
KiB Swap: 0 total, 0 used, 0 free, 12236 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23 root 20 0 0 0 0 R 12.6 0.0 2:11.61 kswapd0
10 root rt 0 0 0 0 S 2.5 0.0 0:52.92 watchdog/0
2315 root 20 0 192m 17m 0 S 2.1 1.0 47:58.74 kubelet
2993 root 20 0 6116m 1.2g 0 S 0.9 70.2 318:41.51 python2.7
6644 root 20 0 55452 12m 0 S 0.9 0.7 0:00.81 python
2011 root 20 0 761m 9924 0 S 0.7 0.6 12:23.44 docker
6624 root 20 0 4176 132 0 D 0.5 0.0 0:00.24 du
140 root 0 -20 0 0 0 S 0.4 0.0 0:08.64 kworker/0:1H
2472 root 20 0 39680 5616 296 D 0.4 0.3 0:27.43 python
1 root 20 0 10656 132 0 S 0.2 0.0 0:02.61 init
3 root 20 0 0 0 0 S 0.2 0.0 2:02.17 ksoftirqd/0
22 root 20 0 0 0 0 R 0.2 0.0 0:24.61 kworker/0:1
1834 root 20 0 53116 756 0 S 0.2 0.0 0:01.79 rsyslogd
1859 root 20 0 52468 9624 0 D 0.2 0.6 0:29.36 supervisord
2559 root 20 0 349m 172m 0 S 0.2 10.1 25:56.31 ruby
Again, I see the python27 process has claimed to 70% (which when combined with the 10% from Ruby, puts me into dangerous territory). Buy why is kswapd going crazy with my 10% CPU when the above vmstat shows 0?
Should I just not trust vmstat's swapd?

Data frame column value with range condition [duplicate]

This question already has answers here:
R add new column with predefined pattern
(2 answers)
Closed 8 years ago.
I want to create the columns below wherein I can specify the range to apply the value 1 to one column and the rest 0. I know this has been asked and answered perfectly here before but I can't find that particular one right now.
time1 time2 time3 time4 time5
1 1 0 0 0 0
2 1 0 0 0 0
3 1 0 0 0 0
4 1 0 0 0 0
5 1 0 0 0 0
6 0 1 0 0 0
7 0 1 0 0 0
8 0 1 0 0 0
9 0 1 0 0 0
10 0 1 0 0 0
11 0 0 1 0 0
12 0 0 1 0 0
13 0 0 1 0 0
14 0 0 1 0 0
15 0 0 1 0 0
16 0 0 0 1 0
17 0 0 0 1 0
18 0 0 0 1 0
19 0 0 0 1 0
20 0 0 0 1 0
21 0 0 0 0 1
22 0 0 0 0 1
23 0 0 0 0 1
24 0 0 0 0 1
25 0 0 0 0 1
I can't recall how this was generated but the answer included an n option to specify the intervals per column.
You could do
cols <- 4
diag(cols)[rep(1:cols, each=cols), ]
[,1] [,2] [,3] [,4]
[1,] 1 0 0 0
[2,] 1 0 0 0
[3,] 1 0 0 0
[4,] 1 0 0 0
[5,] 0 1 0 0
[6,] 0 1 0 0
[7,] 0 1 0 0
[8,] 0 1 0 0
[9,] 0 0 1 0
[10,] 0 0 1 0
[11,] 0 0 1 0
[12,] 0 0 1 0
[13,] 0 0 0 1
[14,] 0 0 0 1
[15,] 0 0 0 1
[16,] 0 0 0 1
I finally found the exact question: "R add new column with predefined pattern" and the solution I was looking for is:
range <- 5
cols <- 5
y <- gl(cols, range)
mat <- model.matrix(~y-1) # -1 is for remove the intercept
colnames(mat) <- paste0('var', 1:cols)
mat
It provides a dataframe ready to be included in other dataframes (via merge(old, new)) and specifying a name for the columns.
The output is:
var1 var2 var3 var4 var5
1 1 0 0 0 0
2 1 0 0 0 0
3 1 0 0 0 0
4 1 0 0 0 0
5 1 0 0 0 0
6 0 1 0 0 0
7 0 1 0 0 0
8 0 1 0 0 0
9 0 1 0 0 0
10 0 1 0 0 0
11 0 0 1 0 0
12 0 0 1 0 0
13 0 0 1 0 0
14 0 0 1 0 0
15 0 0 1 0 0
16 0 0 0 1 0
17 0 0 0 1 0
18 0 0 0 1 0
19 0 0 0 1 0
20 0 0 0 1 0
21 0 0 0 0 1
22 0 0 0 0 1
23 0 0 0 0 1
24 0 0 0 0 1
25 0 0 0 0 1

Resources