It's taking about 15 minutes to insert a 500MB ndjson file with 100,000 rows into MS SQL Server table. I am running Spark locally on a machine with good specs - 32GB RAM, i9-10885H CPU with 8 cores. I doubt that the machine is being used to its full capabilities. Here is what I am trying.
master = "local[16]"
conf = SparkConf() \
.setAppName(appName) \
.set("spark.driver.memory", "16g") \
.set("spark.executor.memory", "1g") \
.set('spark.executor.cores', '5') \
.set("spark.driver.extraClassPath","./mssql-jdbc-9.2.1.jre11.jar") \
.setMaster(master)
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
spark = sqlContext.sparkSession
def insert_into_ss(start):
for i in range(start, len(files)):
item = files[i]
print(item)
start = datetime.now()
spark_df = sqlContext.read.json(upload_dir + '/' +item)
spark_df = spark_df.select([col(c).cast("string") for c in spark_df.columns])
print('Casting time', datetime.now() - start)
spark_df.write.mode("append") \
.format("jdbc") \
.option("url", url) \
.option("dbtable", table) \
.option("batchsize", 20000) \
.option("reliabilityLevel", 'NO_DUPLICATES') \
.option("tableLock", 'true') \
.option("numPartitions", 16) \
.option("bulkCopyTimeout", 600000) \
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
.save()
end = datetime.now()
print(end-start)
insert_into_ss()
I'm trying to run Qt5 framework through eglfs instead of X11 or wayland.
I'm trying to install Qt5 for qemuarm emualting Raspberry pi 3 based on yocto Rocko.
This is my bblayers.conf:
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"
BBPATH = "${TOPDIR}"
SRCPATH = "/home/yocto/yocto"
BBFILES ?= ""
BBLAYERS ?= " \
${SRCPATH}/meta \
${SRCPATH}/meta-poky \
${SRCPATH}/meta-openembedded/meta-oe \
${SRCPATH}/meta-openembedded/meta-multimedia \
${SRCPATH}/meta-openembedded/meta-networking \
${SRCPATH}/meta-openembedded/meta-perl \
${SRCPATH}/meta-openembedded/meta-python \
${SRCPATH}/meta-qt5 \
${SRCPATH}/meta-raspberrypi \
${SRCPATH}/meta-security \
and this is my local.conf:
MACHINE ??= "qemuarm"
DL_DIR ?= "${TOPDIR}/../downloads"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_deb"
EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
BB_DISKMON_DIRS = "\
STOPTASKS,${TMPDIR},1G,100K \
STOPTASKS,${DL_DIR},1G,100K \
STOPTASKS,${SSTATE_DIR},1G,100K \
STOPTASKS,/tmp,100M,100K \
ABORT,${TMPDIR},100M,1K \
ABORT,${DL_DIR},100M,1K \
ABORT,${SSTATE_DIR},100M,1K \
ABORT,/tmp,10M,1K"
LICENSE_FLAGS_WHITELIST = "commercial"
CONF_VERSION = "1"
PREFERRED_VERSION_linux-raspberrypi = "4.%"
DISTRO_FEATURES_remove = "x11 wayland"
DISTRO_FEATURES_append = " systemd opengl pam ${DISTRO_FEATURES_LIBC}"
VIRTUAL-RUNTIME_init_manager = "systemd"
EXTRA_IMAGE_FEATURES += "package-management splash"
INHERIT+="toaster buildhistory"
CORE_IMAGE_EXTRA_INSTALL += "openssh"
ENABLE_UART="1"
#PACKAGECONFIG_append_qtbase = " accessibility eglfs fontconfig gles2 linuxfb"
################### QT ######################
QT_DEV_TOOLS = " \
qtbase-dev \
qtbase-mkspecs \
qtbase-plugins \
qtbase-tools \
qtserialport-dev \
qtserialport-mkspecs \
"
QT_TOOLS = " \
qtbase \
qtserialport \
"
FONTS = " \
fontconfig \
fontconfig-utils \
ttf-bitstream-vera \
"
TSLIB = " \
tslib \
tslib-conf \
tslib-calibrate \
tslib-tests \
tspress \
"
QT5_PKGS = " \
qt3d \
qtcharts \
qtdeclarative \
qtdeclarative-plugins \
qtdeclarative-qmlplugins \
qtgraphicaleffects \
qtlocation-plugins \
qtmultimedia \
qtquickcontrols2 \
qtsensors-plugins \
qtserialbus \
qtsvg \
qtwebsockets-qmlplugins \
qtvirtualkeyboard \
qtxmlpatterns \
"
QML_APPS = " \
qqtest \
"
CORE_IMAGE_EXTRA_INSTALL += " \
${FONTS} \
${QT_TOOLS} \
${QT5_PKGS} \
cinematicexperience \
"
I'm trying to build this image bitbake rpi-hwup-image
the problem is with qtbase, it fails with this error:
| ERROR: Feature 'opengl-desktop' was enabled, but the pre-condition '(config.win32 && !config.winrt && !features.opengles2 && (config.msvc || libs.opengl))
| || (!config.watchos && !config.win32 && libs.opengl)' failed.
|
| ERROR: The OpenGL functionality tests failed!
| You might need to modify the include and library search paths by editing QMAKE_INCDIR_OPENGL[_ES2],
| QMAKE_LIBDIR_OPENGL[_ES2] and QMAKE_LIBS_OPENGL[_ES2] in the mkspec for your platform.
Update
This problem is solved by uncommenting PACKAGECONFIG_append_qtbase and it has a typo so, it's been updated to be PACKAGECONFIG_append_pn-qtbase.
I added those lines too:
PACKAGECONFIG_append_pn-qemu-native = " sdl"
PACKAGECONFIG_append_pn-nativesdk-qemu = " sdl".
I comment out this line LICENSE_FLAGS_WHITELIST = "commercial".
but it fails again at qtbase build by this error (this is the tail of the log file) (I deleted the tmp folder and started bitbake rpi-hwup-image again but it went to the same error)
| cd windowflags/ && ( test -e Makefile || /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/recipe-sysroot-native/usr/bin/qt5/qmake -o Makefile /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/git/examples/widgets/widgets/windowflags/windowflags.pro -qtconf /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/bin/qt.conf ) && make -f Makefile
| make[4]: Entering directory '/home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/examples/widgets/widgets/windowflags'
| compiling /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/git/examples/widgets/widgets/windowflags/controllerwindow.cpp
| compiling /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/git/examples/widgets/widgets/windowflags/previewwindow.cpp
| linking wiggly
| make[4]: Leaving directory '/home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/examples/widgets/widgets/wiggly'
| compiling /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/git/examples/widgets/widgets/windowflags/main.cpp
| linking validators
| generating .moc/moc_predefs.h
| moc /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/git/examples/widgets/widgets/windowflags/controllerwindow.h
| make[4]: Leaving directory '/home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/examples/widgets/widgets/validators'
| moc /home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/git/examples/widgets/widgets/windowflags/previewwindow.h
| compiling .moc/moc_controllerwindow.cpp
| compiling .moc/moc_previewwindow.cpp
| linking windowflags
| make[4]: Leaving directory '/home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/examples/widgets/widgets/windowflags'
| make[3]: Leaving directory '/home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/examples/widgets/widgets'
| make[2]: Leaving directory '/home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/examples/widgets'
| make[1]: Leaving directory '/home/yocto/yocto/build/RP3_Qt/tmp/work/armv5e-poky-linux-gnueabi/qtbase/5.9.3+gitAUTOINC+4d8ae444c2-r0/build/examples'
| ERROR: oe_runmake failed
| WARNING: exit code 1 from a shell command.
I found the solution according to this page.
The configuration part -at this page- says something about ommiting the tests part at this option -nomake tests, it's the same part that fails at the compiling stage of qtbase.
so after cleaning qtbase and adding this part to my local.conf
PACKAGECONFIG_remove_pn-qtbase = " tests gl"
gl is omitted too because we want to build qtbase for eglfs not for desktop.
I am working on an AutoIt script that will take in a text file. The text file is made up of a series of test results in groups of 5 lines. This script is to go through the file and output the median score for each group. the score is displayed after the 11th ",". For example with this sample file:
a,a,a,a,a,a,a,a,a,a,100.2
a,a,a,a,a,a,a,a,a,a,300.2
a,a,a,a,a,a,a,a,a,a,160.2
a,a,a,a,a,a,a,a,a,a,301.2
a,a,a,a,a,a,a,a,a,a,161.2
b,b,b,b,b,b,b,b,b,b,110.5
b,b,b,b,b,b,b,b,b,b,87.5
b,b,b,b,b,b,b,b,b,b,89.5
b,b,b,b,b,b,b,b,b,b,190.5
b,b,b,b,b,b,b,b,b,b,170.5
c,c,c,c,c,c,c,c,c,c,90.2
c,c,c,c,c,c,c,c,c,c,190.2
c,c,c,c,c,c,c,c,c,c,40.2
c,c,c,c,c,c,c,c,c,c,20.2
c,c,c,c,c,c,c,c,c,c,80.2
I am expecting median scores of 161.2, 110.5 and 80.2 for groups a, b and c respectively. However, the output result is as follows:
median is = 161.2 ----- sorted array : 100.2 , 160.2 , 161.2 , 300.2 , 301.2
median is = 190.5 ----- sorted array : 110.5 , 170.5 , 190.5 , 87.5 , 89.5
median is = 40.2 ----- sorted array : 190.2 , 20.2 , 40.2 , 80.2 , 90.2
The first group gets sorted correctly but after that the array is no longer being sorted. And this is despite _ArraySort(...) being called directly before the output. Any idea why this could be happening?
The code is as follows:
Func CondenseResults()
$size = 5 ;temporary, variable to be passed on from other function
$gameDetail = ""
$rawResult = "D:\RTG_Benchmark\results\results.txt"
local $openResults = FileOpen("D:\RTG_Benchmark\results\results2.txt", 1)
Local $i = 1
Local $j = 0
Local $spaceCount = 0
Local $score[$size]
Local $runMultiplier[2] ; 0 = previous, 1 = current
Local $subString
Do
$resultLine = FileReadLine($rawResult, $i) ; read line from raw result
$subString = StringSplit($resultLine,",") ; split
If StringLen($resultLine) = 0 Then
$spaceCount += 1
$j = 0
FileWriteLine ( $openResults, "" & #CRLF)
Else
$gameDetail = $subString[1] & $subString[2] & $subString[3] & $subString[4] & $subString[5] & $subString[6] & $subString[7] & $subString[8] & $subString[9] & $subString[10] ; substring 1 - 10
$runMultiplier[1] = $gameDetail
If $i = 1 Then ;
$score[$j] = $subString[11]
Else
If $runMultiplier[1] <> $runMultiplier[0] Then
$j = 0
$score[$j] = $subString[11]
Else
$j += 1
If $j >= $size Then
$j = 0
EndIf
$score[$j] = $subString[11]
EndIf
$resultLine = FileReadLine($rawResult, $i+1)
$subString = StringSplit($resultLine,",") ; split
If StringLen($resultLine) = 0 Or $gameDetail <> $subString[1] & $subString[2] & $subString[3] & $subString[4] & $subString[5] & $subString[6] & $subString[7] & $subString[8] & $subString[9] & $subString[10] Then
_ArraySort($score, 0,0,0,0,1)
ConsoleWrite("median is = " & ($score[$size/2] & " ----- sorted array : " & $score[0] & " , " & $score[1] & " , " & $score[2] & " , " & $score[3] & " , " & $score[4]& #CRLF))
$j = 0
EndIf
EndIf
EndIf
$i += 1
$runMultiplier[0] = $runMultiplier[1]
Until $i >=_FileCountLines($rawResult)
$j += 1
$score[$j] = $subString[11]
_ArraySort($score)
ConsoleWrite("median is =" & ($score[$size/2] & " ----- sorted array : " & $score[0] & " , " & $score[1] & " , " & $score[2] & " , " & $score[3] & " , " & $score[4]& #CRLF))
FileClose($openResults)
sleep(100000)
Return 0
EndFunc
I found the issue. AutoIt was treating the numbers as strings and was sorting it one digit at a time from left to right.
Number() function took care of this issue.
This is the exact code I am running in my system with sh lookup.sh. I don't see any details within nawk block printed or written to the file abc.txt. Only I am here 0 and I am here 1 are printed. Even the printf in nawk is not working. Please help.
processbody() {
nawk '
NR == FNR {
split($0, x, "#")
country_code[x[2]] = x[1]
next
system(" echo " I am here ">>/tmp/abc.txt")
}
{
CITIZEN_COUNTRY_NAME = "INDIA"
system(" echo " I am here 1">>/tmp/abc.txt")
if (CITIZEN_COUNTRY_NAME in country_code) {
value = country_code[CITIZEN_COUNTRY_NAME]
system(" echo " I am here 2">>/tmp/abc.txt")
} else {
value = "null"
system(" echo " I am here 3">>/tmp/abc.txt")
}
system(" echo " I am here 4">>/tmp/abc.txt")
print "found " value " for country name " CITIZEN_COUNTRY_NAME >> "/tmp/standalone.txt"
} ' /tmp/country_codes.config
echo "I am here 5" >> /tmp/abc.txt
}
# Main program starts here
echo "I am here 0" >> /tmp/abc.txt
processbody
And my country_codes.config file:
$ cat country_codes.config
IND#INDIA
IND#INDIB
USA#USA
CAN#CANADA
That's some pretty interesting awk code. The problem is that your first condition, the NR == FNR one, is active for each record read from the first file - the country_codes.config file, but the processing action contains next so after it reads a record and splits it and saves it, it goes and reads the next record - not executing the second block of the awk script. At the end, it is done - nothing more to do, so it never prints anything.
This works sanely:
processbody()
{
awk '
{
split($0, x, "#")
country_code[x[2]] = x[1]
#next
}
END {
CITIZEN_COUNTRY_NAME = "INDIA"
if (CITIZEN_COUNTRY_NAME in country_code) {
value = country_code[CITIZEN_COUNTRY_NAME]
} else {
value = "null"
}
print "found " value " for country name " CITIZEN_COUNTRY_NAME
} ' /tmp/country_codes.config
}
# Main program starts here
processbody
It produces the output:
found IND for country name INDIA
As Hai Vu notes, you can use awk's intrinsic record splitting facilities to simplify life:
processbody()
{
awk -F# '
{ country_code[$2] = $1 }
END {
CITIZEN_COUNTRY_NAME = "INDIA"
if (CITIZEN_COUNTRY_NAME in country_code) {
value = country_code[CITIZEN_COUNTRY_NAME]
} else {
value = "null"
}
print "found " value " for country name " CITIZEN_COUNTRY_NAME
} ' /tmp/country_codes.config
}
# Main program starts here
processbody
I don't know what you want to accomplish, but let me guess: if country is INDIA, then print the following output:
found IND for country name INDIA
If that is the case, the following code will accomplish that goal:
awk -F# '/INDIA/ {print "found " $1 " for country name " $2 }' /tmp/country_codes.config
The -F# flag tells awk (or nawk) to use # as the field separator.
#user549432 I think that you want one awk script that first reads in the country codes file and builds the associative array, and then reads in the input files (not # delimited) and does a substitution?
if so, let's assume that /tmp/country_codes.config has:
IND#INDIA
IND#INDIB
USA#USA
CAN#CANADA
and /tmp/input_file (not # delimited) has:
I am from INDIA
I am from INDIB
I am from CANADA
Then, we can have a nawk script like this:
nawk '
BEGIN {
while (getline < "/tmp/country_codes.config")
{
split($0,x,"#")
country_code[x[2]] = x[1]
}
}
{ print $1,$2,$3,country_code[$4]}
' /tmp/input_file
The output will be:
I am from IND
I am from IND
I am from CAN