I run the following code in python:
exchange_path = f"dfs://market/{exchange.value}"
script = f'''if(existsDatabase(\"{exchange_path}\")) {{ dropDatabase(\"{exchange_path}\") }}'''
session.run(script)
There is an error:
RuntimeError: <Server Exception> in run: dropDatabase("dfs://market/BINANCE") => deleteSubChunks failed on '/market/BINANCE', chunk 17a19f25-cc16-15ae-714b-5ee30d5d6795 is not in COMPLETE state
Firstly drop the partitions by dropPartition and then use dropDatabase to delete the database.
dbName=exchange_path
fileCond=dbName + "%"
t=exec substr(file,strlen(dbName)) from rpc(getControllerAlias(),getClusterChunksStatus) where file like fileCond, state != "COMPLETE"
dropPartition(database("dfs:/"+dbName),t,,true)
Related
According to the Snowflake docs, when a user executes a copy command it will return 1 of 3 status values:
loaded
load failed
partially loaded
My question is if I use the Python Snowflake Connector (see example code below) to execute a copy command is an exception raised if the status returned is load failed or partially loaded?
Thank you!
copy_dml = 'copy into database.schema.table ' \
'from #fully_qualified_stage pattern = \'.*'+ table_name +'.*[.]json\' ' \
'file_format = (format_name = fully_qualified_json_format) ' \
'force = true;'
try:
import snowflake.connector
#-------------------------------------------------------------------------------------------------------------------------------
#snowflake variables
snowflake_warehouse = credentials.iloc[0]['snowflake_warehouse']
snowflake_account = credentials.iloc[0]['snowflake_account']
snowflake_role = credentials.iloc[0]['snowflake_role']
snowflake_username = credentials.iloc[0]['Username']
snowflake_password = credentials.iloc[0]['Password']
snowflake_connection = ''
cs = ''#snowflake connection cursor
exec_copy_dml = ''
copy_result_field_metadata = ''
copy_result = ''
snowflake_copy_result_df = ''
#-------------------------------------------------------------------------------------------------------------------------------
# load JSON file(s) into Snowflake
snowflake_connection = snowflake.connector.connect(
user = snowflake_username,
password = snowflake_password,
account = snowflake_account,
warehouse = snowflake_warehouse,
role = snowflake_role)
cs = snowflake_connection.cursor()
exec_copy_dml = cs.execute(copy_dml)
copy_result = exec_copy_dml.fetchall()
copy_result_field_metadata = cs.description
snowflake_copy_result_df = snowflake_results_df(copy_result_field_metadata,copy_result)
except snowflake.connector.errors.ProgrammingError as copy_error:
copy_exception_message = "There was a problem loading JSON files to Snowflake," + \
"a snowflake.connector.errors.ProgrammingError exception was raised."
print(copy_exception_message)
raise
except Exception as error_message:
raise
finally:
snowflake_connection.close()
I believe it won't raise exception for load status, you have to check the load status and take necessary action if required.
After you issue your COPY INTO dml, you can run the following query -
SELECT * FROM TABLE(VALIDATE(TABLE_NAME, job_id => '_last'))
This will give you details on the files that you were trying to load. It will normally return empty, unless you encountered issues upload.
You can save this save results in an object and make necessary control adjustments.
I would like to use Slick (3.2.3) to connect to a MSSQL database.
Currently, my project is the following.
In application.conf, I have
somedbname = {
driver = "slick.jdbc.SQLServerProfile$"
db {
host = "somehost"
port = "someport"
databaseName = "Recupel.Datawarehouse"
url = "jdbc:sqlserver://"${somedbname.db.host}":"${somedbname.db.port}";databaseName="${somedbname.db.databaseName}";"
user = "someuser"
password = "somepassword"
}
}
The "somehost" looks like XX.X.XX.XX where X's are numbers.
My build.sbt contains
name := "test-slick"
version := "0.1"
scalaVersion in ThisBuild := "2.12.7"
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.2.3",
"com.typesafe.slick" %% "slick-hikaricp" % "3.2.3",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.microsoft.sqlserver" % "mssql-jdbc" % "7.0.0.jre10"
)
The file with the "main" object contains
import slick.basic.DatabaseConfig
import slick.jdbc.JdbcProfile
import slick.jdbc.SQLServerProfile.api._
import scala.concurrent.Await
import scala.concurrent.duration._
val dbConfig: DatabaseConfig[JdbcProfile] = DatabaseConfig.forConfig("somedbname")
val db: JdbcProfile#Backend#Database = dbConfig.db
def main(args: Array[String]): Unit = {
try {
val future = db.run(sql"SELECT * FROM somettable".as[(Int, String, String, String, String,
String, String, String, String, String, String, String)])
println(Await.result(future, 10.seconds))
} finally {
db.close()
}
}
}
This, according to all the documentation that I could find, should be enough to connect to the database. However, when I run this, I get
[error] (run-main-0) java.sql.SQLTransientConnectionException: somedbname.db - Connection is not available, request timed out after 1004ms.
[error] java.sql.SQLTransientConnectionException: somedbname.db - Connection is not available, request timed out after 1004ms.
[error] at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
[error] at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
[error] at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
[error] at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
[error] at slick.jdbc.hikaricp.HikariCPJdbcDataSource.createConnection(HikariCPJdbcDataSource.scala:14)
[error] at slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
[error] at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
[error] at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
[error] at slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)
[error] at slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)
[error] at slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
[error] at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] at java.base/java.lang.Thread.run(Thread.java:844)
[error] Nonzero exit code: 1
Perhaps related and also annoying, when I run this code for the second (and subsequent) times, I get the following error instead:
Failed to get driver instance for jdbcUrl=jdbc:sqlserver://[...]
which forces me to kill and reload sbt each time.
What am I doing wrong? Worth noting: I can connect to the database with the same credential from a software like valentina.
As suggested by #MarkRotteveel, and following this link, I found a solution.
First, I explicitly set the driver, adding the line
driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
in the db dictionary, after password = "somepassword".
Secondly, the default timeout (after one second) appears to be too short for my purposes, and therefore I added the line
connectionTimeout = "30 seconds"
after the previous driver line, still in the db dictionary.
Now it works.
I am just getting started with the yocto project and trying to build an image for x86 architecture to be emulated using QEMU emulator (running on Ubuntu 16.04 ).I am getting the following error while building the OS image.
ERROR: core-image-sato-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_rootfs(d)
0003:
File: '/home/rahul/poky/poky/meta/classes/image.bbclass', lineno: 258, function: do_rootfs
0254: progress_reporter.next_stage()
0255:
0256: # generate rootfs
0257: d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
*** 0258: create_rootfs(d, progress_reporter=progress_reporter, logcatcher=logcatcher)
0259:
0260: progress_reporter.finish()
0261:}
0262:do_rootfs[dirs] = "${TOPDIR}"
File: '/home/rahul/poky/poky/meta/lib/oe/rootfs.py', lineno: 1010, function: create_rootfs
1006: env_bkp = os.environ.copy()
1007:
1008: img_type = d.getVar('IMAGE_PKGTYPE')
1009: if img_type == "rpm":
*** 1010: RpmRootfs(d, manifest_dir, progress_reporter, logcatcher).create()
1011: elif img_type == "ipk":
1012: OpkgRootfs(d, manifest_dir, progress_reporter, logcatcher).create()
1013: elif img_type == "deb":
1014: DpkgRootfs(d, manifest_dir, progress_reporter, logcatcher).create()
File: '/home/rahul/poky/poky/meta/lib/oe/rootfs.py', lineno: 201, function: create
0197: if self.progress_reporter:
0198: self.progress_reporter.next_stage()
0199:
0200: # call the package manager dependent create method
*** 0201: self._create()
0202:
0203: sysconfdir = self.image_rootfs + self.d.getVar('sysconfdir')
0204: bb.utils.mkdirhier(sysconfdir)
0205: with open(sysconfdir + "/version", "w+") as ver:
File: '/home/rahul/poky/poky/meta/lib/oe/rootfs.py', lineno: 450, function: _create
0446: rpm_pre_process_cmds = self.d.getVar('RPM_PREPROCESS_COMMANDS')
0447: rpm_post_process_cmds = self.d.getVar('RPM_POSTPROCESS_COMMANDS')
0448:
0449: # update PM index files
*** 0450: self.pm.write_index()
0451:
0452: execute_pre_post_process(self.d, rpm_pre_process_cmds)
0453:
0454: if self.progress_reporter:
File: '/home/rahul/poky/poky/meta/lib/oe/package_manager.py', lineno: 543, function: write_index
0539:
0540: def write_index(self):
0541: lockfilename = self.d.getVar('DEPLOY_DIR_RPM') + "/rpm.lock"
0542: lf = bb.utils.lockfile(lockfilename, False)
*** 0543: RpmIndexer(self.d, self.rpm_repo_dir).write_index()
0544: bb.utils.unlockfile(lf)
0545:
0546: def insert_feeds_uris(self, feed_uris, feed_base_paths, feed_archs):
0547: from urllib.parse import urlparse
File: '/home/rahul/poky/poky/meta/lib/oe/package_manager.py', lineno: 105, function: write_index
0101: else:
0102: signer = None
0103:
0104: createrepo_c = bb.utils.which(os.environ['PATH'], "createrepo_c")
*** 0105: result = create_index("%s --update -q %s" % (createrepo_c, self.deploy_dir))
0106: if result:
0107: bb.fatal(result)
0108:
0109: # Sign repomd
File: '/home/rahul/poky/poky/meta/lib/oe/package_manager.py', lineno: 21, function: create_index
0017:def create_index(arg):
0018: index_cmd = arg
0019:
0020: bb.note("Executing '%s' ..." % index_cmd)
*** 0021: result = subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
0022: if result:
0023: bb.note(result)
0024:
0025:"""
File: '/usr/lib/python3.5/subprocess.py', lineno: 626, function: check_output
0622: # empty string. That is maintained here for backwards compatibility.
0623: kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
0624:
0625: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
*** 0626: **kwargs).stdout
0627:
0628:
0629:class CompletedProcess(object):
0630: """A process that has finished running.
File: '/usr/lib/python3.5/subprocess.py', lineno: 708, function: run
0704: raise
0705: retcode = process.poll()
0706: if check and retcode:
0707: raise CalledProcessError(retcode, process.args,
*** 0708: output=stdout, stderr=stderr)
0709: return CompletedProcess(process.args, retcode, stdout, stderr)
0710:
0711:
0712:def list2cmdline(seq):
Exception: subprocess.CalledProcessError: Command '/home/rahul/poky/poky/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/recipe-sysroot-native/usr/bin/createrepo_c --update -q /home/rahul/poky/poky/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/oe-rootfs-repo' returned non-zero exit status 1
Subprocess output:
Temporary repodata directory /home/rahul/poky/poky/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/oe-rootfs-repo/.repodata/ already exists! (Another createrepo process is running?)
ERROR: core-image-sato-1.0-r0 do_rootfs: Function failed: do_rootfs
ERROR: Logfile of failure stored in: /home/rahul/poky/poky/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/temp/log.do_rootfs.5019
ERROR: Task (/home/rahul/poky/poky/meta/recipes-sato/images/core-image-sato.bb:do_rootfs) failed with exit code '1'
Build process runs upto almost 90 percent after which this error comes up which terminates the process.What could be the issue ?
I got the same error when my host machine shut down abruptly, but everything worked well after I delete the .repodata folder with sudo rm -r build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/oe-rootfs-repo/.repodata/ and then build again.
I stopped a build using Ctrl-C and got the python error described in the original question.
The .repodata folder (please see the answer from jmiranda) was empty. So I deleted the oe-rootfs-repo folder and this worked for me.
I get the same issue, but with error "Directory not empty" instead, when building in Docker container. Deleting the destination directory using rm -r, and running the build again, works.
None of these methods worked for me.
I then clean the build using bitbake -c clean mybuildname and then again made the build and it worked flawlessly, i hope it helps someone.
I have a perl script which prepares a statement with params 1,2,3,4 using DBI interface .The prepare statement executes a stored procedure with 4 params.Then it runs a foreach loop and does execute
If the parameters are not right the stored procedure throws error and execute fails. I am interested in printing a warning and proceeding to next set of params.
After the first execute failure for an "invalid parameter", I get "Attempt to initiate a new Adaptive Server operation with results pending" as errmsg in SQL execute for subsequent loop iteration.
foreach my $file (#filelist)
{
#.. get param1 , param2 , param3, param4 from $file
unless ( dbh->execute($param1,$param2,$param3,$param4) )
{
#print some warning
next;
}
}
How do I continue processing with this error?
These are the version I am using
>perl -MDBI -e 'DBI-> installed_versions;'
Perl : 5.010001 (x86_64-linux-thread-multi)
OS : linux (2.6.18-348.12.1.el5)
DBI : 1.609
DBD::Sybase : 1.15
DBD::Sponge : 12.010002
DBD::SQLite : 1.27
DBD::Proxy : install_driver(Proxy) failed: Can't locate RPC/PlClient.pm in #INC
DBD::Informix : 2013.0521
DBD::Gofer : 0.011565
DBD::File : 0.37
DBD::ExampleP : 12.010007
DBD::DBM : 0.03
I added { RaiseError => 0, PrintWarn => 1, PrintError => 1 } in the connect as suggested by mekazu, (The code is exact same as yours mekazu ) but it is not helping still getting the same error.
In foreach I get success,
iteration 1 no error
iteration 2 gives a stored procedure error
" Database execute failed. ERROR MESSAGE: Server message number=52001
severity=16 state=1 line=124 server=dev procedure=sp1 text=Data
Error:Invalid parameter: P1
iteration 3
Database execute failed. ERROR MESSAGE: OpenClient message: LAYER =
(0) ORIGIN = (0) SEVERITY = (78) NUMBER = (51) Server dev, database
dev Message String: Attempt to initiate a new Adaptive Server
operation with results pending
iteration 4
Database execute failed. ERROR MESSAGE: OpenClient message: LAYER =
(0) ORIGIN = (0) SEVERITY = (78) NUMBER = (51) Server dev, database
dev Message String: Attempt to initiate a new Adaptive Server
operation with results pending
Ensure you disable RaiseError (default: false) when creating the handle. You can use PrintWarn (default: false) and PrintError (default: true) to save you doing it yourself:
$dbh = DBI->connect($dsn, $user, $password,
{ RaiseError => 0, PrintWarn => 1, PrintError => 1 });
my $sth = $dbh->prepare($query);
foreach my $file (#filelist)
{
#.. get param1 , param2 , param3, param4 from $file
unless ( $sth->execute($param1,$param2,$param3,$param4) )
{
#warning already printed
next;
}
}
i have many data.frames() that i am trying to send to MySQL database via RMySQL().
# Sends data frame to database without a problem
dbWriteTable(con3, name="SPY", value=SPY , append=T)
# stock1 contains a character vector of stock names...
stock1 <- c("SPY.A")
But when I try to loop it:
i= 1
while(i <= length(stock1)){
# converts "SPY.A" into SPY
name <- print(paste0(str_sub(stock1, start = 1, end = -3))[i], quote=F)
# sends data.frame to database
dbWriteTable(con3,paste0(str_sub(stock1, start = 1, end = -3))[i], value=name, append=T)
i <- 1+i
}
The following warning is returned & nothing was sent to database
In addition: Warning message:
In file(fn, open = "r") :
cannot open file './SPY': No such file or directory
However, I believe that the problem is with pasting value onto dbWriteTable() since writing dbWriteTable(con3, "SPY", SPY, append=T) works but dbWriteTable(con3, "SPY", name, append=T) will not...
You are probably using a non-base package for str_sub and I'm guessing you get the same behavior with substr. Does this succeed?
dbWriteTable(con3, substr( stock1, 1,3) , get(stock1), append=T)