Snowflake Python Connector: Copy Command Status and Error Handling - snowflake-cloud-data-platform

According to the Snowflake docs, when a user executes a copy command it will return 1 of 3 status values:
loaded
load failed
partially loaded
My question is if I use the Python Snowflake Connector (see example code below) to execute a copy command is an exception raised if the status returned is load failed or partially loaded?
Thank you!
copy_dml = 'copy into database.schema.table ' \
'from #fully_qualified_stage pattern = \'.*'+ table_name +'.*[.]json\' ' \
'file_format = (format_name = fully_qualified_json_format) ' \
'force = true;'
try:
import snowflake.connector
#-------------------------------------------------------------------------------------------------------------------------------
#snowflake variables
snowflake_warehouse = credentials.iloc[0]['snowflake_warehouse']
snowflake_account = credentials.iloc[0]['snowflake_account']
snowflake_role = credentials.iloc[0]['snowflake_role']
snowflake_username = credentials.iloc[0]['Username']
snowflake_password = credentials.iloc[0]['Password']
snowflake_connection = ''
cs = ''#snowflake connection cursor
exec_copy_dml = ''
copy_result_field_metadata = ''
copy_result = ''
snowflake_copy_result_df = ''
#-------------------------------------------------------------------------------------------------------------------------------
# load JSON file(s) into Snowflake
snowflake_connection = snowflake.connector.connect(
user = snowflake_username,
password = snowflake_password,
account = snowflake_account,
warehouse = snowflake_warehouse,
role = snowflake_role)
cs = snowflake_connection.cursor()
exec_copy_dml = cs.execute(copy_dml)
copy_result = exec_copy_dml.fetchall()
copy_result_field_metadata = cs.description
snowflake_copy_result_df = snowflake_results_df(copy_result_field_metadata,copy_result)
except snowflake.connector.errors.ProgrammingError as copy_error:
copy_exception_message = "There was a problem loading JSON files to Snowflake," + \
"a snowflake.connector.errors.ProgrammingError exception was raised."
print(copy_exception_message)
raise
except Exception as error_message:
raise
finally:
snowflake_connection.close()

I believe it won't raise exception for load status, you have to check the load status and take necessary action if required.

After you issue your COPY INTO dml, you can run the following query -
SELECT * FROM TABLE(VALIDATE(TABLE_NAME, job_id => '_last'))
This will give you details on the files that you were trying to load. It will normally return empty, unless you encountered issues upload.
You can save this save results in an object and make necessary control adjustments.

Related

Db::getInstance()->ExecuteS error 'must be used only with select' when i'm doing it

I have the same problem when i try to make a custom sql. My Prestashop version 1.7.3.4
$sql_order_detail = 'SELECT `id_order_detail`, `product_id`, `product_quantity`, `unit_price_tax_incl` FROM '._DB_PREFIX_.'order_detail WHERE `id_order` = ' . $id_order;
$order_details = Db::getInstance()->ExecuteS($this->sql_order_detail, $array = true, $use_cache = 0);
I test my sql through logs and phpMyAdmin:
SELECT `id_order_detail`, `product_id`, `product_quantity`, `unit_price_tax_incl` FROM ps_order_detail WHERE `id_order` = 24
The error is:
stderr: Db->executeS() must be used only with select, show, explain or describe queries,
Solved using:
(mysteriously)
$sql = new DbQuery();
$sql->select('*');
$sql->from('order_detail');
$sql->where('id_order = ' . $id_order);
$order_details = Db::getInstance()->executeS($sql);

Passing na.strings to dbWriteTable

I'm trying to run the following command in R in order to read a local tab-delimited file as a SQLite database:
library(RSQLite)
banco <- dbConnect(drv = "SQLite",
dbname = "data.sqlite")
dbWriteTable(conn = banco,
name = "Tarefas",
value = "data.tsv",
sep = "\t",
dec = ",",
na.strings = c("", NA),
row.names = FALSE,
header = TRUE)
However, the statements above yield the following error:
Error in read.table(fn, sep = sep, header = header, skip = skip, nrows
= nrows, : formal argument "na.strings" matched by multiple actual arguments
Which makes me think I'm not being able to pass na.strings explicitly as a read.delim argument. Running dbWriteTable without this argument gives me "RS-DBI driver: (RS_sqlite_import: ./data.tsv line 17696 expected 20 columns of data but found 18)". This is understandable, since I've checked line 17696 and it is almost completely blank.
Another test run using sqldf also gives me an error:
> read.csv2.sql(file = "data.tsv",
+ sql = "CREATE TABLE Tarefas AS SELECT * FROM FILE LIMIT 5",
+ dbname = "data.sqlite",
+ header = TRUE,
+ row.names = FALSE)
Error in sqliteExecStatement(con, statement, bind.data) :
RS-DBI driver: (error in statement: no such table: FILE)
Which I believe is an unrelated error, but still very confusing for someone who's pretty much an absolute SQL noob such as myself. Runnin read.csv.sql instead gives me this error:
Error in read.table(fn, sep = sep, header = header, skip = skip, nrows = nrows, :
more columns than column names
So is there a way to pass na.strings = c("", NA) at dbWriteTable? Is there a better way to read 10 GB tab-delimited files into R aside from sqldf and RSQLite? I've already tried data.table and ff.

String or binary data would be truncated when updating datatable

I am trying to update customer info. I use this code to load 5 fields:
cust_cn.Open()
Dim cust_da As New SqlDataAdapter("SELECT * FROM [customers] where [custID]=" & txtCustPhone.Text, cust_cn)
cust_da.Fill(cust_datatable)
txtCustPhone.Text = cust_datatable.Rows(0).Item("custID")
txtCustFirstName.Text = cust_datatable.Rows(0).Item("first")
txtCustLastName.Text = cust_datatable.Rows(0).Item("last")
txtCustAddress.Text = cust_datatable.Rows(0).Item("address")
txtCustZip.Text = cust_datatable.Rows(0).Item("zip")
and this works fine. When I try to modify one of the fields (change zip code on an existing customer)
with this code:
If cust_datatable.Rows.Count <> 0 Then
cust_datatable.Rows(0).Item("custID") = txtCustPhone.Text
cust_datatable.Rows(0).Item("first") = txtCustFirstName.Text
cust_datatable.Rows(0).Item("last") = txtCustLastName
cust_datatable.Rows(0).Item("address") = txtCustAddress.Text
cust_datatable.Rows(0).Item("zip") = txtCustZip.Text
'cust_datatable.Rows(custrecord)("custID") = txtCustPhone.Text
'cust_datatable.Rows(custrecord)("first") = txtCustFirstName.Text
'cust_datatable.Rows(custrecord)("last") = txtCustLastName.Text
'cust_datatable.Rows(custrecord)("address") = txtCustAddress.Text
'cust_datatable.Rows(custrecord)("zip") = txtCustZip.Text
cust_DA.Update(cust_datatable)
End If
I get the error: "String or binary data would be truncated"
I originally tried to update using the commented section, but it was only modifying the first record in the database.
Any thoughts?

Setting a not-default db adapter in zend framework 1.12

i've a very specific problem, but i'm a niewbie with zend framework so i don't have idea of how exctly this db adapter works as a configuration, but i've already made a db connection with the default adapter of zend, and it was successful. Now i've to set two different database connections for two different db in the same application. So i've taken my application.ini and i've written the following lines:
;connessione al db
resources.db.adapter = pdo_mssql
resources.db.params.host = "ip"
resources.db.params.username = user
resources.db.params.password = pwd
resources.db.params.dbname = NAME
resources.db.isDefaultTableAdapter = true
resources.db.params.pdoType = dblib
;connessione al db1
resources.db1.adapter = pdo_mssql
resources.db1.params.host = "ip"
resources.db1.params.username = user
resources.db1.params.password = pwd
resources.db1.params.dbname = NAME
resources.db1.isDefaultTableAdapter = false
resources.db1.params.pdoType = dblib
then i went to my action controller and i wrote:
$db = Zend_Registry::get ( 'db' );
$result = $db->fetchRow("SELECT [Sell-to Customer No_] FROM dbo.SyncroPlanningTable WHERE id='".$id);
$rag_soc=$result->{"Sell-to Customer No_"};
$db1 = Zend_Registry::get ( 'db1' );
$result1 = $db1->fetchRow("SELECT [No_],[Name],[Address],[City],[Contact],[Name],[Phone] FROM `dbo.SOS$Customer` WHERE No_ = '".$rag_soc."'");
The error i'm getting is the following:
Fatal error: Uncaught exception 'Zend_Application_Bootstrap_Exception' with message 'Unable to resolve plugin "db1";
UPDATE:
My bootstrap.php is:
$resource = $this->getPluginResource ( "db" );
$db = $resource->getDbAdapter ();
$db->setFetchMode ( Zend_Db::FETCH_OBJ );
Zend_Db_Table_Abstract::setDefaultAdapter ( $db );
Zend_Registry::set ( "db", $db );
How can i change it? it is not mentioned in the manual page you gave me.
resources.db refers to Zend_Application_Resource_Db, so "db" here is not a variable name.
You should use Zend_Application_Resource_Multidb to support multiple database connections:
http://framework.zend.com/manual/1.12/en/zend.application.available-resources.html#zend.application.available-resources.multidb
Your code is expecting the DB adapters to be in the registry, so you need to grab them from the multiDB resource and store them:
$multiDB = $this->getPluginResource('multidb');
Zend_Registry::set('db1', $multiDB->getDb('db1');
Zend_Registry::set('db2', $multiDB->getDb('db2');
also, this line:
Zend_Db_Table_Abstract::setDefaultAdapter ( $db );
can be removed, as you're specifying the default adapter in the application.ini.

Ldap error code 32

I'm trying to synchronize OpenLDAP and Active directory together. To do so I'm using a program called LSC-Project which is specified to do this sort of thing.
I have configured the program the best I can however I can't find a way to shake off the following error:
javax.naming.NameNotFoundException: [LDAP: error code 32 - 0000208D: NameErr: DSID-
031001CD,
problem 2001 (NO_OBJECT), data 0, best match of:
'DC=domname,DC=com'
]; remaining name
'uid=user1,ou=Users'
May 09 15:19:25 - ERROR - Error while synchronizing ID uid=user1,ou=Users:
java.lang.Exception:
Technical problem while applying modifications to directory
dn: uid=user1,ou=Users,dc=domname,dc=com
changetype: add
userPassword: 3+kU2th/WMo/v553A24a3SBw2kU=
objectClass: uid
This is the configuration file that the program runs on:
###############################
Destination LDAP directory #
##############################
dst.java.naming.provider.url = ldap://192.168.1.3:389/dc=Windows,dc=com
dst.java.naming.security.authentication = simple
dst.java.naming.security.principal = cn=Administrator,cn=Users,dc=Windows,dc=com
dst.java.naming.security.credentials = 11111
dst.java.naming.referral = ignore
dst.java.naming.ldap.derefAliases = never
dst.java.naming.factory.initial = com.sun.jndi.ldap.LdapCtxFactory
dst.java.naming.ldap.version = 3
dst.java.naming.ldap.pageSize = 1000
#########################
Source LDAP directory
#########################
src.java.naming.provider.url = ldap://192.168.1.2:389/dc=Linux,dc=com
src.java.naming.security.authentication = simple
src.java.naming.security.principal = uid=root,ou=users,dc=Linux,dc=com
src.java.naming.security.credentials = 11111
src.java.naming.referral = ignore
src.java.naming.ldap.derefAliases = never
src.java.naming.factory.initial = com.sun.jndi.ldap.LdapCtxFactory
src.java.naming.ldap.version = 3
#######################
Tasks configuration
#######################
lsc.tasks = Administrator
lsc.tasks.Administrator.srcService = org.lsc.jndi.SimpleJndiSrcService
lsc.tasks.Administrator.srcService.baseDn = ou=users
lsc.tasks.Administrator.srcService.filterAll = (&(objectClass=person))
lsc.tasks.Administrator.srcService.pivotAttrs = uid
lsc.tasks.Administrator.srcService.filterId = (&(objectClass=person)(uid={uid}))
lsc.tasks.Administrator.srcService.attrs = description uid userPassword
lsc.tasks.Administrator.dstService = org.lsc.jndi.SimpleJndiDstService
lsc.tasks.Administrator.dstService.baseDn = cn=Users
lsc.tasks.Administrator.dstService.filterAll = (&(cn=*)(objectClass=organizationalPerson))
lsc.tasks.Administrator.dstService.pivotAttrs = cn, top, person, user, organizationalPerson
lsc.tasks.Administrator.dstService.filterId = (&(objectClass=user) (sAMAccountName={cn}))
lsc.tasks.Administrator.dstService.attrs = description cn userPassword objectClass
lsc.tasks.Administrator.bean = org.lsc.beans.SimpleBean
lsc.tasks.Administrator.dn = "uid=" + srcBean.getAttributeValueById("uid") + ",ou=Users"
dn.real_root = dc=Domname,dc=com
#############################
Syncoptions configuration
#############################
lsc.syncoptions.Administrator = org.lsc.beans.syncoptions.PropertiesBasedSyncOptions
lsc.syncoptions.Administrator.default.action = M
lsc.syncoptions.Administrator.objectClass.action = M
lsc.syncoptions.Administrator.objectClass.force_value = srcBean.getAttributeValueById("cn").toUpperCase()
lsc.syncoptions.Administrator.userPassword.default_value = SecurityUtils.hash(SecurityUtils.HASH_SHA1, "defaultPassword")
lsc.syncoptions.Administrator.default.delimiter=;
lsc.syncoptions.Administrator.objectClass.force_value = "top";"user";"person";"organizationalPerson"
lsc.syncoptions.Administrator.userPrincipalName.force_value = srcBean.getAttributeValueById("uid") + "#Domname.com"
lsc.syncoptions.Administrator.userAccountControl.create_value = AD.userAccountControlSet ( "0", [AD.UAC_SET_NORMAL_ACCOUNT])
I'm suspecting that it has something to do with the baseDn of the Task configuration in the part of the source configuration.
The OSs is ubuntu 10.04 and Windows2K3
Someone suggested to me to make a manual sync between them but I have not found any guides to do so. And this program is pretty much the only thing that says that is does this kind of job without costs.
The baseDn should be the distinguished name of the base object of the search, for example, ou=users,dc=domname,dc=com.
see also
LDAP: Mastering Search Filters
LDAP: Search best practices
LDAP: Programming practices
The main reason for NameNotFoundException is that the object which you're searching doesn't exist or the container in which you are searching is not correct.
In case of Spring-ldap, we used to get this error when we specify the baseDn in the context file(LdapContextSource bean) and also in createUser code to build userDn.we need not specify the dc again in the buildUserDn()
protected Name buildUserDn(String userName) {
DistinguishedName dn = new DistinguishedName();
//only cn is required as the base dn is already specified in context file
dn.add("cn", userName);
return dn;
}
In Active Directory: Users catalog is container class, not OrganizationalUnit, so you should use: cn=users,dc=domname,dc=com

Resources