Issue :
Someone has added a junk column in one of my table.I want to figure it out from the logs as when and from where this activity has been performed.
Please Help regarding this issue.
Make sure enable logging in postgresql.conf
1.log_destination = 'stderr' #log_destination = 'stderr,csvlog,syslog'
2.logging_collector = on #need restart
3.log_directory = 'pg_log'
4.log_file_name = 'postgresql-%Y-%m-%d_%H%M%S.log'
5.log_rotation_age = 1d
6.log_rotation_size = 10MB
7.log_min_error_statement = error
8.log_min_duration_statement = 5000 # -1 = disable ; 0 = ALL ; 5000 = 5sec
9.log_line_prefix = '|%m|%r|%d|%u|%e|'
10.log_statment = 'ddl' # 'none' | 'ddl' | 'mod' | 'all'
#prefer 'ddl' because the log output will be 'ddl' and 'query min duration'
If you don't enable it, make sure enable it now.
if you don't have log the last attempt is pg_xlogdump your xlog file under pg_xlog and look for DDL
Related
According to the Snowflake docs, when a user executes a copy command it will return 1 of 3 status values:
loaded
load failed
partially loaded
My question is if I use the Python Snowflake Connector (see example code below) to execute a copy command is an exception raised if the status returned is load failed or partially loaded?
Thank you!
copy_dml = 'copy into database.schema.table ' \
'from #fully_qualified_stage pattern = \'.*'+ table_name +'.*[.]json\' ' \
'file_format = (format_name = fully_qualified_json_format) ' \
'force = true;'
try:
import snowflake.connector
#-------------------------------------------------------------------------------------------------------------------------------
#snowflake variables
snowflake_warehouse = credentials.iloc[0]['snowflake_warehouse']
snowflake_account = credentials.iloc[0]['snowflake_account']
snowflake_role = credentials.iloc[0]['snowflake_role']
snowflake_username = credentials.iloc[0]['Username']
snowflake_password = credentials.iloc[0]['Password']
snowflake_connection = ''
cs = ''#snowflake connection cursor
exec_copy_dml = ''
copy_result_field_metadata = ''
copy_result = ''
snowflake_copy_result_df = ''
#-------------------------------------------------------------------------------------------------------------------------------
# load JSON file(s) into Snowflake
snowflake_connection = snowflake.connector.connect(
user = snowflake_username,
password = snowflake_password,
account = snowflake_account,
warehouse = snowflake_warehouse,
role = snowflake_role)
cs = snowflake_connection.cursor()
exec_copy_dml = cs.execute(copy_dml)
copy_result = exec_copy_dml.fetchall()
copy_result_field_metadata = cs.description
snowflake_copy_result_df = snowflake_results_df(copy_result_field_metadata,copy_result)
except snowflake.connector.errors.ProgrammingError as copy_error:
copy_exception_message = "There was a problem loading JSON files to Snowflake," + \
"a snowflake.connector.errors.ProgrammingError exception was raised."
print(copy_exception_message)
raise
except Exception as error_message:
raise
finally:
snowflake_connection.close()
I believe it won't raise exception for load status, you have to check the load status and take necessary action if required.
After you issue your COPY INTO dml, you can run the following query -
SELECT * FROM TABLE(VALIDATE(TABLE_NAME, job_id => '_last'))
This will give you details on the files that you were trying to load. It will normally return empty, unless you encountered issues upload.
You can save this save results in an object and make necessary control adjustments.
I'm really new to Flume. I prefer Flume than Sqoop because data is continued to be imported to MS SQL Server in my case, therefore I think Flume is a better choice which is able to transfer data in real time.
I just followed some online example and then editing my own flume config file which tells something about the source, channel, and sink. However, it seemed that Flume didn't work successfully. There was no data being transferred to HBase.
mssql-hbase.conf
# source, channel, sink
agent1.sources = src1
agent1.channels = ch1
agent1.sinks = sk1
# declare source type
agent1.sources.src1.type = org.keedio.flume.source.SQLSource
agent1.sources.src1.hibernate.connection.url = jdbc:sqlserver://xx.xx.xx.xx:1433;DatabaseName=xxxx
agent1.sources.src1.hibernate.connection.user = xxxx
agent1.sources.src1.hibernate.connection.password = xxxx
agent1.sources.src1.table = xxxx
agent1.sources.src1.hibernate.connection.autocommit = true
# declare mysql hibernate dialect
agent1.sources.src1.hibernate.dialect = org.hibernate.dialect.SQLServerDialect
agent1.sources.src1.hibernate.connection.driver_class = com.microsoft.sqlserver.jdbc.SQLServerDriver
#agent1.sources.src1.hibernate.provider_class=org.hibernate.connection.C3P0ConnectionProvider
#agent1.sources.src1.columns.to.select = *
#agent1.sources.src1.incremental.column.name = PK, name, machine, time
#agent1.sources.src1.start.from=0
#agent1.sources.src1.incremental.value = 0
# query time interval
agent1.sources.src1.run.query.delay = 5000
# declare the folder loaction where flume state is saved
agent1.sources.src1.status.file.path = /home/user/flume-source-state
agent1.sources.src1.status.file.name = src1.status
agent1.sources.src1.batch.size = 1000
agent1.sources.src1.max.rows = 1000
agent1.sources.src1.delimiter.entry = |
# set the channel to memory mode
agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 10000
agent1.channels.ch1.transactionCapacity = 10000
agent1.channels.ch1.byteCapacityBufferPercentage = 20
agent1.channels.ch1.byteCapacity = 800000
# declare sink type
agent1.sinks.sk1.type = org.apache.flume.sink.hbase.HBaseSink
agent1.sinks.sk1.table = yyyy
agent1.sinks.sk1.columnFamily = yyyy
agent1.sinks.sk1.hdfs.batchSize = 100
agent1.sinks.sk1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
agent1.sinks.sk1.serializer.regex = ^\"(.*?)\",\"(.*?)\",\"(.*?)\"$
agent1.sinks.sk1.serializer.colNames = PK, name, machine, time
# bind source, channel, sink
agent1.sources.src1.channels = ch1
agent1.sinks.sk1.channel = ch1
But, I use a similar config file to transfer data from MySql to HBase. Luckily, it worked.
mysql-hbase.conf
# source, channel, sink
agent1.sources = src1
agent1.channels = ch1
agent1.sinks = sk1
# declare source type
agent1.sources.src1.type = org.keedio.flume.source.SQLSource
agent1.sources.src1.hibernate.connection.url = jdbc:mysql://xxxx:3306/userdb
agent1.sources.src1.hibernate.connection.user = xxxx
agent1.sources.src1.hibernate.connection.password = xxxx
agent1.sources.src1.table = xxxx
agent1.sources.src1.hibernate.connection.autocommit = true
# declare mysql hibernate dialect
agent1.sources.src1.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
agent1.sources.src1.hibernate.connection.driver_class = com.mysql.jdbc.Driver
#agent1.sources.src1.hibernate.provider_class=org.hibernate.connection.C3P0ConnectionProvider
#agent1.sources.src1.columns.to.select = *
#agent1.sources.src1.incremental.column.name = id
#agent1.sources.src1.incremental.value = 0
# query time interval
agent1.sources.src1.run.query.delay = 5000
# declare the folder loaction where flume state is saved
agent1.sources.src1.status.file.path = /home/user/flume-source-state
agent1.sources.src1.status.file.name = src1.status
#agent1.sources.src1.interceptors=i1
#agent1.sources.src1.interceptors.i1.type=search_replace
#agent1.sources.src1.interceptors.i1.searchPattern="
#agent1.sources.src1.interceptors.i1.replaceString=,
# Set the channel to memory mode
agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 10000
agent1.channels.ch1.transactionCapacity = 10000
agent1.channels.ch1.byteCapacityBufferPercentage = 20
agent1.channels.ch1.byteCapacity = 800000
# declare sink type
agent1.sinks.sk1.type = org.apache.flume.sink.hbase.HBaseSink
agent1.sinks.sk1.table = user_test_2
agent1.sinks.sk1.columnFamily = user_hobby
agent1.sinks.sk1.hdfs.batchSize = 100
agent1.sinks.sk1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
agent1.sinks.sk1.serializer.regex = ^\"(.*?)\",\"(.*?)\",\"(.*?)\",\"(.*?)\"$
agent1.sinks.sk1.serializer.colNames = id,name,age,hobby
# bind source, channel, sink
agent1.sources.src1.channels = ch1
agent1.sinks.sk1.channel = ch1
Does anyone know is there something wrong in the config file? Thanks.
Please help I was trying to call watson assistant endpoint
https://gateway.watsonplatform.net/assistant/api/v1/workspaces/myworkspace/logs?version=2018-09-20 to get all the list of events
and filter by date range using this params
var param =
{ workspace_id: '{myworkspace}',
page_limit: 100000,
filter: 'response_timestamp%3C2018-17-12,response_timestamp%3E2019-01-01'}
apparently I got any empty response below.
{
"logs": [],
"pagination": {}
}
Couple of things to check.
1. You have 2018-17-12 which is a metric date. This translates to "12th day of the 17th month of 2018".
2. Assuming the date should be a valid one, your search says "Documents that are Before 17th Dec 2018 and after 1st Jan 2019". Which would return no documents.
3. Logs are only generated when you call the message() method through the API. So check your logging page in the tooling to see if you even have logs.
4. If you have a lite account logs are only stored for 7 days and then deleted. To keep logs longer you need to upgrade to a standard account.
Although not directly related to your issue, be aware that page_limit has an upper hard coded limit (IIRC 200-300?). So you may ask for 100,000 records, but it won't give it to you.
This is sample python code (unsupported) that is using pagination to read the logs:
from watson_developer_cloud import AssistantV1
username = '...'
password = '...'
workspace_id = '....'
url = '...'
version = '2018-09-20'
c = AssistantV1(url=url, version=version, username=username, password=password)
totalpages = 999
pagelimit = 200
logs = []
page_count = 1
cursor = None
count = 0
x = { 'pagination': 'DUMMY' }
while x['pagination']:
if page_count > totalpages:
break
print('Reading page {}. '.format(page_count), end='')
x = c.list_logs(workspace_id=workspace_id,cursor=cursor,page_limit=pagelimit)
if x is None: break
print('Status: {}'.format(x.get_status_code()))
x = x.get_result()
logs.append(x['logs'])
count = count + len(x['logs'])
page_count = page_count + 1
if 'pagination' in x and 'next_url' in x['pagination']:
p = x['pagination']['next_url']
u = urlparse(p)
query = parse_qs(u.query)
cursor = query['cursor'][0]
Your logs object should contain the logs.
I believe the limit is 500, and then we return a pagination URL so you can get the next 500. I dont think this is the issue but once you start getting logs back its good to know
I have Apache2 with APC.
When I change something, I have to restart Apache to see the effect. I know 100% for sure it's because of APC.
What is wrong in my settings? (thanks for the help!)
extension=apc.so
apc.enabled = On
apc.optimization = 0
apc.shm_segments = 1
apc.shm_size = 2.6G
apc.ttl = 7200
apc.user_ttl = 720
apc.num_files_hint = 102400
apc.mmap_file_mask = /tmp/apc.XXXXXX
apc.enable_cli = 1
apc.cache_by_default = 1
apc.max_file_size = 220M
apc.stat = 0
You have apc.stat set to 0. This means APC will not check whether the file is modified when it's requested, it will always serve it from cache after the first compilation.
To fix you problem either remove apc.stat = 0 or change it back to default apc.stat = 1
I'm trying to synchronize OpenLDAP and Active directory together. To do so I'm using a program called LSC-Project which is specified to do this sort of thing.
I have configured the program the best I can however I can't find a way to shake off the following error:
javax.naming.NameNotFoundException: [LDAP: error code 32 - 0000208D: NameErr: DSID-
031001CD,
problem 2001 (NO_OBJECT), data 0, best match of:
'DC=domname,DC=com'
]; remaining name
'uid=user1,ou=Users'
May 09 15:19:25 - ERROR - Error while synchronizing ID uid=user1,ou=Users:
java.lang.Exception:
Technical problem while applying modifications to directory
dn: uid=user1,ou=Users,dc=domname,dc=com
changetype: add
userPassword: 3+kU2th/WMo/v553A24a3SBw2kU=
objectClass: uid
This is the configuration file that the program runs on:
###############################
Destination LDAP directory #
##############################
dst.java.naming.provider.url = ldap://192.168.1.3:389/dc=Windows,dc=com
dst.java.naming.security.authentication = simple
dst.java.naming.security.principal = cn=Administrator,cn=Users,dc=Windows,dc=com
dst.java.naming.security.credentials = 11111
dst.java.naming.referral = ignore
dst.java.naming.ldap.derefAliases = never
dst.java.naming.factory.initial = com.sun.jndi.ldap.LdapCtxFactory
dst.java.naming.ldap.version = 3
dst.java.naming.ldap.pageSize = 1000
#########################
Source LDAP directory
#########################
src.java.naming.provider.url = ldap://192.168.1.2:389/dc=Linux,dc=com
src.java.naming.security.authentication = simple
src.java.naming.security.principal = uid=root,ou=users,dc=Linux,dc=com
src.java.naming.security.credentials = 11111
src.java.naming.referral = ignore
src.java.naming.ldap.derefAliases = never
src.java.naming.factory.initial = com.sun.jndi.ldap.LdapCtxFactory
src.java.naming.ldap.version = 3
#######################
Tasks configuration
#######################
lsc.tasks = Administrator
lsc.tasks.Administrator.srcService = org.lsc.jndi.SimpleJndiSrcService
lsc.tasks.Administrator.srcService.baseDn = ou=users
lsc.tasks.Administrator.srcService.filterAll = (&(objectClass=person))
lsc.tasks.Administrator.srcService.pivotAttrs = uid
lsc.tasks.Administrator.srcService.filterId = (&(objectClass=person)(uid={uid}))
lsc.tasks.Administrator.srcService.attrs = description uid userPassword
lsc.tasks.Administrator.dstService = org.lsc.jndi.SimpleJndiDstService
lsc.tasks.Administrator.dstService.baseDn = cn=Users
lsc.tasks.Administrator.dstService.filterAll = (&(cn=*)(objectClass=organizationalPerson))
lsc.tasks.Administrator.dstService.pivotAttrs = cn, top, person, user, organizationalPerson
lsc.tasks.Administrator.dstService.filterId = (&(objectClass=user) (sAMAccountName={cn}))
lsc.tasks.Administrator.dstService.attrs = description cn userPassword objectClass
lsc.tasks.Administrator.bean = org.lsc.beans.SimpleBean
lsc.tasks.Administrator.dn = "uid=" + srcBean.getAttributeValueById("uid") + ",ou=Users"
dn.real_root = dc=Domname,dc=com
#############################
Syncoptions configuration
#############################
lsc.syncoptions.Administrator = org.lsc.beans.syncoptions.PropertiesBasedSyncOptions
lsc.syncoptions.Administrator.default.action = M
lsc.syncoptions.Administrator.objectClass.action = M
lsc.syncoptions.Administrator.objectClass.force_value = srcBean.getAttributeValueById("cn").toUpperCase()
lsc.syncoptions.Administrator.userPassword.default_value = SecurityUtils.hash(SecurityUtils.HASH_SHA1, "defaultPassword")
lsc.syncoptions.Administrator.default.delimiter=;
lsc.syncoptions.Administrator.objectClass.force_value = "top";"user";"person";"organizationalPerson"
lsc.syncoptions.Administrator.userPrincipalName.force_value = srcBean.getAttributeValueById("uid") + "#Domname.com"
lsc.syncoptions.Administrator.userAccountControl.create_value = AD.userAccountControlSet ( "0", [AD.UAC_SET_NORMAL_ACCOUNT])
I'm suspecting that it has something to do with the baseDn of the Task configuration in the part of the source configuration.
The OSs is ubuntu 10.04 and Windows2K3
Someone suggested to me to make a manual sync between them but I have not found any guides to do so. And this program is pretty much the only thing that says that is does this kind of job without costs.
The baseDn should be the distinguished name of the base object of the search, for example, ou=users,dc=domname,dc=com.
see also
LDAP: Mastering Search Filters
LDAP: Search best practices
LDAP: Programming practices
The main reason for NameNotFoundException is that the object which you're searching doesn't exist or the container in which you are searching is not correct.
In case of Spring-ldap, we used to get this error when we specify the baseDn in the context file(LdapContextSource bean) and also in createUser code to build userDn.we need not specify the dc again in the buildUserDn()
protected Name buildUserDn(String userName) {
DistinguishedName dn = new DistinguishedName();
//only cn is required as the base dn is already specified in context file
dn.add("cn", userName);
return dn;
}
In Active Directory: Users catalog is container class, not OrganizationalUnit, so you should use: cn=users,dc=domname,dc=com