In application.conf, parameters are set:
url = "jdbc:mysql://.../table_name"
user = ...
password = ...
driver = "com.mysql.jdbc.Driver"
connectionPool = HikariCP
queueSize = 25000
I am still receiving an error whenever the queue reaches 1000 items, meaning that the queueSize property is still the default value.
Task scala.slick.backend.DatabaseComponent$DatabaseDef$...
rejected from java.util.concurrent.ThreadPoolExecutor...
[Running, pool size = 20,
active threads = 20,
queued tasks = 1000,
completed tasks = 7507]
Not sure why it's not picking up your value, but you might want to try a different way of configuring.. you didn't say what version of Slick you are using. But please refer to the Slick 3.0.0 documentation. Try it using TypeSafe Config:
In your application.conf:
database {
dataSourceClass = "org.postgresql.ds.PGSimpleDataSource" // replace with mysql driver
properties = {
databaseName = "mydb"
user = "myuser"
password = "secret"
}
queueSize = 25000 // I've never changed that property, so not tested.
}
Then in scala:
val db = Database.forConfig("database")
Hope this works for you.
Related
After running this code, the data will be inserted into the database. when I check the database, I find a row added with values: int_field=0 and other="" (empty string)
My question is how to throw exception in case of Null/empty insert.
from datetime import datetime
db = pw.MySQLDatabase('tests', user = 'root', password = 'root', host = 'localhost', port = 3306)
class Model(pw.Model):
name = pw.CharField()
created_at = pw.DateTimeField(default = datetime.now())
other = pw.CharField(null = False)
int_field = pw.IntegerField(null = False)
class Meta:
database = db
db_table = 'nmodel'
Model.create_table()
Model.create(name = "tests")
Its a MySQL strict_mode issue, you should enable it in peewee Database Instance Like below.
Add this (sql_mode='STRICT_ALL_TABLES') to the constrcutor call:
db=pw.MySQLDatabase('database_name', user='user_name', password='user_psswd',
host='host_name', port=port_num,sql_mode='STRICT_ALL_TABLES')
This will throw a mysql exception when null values are passed.
I think you need to set your sql_mode to be more strict. See:
https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sql-mode-setting
Specifically, I think you're running into:
https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sqlmode_no_auto_value_on_zero
I am using Akka.Net (v 1.3.2) and am trying to query the event journal for all events with a specific tag. I only want the events that exist at the time I query the journal. Inside an actor, I have the following code:
var readJournal = PersistenceQuery.Get(Context.System).ReadJournalFor<SqlReadJournal>(SqlReadJournal.Identifier);
var stream = readJournal.CurrentEventsByTag("The Tag Name", Offset.NoOffset());
var materializer = ActorMaterializer.Create(Context.System);
stream.RunForeach(envelope =>
{
// Do some stuff with the EventEnvelope
}, materializer).Wait();
This will successfully query the event journal. However, the problem is it will only return the first 100 events. I need all of them that match the query!
Question: How do I remove the limit/filter that exists when querying the event journal by tag name?
If you need it, here is my akka.persistence configuration:
var config = Akka.Configuration.ConfigurationFactory.ParseString(#"
akka.persistence {
journal {
plugin = ""akka.persistence.journal.sql-server""
sql-server {
class = ""Akka.Persistence.SqlServer.Journal.SqlServerJournal, Akka.Persistence.SqlServer""
connection-string = """ + connectionString + #"""
schema-name = dbo
table-name = __akka_EventJournal
metadata-table-name = __akka_Metadata
auto-initialize = on
}
}
snapshot-store {
plugin = ""akka.persistence.snapshot-store.sql-server""
sql-server {
class = ""Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer""
connection-string = """ + connectionString + #"""
schema-name = dbo
table-name = __akka_SnapshotStore
auto-initialize = on
}
}
}"
);
There are two things to check out:
You can set the maximum number of messages returned in one query by setting up akka.persistence.query.journal.sql.max-buffer-size value (see: reference.conf).
Use readJournal.EventsByTag instead of readJournal.CurrentEventsByTag to get a continuous stream of events. Just keep in mind, that it won't complete by itself, but will live on waiting for new events to arrive. You can stop it explicitly i.e. by using KillSwitch.
I use classic connecion, which created in bootstap by read parameters from application.ini file. I would like use more external DBs, but I can´t read them from application.ini. I prefer read parameters from main DB (which external DB it is, depend on website). So how effectivelly make connection in model? Now I set that connection every time when I need use it. And when I need use main DB, is neccessary make connection again. It´s very uneffective solution.
function joinClientDB($id)
{
$web = $this->getById($id);
$dbSettings = array();
$dbSettings['host'] = $web['web_dbHost'];
$dbSettings['username'] = $web['web_dbUsername'];
$dbSettings['password'] = $web['web_dbPassword'];
$dbSettings['dbname'] = $web['web_dbName'];
$this->_db = Zend_Db::factory('pdo_mysql', $dbSettings);
$this->_db->query('SET CHARACTER SET ' . $web['web_dbCharset']);
}
function joinDefaultDb()
{
$this->_db = Zend_Registry::get('db');
}
Has anybody easy solution for me?
I would suggest to use the Zend MULTI DB Plugin. Add to your applications.ini all database connections:
; Database ONE
resources.multidb.test.adapter =
resources.multidb.test.dbname =
resources.multidb.test.username =
resources.multidb.test.password = ""
resources.multidb.test.host = "your.local.host"
resources.multidb.test.default = true
; Database TWO
resources.multidb.live.adapter =
resources.multidb.live.dbname =
resources.multidb.live.username =
resources.multidb.live.password = ""
resources.multidb.live.host = "your.live.host.com"
And now you could put them in your registry again (bootstrap.php):
protected function _initDbAdapters()
{
$this->bootstrap('multidb');
$resource = $this->getPluginResource('multidb');
$resource>init();
$testDB = $resource->getDb('test');
$liveDB = $resource->getDb('live');
Zend_Registry::set('DB_TEST', $testDB);
Zend_Registry::set('DB_LIVE', $liveDB);
}
I am new to SalesForce (3 months).
Thus far I have been able to create an application in C# that I can use to preform Inserts and Updates to the SalesForce database. These transactions are one at a time.
No I have the need to preform large scale transactions. For example updating thousands of records at a time. Doing them one by one would quickly put us over our allotted API calls per 24 hour period.
I want to utilize the available bulk transactions process to cut down on the number of API calls. Thus far I have not had much luck coding this nor have I found any such documentation.
If anyone could either provide some generic examples or steer me to reliable documentation on the subject I would greatly appreciate it.
FYI, the data I need to use to do the updates and inserts comes from an IBM Unidata database sitting on an AIX machine. So direct web services communication is not realy possible. Getting the data from Unidata has been my headache. I have that worked out. Now the bulk api to SalesForce is my new headache.
Thanks in advance.
Jeff
You don't mention which API you're currently using, but using the soap partner or enterprise APIs you can write records to salesforce 200 at a time. (the create/update/upsert calls all take an array of SObjects).
Using the bulk API you can send data in chunks of thousands of rows at a time.
You can find the documentation for both sets of APIs here
The answers already given are a good start; however, are you sure you need to actually write a custom app that uses the bulk API? The salesforce data loader is a pretty robust tool, includes a command line interface, and can use either the "normal" or bulk data API's. Unless you are needing to do fancy logic as part of your insert/updates, or some sort of more real-time / on-demand loading, the data loader is going to be a better option than a custom app.
(this is the SOAP code though, not the Salesforce "Bulk API" ; careful not to confuse the two)
Mighy be below code provide clear insight on how to do bulk insertion.
/// Demonstrates how to create one or more Account records via the API
public void CreateAccountSample()
{
Account account1 = new Account();
Account account2 = new Account();
// Set some fields on the account1 object. Name field is not set
// so this record should fail as it is a required field.
account1.BillingCity = "Wichita";
account1.BillingCountry = "US";
account1.BillingState = "KA";
account1.BillingStreet = "4322 Haystack Boulevard";
account1.BillingPostalCode = "87901";
// Set some fields on the account2 object
account2.Name = "Golden Straw";
account2.BillingCity = "Oakland";
account2.BillingCountry = "US";
account2.BillingState = "CA";
account2.BillingStreet = "666 Raiders Boulevard";
account2.BillingPostalCode = "97502";
// Create an array of SObjects to hold the accounts
sObject[] accounts = new sObject[2];
// Add the accounts to the SObject array
accounts[0] = account1;
accounts[1] = account2;
// Invoke the create() call
try
{
SaveResult[] saveResults = binding.create(accounts);
// Handle the results
for (int i = 0; i < saveResults.Length; i++)
{
// Determine whether create() succeeded or had errors
if (saveResults[i].success)
{
// No errors, so retrieve the Id created for this record
Console.WriteLine("An Account was created with Id: {0}",
saveResults[i].id);
}
else
{
Console.WriteLine("Item {0} had an error updating", i);
// Handle the errors
foreach (Error error in saveResults[i].errors)
{
Console.WriteLine("Error code is: {0}",
error.statusCode.ToString());
Console.WriteLine("Error message: {0}", error.message);
}
}
}
}
catch (SoapException e)
{
Console.WriteLine(e.Code);
Console.WriteLine(e.Message);
}
}
Please find the small code which may help you to insert the data into salesforce objects using c# and WSDL APIs. I stuck to much to write code in c#. I assigned using direct index after spiting you can use your ways.
I split the column using | (pipe sign). You may change this and also <br>, \n, etc. (row and column breaking)
Means you can enter N rows which are in your HTML/text file. I wrote the program to add order by my designers who put the order on other website and fetch the data from e-commerce website and who has no interface for the salesforce to add/view the order records. I created one object for the same. and add following columns in the object.
Your suggestions are welcome.
private SforceService binding; // declare the salesforce servive using your access credential
try
{
string stroppid = "111111111111111111";
System.Net.HttpWebRequest fr;
Uri targetUri = new Uri("http://abc.xyz.com/test.html");
fr = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(targetUri);
if ((fr.GetResponse().ContentLength > 0))
{
System.IO.StreamReader str = new System.IO.StreamReader(fr.GetResponse().GetResponseStream());
string allrow = str.ReadToEnd();
string stringSeparators = "<br>";
string[] row1 = Regex.Split(allrow, stringSeparators);
CDI_Order_Data__c[] cord = new CDI_Order_Data__c[row1.Length - 1];
for (int i = 1; i < row1.Length-1; i++)
{
string colstr = row1[i].ToString();
string[] allcols = Regex.Split(colstr, "\\|");
cord[i] = new CDI_Order_Data__c(); // Very important to create object
cord[i].Opportunity_Job_Order__c = stroppid;
cord[i].jobid__c = stroppid;
cord[i].order__c = allcols[0].ToString();
cord[i].firstname__c = allcols[1].ToString();
cord[i].name__c = allcols[2].ToString();
DateTime dtDate = Convert.ToDateTime(allcols[3]);
cord[i].Date__c = new DateTime(Convert.ToInt32(dtDate.Year), Convert.ToInt32(dtDate.Month), Convert.ToInt32(dtDate.Day), 0, 0, 0); //sforcedate(allcols[3]); //XMLstringToDate(allcols[3]);
cord[i].clientpo__c = allcols[4].ToString();
cord[i].billaddr1__c = allcols[5].ToString();
cord[i].billaddr2__c = allcols[6].ToString();
cord[i].billcity__c = allcols[7].ToString();
cord[i].billstate__c = allcols[8].ToString();
cord[i].billzip__c = allcols[9].ToString();
cord[i].phone__c = allcols[10].ToString();
cord[i].fax__c = allcols[11].ToString();
cord[i].email__c = allcols[12].ToString();
cord[i].contact__c = allcols[13].ToString();
cord[i].lastname__c = allcols[15].ToString();
cord[i].Rep__c = allcols[16].ToString();
cord[i].sidemark__c = allcols[17].ToString();
cord[i].account__c = allcols[18].ToString();
cord[i].item__c = allcols[19].ToString();
cord[i].kmatid__c = allcols[20].ToString();
cord[i].qty__c = Convert.ToDouble(allcols[21]);
cord[i].Description__c = allcols[22].ToString();
cord[i].price__c = Convert.ToDouble(allcols[23]);
cord[i].installation__c = allcols[24].ToString();
cord[i].freight__c = allcols[25].ToString();
cord[i].discount__c = Convert.ToDouble(allcols[26]);
cord[i].salestax__c = Convert.ToDouble(allcols[27]);
cord[i].taxcode__c = allcols[28].ToString();
}
try {
SaveResult[] saveResults = binding.create(cord);
}
catch (Exception ce)
{
Response.Write("Buld order update errror" +ce.Message.ToString());
Response.End();
}
if (str != null) str.Close();
}
I been trying to get sqlcachedependecy working, but it doesn't appear to work
I got the proper settings in my web.config and also global.asa, however when I run this query and the changes are made to the database from either with in or outside the web site the cached objects are not updated please someone help? I know its not because this query is querying a view, because I tested this using straight SqlDependecy and the notification works fine.
public IQueryable<VictoryList> GetVictoryList()
{
string cacheKey = HttpContext.Current.User.Identity.Name + "victoryCacheKey";
IQueryable<VictoryList> cachednews = (IQueryable<VictoryList>)HttpContext.Current.Cache.Get(cacheKey);
if (cachednews == null)
{
var results = from v in _datacontext.ViewVictoryLists
orderby _datacontext.GetNewId()
select new VictoryList
{
MemberID = v.MemberID,
Username = v.Aspnetusername,
Location = v.Location,
DaimokuGoal = v.DaimokuGoal,
PreviewImageID = v.PreviewImageID,
TotalDaimoku = v.TotalDaimoku,
TotalDeterminations = v.TotalDeterminations,
DeterminationID = v.DeterminationID,
DeterminationName = v.DeterminationName
};
results = results.ToList().AsQueryable();
SqlCacheDependencyAdmin.EnableNotifications(_datacontext.Connection.ConnectionString);
SqlCacheDependency dependency =
new SqlCacheDependency(_datacontext.GetCommand(results) as SqlCommand);
HttpContext.Current.Cache.Insert(cacheKey, results, dependency);
return results;
}
return cachednews;
}
According to the stated Limitations for creating a query for notification, listed at msdn...
The statement must not reference a view.