I'm currently working with Integration Services (SSIS) to load a big amount of XML files into a SQL server database. Each XML file content needs to be dispatched in several tables. I have at least 10000 xml files to load using the process. Everything works fine till 6000 files are loaded. After 6000 treatment, I always got an OutOfMemoryException from my first dataflow task, the first in the process.
In this script component, I just check if a value from a XML file is already present in a specific database table. If it is present, I return the matched ID, otherwise, I add a new record. To achieve it, I use a Lookup component. I use it with No cache option, for memory matter. Then in case of matching, I process the return ID in a script component. Like I said, everything works fine until more or less 6000 files are processed. After I got :
Description : System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.HandleUserException(Exception e)
at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.PreExecute()
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPreExecute(IDTSManagedComponentWrapper100 wrapper)
Do you have some suggestions or some ressources which deal with performance and memory issue in SSIS ? Do you encountered similar problem ? DO you have an idea from where this memory problem could come from?
Thanks :)
EDIT:
Here is the code to check an XML against an XSD file. Can you see any memory leak?
public void Main()
{
try
{
XmlReaderSettings settings = new XmlReaderSettings();
settings.Schemas.Add("", Dts.Variables["XSDFilePath"].Value.ToString());
settings.ValidationType = ValidationType.Schema;
using (XmlReader reader = XmlReader.Create(Dts.Variables["XMLFilePath"].Value.ToString(), settings))
{
XmlDocument document = new XmlDocument();
document.Load(reader);
ValidationEventHandler eventHandler = new ValidationEventHandler(XMLValidationHandler);
document.Validate(eventHandler);
}
if (ValidationResult)
{
Dts.TaskResult = (int)ScriptResults.Success;
}
else
{
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
catch (Exception ex)
{
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
private void XMLValidationHandler(object sender, ValidationEventArgs e)
{
switch (e.Severity)
{
case XmlSeverityType.Error:
Console.WriteLine("Warning {0}", e.Message);
ValidationResult = false;
break;
case XmlSeverityType.Warning:
Console.WriteLine("Warning {0}", e.Message);
break;
}
}
Don't use the XMLDocument Object, It will load the entire document into memory and is likely the cause of your memory problem. Try instead to use the XmlValidatingReader
http://msdn.microsoft.com/en-us/library/system.xml.xmlvalidatingreader.aspx
It should provide a more efficient use of memory.
Refer to the 'answers' in this.,
http://social.msdn.microsoft.com/Forums/en-US/sqlintegrationservices/thread/4d90afc8-0a1c-418f-9e3a-0b766f581b2c/
Related
I am trying to build an inventory management system and have recently asked a question about how to be able to store images in a package within my src file. I was told that you should not store images where class files are stored but have not been told what the best practices are for file systems. I have created a new page that allows the user to input all the data about a new part that they are adding to the system and upload an image associated with the part. When they save, everything worked fine until you try to reload the parts database. If you 'refresh' eclipse and then update the database, everything was fine because you could see the image pop into the package when refreshed. (All database info was updated properly as well.
I was told not to store these types of 'new' images with the program files but to create a separate file system to store these types of images. Is there a best practice for these types of file systems? My confusion is when the program gets saved where ever it is going to be saved, I can't have it point to an absolute path because it might not be saved on a C drive or K drive and I wouldn't want an images folder just sitting on the C drive that has all of the parts images for anyone to mess with. Please give me some good resources on how to build these file systems. I would like the images folder 'packaged' with the program when I compile it and package all the files together, I have not been able to find any good information on this, thanks!
To answer this question, probably not in the best way, but works pretty well.
I ended up making another menuItem and menu that you can see at the top 'Image Management', where it lets the user set the location that they would like to save all the images as well as a location to back up the images. it creates the directory if it is not there or it will save over the images if the directory is already there. This menu will only appear if the user has admin privileges. I would think that this could be set up with an install wizard, but I have no idea how to make one, where it only runs on installation. I am also going to add an autosave feature to save to both locations if a backup location has been set. This is the best way I can think of managing all the parts images, if anyone has some good input, please let me know. I considered a server, but think that is too much for this application and retrieving images every time the tableView populates would take a lot of time. If interested the code I used is:
public class ImageDirectoryController implements Initializable{
#FXML private AnchorPane imageDirectory;
#FXML private Label imageDirLbl, backupLbl;
#FXML private Button setLocationButton, backupButton;
#FXML private TextField imageDirPathTxtField;
Stage window;
String image_directory;
#Override
public void initialize(URL location, ResourceBundle resources) {
// TODO Auto-generated method stub
}
public void setImageDirectory(String image_address, String backup_address) {
imageDirLbl.setText(image_address);
backupLbl.setText(backup_address);
}
#FXML
public void setLocationButtonClicked () {
String imagesPath = imageDirPathTxtField.getText() + "tolmarImages\\";
File files = new File(imagesPath + "asepticImages");
File generalFiles = new File(imagesPath + "generalImages");
File facilitiesFiles = new File(imagesPath + "facilitiesImages");
boolean answer = ConfirmBox.display("Set Image", "Are you sure you want to set this location?");
if(answer) {
if (!files.exists()) {
if (files.mkdirs() && generalFiles.mkdirs() && facilitiesFiles.mkdirs()) {
JOptionPane.showMessageDialog (null, "New Image directories have been created!", "Image directory created", JOptionPane.INFORMATION_MESSAGE);
} else {
JOptionPane.showMessageDialog (null, "Failed to create multiple directories!", "Image directory not created", JOptionPane.INFORMATION_MESSAGE);
}
}
DBConnection dBC = new DBConnection();
Connection con = dBC.getDBConnection();
String updateStmt = "UPDATE image_address SET image_address = ? WHERE rowid = ?";
try {
PreparedStatement myStmt = con.prepareStatement(updateStmt);
myStmt.setString(1, imageDirPathTxtField.getText());
myStmt.setInt(2, 1);
myStmt.executeUpdate();
myStmt.close();
imageDirPathTxtField.clear();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
#FXML
public void backupButtonClicked () {
String backupStatus = null;
if (backupLbl.getText().equals("")&& !imageDirPathTxtField.getText().equals("")) {
backupStatus = imageDirPathTxtField.getText();
} else if (!imageDirPathTxtField.getText().equals("")) {
backupStatus = imageDirPathTxtField.getText();
} else if (!backupLbl.getText().equals("")){
backupStatus = backupLbl.getText();
} else {
JOptionPane.showMessageDialog(null, "You must create a directory.", "No directory created", JOptionPane.INFORMATION_MESSAGE);
return;
}
boolean answer = ConfirmBox.display("Set Image", "Are you sure you want to backup the images?");
if(answer) {
DBConnection dBC = new DBConnection();
Connection con = dBC.getDBConnection();
String updateStmt = "UPDATE image_address SET image_address = ? WHERE rowid = 2";
try {
PreparedStatement myStmt = con.prepareStatement(updateStmt);
myStmt.setString(1, backupStatus);
myStmt.executeUpdate();
myStmt.close();
String source = imageDirLbl.getText() + "tolmarImages";
File srcDir = new File(source);
String destination = backupStatus + "tolmarImages";
File destDir = new File(destination);
try {
FileUtils.copyDirectory(srcDir, destDir);
JOptionPane.showMessageDialog(null, "Images copied successfully.", "Images copied", JOptionPane.INFORMATION_MESSAGE);
} catch (IOException e) {
e.printStackTrace();
}
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
Is there a way to change the tables that logback writes its data to using the dbAppender, It has three default tables that must be created before using dbAppender, but I want to customise it to write to one table of my choosing. Something similar to Log4J where I can specify the SQL that gets executed when inserting the log to the database.
Tomasz, maybe I'm missing something but I don't see how just using custom DBNameResolver could be the answer to what Magezy asked. DBNameResolver is used by DBAppender via SQLBuilder to construct 3 SQL insert querys - via DBNameResolve one can only affect names of tables and columns where data will be inserted, but can not limit inserting to just one table, not to mention that by just implementing DBNameResolver there are no means to control what actually gets inserted.
To match log4j's JDBCAppender IMO one has to extend logback's DBAppender, or DBAppenderBase, or maybe even implement completely new custom Appender.
The easiest way for me was to make an appender from scratch. I'm appending to a single table, using Spring JDBC. It works something like this:
public class MyAppender extends AppenderBase<ILoggingEvent>
{
private String _jndiLocation;
private JDBCTemplate _jt;
public void setJndiLocation(String jndiLocation)
{
_jndiLocation = jndiLocation;
}
#Override
public void start()
{
super.start();
if (_jndiLocation == null)
{
throw new IllegalStateException("Must have the JNDI location");
}
DataSource ds;
Context ctx;
try
{
ctx = new InitialContext();
Object obj = ctx.lookup(_jndiLocation);
ds= (DataSource) obj;
if (ds == null)
{
throw new IllegalStateException("Failed to obtain data source");
}
_jt = new JDBCTemplate(ds);
}
catch (Exception ex)
{
throw new IllegalStateException("Unable to obtain data source", ex);
}
}
#Override
protected void append(ILoggingEvent e)
{
// log to database here using my JDBCTemplate instance
}
}
I ran into trouble with SLF4J - the substitute logger error described here:
http://www.slf4j.org/codes.html#substituteLogger
This thread on multi-step configuration enabled me to work around that issue.
You need to implement ch.qos.logback.classic.db.names.DBNameResolver and use it in the configuration:
<appender name="DB" class="ch.qos.logback.classic.db.DBAppender">
<dbNameResolver class="com.example.MyDBNameResolver"/>
<!-- ... -->
</appender>
<appender name="CUSTOM_DB_APPENDER" class="com.....MyDbAppender">
<filter class="com......MyFilter"/>
<param name="jndiLocation" value="java:/comp/env/jdbc/....MyPath"/>
</appender>
And your java MyDbAppender should have a string jndiLocation with setter.
Now do a jndi lookup (see the solution answered in Oct 17 '11 at 16:03)
I am REALLY new to Eclipse plugin developement, so maybe my question sounds dumb, but I've been googling for hours...
Whats the way to load an SQLite db into my Eclipse plugin?
First I created a standalone Java application, it loaded, and processed the database file just file, but when I added the code (well copied the files to the new project) it says:
java.sql.SQLException: no such table: measurement
The db file is in the projects root dir.
I thought this issue is caused becouse the applicaiton cannot find the database file, so I changed the filename from test.db to asdftest.db, and I got the same error, so the problem was this indeed.
Thanks!
Here is the code:
public List<Coord> generateModel() {
try {
Class.forName("org.sqlite.JDBC");
Connection conn = DriverManager
.getConnection("jdbc:sqlite:test.db");
Statement stat = conn.createStatement();
ResultSet rs = stat
.executeQuery("select * from measurement where has_gps = 'TRUE';");
LinkedList<Coord> nodes = new LinkedList<Coord>();
while (rs.next()) {
nodes.add(new Coord(rs.getString("_id"), rs.getString("lat"),
rs.getString("lon")));
}
rs.close();
conn.close();
return nodes;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
I put my database field in "assets" folder. And use the code from this blog to copy the database to "/data/data/my_packname/databases/", (This copy code i run it in the onCreate() method when i run this app) then use select * from ... to get data. But it gives me the exception: no such table.
Someone told me that if i am attempting to copy the file in SQLiteOpenHelper's onCreate(), it's too late. So the copy file code can not copy the complete file.
So i need to use adb or ddms to pull the database first?
So, Anyone can teach me how to use my own databse?
Can you tell me the setup?
I've used the instructions in that blog post and found them, while on the right track, to severely complicate the issue by unnecessarily extending SQLiteOpenHelper. I've had much better luck doing the following:
Create a utility class that creates the static db by copying it into the correct directory from assets, but doesn't get itself so hung up on following the SQLiteOpenHelper format.
Using the same utility class to open the db by using SQLiteDatabase.openDatabase()
Edit: Here is a version of this utility class I've created; it's not quite complete, but you'll get the drift.
public class DbUtils {
private static final String DB_PATH = "/data/data/com.mypackage.myapp/databases/";
private static final String DB_NAME = "my.db";
public static void createDatabaseIfNotExists(Context context) throws IOException {
boolean createDb = false;
File dbDir = new File(DB_PATH);
File dbFile = new File(DB_PATH + DB_NAME);
if (!dbDir.exists()) {
dbDir.mkdir();
createDb = true;
}
else if (!dbFile.exists()) {
createDb = true;
}
else {
// Check that we have the latest version of the db
boolean doUpgrade = false;
// Insert your own logic here on whether to upgrade the db; I personally
// just store the db version # in a text file, but you can do whatever
// you want. I've tried MD5 hashing the db before, but that takes a while.
// If we are doing an upgrade, basically we just delete the db then
// flip the switch to create a new one
if (doUpgrade) {
dbFile.delete();
createDb = true;
}
}
if (createDb) {
// Open your local db as the input stream
InputStream myInput = context.getAssets().open(DB_NAME);
// Open the empty db as the output stream
OutputStream myOutput = new FileOutputStream(dbFile);
// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024];
int length;
while ((length = myInput.read(buffer)) > 0) {
myOutput.write(buffer, 0, length);
}
// Close the streams
myOutput.flush();
myOutput.close();
myInput.close();
}
}
public static SQLiteDatabase getStaticDb() {
return SQLiteDatabase.openDatabase(DB_PATH + DB_NAME, null, SQLiteDatabase.OPEN_READONLY);
}
}
After you've copied the database, you should try closing and reopening the SQLiteDatabase object before executing any query on it. I had a similar problem with copying a db from an input stream and that's what solved it for me.
here is my Version from "Silvio Donnini" Code :),
now you can update the Database easily.
private static final String DB_PATH = "/data/data/pakagename/databases/";
private static final String DB_NAME = "databaseName";
private static SQLiteDatabase db;
public static void createDatabaseIfNotExists(Context context,int version) throws IOException {
boolean createDb = false;
File dbDir = new File(DB_PATH);
File dbFile = new File(DB_PATH + DB_NAME);
if (!dbDir.exists()) {
dbDir.mkdir();
createDb = true;
}
else if (!dbFile.exists()) {
createDb = true;
}
else {
// Check that we have the latest version of the db
db = SQLiteDatabase.openDatabase(DB_PATH + DB_NAME, null, SQLiteDatabase.OPEN_READONLY);
if (db.getVersion() != version) {
dbFile.delete();
createDb = true;
}
}
if (createDb) {
// Open your local db as the input stream
InputStream myInput = context.getResources().openRawResource(R.raw.database);
// Open the empty db as the output stream
OutputStream myOutput = new FileOutputStream(dbFile);
// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024];
int length;
while ((length = myInput.read(buffer)) > 0) {
myOutput.write(buffer, 0, length);
}
// Close the streams
myOutput.flush();
myOutput.close();
myInput.close();
SQLiteDatabase dbwrite = SQLiteDatabase.openDatabase(DB_PATH + DB_NAME, null, SQLiteDatabase.OPEN_READWRITE);
dbwrite.setVersion(version);
dbwrite.close();
if (db != null)
db = SQLiteDatabase.openDatabase(DB_PATH + DB_NAME, null, SQLiteDatabase.OPEN_READONLY);
}
}
public static SQLiteDatabase getStaticDb() {
if (db != null)
return db;
return SQLiteDatabase.openDatabase(DB_PATH + DB_NAME, null, SQLiteDatabase.OPEN_READONLY);
}
I know this is an old question, but I lost a lot of time figuring it out, with the help of all the replies.
The issue is that a device stores a database in his data/data/.../databases folder. So, when you change sth about the database (name, add android metadata table, size..) it wont make any difference because of the stored database and the method that checked for existing database.
To get the newest database, after changing it, you must run the program without checking for existing databses (e.g. instead of dbExist = checkDataBase(); make it just false).
dbExist = checkDataBase();
Change to:
dbExists = false;
After you picked up the "new" databse you can return to checking the existing ones.
Hope it helps someone,
dina
I know this is an old post, but for those who still get here after a Google or a Bing search, this is the solution to the stated problem:
in createDataBase() there is the following check;
this.getReadableDatabase();
This checks if there is already a database with the provided name and if not creates an empty database such that it can be overwritten with the one in the assets folder. On newer devices this works flawlessly but there are some devices on which this doesn't work. Mainly older devices. I do not know exactly why, but it seems like the getReadableDatabase() function not only gets the database but also opens it. If you then copy the database from the assets folder over it, it still has the pointer to an empty database and you will get table does not exist errors.
So in order to make it work on all devices you should modify it to the following lines:
SQLiteDatabase db = this.getReadableDatabase();
if (db.isOpen()){
db.close();
}
Even if the database is opened in the check, it is closed thereafter and it will not give you any more trouble.
Calm down guys,After long research finally found silly mistake for "no such table" error
Check name of database in Assets folder if it's like "DATABASE_NAME.EXTENSION" then put full name in Helper class with extension its solved my problem.
like say in Assets name of database is login.sqlite or login.db anything. put DB_NAME=login.sqlite fully with extention.
this tutorial now works perfectly.
The way of creating database from article you've posted is slightly diffrent from that how it's done in android examples (I don't want to say if it's good or bad).
I've learned how to use databases from SDKs NotePad sample
It's good example to start from, becouse it covers both database creation topic and database access through ContentProvider (it's really the only good way to get data from db, otherwise you will have problems when trying to get data simultaneusly from many places of your code).
You should note that SQLiteOpenHelper is really powerful and "it will help you" if you will use it properly. For example it stores current database version (not sqlite version but number you assingn with database schema version) and when you create new application version with new database structure you can update current schema to the new version in onUpdate.
I'm using Visual C#2008 Express Edition and an Express SQL database. Every time I build my solution, I get an error like the one above. Obviously the file name changes. A new file is also created every time I hit a debug point.
I have a stored proc that gets every row from a database table, it gets these rows every time the main form initialises and Adds them to a Generics list. Without inserting or deleting from the table, it gets a different number of rows each time I start my windows application. The error started happening at the same time as the weird data retrieval issue. Any ideas at all about what can cause this?
Thanks
Jose,
Sure, here's my c# method, it retrieves every row in my table, each row has an int and and Image ....
private List<ImageNumber> GetListOfKnownImagesAndNumbers()
{
//ImageNumber imNum = new ImageNumber();
SqlCommand sqlCommand = new SqlCommand();
sqlCommand.Connection = _conn;
try
{
MemoryStream ms = new MemoryStream();
sqlCommand.CommandText = "usp_GetKnownImagesAndValues";
_conn.Open();
using (IDataReader dr = sqlCommand.ExecuteReader())
{
while (dr.Read())
{
ImageNumber imNum = new ImageNumber();
imNum.Value = dr.IsDBNull(dr.GetOrdinal("ImageValue")) ? 0 : Convert.ToInt32(dr["ImageValue"]);
//Turn the bitmap into a byte array
byte[] barrImg = (byte[])dr["ImageCaptured"];
string strfn = Convert.ToString(DateTime.Now.ToFileTime());
FileStream fs = new FileStream(strfn,
FileMode.CreateNew, FileAccess.Write);
fs.Write(barrImg, 0, barrImg.Length);
fs.Flush();
fs.Close();
imNum.Image = (Bitmap)Image.FromFile(strfn);
_listOfNumbers.Add(imNum);
}
dr.Close();
_conn.Close();
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
_conn.Close();
}
return _listOfNumbers;
}
And here's my stored proc....
ALTER PROCEDURE dbo.usp_GetKnownImagesAndValues
AS
BEGIN
select ImageCaptured, ImageValue
from CapturedImages
END
Thanks for looking at this. The answer in the end was to put a Thread.Sleep inside the while loop and it started working perfectly. There may be something else I could do, I am obviously waiting for something to complete which is why allowing more time has helped here. If I knew what needed to complete and how to detect when it had completed then I could check for that instead of simply waiting for a short time.