read db as all text - database

Im reading a xlsx file as a db to do some work
i noticed that its reading some feilds in as int and date even though i just want it all to come in as text . is there anyway to override this feature?
Code below
(feel free to point out anything i could be doing better with my code as well)
private DataSet ExceltoDT(OpenFileDialog dialog)
{
try
{
string connst = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + dialog.FileName + ";Extended Properties=\"Excel 12.0 Xml;HDR=NO\";";
string sheet = "Sheet1$";
string strSQL = "SELECT * FROM [" + sheet + "]";
//string Table = "081710";
OleDbConnection xlsdb = new OleDbConnection(connst);
xlsdb.Open();
OleDbDataAdapter adp = new OleDbDataAdapter(strSQL, xlsdb);
DataSet ds2 = new DataSet();
adp.Fill(ds2);
adp.Dispose();
xlsdb.Close();
xlsdb.Dispose();
return ds2;
}
catch (StackOverflowException stack_ex2)
{
MessageBox.Show("(2007 Excel file) Stack Overflowed!" + "\n" + stack_ex2.Message);
return null;
}
catch (OleDbException ex_oledb2)
{
MessageBox.Show("An OleDb Error Thrown!" + "\n" + ex_oledb2.Message);
return null;
}
}

Add a ' (apostrophe) in front of every cell value. That will tell Excel "Treat this as text even when it looks like a number/date/whatever".
Not what you want? Then don't use the DB connector because it's badly broken. You'll notice that when you have a column with cells that are mixed. In that case, the DB driver will look at the first 8 rows and set the type to the majority of types it finds and return NULL for anything in that column that doesn't fit. You can fix that by hacking your registry.
Instead use the OLE API to open the Workbook and then start from there, reading row by row, converting the data as you need (this long list of posts should contain about every possible way to access Excel from C# plus all the bugs and problems you can encounter).

Related

How to load multiple sheets of an Excel File in SSIS with header information in the sheets

I have a file with 2 sheets in it. It has the header information in it. I tried foreach loop container to load the data
Error i got is:
[SSIS.Pipeline] Error: "Excel Source" failed validation and returned validation status "VS_NEEDSNEWMETADATA".
I tried removing the header row manually from the sheets and ran foreach loop container and it worked perfectly fine.
But in my requirement i will be getting the header row followed by blank row in each sheet.
How do i do in this case.
I believe we need to use script task to to eliminate header and followed null row from the file and read the rest of the records.
My problem is i am bad at c# code logic.
Your help is much appreciated.
Thank you,
swathi
The following Script Task will delete the top 2 rows from every worksheet in the file (you'll need to create the variable 'ExcelFilePath' in SSIS and pass that in to the task, along with 'System::TaskName'):
public void Main()
{
MainTask();
GC.Collect();
GC.WaitForPendingFinalizers();
}
private void MainTask()
{
xl.Application xlApp = null;
xl.Workbook excelFile = null;
string excelFilePath = Dts.Variables["User::ExcelFilePath"].Value.ToString();
string thisTask = Dts.Variables["System::TaskName"].Value.ToString();
try
{
xlApp = new xl.Application();
excelFile = xlApp.Workbooks.Open(excelFilePath);
xlApp.DisplayAlerts = false;
foreach (xl.Worksheet ws in excelFile.Worksheets)
{
ws.Rows["1:2"].EntireRow.Delete();
}
xlApp.DisplayAlerts = true;
excelFile.Save();
excelFile.Close();
xlApp.Quit();
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
Dts.Events.FireError(0, thisTask, ex.Message, String.Empty, 0);
if (excelFile != null) excelFile.Close(SaveChanges:false);
if (xlApp != null) xlApp.Quit();
}
}
You will need to add references to 'COM' > 'Microsoft Excel [version number] Object Library' (whichever version you have) and '.NET' > 'Microsoft.CSharp'. You'll then need to declare using xl = Microsoft.Office.Interop.Excel; in your 'Namespaces' region.

Npgsql 3 Geometry x Invalid endian flag value encountered using writer.Write<NpgsqlLine>

I have using previous version of npgsql (2.0.7), it works fine.
Now I have upgrade to npgsql 3.0.5
and with new rewrite of copy method of 3.0.5
I have to change code for geometry
I try to using
while
{
var line = new NpgsqlLine(122149.006850, 483672.683450, 122156.366150);
writer.Write<NpgsqlLine>(line, NpgsqlDbType.Line)
}
writer.Close();
at debug mode : in the loop is ok, but when writer.Close()
Error!! with this message
XX000: Invalid endian flag value encountered.
Need help on this, any suggestion are highly appreciate.
Thanks in advance.
Hope it helps you as I was having the same problem.
In my case, it has been resolved by taking away my "using" directive :
instead of:
using (var writer = conn.BeginBinaryImport("sql copy from stdin command"){ ///writer.StartRow(); writer.Write(...); writer.Write();}
Take away using :
var writer = conn.BeginBinaryImport("sql copy from stdin command");
then in while statement:
while(condition){writer.StartRow(); writer.Write(...); writer.Write();}
I think the problem comes from writer.Dispose() managed by using directive : In fact, without using directive, if you call your writer.Dispose(), the same Npgsql exception is raised.
Good luck!
Above does not work - in 4.05 you have to do
dbimporter.Complete()
but this works for all geometry types loading into a geometry column in postgres
using
Dim conn As New NpgsqlConnection
Dim ct As String = "COPY " + datatab + "( " + datacolumns + ") FROM STDIN (FORMAT BINARY)"
Dim dbimporter = conn.BeginBinaryImport(ct)
when you need to import geometry data use this, you will need sqlserverdatatypes dll
dim wkt1 as string ="POINT(7 7)" - any wkt geometry representation
Dim udtText1 As New System.Data.SqlTypes.SqlChars(wkt1)
Dim sqlGeometry11 As Microsoft.SqlServer.Types.SqlGeometry = Microsoft.SqlServer.Types.SqlGeometry.STGeomFromText(udtText1, srid)
Dim ms1 As New MemoryStream()
Dim bw1 As New BinaryWriter(ms1)
Dim WKB1() As Byte = sqlGeometry11.STAsBinary().Buffer
bw1.Write(WKB1)
dbimporter.Write(WKB1, NpgsqlTypes.NpgsqlDbType.Bytea)

my combo box is duplicating the strings

help, combobox just keep adding items, i tried using removeallitems but after that i cant put anything on the first combobox
public class Function {
public void combofillsect(JComboBox section, String year){
Connection conn = null;
PreparedStatement pst = null;
ResultSet rs = null;
String query;
try{
query = "Select Section from asd where Year=?";
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/test","root","");
pst = conn.prepareStatement(query);
pst.setString(1, year);
rs = pst.executeQuery();
while(rs.next()){
section.addItem(rs.getString("Section"));
}
}catch(Exception e){
JOptionPane.showMessageDialog(null, e);
section.addItem(e.toString());
};
}
Function funct= new Function();
{funct.combofillsect(jComboBox1,String.valueOf(jComboBox2.getSelectedItem())); }
why cant I post image?
Are you programming in C# ? If it's the case then you can use the function Clear like that : yourComboBox.Items.Clear() to delete all the current items. I don't know if it will solve your problem but your technique of getting the data from your database seems weird to me, if you used a dataset you could have done dataset.Tables(0).Rows.Count() to get the number of entries and then set the exit condition of your loop like this -> counter < dataset.Tables(0).Rows.Count(), and set a counter++ at the end of your while (maybe that's why you say your combobox won't stop filling, but I don't know what do the next() function).
I don't know the C# code but there is my VB.NET function :
Public Function getAll() As DataSet
ConnectionDB()
Dim cmd As SqlClient.SqlCommand
cmd = New SqlClient.SqlCommand("SELECT * FROM table", Connect)//Connect is a System.Data.SqlClient.SqlConnection, or my connection string
Dim adapter As New Data.SqlClient.SqlDataAdapter
Dim dataset As New DataSet
adapter.SelectCommand = cmd
adapter.Fill(dataset)
adapter.Dispose()
cmd.Dispose()
Connect.Close()
Return dataset
End Function
I don't know if I helped you but I didn't really understood what your problem was and you didn't even mentioned the language you use ^^ Good luck
Edit : and if you can't post images, that's because you don't have yet 10 points of reputation, you can get informations about reputation here : https://stackoverflow.com/help/whats-reputation, but you still can post the link of a picture it will allow users to click on it

Programatically add document to Hummingbird/OpenText eDocs database

I am working with the the (formerly Hummingbird Enterprise) OpenText eDocs document management system.
http://www.opentext.com/2/global/products/products-opentext-edocs-products/products-opentext-edocs-document-management.htm
We are still using Hummingbird 5.1.0.5.
I have been reviewing the API docs for this software, but some areas are slightly vague.
So far, I can create my Profile form, populate some values.
DOCSObjects.Application docApp = null;
DOCSObjects.IProfile profile = null;
Type fType = Type.GetTypeFromProgID("DOCSObjects.Application");
docApp = (DOCSObjects.Application)Activator.CreateInstance(fType);
try { profile = docApp.CurrentLibrary.CreateProfile("DEF_PROF"); }
catch (Exception ex) { System.Diagnostics.Debug.WriteLine(ex.Message); }
if (profile != null)
{
try
{
profile.Columns["DOCNAME"].Value = "New PDF Document";
profile.Columns["APP_ID"].Value = "ACROBAT";
profile.ShowProfile(1);
// not sure how to set a document here
profile.SetDocument(docApp.CurrentLibrary.Name, document);
profile.Save(); // requires a short flag, but what?
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.Message);
}
}
else
{
MessageBox.Show("Profile is null");
}
Where I am having trouble is how to save a document with the profile.
I am using C# and the API docs and intellisense simply ask for on object for the document.
Does that mean the path or do I need to load the PDF into some specific DOCSObjects type?
Also, the API docs references a Constant such as OF_NORMAL when saving the document. I assume this is 0, but are there others I should know about? There are many Constants referenced in the docs that have no values defined. (All examples are in C++/VB).
I know it's a long shot anyone is using this software, but thought I would give it a try.
Thank you and any assistance is appreciated.
I have done it in VB - using an API wrapper that I created. You should use the PCDClient under DM API folder instead of the DOCSObjects.
This code here probably won't work right away for you because it is heavily customized, but play around with it and you can probably figure it out. Good Luck!
Public Sub CreateProfile(ByRef Doc As Profile)
Try
'SET THE STATIC META DATA
Doc.objDoc.SetProperty("TYPE_ID", "DOCS") ' DOCUMENT TYPE IS ALWAYS DOCS
Doc.objDoc.SetProperty("TYPIST_ID", RDIMSAPI._UserID)
Doc.objDoc.SetProperty("APP_ID", RDIMSData.GetApp(Doc.FileToImport)) ' FILE TO IMPORT
'CREATE THE DOCUMENT
Doc.objDoc.Create()
If Doc.objDoc.ErrNumber <> 0 Then
Throw New Exception(Doc.objDoc.ErrNumber & " - " & Doc.objDoc.ErrDescription)
End If
'RETRIEVE THE NEW DOCUMENT PROFILE
Dim DocNumber As Integer = Doc.objDoc.GetReturnProperty("%OBJECT_IDENTIFIER")
Dim VersionID As Integer = Doc.objDoc.GetReturnProperty("%VERSION_ID")
'ADD THE DOCUMENT TO THE PROFILE
Dim objPutDoc As New PCDClient.PCDPutDoc
objPutDoc.SetDST(RDIMSAPI._sDST)
objPutDoc.AddSearchCriteria("%TARGET_LIBRARY", RDIMSAPI._Library)
objPutDoc.AddSearchCriteria("%DOCUMENT_NUMBER", DocNumber)
objPutDoc.AddSearchCriteria("%VERSION_ID", VersionID)
objPutDoc.Execute()
If objPutDoc.ErrNumber <> 0 Then
Throw New Exception(Doc.objDoc.ErrNumber & " - " & Doc.objDoc.ErrDescription)
End If
objPutDoc.NextRow()
'UPLOAD THE DOCUMENT
Dim objPutStream As PCDClient.PCDPutStream = objPutDoc.GetPropertyValue("%CONTENT")
Dim fs As FileStream = System.IO.File.OpenRead(Doc.FileToImport)
Dim fi As FileInfo = New System.IO.FileInfo(Doc.FileToImport)
Dim br As BinaryReader = New BinaryReader(fs)
Dim addDocBytes As Byte() = br.ReadBytes(CInt(fs.Length))
br.Read(addDocBytes, 0, addDocBytes.Length)
br.Close()
Dim bytesWritten As Integer = 0
objPutStream.Write(addDocBytes, addDocBytes.Length, bytesWritten)
objPutStream.SetComplete()
'UNLOCK THE DOCUMENT
Dim objDoc As New PCDClient.PCDDocObject
objDoc.SetDST(RDIMSAPI._sDST)
objDoc.SetObjectType("0_RDIMSPROF_SYS")
objDoc.SetProperty("%TARGET_LIBRARY", RDIMSAPI._Library)
objDoc.SetProperty("%OBJECT_IDENTIFIER", DocNumber)
objDoc.SetProperty("%VERSION_ID", VersionID)
objDoc.SetProperty("%STATUS", "%UNLOCK")
objDoc.Update()
objDoc.Fetch()
objDoc = Nothing
If Doc.objDoc.ErrNumber <> 0 Then
Throw New Exception(Doc.objDoc.ErrNumber & " - " & Doc.objDoc.ErrDescription)
End If
'RELEASE ALL OBJECTS AND RETURN DOCUMENT NUMBER
objPutDoc = Nothing
Catch ex As Exception
'IF EXCEPTION, LOG ERROR AND DISPLAY MESSAGE
Throw New Exception("(" & Me.GetType().FullName & "." & New StackTrace(0).GetFrame(0).GetMethod.Name & ") " & ex.Message)
Exit Sub
End Try
End Sub
I don't know if you're still trying. But here's my C# code for this. It's part of a larger module, so it won't work immediately. The profile parameter would be for example "DEF_PROF".
This also uses the PCDClientLib. My understanding is that these are serverside libraries, wich you should use only on the server. And that you should use the lib you've already used for clientside code.
// All variable prepended with an underscore are class fields etc...
// DMImportException is a custom exception, nothing special really
/// <summary>
/// Import a file into the library previously logged in to.
/// </summary>
/// <param name="profile">The name of the used profile.</param>
/// <param name="profileNameValues">A dictionary of strings containing the profile values wich should be saved for the document.</param>
/// <param name="FileName">The path and filename of the file to import.</param>
public virtual void ImportFile(string profile, Dictionary<string, string> profileNameValues, string FileName)
{
if (!_isLoggedIn)
{
throw new DMImportException("Trying to import a file while not logged in into DM.");
}
int totalbyteswritten;
byte[] bdata;
bdata = file.readallbytes(filename);
pcddocobject objdoc = new pcddocobject();
objdoc.setproperty("%target_library", _library);
objdoc.setdst(_dst);
objdoc.setobjecttype(profile);
foreach(var profilenamevaluepair in profilenamevalues)
{
objdoc.setproperty(profilenamevaluepair.key, profilenamevaluepair.value);
}
objdoc.create();
if (objdoc.errnumber != 0)
{
throw new dmimportexception("error while creating a new objdoc. check the inner error.", objdoc.errnumber, objdoc.errdescription);
}
_docnumber = objDoc.GetReturnProperty("%OBJECT_IDENTIFIER").ToString();
_versionID = objDoc.GetReturnProperty("%VERSION_ID").ToString();
PCDPutDoc objPutDoc = new PCDPutDoc();
objPutDoc.SetDST(_dst);
objPutDoc.AddSearchCriteria("%TARGET_LIBRARY", _library);
objPutDoc.AddSearchCriteria("%DOCUMENT_NUMBER", _docNumber);
objPutDoc.AddSearchCriteria("%VERSION_ID", _versionID);
objPutDoc.Execute();
if (objPutDoc.ErrNumber != 0)
{
throw new DMImportException("RecentEdit Failure on Execute: Error while trying to get a handle to the newly created doc. Check the inner error.", objPutDoc.ErrNumber, objPutDoc.ErrDescription);
}
objPutDoc.NextRow();
PCDPutStream objPutStream = (PCDPutStream)objPutDoc.GetPropertyValue("%CONTENT");
objPutStream.Write((object)bdata, (int)bdata.Length, out TotalBytesWritten);
objPutStream.SetComplete();
objPutStream = null;
objDoc = null;
objDoc = new PCDDocObject();
objDoc.SetDST(_dst);
objDoc.SetObjectType(profile);
objDoc.SetProperty("%TARGET_LIBRARY", _library);
objDoc.SetProperty("%OBJECT_IDENTIFIER", _docNumber);
objDoc.SetProperty("%VERSION_ID", _versionID);
objDoc.SetProperty("%STATUS", "%UNLOCK");
objDoc.Update();
if (objDoc.ErrNumber != 0)
{
throw new DMImportException("Error while trying to unlock the just imported file. Check the inner error.", objDoc.ErrNumber, objDoc.ErrDescription);
}
objPutDoc = null;
objDoc = null;
return;
}
P.S. I'd recommend you update to a later version of eDocs (we're upgrading from 5.1.0.5 to 5.2.1 end of this week ;-D)
--- EDIT ---
I think you need
Application.CurrentLibrary.CreateProfile("PROF_DEF").CreateVersionFromFile( /* filePath is one of the params */);
if you really need to do this with the DM Ext. API instead of the DM API

How to Bulk Insert csv with double quotes around all values?

I am trying to insert a .csv file into SQL Server 2008 R2.
The .csv is 300+MB from http://ipinfodb.com/ip_database.php Complete
(City), 4.0M records.
Here're the top 5 lines, with 1st line = column headers:
"ip_start";"country_code";"country_name";"region_code";"region_name";"city";"zipcode";"latitude";"longitude";"metrocode"
"0";"RD";"Reserved";;;;;"0";"0";
"16777216";"AU";"Australia";;;;;"-27";"133";
"17367040";"MY";"Malaysia";;;;;"2.5";"112.5";
"17435136";"AU";"Australia";;;;;"-27";"133";
I tried Import and Export Data, and BULK INSERT, but haven't been able to import them correctly yet.
Shall I resort to use bcp? can it handle stripping the ""? how?
Thank you very much.
Got it, forgot to set Text Qualifier as ":
Your data looks pretty inconsistent since NULL values don't also carry a quotation enclosure.
I believe you can create a format file to customize to your particular csv file and its particular terminators in SQL SERVER.
See more here:
http://lanestechblog.blogspot.com/2008/08/sql-server-bulk-insert-using-format.html
Is this a single import or are you wanting to schedule a recurring import? If this is a one-time task, you should be able to use the Import and Export Wizard. The text qualifier will be the quotation mark ("), be sure to select column names in the first data row, and you'll want to convey that the field delimiter is the semicolon (;).
I'm not certain the file is properly formatted - the last semicolon following each of the data rows might be a problem. If you hit any errors, simply add a new column header to the file.
EDIT: I just did a quick test, the semicolons at the end will be treated as part of the final value in that row. I would suggest adding a ;"tempheader" at the end of your header (first) row - that will cause SQL to treat the final semicolon as a delimiter and you can delete that extra column once the import is complete.
In C# you can use this code, working for me
public bool CSVFileRead(string fullPathWithFileName, string fileNameModified, string tableName)
{
SqlConnection con = new SqlConnection(ConfigurationSettings.AppSettings["dbConnectionString"]);
string filepath = fullPathWithFileName;
StreamReader sr = new StreamReader(filepath);
string line = sr.ReadLine();
string[] value = line.Split(',');
DataTable dt = new DataTable();
DataRow row;
foreach (string dc in value)
{
dt.Columns.Add(new DataColumn(dc));
}
while (!sr.EndOfStream)
{
//string[] stud = sr.ReadLine().Split(',');
//for (int i = 0; i < stud.Length; i++)
//{
// stud[i] = stud[i].Replace("\"", "");
//}
//value = stud;
value = sr.ReadLine().Split(',');
if (value.Length == dt.Columns.Count)
{
row = dt.NewRow();
row.ItemArray = value;
dt.Rows.Add(row);
}
}
SqlBulkCopy bc = new SqlBulkCopy(con.ConnectionString, SqlBulkCopyOptions.TableLock);
bc.DestinationTableName = tableName;
bc.BatchSize = dt.Rows.Count;
con.Open();
bc.WriteToServer(dt);
bc.Close();
con.Close();
return true;
}

Resources