I have been trying to insert a geojson into postgis.
Both of my geojson exceed 300 MB.
When I tried
ogr2ogr -f "ESRI Shapefile" name.shp "name.geojson"
or
ogr2ogr -f PostgreSQL PG:"dbname=name user=user" name.geojson -nln newtable
both processes just show "Killed", with no describing error, not anything more.
How should I check why it is killed? How to insert my big geojsons into postgis?
Related
Trying to copy data from Sybase IQ Table to .txt file using bcp
Command
C:\Windows\system32>bcp xxTable out \xx\xx\xxTable.txt -S HSPREP_15 -U xxUser -A 16384 -c -t\t$$ -P xxpwd -F -L 1>\xx\xx\xxTable_rpt.txt
Source Table got about 132 million records
bcp copy about 60 million records to .txt file and then fails with below error:
There is no issue with disk spare or user credentials.
Any clues? where should i be looking into?
Bcp out doesnt create a file bigger than 2GB which might the case please check
I'm trying to transfer table data from one SQL Server to another and wanting to use the bcp utility for it. This is purely to transfer data between two identical schemas, but I'm not able to use something like SSDT; I need something that can be scriptable and portable so it can be run by others with just SQL server and SSMS access.
I am generating a native output file and format file like so:
$> bcp database.TableName OUT c:\data\bcp\TableName.bcp -T -N -S SQLINSTANCE
$> bcp database.TableName format nul -f c:\data\bcp\TableName.fmt -T -N
Then in Management Studio I am trying to in turn read the files like this:
SELECT
*
FROM
OPENROWSET (BULK 'c:\data\bcp\TableName.bcp',
FORMATFILE = 'c:\data\bcp\TableName.fmt') AS t1
But am getting this error:
The bulk load failed. The column is too long in the data file for row 6, column 19. Verify that the field terminator and row terminator are specified correctly.
I have followed this process before successfully, and it works for other tables. But I'm running into issue with this table. The column mentioned is of datatype nvarchar(max). I can inspect what I think is the "problem" record in the source data and it's just a very long string but I don't see anything else special about it.
Is there something else I should be doing when generating the format file or what else am I missing?
If you are only exporting for the purpose of importing to another SQL Server, native format is the way to go. And is this case you don't need to use format files. Just do a native export and import.
Note you are specifying a capital -N and that's not native. Native is lower -n.
You should export using something like:
bcp database.Schema.TableName OUT c:\data\bcp\TableName.bcp -T -n -S SQLINSTANCE
Then on the importing side I sugest using BULK IMPORT, which don't need a format file for native at all:
BULK INSERT TargetDB.dbo.TargetTable
FROM 'c:\data\bcp\TableName.bcp'
WITH (DATAFILETYPE = 'native');
If you can't use BULK INSERT and must absolutely go for OPENROWSET, you need a format file. bcp can generate that for you, but again, lower case -n:
bcp database.Schema.TableName format nul -f c:\data\bcp\TableName.fmt -T -n -S SQLINSTANCE
Now your OPENROWSET should work.
i have a small problem with BCP functionality in SQL Server 2012.
The things is:
im loading .jpg image (167KB in size) using below command:
INSERT [tabela_testowa] ( Data )
SELECT * FROM OPENROWSET (BULK N'C:\foty\ch6_MagicShop.jpg', SINGLE_BLOB) a;
and then im trying to export it back to disk using:
BCP "SELECT data FROM tabela_testowa WHERE ID = 1" queryout "C:\test\file.jpg" -T -n -d test
File gets saved on disk no problem, size is also 167 KB but.. it cant be opened like the original copy.
I dont know whatever some parameter is wrong in BCP export? Or maybe it gets corrupted at import stage?
Anyone had similiar problems?
Thank god, thanks to #user_0 answer and #user3494351's cryptic answer and comment and this ancient forum post I finally figured this out after several hours of banging my head against the wall.
The issue is that BCP likes to add an extra 8 bytes to the file by default. This corrupts the file and makes it unable to be opened if you just use the native -n flag.
However, BCP allows you to specify a format file as output that can allow you to tell it not to add the extra 8 bytes. So I have a table I created (to be used in a cursor) in SQL Server that only has ONE ROW and ONE COLUMN with my binary data. Table must exist when you run the first command.
In command line first you need to do this:
bcp MyDatabase.MySchema.MyTempTable format nul -T -n -f formatfile.fmt
This creates formatfile.fmt in the directory you are in. I did on E:\ drive. Here's what it looks like:
10.0
1
1 SQLBINARY 8 0 "" 1 MyColumn ""
That 8 right there is the variable that bcp says how many bytes to add to your file. It is the bastard that is corrupting your files. Change that sucker to a 0:
10.0
1
1 SQLBINARY 0 0 "" 1 MyColumn ""
Now just run your BCP script, drop the -n flag and include the -f flag:
bcp "SELECT MyColumn FROM MyDatabase.MySchema.MyTempTable" queryout "E:\MyOutputpath" -T -f E:\formatfile.fmt
BCP is adding informations to his file. Just few data, but you are not exporting just a jpg file.
You say 167 KB, but watch the real bytes, not the rounded dimension. There will be a difference.
You cannot export the image via BCP.
OK so i solved the issue.
Format file has to be added using -f and path to the file. It can be create by running bcp without any format and order it to save format file to disk. Then we can use this format file so its no longer needed to answer those questions, and file itself has no additional data and can be opened without problems
i have a small problem with BCP functionality in SQL Server 2012.
The things is:
im loading .jpg image (167KB in size) using below command:
INSERT [tabela_testowa] ( Data )
SELECT * FROM OPENROWSET (BULK N'C:\foty\ch6_MagicShop.jpg', SINGLE_BLOB) a;
and then im trying to export it back to disk using:
BCP "SELECT data FROM tabela_testowa WHERE ID = 1" queryout "C:\test\file.jpg" -T -n -d test
File gets saved on disk no problem, size is also 167 KB but.. it cant be opened like the original copy.
I dont know whatever some parameter is wrong in BCP export? Or maybe it gets corrupted at import stage?
Anyone had similiar problems?
Thank god, thanks to #user_0 answer and #user3494351's cryptic answer and comment and this ancient forum post I finally figured this out after several hours of banging my head against the wall.
The issue is that BCP likes to add an extra 8 bytes to the file by default. This corrupts the file and makes it unable to be opened if you just use the native -n flag.
However, BCP allows you to specify a format file as output that can allow you to tell it not to add the extra 8 bytes. So I have a table I created (to be used in a cursor) in SQL Server that only has ONE ROW and ONE COLUMN with my binary data. Table must exist when you run the first command.
In command line first you need to do this:
bcp MyDatabase.MySchema.MyTempTable format nul -T -n -f formatfile.fmt
This creates formatfile.fmt in the directory you are in. I did on E:\ drive. Here's what it looks like:
10.0
1
1 SQLBINARY 8 0 "" 1 MyColumn ""
That 8 right there is the variable that bcp says how many bytes to add to your file. It is the bastard that is corrupting your files. Change that sucker to a 0:
10.0
1
1 SQLBINARY 0 0 "" 1 MyColumn ""
Now just run your BCP script, drop the -n flag and include the -f flag:
bcp "SELECT MyColumn FROM MyDatabase.MySchema.MyTempTable" queryout "E:\MyOutputpath" -T -f E:\formatfile.fmt
BCP is adding informations to his file. Just few data, but you are not exporting just a jpg file.
You say 167 KB, but watch the real bytes, not the rounded dimension. There will be a difference.
You cannot export the image via BCP.
OK so i solved the issue.
Format file has to be added using -f and path to the file. It can be create by running bcp without any format and order it to save format file to disk. Then we can use this format file so its no longer needed to answer those questions, and file itself has no additional data and can be opened without problems
i have a small problem with BCP functionality in SQL Server 2012.
The things is:
im loading .jpg image (167KB in size) using below command:
INSERT [tabela_testowa] ( Data )
SELECT * FROM OPENROWSET (BULK N'C:\foty\ch6_MagicShop.jpg', SINGLE_BLOB) a;
and then im trying to export it back to disk using:
BCP "SELECT data FROM tabela_testowa WHERE ID = 1" queryout "C:\test\file.jpg" -T -n -d test
File gets saved on disk no problem, size is also 167 KB but.. it cant be opened like the original copy.
I dont know whatever some parameter is wrong in BCP export? Or maybe it gets corrupted at import stage?
Anyone had similiar problems?
Thank god, thanks to #user_0 answer and #user3494351's cryptic answer and comment and this ancient forum post I finally figured this out after several hours of banging my head against the wall.
The issue is that BCP likes to add an extra 8 bytes to the file by default. This corrupts the file and makes it unable to be opened if you just use the native -n flag.
However, BCP allows you to specify a format file as output that can allow you to tell it not to add the extra 8 bytes. So I have a table I created (to be used in a cursor) in SQL Server that only has ONE ROW and ONE COLUMN with my binary data. Table must exist when you run the first command.
In command line first you need to do this:
bcp MyDatabase.MySchema.MyTempTable format nul -T -n -f formatfile.fmt
This creates formatfile.fmt in the directory you are in. I did on E:\ drive. Here's what it looks like:
10.0
1
1 SQLBINARY 8 0 "" 1 MyColumn ""
That 8 right there is the variable that bcp says how many bytes to add to your file. It is the bastard that is corrupting your files. Change that sucker to a 0:
10.0
1
1 SQLBINARY 0 0 "" 1 MyColumn ""
Now just run your BCP script, drop the -n flag and include the -f flag:
bcp "SELECT MyColumn FROM MyDatabase.MySchema.MyTempTable" queryout "E:\MyOutputpath" -T -f E:\formatfile.fmt
BCP is adding informations to his file. Just few data, but you are not exporting just a jpg file.
You say 167 KB, but watch the real bytes, not the rounded dimension. There will be a difference.
You cannot export the image via BCP.
OK so i solved the issue.
Format file has to be added using -f and path to the file. It can be create by running bcp without any format and order it to save format file to disk. Then we can use this format file so its no longer needed to answer those questions, and file itself has no additional data and can be opened without problems