hive command in bash script won't load file into table - file

I am writing a bash script that - among other things - has to create a hive table and load a csv file (whose name is not known a priori) into that table.
I have exported the name of the file foo.csv into the environment variable myfile and have tried the command
hive --hiveconf mf=$myfile -e 'set mf; set hiveconf:mf; load data local inpath ${hiveconf:mf} into table mytable'
It returns the error
FAILED: ParseException line 1:23 mismatched input 'foo' expecting StringLiteral near 'inpath' in load statement
I already tried using the absolute path to the file and it won't work either: if the path is /mypath/foo.csv the error will be
FAILED: ParseException line 1:23 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
Even trying to directly put the file name like this
hive -e 'load data local inpath foo.csv into table mytable'
doesn't work at all, and the thrown error is the same as before.
Does anybody have any idea on what is wrong with these commands? I could really appreciate some help, thanks.


Filename should be placed inside '' :
load data local inpath 'foo.csv' into table mytable
In your script you should probably escape these symbols so you won't get another parse exception.
Also, look at Language Manual on loading

Related

How I can configure flat file name as filename_*.txt in SSIS

I have flat file source and connection manager. I need to configure its connection string as filename_* .txt. I have one variable on package level to read the directory path and then I am using it in expression property to read the file path as #FilePath + "filename__* " + ".txt". This is not working. I am not able to figure out how to configure file name as filename_*.txt.

In a case like this best practice is to use a foreach loop container of type File Enumerator, you can pass it a search string to scan for files and run a process using the found file name.
Note if you have multiple files in that folder it will run the same process for each file
The container will then execute everything inside it for each file found with that search string. You will need to map the found file name to a variable on the Variable Mapping section.
You can now use the variable as an expression in your file connection
Your control flow should look like this

BCP Command to Export Query Results to File with Pipe Delimiter

I have a requirement to create a SQL Job that exports a query to a Pipe / Vertical bar delimited file (|) and save it on a network drive in either *.txt or *.csv format. Right now, I am just trying to get this to work inside SSMS, calling the BCP command and exporting the stored procedure in the proper format to the network location, but not able to get it working.
I have been researching this and there are two methods for this.
Use the export data wizard to create a job and schedule that to run. But this method, if we need to make changes, I believe we cannot change the SSIS package that is created so we lose flexibility
Use the BCP command to export the file.
I greatly prefer to use option #2, the BCP command, but I am having problems. I just cannot seem to get the syntax correct and hoping someone could show me what I am missing:
This is my command:
Exec master..xp_cmdshell 'bcp EXEC [dbo].[usp_Report_1123] ''7786'' -t| out \\networkDrive\Reports\REPORT_1123\report1123.csv -T'
But I get the following messages:
output
'out' is not recognized as an internal or external command,
operable program or batch file.
NULL
The stored procedure does work and returns data. The network path, if I enter it into my computer, finds the path. But I am not sure what I am missing and hoping someone could help.

Export data with bcp utility with double quotes

I'm trying to exporting data from MS sql server by using bcp utility command line. The problem is that in the exported output is missing the first double quote at the first line and I cannot explain the reason.
Below the command that I'm using for the export:
/opt/mssql-tools/bin/bcp db_schema out dump.csv -c -t"\",\"" -r"\"\n\"" -S my_host -U my_user
But the output result is missing the first double quotes on first line (only the first line) of the exported csv file:
801","40116","Hazelnut MT -L","Thursday Promo","Large","","5.9000","","801","1.0000","","3.6500","2.2500",".0000","default","","","","","Chatime","02/06/2014","09125a9cfffd4143a00e73e3b62f15f2","CB01","",".0000","5.9000","6.9000",".0000",".0000",".0000",".0000",".0000",".0000","0","","0","0","0","","","","","","","","","Modern Milk Tea","","","0","","","1","0","","","","","","","","0","Hau Chan","","","","","","","","","","0","","","","","","","-1","","","","","","","","","","","","0","00000000420714AA","2014-06-02","1900-01-01","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","",""
Am I missing something?

If you know the field names, you can try the following:
Create Table [names]
(
Id Int,
fname VarChar(15),
lname VarChar(25)
)
Insert Into names Values
(1, 'Jim', 'Morrison'),
(2,'Robert', 'Plant'),
(3,'Janis', 'Joplin')
BCP command: Using quotename() with char(34)
(This BCP command uses a Trusted connection)
bcp "SELECT quotename(Cast(Id As VarChar(15)),char(34)), quotename(fname,char(34)), quotename(lname,char(34)) FROM names" queryout dump.csv -c -t"," -S SERVER -d DBNAME -T
Result:
"1","Jim","Morrison"
"2","Robert","Plant"
"3","Janis","Joplin"

I'll bet that you also have a line at the end of the file that is just a single double-quote. Why? With the command line switches you've provided, you're saying "end every field with "," and every row with "\n"". And that's what it's doing.
So, the second and subsequent lines start with a double-quote because the previous line ends with one.
Clippy DBA says "it looks like you're trying to produce a CSV". Without knowing why, it's hard for me... er... Clippy to suggest an alternative. Specifically, what is going to be reading this file? Will you be reading this with Excel or something else that's expecting a specific format? WIll you be importing it into another database table?

The quotename answer provided by level3looper will work in your case.
For completeness, Im providing a solution I've given in the past for the same purpose.
I like this one a little better because it keeps the definition of formatting in the format file, which is where I prefer to go to get that info. The quotename is a good solution for quick, adhoc work, but for automation, business process, I would recommand the link below.
Essentially, you just add a "dummy" column to the beginning of the definition of the extract file and delimit that column with a single doublequote. Then you also note in the format file to skip the first column. This gives you just the doublequote at the start of the line.
sql server bcp bulk insert pipe delimited with text qualifier format file

Inputting a .sas7bdat file on SAS On Demand

I am using SAS Studio (the browser version) instead of a desktop SAS program.
I am trying to read a .sas7bdat file that I have uploaded onto a folder on SAS Studio.
libname HW5 '~/home/xxxxxxxxxxxx/sasuser.v94/HW5';
DATA FILE1;
set HW5.orders;
RUN;
I get the error:
ERROR: File HW5.ORDERS.DATA does not exist.
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.FILE1 may be incomplete. When this step was stopped there
were 0 observations and 0 variables.
WARNING: Data set WORK.DISCOUNT_RET was not replaced because this step was stopped.
Here is the image of the folder:
https://ibb.co/hN83ua
I realize this is a simple error but I don't know how to fix it. Thanks! If nothing works, can i read this via infile?

Right click on the HW5 folder in the list on the left and select properties. This should show you the physical location for that folder. Copy it and paste it into your LIBNAME statement enclosed in quotes. Most likely the issue was your inclusion of the ~ in front of the path.
libname HW5 '/home/xxxxxxxxxxxx/sasuser.v94/HW5';

create flat file in ssis package

I'm working on creating a csv export from a SQL Server database and I've been familiar with a process for doing so that admittedly, I've never completely understood. The process involves creating a "template" file, which defines the columns and structure for the file export. Once the "template" file exists, you can use a Data Flow task to fill it and a File System Task to copy it to the final storage destination with whatever file name you'd like (frequently a date/time stamp).
Is there a reason that you can't simply create a file directly, without the intermediate "template" file? I've looked around for a bit and it seems like all the proposed solutions involve connecting to an existing file. I see that there is a "Create File" Usage type for a "File" connection manager, but you can't use it in any File System Task. The only File System Type connection managers you can use relative to a file are "Copy", "Delete", "Move", "Rename", and "Set Attributes".
Is there a way to create a file at package run time and fill it?

The whole point of SSIS is to create a data flow with metadata so that the data can be manipulated - if you just want to go database direct to CSV you are probably better off using bcp (bulk copy program) from the command line. If you want to include it as part of a SSIS package just add an Execute Process Task and add the command line to that. You can dynamically change the included columns or the output file by adding an expression to the task. You could also call bcp though TSQL using an Excute SQL Task.
One other option is to concatenate all your columns in your query inter-spaced with a comma literal and output to a text file with just one very wide column.
For documentation on bcp look here

Resources