unable to insert data in hive having custom delimiter - file

Question

I am trying to learn hive, this may be stupid a question but
I created a table in hive as follows
create table if not exists tweets_table(
tweetdata STRING,
followerscount INT,
friendscount INT,
statuscount INT,
retweetcount INT,
favouritescount INT,
lang STRING,
placefullname STRING,
placename STRING,
countryname STRING,
countrycode STRING,
hashtags STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '^**^'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;
LOAD DATA LOCAL INPATH '/home/cloudera/Desktop/TestDB.txt' INTO TABLE tweets_table5 ;
I have used '^**^' to delimit the text because the tweets has a lot of "\n \r ," (please suggest a standard practice if any)
So I have a text file which I am trying to load
09-09-2016 10:51:33|^**^|#ArvindKejriwal #abpnewstv तुम्हारे दावों का क्या हुआ केजरीवाल।|^**^|74|^**^|30|^**^|0|^**^|98|^**^|0|^**^|49|^**^|en|^**^|Ambikapur, India|^**^|Ambikapur|^**^|India|^**^|IN|^**^|[]
09-09-2016 10:51:37|^**^|#LiveLawIndia It is shocking a judge arrested. I am sure Higher Judiciary will come their rescue , Judges per se cannot be wrong|^**^|0|^**^|14|^**^|0|^**^|32|^**^|0|^**^|2|^**^|en|^**^|Rajasthan, India|^**^|Rajasthan|^**^|India|^**^|IN|^**^|[]
After successfully loading it and querying it,
I get the following output
09-09-2016 10:51:33| NULL NULL NULL NULL NULL |30| ** |0| ** |98| **
09-09-2016 10:51:37| NULL NULL NULL NULL NULL |14| ** |0| ** |32| **
I fail to understand where am I going wrong is it in my tex tfile or the hive table. Please help

Answer 1

Several issues with what you're trying to do:
Using FIELDS TERMINATED BY You can't have a delimiter that is more than 1 char.
Even if that worked, that doesn't solve the problem that your tweets have line delimiters in them - each \n in a tweet starts a new row.
The way you describe your table - it's impossible to parse correctly - you can't have \n as line delimiters and in the tweet data. If you are the one generating this input file, I would suggest replacing all \n and \r in the tweets with spaces.

Answer 2

Create table with regex serde instead of default hive serde.
Modify below regex based on number of columnns.:
^(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)$
Table:
create external table if not exists tweets_table(
tweetdata STRING,
followerscount INT,
friendscount INT,
statuscount INT,
retweetcount INT,
favouritescount INT,
lang STRING,
placefullname STRING,
placename STRING,
countryname STRING,
countrycode STRING,
hashtags STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = " ^(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)\|\^\*\*\^\|(.+)$",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s %10$s %11$s %12$s %13$s"
)
STORED AS TEXTFILE;
Load data:
LOAD DATA LOCAL INPATH '/home/cloudera/Desktop/TestDB.txt' INTO TABLE tweets_table ;
If you get RegexSerDe classNotFoundException then add regex serde jar:
ADD JAR hive-contrib=x.x.x.jar