After creating super table and tables, call taos_load_table_info to load the table information. Then initialize stmt by calling taos_stmt_init and taos_stmt_set_tbname to set up table name.
Create the TAOS_BIND object with the following attributes:
buffer_type = TSDB_DATA_TYPE_NCHAR
buffer_length = sizeof(str)
buffer = &str
length = sizeof(str)
Then call taos_stmt_bind_param and taos_stmt_add_batch, and finally execute with taos_stmt_execute.
The problem is that the insertion failed because I check the shell and use select * to look for the data but it only shows an empty column.
I strongly recommend you first try to insert a simple nchar type data to check whether it is the taos_stmt API's problem. If that insertion success, then you can also check if the insert nchar string has the same length as str variable. Sometimes, buffer_length is greater than or equal to length. If the actual size of your nchar data is less than the length value in TAOS_BIND, then tdengine will still analyze the binding value with other extra empty values and will fail to insert.
Related
I've created a CLR funtion for SQL Server 2014 that should calculate subtraction between the first and the last value in column [Value].
Here is the table:
Date_Time Value
-------------------------------------
2018-03-29 09:30:02.533 6771
2018-03-29 10:26:23.557 6779
2018-03-29 13:12:04.550 6787
2018-03-29 13:55:44.560 6795
Here is the code:
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
[Serializable]
[SqlUserDefinedAggregate(Format.Native,
IsInvariantToDuplicates = false,
IsInvariantToNulls = true,
IsInvariantToOrder = false,
IsNullIfEmpty = true,
Name = "SUBT")]
public struct SUBT
{
private double first;
private double last;
private int count;
public void Init()
{
first = 0.0;
last = 0.0;
count = 0;
}
public void Accumulate(SqlDouble Value)
{
if (!Value.IsNull)
{
if (count == 0)
first = (double)Value;
else
last = (double)Value;
count += 1;
}
}
public void Merge(SUBT Group)
{
first = Group.first;
last = Group.last;
count += 1;
}
public SqlDouble Terminate()
{
double value = (double)(last - first);
return new SqlDouble(value);
}
}
So the result should be [Value]=24, i.e. 6795 - 6771, but I get 6795 :(
Where is the error?
This aggregate function is dependent on order but there is no ordering guarantee for the aggregation input stream. Consequently, the results are execution plan dependent.
Assuming the Date_Time value is the desired ordering, you could provide both Date_Time and Value as function arguments, save the value with the lowest and highest Date_time values and use those in the Merge and Terminate methods.
There are a couple of problems here, I believe:
In the Merge method you are making two bad assumptions:
You are assuming that the incoming Group has a value, yet it could have been called on one or more NULL values, in which case all 3 internal variables are 0. Yet you are overwriting the current values of first and last, which could have non-0 values prior to Merge being called, but then will end up back at 0 due to being overwritten.
You are assuming that at least one of the instances — the current one or the incoming Group — has values set (i.e. has been called at least once on a non-NULL value). In the case that both instances have only been called with a NULL value, you will have 0 for first and last, yet you will increment counter. I am not sure if Accumulate will be called again once things are being merged, but if it does, you will skip setting first. This is not your problem at the moment since you do not have multiple (or any) NULL values, but it is a potential problem for real data sets.
In the case where both instances have been called on non-NULL values, both will at least have first set, and maybe last (or maybe not). By overwriting the current instance with the incoming Group, you could be losing the real first value, losing the real last value.
As #DanGuzman mentioned in his answer, there is no guaranteed ordering of User-Defined Aggregates (the IsInvariantToOrder property of the SqlUserDefinedAggregate attribute is ignored / unused). And, as he noted, you will need to pass in the Date_Time value in order to handle this aspect of the operation manually. However, it won't be used in the Terminate method. It will instead be used to compare to two new variables: firstDate and lastDate, initialized to a future and past, respectively (this will likely require changing the Format to UserDefined and then adding custom Read and Write methods -- unless you can store the full DateTime values as ticks, perhaps).
Get rid of the counter variable
In the Accumulate method, you will need to:
IF the incoming Date_Value against firstDate. If it is before to firstDate then store this new value as firstDate and store Value as first, ELSE
IF the incoming Date_Value against lastDate. If it is after to lastDate then store this new value as lastDate and store Value as last, ELSE do nothing
In the Merge method, do a similar comparison for firstDate between both instances and keep the earlier one (date and value). Do the same with lastDate and keep the later one (date and value). (Note: these changes should fix all of the Merge issues noted above in # 1)
The terminate method shouldn't change
For what it's worth, I ran the code exactly as you have posted in the question, and it returns the expected value, using the following test query:
CREATE TABLE #Test ([Date_Time] DATETIME, [Value] FLOAT);
-- TRUNCATE TABLE #Test;
INSERT INTO #Test VALUES ('2018-03-29 09:30:02.533', 6771);
INSERT INTO #Test VALUES ('2018-03-29 10:26:23.557', 6779);
INSERT INTO #Test VALUES ('2018-03-29 13:12:04.550', 6787);
INSERT INTO #Test VALUES ('2018-03-29 13:55:44.560', 6795);
SELECT dbo.SUBT([Value])
FROM #Test;
-- 24
So, if you are still having issues, then you will need to post more info, such as the test query (and maybe table) that you are using. But even if it appears to work, like it does on my system, it still has the potential ordering problem and will need to be updated as noted above regardless.
Other notes:
In the Accumulate method you have (double)Value. There is no need to cast the incoming parameter. All Sql* types have a Value property that returns the value in the native .NET type. In this case, just use Value.Value. That isn't great for readability, so consider changing the name of the input paramter ;-).
You never use the value of counter, so why increment it? You could instead just use a bool, and set it to true here. Setting it to true for each non-NULL value will not change the operation. However, this is a moot point since you truly need to set either first or last in each call to this UDA, based on the current Date_Time value.
How do we concatenate fields of a dynamic work area? The idea is in the below code:
LOOP AT lt_final INTO DATA(ls_final).
CONCATENATE ls_final-field1
ls_final-field2
ls_final-field3
ls_final-field4
ls_final-field5
INTO ls_attachment SEPARATED BY lc_tab. "lc_tab is horizontal tab
APPEND ls_attachment TO lt_attachment.
CLEAR: ls_attachment.
ENDLOOP.
(This code will be used for sending email attachment.) Now, my problem is, the internal table in the above code is a dynamic internal table, therefore I am not sure how many fields will be there and the field names as well.
How do I concatenate the fields? Any idea, please help..
LOOP AT <dynamic_table> INTO DATA(ls_final).
CONCATENATE ls_final-(?)
ls_final-(?)
ls_final-(?)
ls_final-(?)
ls_final-(?)
"or more fields insert here depending on dynamic table
INTO ls_attachment SEPARATED BY lc_tab. "lc_tab is horizontal tab
APPEND ls_attachment TO lt_attachment.
CLEAR: ls_attachment.
ENDLOOP.
FIELD-SYMBOLS: <lv_field> TYPE ANY.
LOOP AT lt_final
ASSIGNING FIELD-SYMBOL(<ls_final>).
DO.
ASSIGN COMPONENT sy-index
OF STRUCTURE <ls_final>
TO <lv_field>.
IF sy-subrc EQ 0.
IF sy-index EQ 1.
ls_attachment = <lv_field>.
ELSE.
ls_attachment = ls_attachment && lc_tab && <lv_field>.
ENDIF.
ELSE.
EXIT.
ENDIF.
ENDDO.
ENDLOOP.
I hope it is self explaining, but:
You can use the system variable (sy-index), it is incremented automatically by SAP.
In the first step, just copy the value, there is nothing to concatenate yet (otherwise there will be an unnecessary lc_tab at the beginning of the string).
Just read your structure by index.
data :
lv_attachment type string.
lv_index type i value 1.
field-symbols:
<lv_value> type any.
while 1 = 1.
assign component lv_index of structure ls_final to <lv_value>.
if sy-subrc <> 0.
exit.
endif.
concatenate lv_attachment <lv_value> into lv_attachment separated by lc_tab.
lv_index = lv_index + 1.
endwhile.
Hope it helps.
You can use CL_ABAP_CONTAINER_UTILITIES class for that task, method FILL_CONTAINER_C.
Here is the sample of populating dynamic table and concatenating its fields into container field:
PARAMETERS: p_tab TYPE string.
FIELD-SYMBOLS: <fs_tab> TYPE STANDARD TABLE.
DATA tab TYPE REF TO data.
CREATE DATA tab TYPE TABLE OF (p_tab).
ASSIGN tab->* TO <fs_tab>.
SELECT * UP TO 100 ROWS
INTO TABLE <fs_tab>
FROM (p_tab).
LOOP AT <fs_tab> ASSIGNING FIELD-SYMBOL(<fs_line>).
CALL METHOD CL_ABAP_CONTAINER_UTILITIES=>FILL_CONTAINER_C
EXPORTING
IM_VALUE = <fs_line>
IMPORTING
EX_CONTAINER = DATA(container)
EXCEPTIONS
ILLEGAL_PARAMETER_TYPE = 1
others = 2.
CONDENSE container.
" do smth
ENDLOOP.
I have some sql function that returns character varying type. The output is something like this:
'TTFFFFNN'. I need to get this characters by index. How to convert character varying to array?
Use string_to_array() with NULL as delimiter (pg 9.1+):
SELECT string_to_array('TTFFFFNN'::text, NULL) AS arr;
Per documentation:
In string_to_array, if the delimiter parameter is NULL, each character
in the input string will become a separate element in the resulting array.
In older versions (pg 9.0-), the call with NULL returned NULL. (Fiddle.)
To get the 2nd position (example):
SELECT (string_to_array('TTFFFFNN'::text, NULL))[2] AS item2;
Alternatives
For single characters I would use substring() directly, like #a_horse commented:
SELECT substring('TTFFFFNN'::text, 2, 1) AS item2;
SQL Fiddle showing both.
For strings with actual delimiters, I suggest split_part():
Split comma separated column data into additional columns
Only use regexp_split_to_array() if you must. Regular expression processing is more expensive.
In addition to the solutions outlined by Erwin and a horse, you can use regexp_split_to_array() with an empty regexp instead:
select regexp_split_to_array('TTFFFFNN'::text, '');
With an index, that becomes:
select (regexp_split_to_array('TTFFFFNN'::text, ''))[2];
Below is what I think of the hadoop framework processing text files. Please correct me if I am going wrong somewhere.
Each mapper acts on an input split which contains some records.
For each input split a record reader is getting created which starts reading records from the input split.
If there are n records in an input split the map method in the mapper is called n times which in turn reads a key-value pair using the record reader.
Now coming to the databases perspective
I have a database on a single remote node. I want to fetch some data from a table in this database. I would configure the parameters using DBConfigure and mention the input table using DBInputFormat. Now say if my table has 100 records in all, and I execute an SQL query which generates 70 records in the output.
I would like to know :
How are the InputSplits getting created in the above case (database) ?
What does the input split creation depend on, the number of records which my sql query generates or the total number of records in the table (database) ?
How many DBRecordReaders are getting created in the above case (database) ?
How are the InputSplits getting created in the above case (database)?
// Split the rows into n-number of chunks and adjust the last chunk
// accordingly
for (int i = 0; i < chunks; i++) {
DBInputSplit split;
if ((i + 1) == chunks)
split = new DBInputSplit(i * chunkSize, count);
else
split = new DBInputSplit(i * chunkSize, (i * chunkSize)
+ chunkSize);
splits.add(split);
}
There is the how, but to understand what it depends on let's take a look at chunkSize:
statement = connection.createStatement();
results = statement.executeQuery(getCountQuery());
results.next();
long count = results.getLong(1);
int chunks = job.getConfiguration().getInt("mapred.map.tasks", 1);
long chunkSize = (count / chunks);
So chunkSize takes the count = SELECT COUNT(*) FROM tableName and divides this by chunks = mapred.map.tasks or 1 if it is not defined in the configuration.
Then finally, each input split will have a RecordReader created to handle the type of database you are reading from for instance: MySQLDBRecordReader for MySQL database.
For more info check out the source
It appears #Engineiro explained it well by taking the actual hadoop source. Just to answer, number of DBRecordReader is equal to number of map tasks.
To explain further, the Hadoop Map side framework creates an instance of DBRecordReader for each Map task, in case where the child JVM is not reused for further Map tasks. In other words, the number of input splits is equals to the value of map.reduce.tasks in case of DBInputFormat. So, each map Task's record Reader has the meta information to construct the query to get subset of data from the table. Each Record Reader executes a pagination type of SQL which is similar to the below.
SELECT * FROM (SELECT a.*,ROWNUM dbif_rno FROM ( select * from emp ) a WHERE rownum <= 6 + 7 ) WHERE dbif_rno >= 6
The above SQL is for the second Map tasks to return the rows between 6 and 13
To generalize for any type of Input formats, the number of Record Readers is equals to the number of Map Tasks.
This post talks about all that you want : http://blog.cloudera.com/blog/2009/03/database-access-with-hadoop/
I'm reading info from a pqsql database (and changed my engine from mysql)
there is something weird in the string data returned,
if the field length in the db is 50 chars the returned value has 50 chars length, so
even if the value in the db is inserted with fewer elements the returned value size is 50.
in mysql the returned value size was the same as it was inserted in the db.
example
create table values(data char(50))
insert into table values('info');
when i pick up this data with PQgetvalue
res=PQexec(conn,"select * from values");
printf(" the value has the lenght of %d chars",strlen(PQgetvalue(res,0,0)));
it will display
the value has the lenght of 50
im kinda surprised of this because now i need to store o calculate in some way the real size of the field, no the max field size
I'm doing something wrong?
(sorry for the typos)
The field is declared as char, not varchar. A char field's value will always have the declared length, probably after padding the value from the right with spaces. From your description, I'd consider changing it to varchar.
See http://www.postgresql.org/docs/8.2/static/datatype-character.html