I want to read a serail port every 0.1s and append the incoming data to an array, I can show the data this time but the array seems only store the newest data. Anyone can tell me why? Thanks.
Here is some code:
function wtsMat_OpeningFcn(hObject, eventdata, handles, varargin)
.....
%%tact is the array to store data
tact=ones(1,84);
handles.tact=tact;
% Update handles structure
guidata(hObject, handles);
Here is setting of scom
scom=serial(com_cur,'BaudRate',baud_curNum,'Parity','none','DataBits',8,'StopBits',1,...
'InputBufferSize',1000,...
'TimeOut',1,...
'TimerPeriod',0.1,...
'timerfcn',{#mycallback,handles});
fopen(scom);
handles.scom=scom;
guidata(hObject,handles);
here is mycallback function
function mycallback(scom,BytsAvailable,handles)
%start single frame acquisition
showData=ones(84,1);
showWin=ones(14,6);
%%get previous data from handles
tact=handles.tact;
fwrite(scom,uint8(hex2dec(['AA';'AA';'AA';'20';'01';'00';'00';'8F';'83'])));
myData = fread(scom,183);%read raw data from sensor
for i=1:84
showData(i,1)=myData(13+i*2)*16*16+myData(12+i*2);
end
%%append to tact array
tact=[tact;showData'];
handles.tact=tact;
....
save tact when close the scom
function pb_close_Callback(hObject, eventdata, handles)
% hObject handle to pb_close (see GCBO)
% eventdata reserved - to be defined in a future version of MATLAB
% handles structure with handles and user data (see GUIDATA)
scom=handles.scom;
%stop acquising
fwrite(scom,uint8(hex2dec(['AA';'AA';'AA';'22';'00';'00';'0E';'76'])));
fclose(scom);
tact=handles.tact;
save('tact.mat','tact');
Try creating a buffer and saving the new data in the last position at every iteration
buf_len = 100;
index = 1:buf_len;
%Initialize array
arrayOfData = zeros(buf_len,1);
% get Data here. Let's say the new value is theta
arrayOfData = [ arrayOfData (2:end) ; theta ];
plot(index,Tdata,'r','LineWidth',2);
Related
I'm learning/experimenting with Flink, and I'm observing some unexpected behavior with the DataStream join, and would like to understand what is happening...
Let's say I have two streams with 10 records each, which I want to join on a id field. Let's assume that for each record in one stream had a matching one in the other, and the IDs are unique in each stream. Let's also say I have to use a global window (requirement).
Join using DataStream API (my simplified code in Scala):
val stream1 = ... // from a Kafka topic on my local machine (I tried with and without .keyBy)
val stream2 = ...
stream1
.join(stream2)
.where(_.id).equalTo(_.id)
.window(GlobalWindows.create()) // assume this is a requirement
.trigger(CountTrigger.of(1))
.apply {
(row1, row2) => // ...
}
.print()
Result:
Everything is printed as expected, each record from the first stream joined with a record from the second one.
However:
If I re-send one of the records (say, with an updated field) from one of the stream to that stream, two duplicate join events get emitted 😞
If I repeat that operation (with or without updated field), I will get 3 emitted events, then 4, 5, etc... 😞
Could someone in the Flink community explain why this is happening? I would have expected only 1 event emitted each time. Is it possible to achieve this with a global window?
In comparison, the Flink Table API behaves as expected in that same scenario, but for my project I'm more interested in the DataStream API.
Example with Table API, which worked as expected:
tableEnv
.sqlQuery(
"""
|SELECT *
| FROM stream1
| JOIN stream2
| ON stream1.id = stream2.id
""".stripMargin)
.toRetractStream[Row]
.filter(_._1) // just keep the inserts
.map(...)
.print() // works as expected, after re-sending updated records
Thank you,
Nicolas
The issue is that records are never removed from your global window. So you trigger the join operation on the global window, whenever a new record has arrived, but the old records are still present.
Thus, to get it running in your case, you'd need to implement a custom evictor. I expanded your example in a minimal working example and added the evictor, which I will explain after the snippet.
val data1 = List(
(1L, "myId-1"),
(2L, "myId-2"),
(5L, "myId-1"),
(9L, "myId-1"))
val data2 = List(
(3L, "myId-1", "myValue-A"))
val stream1 = env.fromCollection(data1)
val stream2 = env.fromCollection(data2)
stream1.join(stream2)
.where(_._2).equalTo(_._2)
.window(GlobalWindows.create()) // assume this is a requirement
.trigger(CountTrigger.of(1))
.evictor(new Evictor[CoGroupedStreams.TaggedUnion[(Long, String), (Long, String, String)], GlobalWindow](){
override def evictBefore(elements: lang.Iterable[TimestampedValue[CoGroupedStreams.TaggedUnion[(Long, String), (Long, String, String)]]], size: Int, window: GlobalWindow, evictorContext: Evictor.EvictorContext): Unit = {}
override def evictAfter(elements: lang.Iterable[TimestampedValue[CoGroupedStreams.TaggedUnion[(Long, String), (Long, String, String)]]], size: Int, window: GlobalWindow, evictorContext: Evictor.EvictorContext): Unit = {
import scala.collection.JavaConverters._
val lastInputTwoIndex = elements.asScala.zipWithIndex.filter(e => e._1.getValue.isTwo).lastOption.map(_._2).getOrElse(-1)
if (lastInputTwoIndex == -1) {
println("Waiting for the lookup value before evicting")
return
}
val iterator = elements.iterator()
for (index <- 0 until size) {
val cur = iterator.next()
if (index != lastInputTwoIndex) {
println(s"evicting ${cur.getValue.getOne}/${cur.getValue.getTwo}")
iterator.remove()
}
}
}
})
.apply((r, l) => (r, l))
.print()
The evictor will be applied after the window function (join in this case) has been applied. It's not entirely clear how your use case exactly should work in case you have multiple entries in the second input, but for now, the evictor only works with single entries.
Whenever a new element comes into the window, the window function is immediately triggered (count = 1). Then the join is evaluated with all elements having the same key. Afterwards, to avoid duplicate outputs, we remove all entries from the first input in the current window. Since, the second input may arrive after the first inputs, no eviction is performed, when the second input is empty. Note that my scala is quite rusty; you will be able to write it in a much nicer way. The output of a run is:
Waiting for the lookup value before evicting
Waiting for the lookup value before evicting
Waiting for the lookup value before evicting
Waiting for the lookup value before evicting
4> ((1,myId-1),(3,myId-1,myValue-A))
4> ((5,myId-1),(3,myId-1,myValue-A))
4> ((9,myId-1),(3,myId-1,myValue-A))
evicting (1,myId-1)/null
evicting (5,myId-1)/null
evicting (9,myId-1)/null
A final remark: if the table API offers already a concise way of doing what you want, I'd stick to it and then convert it to a DataStream when needed.
I'm working on building a Google Sheets-based tool to calculate the cost of making various machined and fabricated parts. As it currently sits, there are about 60 different variables that I modify each time I build an estimate. Things like "number of parts," "length of bar to cut each part from," "cost/bar," "machining time," "machining rate," etc. All of these values I have populated on one sheet, and laid out in a way like. I want to make a button that takes a "snapshot" of all of these values, and stores them on another sheet for later reference. I'd then, ideally create another button, that allows me to re-populate all of the cells based off of a unique ID (such as Part #). This would let me tweak an estimate, or even refer back to material sizes etc in a meaningful way.
So far, I've created a "Named Range" for each of the values, so that as I change the layout, or add values, my script code should update accordingly, instead of using direct cell references.
I've built a few functions to get and set the value's of these named ranges. They're working as expected(i think) for what I'm trying to do. But when I try to place the array of Named Ranges inside of a multi-dimensional array of the named ranges WITH their respective values, I'm running into an issue where each named range is a ROW and their respective value is a second Column. And I need it swapped
I'm not super comfortable with multi-dimensional arrays and am thinking myself in circles trying to figure out how to transpose this logically. My gut says the way I'm attempting to build the arrays is my problem, not just how I'm iterating through them.
function saveCurrentValues(){
//set master spreadhseet
var ss = SpreadsheetApp.getActiveSpreadsheet();
//set calc and save sheets to vars
var calcSheet = ss.getSheetByName('Part Cost Calculator')
var saveSheet = ss.getSheetByName('Saved Parts');
//set named ranges from calcSheet to array
var namedRanges = calcSheet.getNamedRanges();
var savedValues = new Array();
//find next available row for save data (currently troubleshooting)
var nextAvailSaveRange = saveSheet.getRange(1, 1, 60, 2);
//iterate through array and call getNamedRange() function to return name and current value
for(i = 0; i < namedRanges.length; i++){
savedValues[i] = getNamedRange(namedRanges[i].getName());
}
nextAvailSaveRange.setValues(savedValues);
}
function getNamedRange(name){
var ss = SpreadsheetApp.getActiveSheet();
var value = ss.getRange(name).getValue();
Logger.log([name,value]);
return [name, value];
}
As you can see by how I had to temporarily format the nextAvailSaveRange, it needs 60 ROWS, and only two columns, because of how the array is constructed. I'd like to better understand how I'm creating this multi-dimensional array vertically instead of horizontally, and how to fix it!
Once this is done, I'd like to create headers that match the Named Ranges on my save sheet, to allow me to iterate through functions and look for a match to the appropriate column by name. That way if I add more values or change their order, or the order of the array, it wont matter. I think I'll be able to figure that out pretty easily if I can control these damn arrays better!
I agree with the OP. Array building AND iteration are the immediate problems and they are the stumbling block to the development of the spreadsheet.
The OP has a raised number of issues, however the most immediate, and the one to be resolved under this answer, is the copying of a list of parts from one sheet to another. In the OP's code, named ranges were retrieved and used as a basis for creating the copy of the list of parts. However, this also creates a duplicate set of named ranges on the target sheet. In my view this was unnecessarily complicating the duplication of the parts list since it is easy to programmatically create/update a list of named ranges.
The following code consists of three functions:
so_5466573501() - Copies the list of parts from one sheet to another.
Named Ranges are ignored; the OP's stumbling block is the iteration of the raw data and management of arrays. This code deals only with that aspect as a means of simplifying this issue.
createnamedranges() - Programmatically creates/updates Named ranges.
This code is included to assure the OP that it is not important to make named ranges the focus of the duplication by showing how easy it is to programmatically turn a list of parts into a series of Named Ranges (for development, I created 60 Parts and the entire code executes in under a 1 second). The code assumes a list in two columns (Column A = Parameter Name, Column B = Parameter value). The code loops through the list creating/updating a set of named ranges - the range name is the Parameter Name in Column A, and the range itself is the the corresponding row in Column B. The name of the sheet is set in a variable, so this function can be easily adapted.
deletenamedranges() - Programmatically deletes Named ranges.
This code deletes all the Named Ranges from a given sheet. This function is included because the OP's existing code creates duplicate named ranges, and it might be necessary to quickly delete them from a sheet. The sheet name is stored as a variable, so the function can be easily adapted.
function so_5466573501() {
//set master spreadsheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
//create variables for calc and save sheets
var calcSheet = ss.getSheetByName('Part Cost Calculator')
var saveSheet = ss.getSheetByName('Saved Parts');
//get the Parts Parameters from Part Cost Calculator
//var namedRanges = calcSheet.getNamedRanges();
//Logger.log("DEBUG: Number of named ranges on Parts Cost Calculator = "+namedRanges.length);
// get the number of parts in the list on Parts Cost Calculator
var Avals = calcSheet.getRange("A1:A").getValues();
var Alast = Avals.filter(String).length;
//Logger.log("DEBUG: Number of parts in the list: "+Alast); //DEBUG
// get the parts list
var partsRange = calcSheet.getRange(1, 1, Alast, 2);
var partsRangeValues = partsRange.getValues();
//Logger.log("DEBUG: The parts range is: "+partsRange.getA1Notation());//DEBUG
//Logger.log("DEBUG: Parts List Row #1: Name: "+partsRangeValues[0][0]+", Value: "+partsRangeValues[0][1]);//DEBUG
// create an array to use for saving results and updating new Saved Parts sheet
var savedValues = new Array();
// Loop through the Parts List, row by row
for (i = 0; i < Alast; i++) {
// push the part name and part value onto the array
savedValues.push([partsRangeValues[i][0], partsRangeValues[i][1]]);
//Logger.log("DEBUG: Parts List: i = "+i+", Name: "+partsRangeValues[i][0]+", Value: "+partsRangeValues[i][1]);//DEBUG
}
// identify the range on the Saved Parts sheet to copy the parts list array.
var saveRange = saveSheet.getRange(1, 1, Alast, 2);
saveRange.setValues(savedValues);
}
function createnamedranges() {
//set master spreadhseet
var ss = SpreadsheetApp.getActiveSpreadsheet();
//create variables for calc and save sheets
var calcSheetName = "Part Cost Calculator";
var calcSheet = ss.getSheetByName(calcSheetName);
// get the number of parts in the list on Parts Cost Calculator
var AVals = calcSheet.getRange("A1:A").getValues();
var ALast = AVals.filter(String).length;
// get the parts range and values
var partsRange = calcSheet.getRange(1, 1, ALast, 2);
//Logger.log("DEBUG: The Parts range is "+partsRange.getA1Notation());//DEBUG
var partsRangeValues = partsRange.getValues();
// Loop through the parts list row by row
for (var i = 0; i < ALast; i++) {
// get the Part name and assign as the range name
var nrpartname = partsRangeValues[i][0];
//Logger.log("DEBUG: PartName = "+nrpartname+", value: "+partsRangeValues[i][1]);//DEBUG
// get the range to be named -note (i+1) because the loop starts at 0 (zero) but `getrange` starts at 1 (one)
var rng_to_name = ss.getSheetByName(calcSheetName).getRange((i + 1), 2);
//Logger.log("DEBUG: rng_to_name: "+rng_to_name+", range details: "+rng_to_name.getA1Notation());
// set (and/or update) the named range
ss.setNamedRange(nrpartname, rng_to_name);
// DEBUG: check that the range was created //DEBUG
// var rangeCheck = ss.getRangeByName(nrpartname);//DEBUG
// var rangeCheckName = rangeCheck.getA1Notation(); //DEBUG
// Logger.log("DEBUG: Rangename: "+nrpartname+", Range: "+rangeCheckName);//DEBUG
// credit megabyte1024 https://stackoverflow.com/a/12325103/1330560 "setNamedRange() outside of the spreadsheet container?"
}
}
function deletenamedranges() {
//set master spreadhseet
var ss = SpreadsheetApp.getActiveSpreadsheet();
//create variables for calc and save sheets
var calcSheet = ss.getSheetByName('Part Cost Calculator');
// get the named ranges
var namedRanges = calcSheet.getNamedRanges();
// loop through the list of named ranges and delete them
for (var i = 0; i < namedRanges.length; i++) {
namedRanges[i].remove();
}
}
ADDENDUM: - Copy based on Named Ranges
The original so_5466573501 assumes that the parts are in a simple 2 column-list; in which case, Named Ranges are irrelevant.
The following code assumes that the parts are not in a list but scattered, in no particular order, throughout the sheet "Part Cost Calculator". This code is based on obtaining the NamedRanges, identifying the respective Named Range row and column, correlating said row and column to the ENTIRE data range, and then copying the results to the "Saved Parts" sheet. No Named Ranges are created by default on the "Saved Parts" sheet but this can be easily done by using the createnamedranges function (appropriately edited for the correct sheet name).
function so_5466573502() {
//set master spreadhseet
var ss = SpreadsheetApp.getActiveSpreadsheet();
//create variables for calc and save sheets
var calcSheet = ss.getSheetByName('Part Cost Calculator')
var saveSheet = ss.getSheetByName('Saved Parts');
//get the Parts Parameters from Part Cost Calculator
var namedRanges = calcSheet.getNamedRanges();
var numNR = namedRanges.length
//Logger.log("DEBUG: Number of named ranges on Parts Cost Calculator = "+numNR);
// get all the data
var dataRangeValues = calcSheet.getDataRange().getValues();
// create an array to temporarily store results
var resultsarray = [];
// Loop through the array of Named Ranges
for (var x = 0; x < numNR; x++) {
var nrName = namedRanges[x].getName();
var nrRange = namedRanges[x].getRange();
var nrRangerow = nrRange.getRow();
var nrRangecol = nrRange.getColumn();
var nrRangeValue = dataRangeValues[nrRangerow - 1][nrRangecol - 1];
//Logger.log("DEBUG: Named Range-Name: "+nrName+", Range: "+nrRange.getA1Notation()+", Row: "+nrRangerow+", Column: "+nrRangecol+", Value-"+nrRangeValue);//DEBUG
// populate the array with the part name and the part value
resultsarray.push([nrName, nrRangeValue]);
}
// identify the range on the Saved Parts sheet to copy the parts list array.
var saveRange = saveSheet.getRange(1, 1, numNR, 2);
saveRange.setValues(resultsarray);
// sort the results on "Saved Parts"
saveRange.activate().sort({
column: 1,
ascending: true
});
}
Im new in matlab, Im trying to call a function inside a matlab GUI but I keep getting an error. Can you please help.
function PushMe_Callback(hObject, eventdata, handles)
set(handles.output_line,'String','I Got here');
Get_Length(Note_Vector,output,Counter,Total_num)
%------------------------------------------------------------
function Get_Length(Note_Vector,output,Counter,Total_num)
output = [ ];
while (1)
T = Total_num - Counter;
for j=1:T
[xml_input]=Get_Note(Note_Vector(j));
output = [output xml_input];
end
end
You have to get your data from the gui data. You can retrieve these data from the function "guidata".
In fact what happens, is that you define a variable in the "main function" but you cannot access this one.
What you should do is to add this variable in the handles in the "GUI_OpeningFcn" function:
Note_Vector = [ ];
output = [ ];
Counter = 0
for i=1:Total_num
Note_Vector = [Note_Vector Note+Shift];
end
handles.Note_Vector = Note_Vector;
handles.GLoutput = output; %handles.output is already the handle of your GUI, don't change it.
handles.counter=counter;
handles.Total_num=Total_num;
(...)
guidata(hObject, handles);
then you can either retrieve these data in PushMe_Callback by simply writing handles.Note_Vector OR you can retrieve them by doing the following in Get_Length function
handles = guidata(handle). In that case you'll have to put in input the handle of your GUI
Get_Length(Note_Vector,output,Counter,Total_num,handles.output)
I let you check the matlab help file concerning guidata() if you want more information.
in my one of my callbacks, i load a wav file then plot it in an axes.
in another callback, i try to play the wav file but its not working.
please help, how can i play it??
function btnload_Callback(hObject, eventdata, handles)
[filename, pathname] = uigetfile('*.wav','Select Data File');
[y,fs] = wavread([pathname filename]);
axes(handles.axes1)
plot(y);
title('ORIGINAL AUDIO SIGNAL');
ylabel('t');
guidata(hObject, handles); %updates the handles
function btnplay1_Callback(hObject, eventdata, handles)
soundsc(y,fs);
If you need to maintain application data in GUIDE-generated GUIs, one way is to use the handles structure which gets passed around to all callback functions:
function btnload_Callback(hObject, eventdata, handles)
%# read WAV file
[filename, pathname] = uigetfile('*.wav','Select Data File');
[y,fs] = wavread([pathname filename]);
%# plot wave
axes(handles.axes1)
plot(y);
title('ORIGINAL AUDIO SIGNAL');
ylabel('t');
%# save it to handles structure
handles.y = y;
handles.fs = fs;
guidata(hObject, handles); %# updates the handles
end
function btnplay1_Callback(hObject, eventdata, handles)
%# retrieve the wave data, and play the sound
soundsc(handles.y, handles.fs);
end
I have images of various formats (.png, .jpg, .bmp, etc.) stored as compressed text in a text column in a SQL Server 2005 table. I need to read the row, unzip the image and store it in an image column in another table.
I am using the SharpZip library, and all of the examples deal with file sources and destinations. I can't find anything that covers unzipping from a variable to another variable. A code snippet illustrating this or a link to a relevant resource would be much appreciated.
EDIT: A bit more information - the data is stored in a TEXT column. It appears as follows (text column abbreviated for display):
ImageID ImageData
1 FORMAT-ZIPV3 UEsDBBQAAAAIAOV6wzxdTnDvshs...
2 FORMAT-ZIPV3 UEsDBBQAAAAIAAF2yjxGncjOLgA...
3 FORMAT-ZIPV3 UEsDBBQAAAAIAKd6yjyjnQNr6gg...
4 FORMAT-ZIPV3 UEsDBBQAAAAIALdNyzyrPC8EMJw...
5 FORMAT-ZIPV3 UEsDBBQAAAAIAA1rOD1nZY1t0f0...
6 FORMAT-ZIPV3 UEsDBBQAAAAIANZplj2seyJ+VmM...
7 FORMAT-ZIPV3 UEsDBBQAAAAIAC5vhD27LPbPcv8...
8 FORMAT-ZIPV3 UEsDBBQAAAAIAK1qKz5DJNH3xMg...
9 FORMAT-ZIPV3 UEsDBBQAAAAIAHVkEztC3th/9hs...
10 FORMAT-ZIPV3 UEsDBBQAAAAIAEtXKz7DXHUdvow...
What I know for certain is that the images were compressed at some point in the process using SharpZip before being inserted into the table. It appears that the format information was added to the beginning of the data prior to inserting.
Looking at this data, would anyone have any insight on how this image data has been manipulated? Again, I need to get the uncompressed image data into a column of a data type conducive to reading for display on a web page.
EDIT: Ok, I'm stumped. Executing the following code produces the error, "Failed to convert parameter value from a Int32 to a Byte[]". It appears to be placing the length of the byte array into the byte array's value...
commandUncompressed.Connection = connectionUncompressed;
commandUncompressed.Parameters.Add("#Image_k", SqlDbType.VarChar, 10);
commandUncompressed.Parameters.Add("#ImageContents", SqlDbType.Image);
commandUncompressed.CommandText = sqlSaveImage;
connectionUncompressed.Open();
reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
Console.WriteLine(reader["Image_k"].ToString()); // Merely for testing
String format = reader["ImageContents_Compressed"].ToString().Substring(0, 12);
var offset = 13; //"FORMAT-ZIPV3 ".Length;
var s = reader["ImageContents_Compressed"].ToString().Substring(offset);
var bytes = Convert.FromBase64String(s);
if (format == "FORMAT-ZIPV2 ")
{
bytes = ConvertStringToBytes(s); // Not a Base-64 encoded string? External conversion function utilized.
}
using (var zis = new ZipInputStream(new MemoryStream(bytes)))
{
ZipEntry zipEntry = zis.GetNextEntry(); // Doesn't seem to work unless an entry has been referenced
byte[] buffer = new byte[zis.Length];
commandUncompressed.Parameters["#Image_k"].Value = reader["Image_k"].ToString();
commandUncompressed.Parameters["#ImageContents"].Value = zis.Read(buffer, 0, buffer.Length);
commandUncompressed.ExecuteNonQuery();
}
}
}
It appears to be reading the data from the source text column just fine. I just cannot figure out how to get that into the image type parameter. The value for buffer variable shows the length of the byte array, rather than the actual bytes. Maybe that's what the value property typically shows for byte arrays? I'm so close and yet so far away. :/
EDIT: Ok, I'm a knucklehead. I made the following correction, and it works!
zis.Read(buffer, 0, buffer.Length)
commandUncompressed.Parameters["#ImageContents"].Value = buffer;
At this point I am only able to process FORMAT-ZIPV3 data, as I haven't figured out how to decode the FORMAT-ZIP2 strings yet. Following is a sampling of the V2 data. If anyone is able to determine the encoding, let me know. Would it be different if zipped using BZIP instead of ZIP format?
ImageID ImageData
1 FORMAT-ZIPV2 504B03041400020008005157422A2E25FDBAF26701008D6901000E...
2 FORMAT-ZIPV2 504B03041400020008009159422A7FC94BA2B2540500D35705000E...
3 FORMAT-ZIPV2 504B0304140002000800685A422A0CAA51F4473A0600B97206000E...
4 FORMAT-ZIPV2 504B03041400020008001D5D422A770BD3ED201902002C4A02000E...
5 FORMAT-ZIPV2 504B0304140002000800325E422A4B6C2FB4045001001C6E01000E...
6 FORMAT-ZIPV2 504B03041400020008006F72422A5F793AC1A1F00200ECF302000E...
7 FORMAT-ZIPV2 504B0304140002000800D572422A1B348A731DE5000085EB00000E...
8 FORMAT-ZIPV2 504B03041400020008003D73422A8AEBB7F855640300DD1B04000E...
9 FORMAT-ZIPV2 504B03041400020008006368D528C5D0A6BA794900004A2502000E...
10 FORMAT-ZIPV2 504B03041400020008008E5B6C2A2D9E9C33D7AF05005CEC05000E...
In response to a similar question, someone on sqlmonster.com provided a nifty VarBinaryStream class. It works with a column type of varbinary(max).
If your data is stored in a varbinary(max), and is in zip format, you could use that class to instantiate a VarBinaryStream, then instantiate a ZipInputStream around that, and ba-da-boom, you're there. Just read from the ZipInputStream.
In C# it might look like this
using (var imageSrc = new VarBinarySource(connection,
"Table.Name",
"Column",
"KeyColName",
1))
{
using (var s = new VarBinaryStream(imageSrc))
{
using(var zis = new ZipInputStream(s))
{
....
}
}
}
If the images are small, then you probably wouldn't want all this streaming stuff. If the column is a binary(n) or a varbinary(n) where n is less than 8000, just use the SqlBinary type and read in all the data into memory, then instantiate a MemoryStream around that. Simpler. In VB.NET it looks something like this:
Dim bytes as Bytes()
bytes = dr.GetSqlBinary(columnNumber)
Using ms As New MemoryStream(bytes)
Using zis As New ZipInputStream(ms)
...
End Using
End Using
Finally, I'm going to question the wisdom of applying zip compression to .jpg images, and similar. The jpg format is already compressed; compressing it again before putting the data into SQL Server won't cause the data to become appreciably smaller. It only increases processing time. If possible, I'd suggest you reconsider your design for storage of compressed images.
ok, with the update you provided, containing the data format, you can draw some conclusions.
The data is an actual string. Suspecting that it is a Base64-encoded string, I did a small test and used Convert.ToBase64String() on a byte stream that contains a zip file. It looks like this: UEsDBBQAAAAIAJJyYyk3M56F+QIAA...
Aha! you have a base64-encoded (string) version of the byte data for a bonafide zip file. To decode it, strip the prefix and then use FromBase64String() to get the byte array, insert into a MemoryStream, then read it with ZipInputStream.
something like this:
var offset = "FORMAT-ZIPV3 ".Length();
var s = sqlReader["CompressedImage"].ToString().Substring(offset);
var bytes = Convert.FromBase64String(s);
using (var zis = new ZipInputStream(new MemoryStream(bytes)))
{
...
zis.Read(...);
...
}
If the data is "really long", you're going to want to stream it out of that table, rather than just read it into a big string and convert it. I don't know how large text columns can be, but supposing that it could be 500mb, you don't want a 500mb string, and you don't want to do a conversion of a 500mb string with Convert.FromBase64String(). In that case You need to use a Base64Stream, or the FromBase64Transform class in the System.Security.Cryptography namespace.
Editorial comment. It is sort of backwards to zip-compress image data. The images are probably compressed already. But to compound that backwardsness by then doing a base64 encode, thereby expanding the data... ??? That is triple backwards. That makes noooooo sense at all. I understand that's how your vendor supplied it.
Ok, with your furhter update, using this as the format:
ImageID ImageData
1 FORMAT-ZIPV2 504B03041400020008005157422A2E25FDBAF26701008D6901000E...
2 FORMAT-ZIPV2 504B03041400020008009159422A7FC94BA2B2540500D35705000E...
That data is still zipfile data, but it is encoded as simple hex digits. You need to convert that to a byte array. Here's some code to do it.
public static class ConvertEx
{
static readonly String prefix= "FORMAT-ZIPV2 ";
public static string ToHexString(byte[] b)
{
System.Text.StringBuilder sb1 = new System.Text.StringBuilder();
int i = 0;
for (i = 0; i < b.Length; i++)
{
sb1.Append(System.String.Format("{0:X2}", b[i]));
}
return sb1.ToString().ToLower();
}
public static byte[] ToByteArray(string s)
{
if (s.StartsWith(prefix))
{
System.Console.WriteLine("removing prefix");
s = s.Substring(prefix.Length);
}
s= s.Trim(); // whitespace
System.Console.WriteLine("length: {0}", s.Length);
var r= new byte[s.Length/2];
for (int i = 0; i < s.Length; i+=2)
{
r[i/2] = (byte) Convert.ToUInt32(s.Substring(i,2), 16);
}
return r;
}
}
You can use that this way:
string s = GetStringContentFromDatabase()
var decoded = ConvertEx.ToByteArray(s);
using (var ms = new MemoryStream(decoded))
{
// use DotNetZip to read the zip file
// SharpZipLib is something similar...
using (var zip = ZipFile.Read(ms))
{
// print out the list of entries in the zipfile
foreach (var e in zip)
{
System.Console.WriteLine("{0}", e.FileName);
}
}
}
The examples on the SharpZip Wiki use Stream objects - while the sample does use a File, you could easily use a MemoryStream object here and the sample would work the same.