I have a problem with the ReactiveGridFsTemplate. I am trying to read a GridFS file written with the old GridFS (com.mongodb.gridfs) instead of the new GridFS (com.mongodb.client.gridfs.model.GridFS) with an UUID as an ID instead of the ObjectId. Reading the GridFS file metainfo goes fine, but as soon as I want to get the ReactiveGridFsResource it blows with a nice new MongoGridFSException("Custom id type used for this GridFS file").
The culprit is the code below from ReactiveGridFsTemplate which calls the getObjectId() instead of the getId(). Should it call this method or can that be rewritten to the getId() method?
public Mono<ReactiveGridFsResource> getResource(GridFSFile file) {
Assert.notNull(file, "GridFSFile must not be null!");
return Mono.fromSupplier(() -> {
GridFSDownloadStream stream = this.getGridFs().openDownloadStream(file.getObjectId());
return new ReactiveGridFsResource(file, BinaryStreamAdapters.toPublisher(stream, this.dataBufferFactory));
});
}
I hacked the ReactiveGridFsTemplate to use getId() instead of getObjectId() but now it gives me a stackoverflow exception. Can someone tell me what I'm doing wrong?
ReactiveGridFsTemplate reactiveGridFsTemplate = new ReactiveGridFsTemplate(mongoDbDFactory, operations.getConverter(), "nl.loxia.collectie.buitenlandbladen.dgn", 1024) {
public Mono<ReactiveGridFsResource> getResource(GridFSFile file) {
Assert.notNull(file, "GridFSFile must not be null!");
return Mono.fromSupplier(() -> {
GridFSDownloadStream stream = this.getGridFs().openDownloadStream(file.getId());
return new ReactiveGridFsResource(file, BinaryStreamAdapters.toPublisher(stream, this.dataBufferFactory));
});
}
};
var q = Query.query((Criteria.where("_id").is("5449d9e3-7f6d-47b7-957d-056842f190f7")));
List<DataBuffer> block = reactiveGridFsTemplate
.findOne(q)
.flatMap(reactiveGridFsTemplate::getResource)
.flux()
.flatMap(ReactiveGridFsResource::getDownloadStream)
.collectList()
.block();
The stacktrace: https://gist.github.com/nickstolwijk/fa77681572db1d91941d85f6c845f2f4
Also, this code hangs due to the stackoverflow exception. Is that correct?
Related
In the Flutter/Dart app that I am currently working on need to download large files from my servers. However, instead of storing the file in local storage what I need to do is to parse its contents and consume it one-off. I thought the best way to accomplish this was by implementing my own StreamConsumer and overriding the relvant methods. Here is what I have done thus far
import 'dart:io';
import 'dart:async';
class Accumulator extends StreamConsumer<List<int>>
{
String text = '';
#override
Future<void> addStream(Stream<List<int>> s) async
{
print('Adding');
//print(s.length);
return;
}
#override
Future<dynamic> close() async
{
print('closed');
return Future.value(text);
}
}
Future<String> fileFetch() async
{
String url = 'https://file.io/bse4moAYc7gW';
final HttpClientRequest request = await HttpClient().getUrl(Uri.parse(url));
final HttpClientResponse response = await request.close();
return await response.pipe(Accumulator());
}
Future<void> simpleFetch() async
{
String url = 'https://file.io/bse4moAYc7gW';
final HttpClientRequest request = await HttpClient().getUrl(Uri.parse(url));
final HttpClientResponse response = await request.close();
await response.pipe(File('sample.txt').openWrite());
print('Simple done!!');
}
void main() async
{
print('Starting');
await simpleFetch();
String text = await fileFetch();
print('Finished! $text');
}
When I run this in VSCode here is the output I get
Starting
Simple done!! //the contents of the file at https://file.io/bse4moAYc7gW are duly saved in the file
sample.txt
Adding //clearly addStream is being called
Instance of 'Future<int>' //I had expected to see the length of the available data here
closed //close is clearly being called BUT
Finished! //back in main()
My understanding of the underlying issues here is still rather limited. My expectation
I had thought that I would use addStream to accumulate the contents being downloaded until
There is nothing more to download at which point close would be called and the program would display exited
Why is addStream showing instance of... rather than the length of available content?
Although the VSCode debug console does display exited this happens several seconds after closed is displayed. I thought this might be an issue with having to call super.close() but not so. What am I doing wrong here?
I was going to delete this question but decided to leave it here with an answer for the benefit of anyone else trying to do something similar.
The key point to note is that the call to Accumulator.addStream does just that - it furnishes a stream to be listened to, no actual data to be read. What you do next is this
void whenData(List<int> data)
{
//you will typically get a sequence of one or more bytes here.
for(int value in data)
{
//accumulate the incoming data here
}
return;
}
function void whenDone()
{
//now that you have all the file data *accumulated* do what you like with it
}
#override
Future<void> addStream(Stream<List<int>> s) async
{
s.listen(whenData,onDone:whenDone);
//you can optionally ahandler for `onError`
}
I have implemented creating a folder and file in the device's external storage and writing data into that file using this thread.
Now I am trying to get the details of the file. For that, I have added a new function in the interface like below.
//Interface
public interface IAccessFile
{
void CreateFile(string text);
Java.IO.File GetFileDetails();
}
//Android implementation
public Java.IO.File GetFileDetails()
{
string rootPath = Android.OS.Environment.ExternalStorageDirectory.AbsolutePath;
var filePathDir = Path.Combine(rootPath, "StockPDT");
if (File.Exists(filePathDir))
{
string filePath = Path.Combine(filePathDir, "STOCK.TXT");
Java.IO.File file = new Java.IO.File(filePath);
return file;
}
else
{
return null;
}
}
But the problem is with the interface function part, getting below error":
The type or namespace name 'Java' could not be found (are you missing a using directive or an assembly reference?)
Screenshot:
If I return the file from the android part like above, it is easy to get the file details in the portable project. Instead of File, I try to return the file path, but it is empty. Is there any other way to get the file details other than this?
As per the notes in question, I tried to fetch the file details using its path.
Reference: https://learn.microsoft.com/en-us/answers/questions/319908/xamrin-forms-how-to-read-the-details-of-a-file-sto.html
//Interface
public interface IAccessFile
{
string GetFileDetails();
}
//Android implementation
public string GetFileDetails()
{
string rootPath = Android.OS.Environment.ExternalStorageDirectory.AbsolutePath;
var filePathDir = Path.Combine(rootPath, "StockPDT");
if (!File.Exists(filePathDir))
{
Directory.CreateDirectory(filePathDir);
}
string filePath = Path.Combine(filePathDir, "STOCK.TXT");
return filePath;
}
//Main Project:
string path = DependencyService.Get<IAccessFile>().GetFileDetails();
string fileDetails = File.ReadAllText(path);
I send a document over socket like this:
sendFXML(asByteArray(getRequiredScene(fetchSceneRequest())));
private void sendFXML(byte[] requiredFXML) throws IOException, TransformerException {
dataOutputStream.write(requiredFXML);
dataOutputStream.flush();
}
private Document getRequiredScene(String requiredFile) throws IOException, ParserConfigurationException, SAXException, TransformerException {
return new XMLLocator().getDocumentOrReturnNull(requiredFile);
}
private String fetchSceneRequest() throws IOException, ClassNotFoundException {
return dataInputStream.readUTF();
}
On the side of XMLLocator it finds the correct document and parses it right. I see it by printing the whole doc in console.
But I cannot handle it on the clients side where it's fetch by:
public static void receivePage() throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] data = new byte[989898];
int bytesRead = -1;
while((bytesRead = dataInputStream.read(data)) != -1 ) { //stops here
baos.write(data, 0, bytesRead );
}
Files.write(Paths.get(FILE_TO_RECEIVED), data);
}
After the first iteration in while() cycle it just stops on the commented place.
I don't know if I have an error on the side of the server and I send this in doc in an incorrect format or I read the sent byte array incorrectly. Where is the problem?
Edit:
For the debug purpose, in the receivePage() method, I've chosen a different way of reading the byte array from server which goes like:
int count = inputStream.available();
byte[] b = new byte[count];
int bytes = dataInputStream.read(b);
System.out.println(bytes);
for (byte by : b) {
System.out.print((char)by);
}
And now I'm able to print fetched FXLM in console but a new problem has appeared.
On debug, it normally receives the byte[] from server, writes 2024 for count and displayes the content of the file but if I run the app normally via Shift + f10 it fetches nothing and just writes 0 in console
Edit2:
For some reason, once again, on debug, it's able to even write into a file
for (byte by : b) {
Files.write(Paths.get(FILE_TO_RECEIVED), b);
System.out.print((char)by);
}
But when I try to return this fxml on debug and then show like this:
Parent fxmlToShow = FXMLLoader.load(getClass().getResource("/network/gui.fxml"));
Scene childScene = new Scene(fxmlToShow);
Stage window = (Stage)((Node)ae.getSource()).getScene().getWindow();
window.setScene(childScene);
return window;
It shows only previous files. Like on the first attempt of debug it show a blank page when I asked for the 1st one from server. On the second attempt of debug when i ask for 3rd page from server, it shows me the previously asked one and so on.
To me, it seems absolutely insane cuz the fxml rile actually refreshes before the line
Parent fxmlToShow = FXMLLoader.load(getClass().getResource("/network/gui.fxml"));
is invoked.
Yeah, thank everybody for participating.
So, the issue of incorrect displaying if FXML files was caused by the incorrect FILE_TO_RECEIVED path.
When FXMLLoader.load(getClass().getResource("/network/gui.fxml")); loads gui.fxml it takes it not from D:\\JetBrains\\IdeaProjects\\Client\\src\\network\\gui.fxml,im my case, but from D:\\JetBrains\\IdeaProjects\\Client\\OUT\\PRODUCTION\\Client\\network\\gui.fxml.
As for me, that doesn't seem obvious.
What about different behaviour on debug and on run. In method receivePage() it needs to wait until connection is available.
int count = inputStream.available();
If you read docs for this method you will see
Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream ...
The available method for class InputStream always returns 0...
So, you jext need to wait for connection to be available
while(inputStream.available()==0){
Thread.sleep(100);
}
Otherwise it just prepares byte[] b = new byte[count]; for 0 bytes and you can write in nothing.
I am trying to use FileService to create a file on Blobstore.
Code look as follows:
public static BlobKey save(String mimeType, String value, String filename) throws IOException
{
FileService svc = FileServiceFactory.getFileService();
AppEngineFile file = filename == null ? svc.createNewBlobFile(mimeType) : svc.createNewBlobFile(mimeType, filename);
key = svc.getBlobKey(file); //throws exception
}
But I get the following exception:
java.lang.IllegalArgumentException: creation_handle: String properties must be 500 characters or less.
Instead, use com.google.appengine.api.datastore.Text, which can store strings of any length.
at com.google.appengine.api.datastore.DataTypeUtils.checkSupportedSingleValue(DataTypeUtils.java:242)
at com.google.appengine.api.datastore.DataTypeUtils.checkSupportedValue(DataTypeUtils.java:207)
at com.google.appengine.api.datastore.DataTypeUtils.checkSupportedValue(DataTypeUtils.java:173)
at com.google.appengine.api.datastore.Query$FilterPredicate.<init>(Query.java:900)
at com.google.appengine.api.datastore.Query$FilterOperator.of(Query.java:75)
at com.google.appengine.api.datastore.Query.addFilter(Query.java:351)
at com.google.appengine.api.files.FileServiceImpl.getBlobKey(FileServiceImpl.java:329)
Fixed the problem.
I had to ensure the following:
I call the openWriteChannel
Write some content (optional)
Close the channel (close or closeFinally)
And then call getBlobKey
This ensures that getCachedKey method is called (I guess so) or createHandle is available.
I was, earlier, trying to getBlobKey before writing content.
In a nutshell, since GAE cannot write to a filesystem, I have decided to persist my data into the datastore (using JDO). Now, I will like to retrieve the data byte by byte and pass it to the client as an input stream. There's code from the gwtupload library(http://code.google.com/p/gwtupload/) (see below) which breaks on GAE because it writes to the system filesystem. I'll like to be able to provide a GAE ported solution.
public static void copyFromInputStreamToOutputStream(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[100000];
while (true) {
synchronized (buffer) {
int amountRead = in.read(buffer);
if (amountRead == -1) {
break;
}
out.write(buffer, 0, amountRead);
}
}
in.close();
out.flush();
out.close();
}
One work around I have tried (didn't work) is to retrieve the data from the datastore as a resource like this:
InputStream resourceAsStream = null;
PersistenceManager pm = PMF.get().getPersistenceManager();
try {
Query q = pm.newQuery(ImageFile.class);
lf = q.execute();
resourceAsStream = getServletContext().getResourceAsStream((String) pm.getObjectById(lf));
} finally {
pm.close();
}
if (lf != null) {
response.setContentType(receivedContentTypes.get(fieldName));
copyFromInputStreamToOutputStream(resourceAsStream, response.getOutputStream());
}
I welcome your suggestions.
Regards
Store data in a byte array, and use a ByteArrayInputStream or ByteArrayOutputStream to pass it to libraries that expect streams.
If by 'client' you mean 'HTTP client' or browser, though, there's no reason to do this - just deal with regular byte arrays on your end and send them to/from the user as you would any other data. The only reason to mess around with streams like this is if you have some library that expects them.