Deadline exceeded error when inserting attachments into Google Drive - Java - google-app-engine

When there are more attachments and in the process of adding all into the google drive from the code there is an exception with the below description.
com.google.apphosting.api.DeadlineExceededException
A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)
Using the below code to save the attachments into Google Drive.
public File insertFile(Drive service, String title, String description, String mimeType,String parentId , String filename, byte fileByteArray[],
HttpServletRequest req, HttpServletResponse resp)
throws MessagingException, ParseException, InterruptedException
{
try
{
File body = new File();
body.setTitle(title);
body.setDescription(description);
body.setMimeType(mimeType);
if(parentId != null && parentId.length() > 0)
body.setParents(Arrays.asList(new ParentReference[] {
(new ParentReference()).setId(parentId)
}));
com.google.api.services.drive.Drive.Files.List files = service.files().list();
FileList fileList = (FileList)files.execute();
List fileArray = fileList.getItems();
log.info("fileArray"+fileArray);
log.info("file Array Size"+fileArray.size());
File file=null;
try{
file = (File)service.files().insert(body, new InputStreamContent(mimeType, new ByteArrayInputStream(fileByteArray))).execute();
log.info("-----------File----------"+file);
System.out.println("FileName="+MimeUtility.encodeText(file.getTitle()));
System.out.println("File URL"+file.getAlternateLink());
System.out.println("File ID"+file.getId());
return file;
}
catch(DeadlineExceededException e)
{
log.info((new StringBuilder("An error occured: ")).append(e).toString());
//inserted for attachment saving error
System.out.println("inside exception deadlineexception");
}
catch(Exception e)
{
log.info((new StringBuilder("An error occured: ")).append(e).toString());
}
return null;
}
Can someone suggest a way to optimize the code so that all the attachments say around 10 to 15 can be saved into the google drive without any exception.

Related

Selenium-Extent_Reports: Not able to view the failure screenshots on other Computer/Machine

-Failure Screenshot are visible in Extent_Reports on my local machine. But not able to view the failure screenshot in Extent_Reports on other Computer/Machine.
-When i trigger build from Jenkins, After build successful, Sending email to:Recipient List
To Capture Screenshot
public String captureScreen(String fileName) {
if(fileName =="") {
fileName="Screenshot"; }
File destFile=null;
Calendar calendar =Calendar.getInstance() ;
SimpleDateFormat formater= new SimpleDateFormat("dd_MM_yyy_hh_mm_ss");
File srcFile=((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
try {
String reportDirectory = "/src/main/java/com/test/automation/Demo/screenshot/";
//String reportDirectory= new File(System.getProperty("user.dir")).getAbsolutePath()+"./src/main/java/com/test/automation/Demo/screenshot/";
destFile= new File((String)reportDirectory + fileName +"-" + formater.format(calendar.getTime())+ ".png");
FileUtils.copyFile(srcFile,destFile );
//This will help us to link screen shot in Extent report
Reporter.log("<a href='"+destFile+ "'><img src='" +destFile+"' height='100' width='100'/></a>");
//Reporter.log("<a href='"+destFile.getAbsolutePath()+ "'><img src='" +destFile.getAbsolutePath()+"' height='100' width='100'/></a>");
}
catch(IOException e) {
e.printStackTrace();
}
return destFile.toString();
}
For generating Extent reports with screenshots for Failure test cases
public void getresult(ITestResult result) {
if(result.getStatus()==ITestResult.FAILURE)
{
test.log(LogStatus.ERROR, result.getName()+" Test case FAILED due to below issues: "+result.getThrowable());
String screen = captureScreen("");
test.log(LogStatus.FAIL," Failure Screenshot : "+ test.addScreenCapture(screen));
}}
If You're using remoteWebDriver than it must be augmented before you can use the screenshot capability. Did You try to
WebDriver driver = new RemoteWebDriver();
driver = new Augmenter().augment(driver);
// or for mobile driver
androidDriver.setFileDetector(new LocalFileDetector());
//this is needed when using remoteDriver
Here is how I take screenshot for ExtentReport
File scrFile = driver.getScreenshotAs(OutputType.FILE);
String dest = System.getProperty("user.dir") + "/resources/screenshots/" + dataMethod.getAndroidDriver().getSessionId() + ".png";
File destination = new File(dest);
try {
FileUtils.copyFile(scrFile, destination);
// this is just utility which takes screenshot and copy it to desired destination
dataMethod.setScreenshotPath(destination.getAbsolutePath());
} catch (IOException e) {
e.printStackTrace();
}
And on code failure:
#Override
public synchronized void onTestFailure(ITestResult result) {
setTestEndTime(result);
ExtentTest extentTest = methodData.getExtentTest();
extentTest.addScreenCaptureFromPath(methodData.getScreenshotPath());
}
Hope this will help.
I didn't used Extent reports, i have my own implementation for reports. But i am expecting is there is issue with src
<img src='" +destFile+"' height='100' width='100'/></a>");
Here, destFile brings location of image or screenshot which is related to your machine. the same should not be works for others. We have to use relative path, see this
https://www.w3schools.com/html/html_filepaths.asp
And also make sure that when sharing reports, it should contains all requires files and folders also.
Normally, the issue happens as the local files are not allowed to be loaded. So even we put relative or absolute path, that seems not work for many cases.
So I try to take base64screenshot instead, and it displays quite good in Extent Report.
To have the screenshot in folder report, just need to take screenshot as usual.
public static String getBase64Screenshot(WebDriver driver, String screenshotName) throws IOException {
String encodedBase64 = null;
FileInputStream fileInputStream = null;
TakesScreenshot screenshot = (TakesScreenshot) driver;
File source = screenshot.getScreenshotAs(OutputType.FILE);
String destination = windowsPath + "\\FailedTestsScreenshots\\"+screenshotName+timeStamp+".png";
File finalDestination = new File(destination);
FileUtils.copyFile(source, finalDestination);
try {
fileInputStream =new FileInputStream(finalDestination);
byte[] bytes =new byte[(int)finalDestination.length()];
fileInputStream.read(bytes);
encodedBase64 = new String(Base64.encodeBase64(bytes));
}catch (FileNotFoundException e){
e.printStackTrace();
}
return encodedBase64;
}
Call it in failure cases:
public synchronized void onTestFailure(ITestResult result) {
System.out.println("==="+methodDes + "=== failed!");
try {
WebDriver driver = (WebDriver) result.getTestContext().getAttribute("driver");
String base64Screenshot = ExtentManager.getBase64Screenshot(driver, result.getName());
MediaEntityModelProvider mediaModel = MediaEntityBuilder.createScreenCaptureFromBase64String(base64Screenshot).build();
test.get().fail("image:", mediaModel);
} catch (IOException e) {
e.printStackTrace();
}
test.get().fail(result.getThrowable().getMessage());
}

How can I e-mail a .csv file in Codename One?

In my app, I create a file with a comma-separated array by writing to an OutputStream. Then I want to be able to share this by e-mail so a user can get the data. This is the code I use to create the file:
public String getLogFile(String logName) {
String path = FileSystemStorage.getInstance().getAppHomePath() + "exp " + logName + ".csv";
Set<Long> keys;
OutputStream os = null;
try {
os = FileSystemStorage.getInstance().openOutputStream(path);
Hashtable<Long, Integer> log = (Hashtable<Long, Integer>) dataStorage
.readObject(logName);
keys = log.keySet();
for (Long key : keys) {
String outString = (key + "," + log.get(key) + "\n");
System.out.println(outString);
byte[] buffer = outString.getBytes();
os.write(buffer);
}
} catch (IOException e) {
AnalyticsService.sendCrashReport(e, "Error writing log", false);
e.printStackTrace();
} finally {
try {
os.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
return path;
}
Then, I've created a button that when pressed passes the path of the file to share. I've tried to use MIME types such as "text/plain" and "text/comma-separated-values", but that causes errors. Here is the code executed when the button is pressed.
public void exportLog(String logName) {
String path = dataBuffer.getLogFile(logName);
EmailShare email = new EmailShare();
// email.share("Here is your log.", path, "text/plain");
email.share("Here is your log.", path, "text/comma-separated-values");
}
When pressed (in the simulator). I get this stack after selecting the dummy e-mail contact to send to:
java.lang.NullPointerException
at com.codename1.impl.javase.JavaSEPort.scale(JavaSEPort.java:3483)
at com.codename1.ui.Image.scale(Image.java:963)
at com.codename1.ui.Image.scaledImpl(Image.java:933)
at com.codename1.ui.Image.scaled(Image.java:898)
at com.codename1.impl.javase.JavaSEPort$60.save(JavaSEPort.java:6693)
at com.codename1.share.ShareForm.<init>(ShareForm.java:75)
at com.codename1.share.EmailShare$1$2$1.actionPerformed(EmailShare.java:102)
at com.codename1.ui.util.EventDispatcher.fireActionSync(EventDispatcher.java:455)
at com.codename1.ui.util.EventDispatcher.fireActionEvent(EventDispatcher.java:358)
at com.codename1.ui.List.fireActionEvent(List.java:1532)
at com.codename1.ui.List.pointerReleasedImpl(List.java:2011)
at com.codename1.ui.List.pointerReleased(List.java:2021)
at com.codename1.ui.Form.pointerReleased(Form.java:2560)
at com.codename1.ui.Component.pointerReleased(Component.java:3108)
at com.codename1.ui.Display.handleEvent(Display.java:2017)
at com.codename1.ui.Display.edtLoopImpl(Display.java:1065)
at com.codename1.ui.Display.mainEDTLoop(Display.java:994)
at com.codename1.ui.RunnableWrapper.run(RunnableWrapper.java:120)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
The EmailShare class expects a path to an image file not an arbitrary file as its second argument so loading that fails.
The Message class is better suited for that indeed. You can also use the cloud send option which won't launch the native email app. E.g. the Log class includes that ability directly thru the Log.sendLog API.
It looks like the Messages class is better suited for this task, and should allow attachments, etc.

JavaMail MimeBodyPart.SaveFile provide corrupted files

I'm using JavaMail Library to parser email mime message.
I'm trying to extract the attached files and save them to the local disk but the saved files are not valid and their size is different from the original. only *.txt file are saved ok but *.PDF or *.xlsx are not.
Can you please help me to fix the code?
My code is:
private static void Test3() {
String email_string = File_Reader.Read_File_To_String("D:\\8.txt");
MimeMessage mm = Email_Parser.Get_MIME_Message_From_Email_String(email_string);
Email_Parser.Save_Email_Attachments_To_Folder(mm,"D:\\TEST");
}
public static String Read_File_To_String(String file_path) {
byte[] encoded = new byte[0];
try {
encoded = Files.readAllBytes(Paths.get(file_path));
} catch (IOException exception) {
Print_To_Console(exception.getMessage(), true,false);
}
return new String(encoded, m_encoding);
}
public static MimeMessage Get_MIME_Message_From_Email_String(String email_string) {
MimeMessage mm = null;
try {
Session s = Session.getDefaultInstance(new Properties());
InputStream is = new ByteArrayInputStream(email_string.getBytes());
mm = new MimeMessage(s, is);
} catch (MessagingException exception) {
Print_To_Console(exception.getMessage(), true, false);
}
return mm;
}
public static void Save_Email_Attachments_To_Folder(MimeMessage mm, String output_folder_path) {
ArrayList<Pair<String, InputStream>> attachments_InputStreams = Get_Attachments_InputStream_From_MimeMessage(mm);
String attachment_filename;
String attachment_filename_save_path;
InputStream attachment_InputStream;
MimeBodyPart mbp;
for (Pair<String, InputStream> attachments_InputStream : attachments_InputStreams) {
attachment_filename = attachments_InputStream.getKey();
attachment_filename = Get_Encoded_String(attachment_filename);
attachment_filename_save_path = String.format("%s\\%s", output_folder_path, attachment_filename);
attachment_InputStream = attachments_InputStream.getValue();
try {
mbp = new MimeBodyPart(attachment_InputStream);
mbp.saveFile(attachment_filename_save_path);
} catch (MessagingException | IOException exception) {
Print_To_Console(exception.getMessage(), true, false);
}
}
}
You're doing something very strange in Save_Email_Attachments_To_Folder. (Not to mention the strange naming convention using both camel case and underscores. :-)) I don't know what the InputStreams are you're collecting, but constructing new MimeBodyParts based on them and then using the new MimeBodyPart to save the attachment to the file is almost certainly not what you want to do.
What exactly is Get_Attachments_InputStream_From_MimeMessage doing? Why iterate over the message to collect a bunch of InputStreams, then iterate over the InputStreams to save them? Why not iterate over the message to find the attachments and save them as you find them using the MimeBodyPart.saveFile method? Have you seen the msgshow.java sample program?

How to achieve mp3 stream using java Servlet

Goal: build a servlet so that when I type http://xxx.com/servpage?a.mp3 in browser, I can instantaneously start the playing of this mp3 file. Previously if I put the file on goDaddy as a static file, I can do that. My software can play it right away.
Using Servlet, I can ignore what is after ?, just want this page to return the mp3 dynamically (because in the future I may return any other files). What I got is a long wait (>20 seconds), and then got the player to play it.
I followed some examples, and noticed "attachment" in the example. However, if I remove it, the mp3 won't got played even. I am usign Google App Engine though, but just use the input/outputstream to return the http request. Anyone can help?
public void doGet(HttpServletRequest req, HttpServletResponse res) throws ServletException ,IOException {
res.setContentType("audio/mpeg3");
OutputStream os = res.getOutputStream();
res.setHeader("Content-Disposition", "attachment; filename="" + "a.mp3";");
res.setContentLength(1000000);
FileService fileService = FileServiceFactory.getFileService();
boolean lockForRead = false;
String filename = "/gs/" + BUCKETNAME + "/" + FILENAME;
AppEngineFile readableFile = new AppEngineFile(filename);
try{
FileReadChannel readChannel = fileService.openReadChannel(readableFile, lockForRead);
InputStream is = Channels.newInputStream(readChannel);
int BUFF_SIZE = 1024;
byte[] buffer = new byte[BUFF_SIZE];
try {
do {
int byteCount = is.read(buffer);
if (byteCount == -1)
break;
os.write(buffer, 0, byteCount);
os.flush();
} while (true);
} catch (Exception excp) {
} finally {
os.close();
is.close();
}
readChannel.close();
} catch(Exception e){
}
}
Few notes:
You are not doing "streaming". Just a plain file download.
To do blob (file) serving, you do not need to read the blob from BlobStore as you do with AppEngineFile. Just serve it directly with blobstoreService.serve(blobKey). See Serving a Blob for an example.
You can get the BlobKey needed in 2. via fileService.getBlobKey(readableFile).
Update:
Just realized you are using Google Cloud Storage, not BlobStore.
In GS, if ACLs are properly set, files are publicly visible via: http://commondatastorage.googleapis.com/BUCKETNAME/FILENAME
Since you are not doing any authentication, you could publicly share the file on GS and then in your servlet just do a 301 redirect to public URL of the file.

Get stream from java.sql.Blob in Hibernate

I'm trying to use hibernate #Entity with java.sql.Blob to store some binary data. Storing doesn't throw any exceptions (however, I'm not sure if it really stores the bytes), but reading does. Here is my test:
#Test
public void shouldStoreBlob() {
InputStream readFile = getClass().getResourceAsStream("myfile");
Blob blob = dao.createBlob(readFile, readFile.available());
Ent ent = new Ent();
ent.setBlob(blob);
em.persist(ent);
long id = ent.getId();
Ent fromDb = em.find(Ent.class, id);
//Exception is thrown from getBinaryStream()
byte[] fromDbBytes = IOUtils.toByteArray(fromDb.getBlob().getBinaryStream());
}
So it throws an exception:
java.sql.SQLException: could not reset reader
at org.hibernate.engine.jdbc.BlobProxy.getStream(BlobProxy.java:86)
at org.hibernate.engine.jdbc.BlobProxy.invoke(BlobProxy.java:108)
at $Proxy81.getBinaryStream(Unknown Source)
...
Why? Shouldn't it read bytes form DB here? And what can I do for it to work?
Try to refresh entity:
em.refresh(fromDb);
Stream will be reopened. I suspect that find(...) is closing the blob stream.
It is not at all clear how you are using JPA here, but certainly you do not need to deal with Blob data type directly if you are using JPA.
You just need to declare a field in the entity in question of #Lob somewhat like this:
#Lob
#Basic(fetch = LAZY)
#Column(name = "image")
private byte[] image;
Then, when you retrieve your entity, the bytes will be read back again in the field and you will be able to put them in a stream and do whatever you want with them.
Of course you will need a getter and setter methods in your entity to do the byte conversion. In the example above it would be somewhat like:
private Image getImage() {
Image result = null;
if (this.image != null && this.image.length > 0) {
result = new ImageIcon(this.image).getImage();
}
return result;
}
And the setter somewhat like this
private void setImage(Image source) {
BufferedImage buffered = new BufferedImage(source.getWidth(null), source.getHeight(null), BufferedImage.TYPE_INT_RGB);
Graphics2D g = buffered.createGraphics();
g.drawImage(source, 0, 0, null);
g.dispose();
ByteArrayOutputStream stream = new ByteArrayOutputStream();
try {
ImageIO.write(buffered, "JPEG", stream);
this.image = stream.toByteArray();
}
catch (IOException e) {
assert (false); // should never happen
}
}
}
You need to set a breakpoint on method org.hibernate.engine.jdbc.BlobProxy#getStream on line stream.reset() and examine a reason of IOException:
private InputStream getStream() throws SQLException {
try {
if (needsReset) {
stream.reset(); // <---- Set breakpoint here
}
}
catch ( IOException ioe) {
throw new SQLException("could not reset reader");
}
needsReset = true;
return stream;
}
In my case the reason of IOException was in usage of org.apache.commons.io.input.AutoCloseInputStream as a source for Blob:
InputStream content = new AutoCloseInputStream(stream);
...
Ent ent = new Ent();
...
Blob blob = Hibernate.getLobCreator(getSession()).createBlob(content, file.getFileSize())
ent.setBlob(blob);
em.persist(ent);
While flushing a Session hibernate closes Inpustream content (or rather org.postgresql.jdbc2.AbstractJdbc2Statement#setBlob closes Inpustream in my case). And when AutoCloseInputStream is closed - it rases an IOException in method reset()
update
In your case you use a FileInputStream - this stream also throws an exception on reset method.
There is a problem in test case. You create blob and read it from database inside one transaction. When you create Ent, Postgres jdbc driver closes InputStream while flushing a session. When you load Ent (em.find(Ent.class, id)) - you get the same BlobProxy object, that stores already closed InputStream.
Try this:
TransactionTemplate tt;
#Test
public void shouldStoreBlob() {
final long id = tt.execute(new TransactionCallback<long>()
{
#Override
public long doInTransaction(TransactionStatus status)
{
try
{
InputStream readFile = getClass().getResourceAsStream("myfile");
Blob blob = dao.createBlob(readFile, readFile.available());
Ent ent = new Ent();
ent.setBlob(blob);
em.persist(ent);
return ent.getId();
}
catch (Exception e)
{
return 0;
}
}
});
byte[] fromStorage = tt.execute(new TransactionCallback<byte[]>()
{
#Override
public byte[] doInTransaction(TransactionStatus status)
{
Ent fromDb = em.find(Ent.class, id);
try
{
return IOUtils.toByteArray(fromDb.getBlob().getBinaryStream());
}
catch (IOException e)
{
return new byte[] {};
}
}
});
}
My current and only solution is closing the write session and opening new Hibernate session to get back the streamed data. It works. However I do not know what is the difference. I called inputStream.close(), but that was not enough.
Another way:
I tried to call free() method of blob after session.save(attachment) call too, but it throws another exception:
Exception in thread "main" java.lang.AbstractMethodError: org.hibernate.lob.SerializableBlob.free()V
at my.hibernatetest.HibernateTestBLOB.storeStreamInDatabase(HibernateTestBLOB.java:142)
at my.hibernatetest.HibernateTestBLOB.main(HibernateTestBLOB.java:60)
I am using PostgreSQL 8.4 + postgresql-8.4-702.jdbc4.jar, Hibernate 3.3.1.GA
Is the method IOUtils.toByteArray closing the input stream?

Resources