I am using google app engine's urlfetch sevice to get book description from google play. When I fetch the url
https://play.google.com/store/books
, I can get correct web page html content and parse it in my local developement enviroment. But on production server, Google Play just give me a "Unavailable in your country" page.
This is my test code referenced from GAE document(https://developers.google.com/appengine/docs/java/urlfetch/usingjavanet)
try {
URL url = new URL("https://play.google.com/store/books");
final URLConnection connection = url.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
// connection.setRequestProperty("Accept-Language", "en-US");
String line;
while ((line = reader.readLine()) != null) {
log.warn(line);
}
reader.close();
} catch (MalformedURLException e) {
// ...
} catch (IOException e) {
// ...
}
I also try to change request property "Accept-Language" but not work. Do you have any ideas for this ?
Related
I try to call an external web service (not mine) from my GWT application working with App Engine.
I know it's impossible to do it from the client due to the SOP (Same Origin Policy) and RequestBuilder is not a solution on the server. I followed the tutorial on the web site and using java.net as well
Here is the client
AsyncCallback<CustomObject> callback = new AsyncCallback<CustomObjectCustomObject>() {
#Override
public void onFailure(Throwable caught) {
caught.printStackTrace();
}
#Override
public void onSuccess(CustomObject result) {
// code omitted
}
};
service.callMethod(aString, callback);
And this is the server
try {
String xmlRequest = "xmlToSend";
URL url = new URL("https://www.externalWebService.com");
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestMethod("POST");
conn.setAllowUserInteraction(false);
conn.setDoOutput(true);
conn.setRequestProperty("Content-Type","application/soap+xml");
conn.setRequestProperty( "Content-length", Integer.toString(xmlRequest.length()));
conn.setRequestProperty("charset", "utf-8");
conn.setConnectTimeout(10000);
OutputStream rawOutStream = conn.getOutputStream();
PrintWriter pw = new PrintWriter(rawOutStream);
pw.print(xmlRequest);
pw.flush();
pw.close();
if(conn.getResponseCode() != 200){
// Something...
}
I keep having the same error at conn.getResponseCode() :
java.lang.ClassCastException: com.google.appengine.repackaged.org.apache.http.message.BasicHttpRequest cannot be cast to com.google.appengine.repackaged.org.apache.http.client.methods.HttpUriRequest
Without making a real request, the remote service works well : it's able to serialize and return objects to the client. The issue is not linked to the communication between the client and the server, it's more like AppEngine doesn't support HttpURLConnection. But it should on the server (isn't it?)
Any thoughts would be hightly appreciated! Thanks in advance
Your problem has nothing to do with GWT: as long as you are running on the server, you can use any 'normal' Java and it will work unless AppEngine has restrictions.
It seems you have imported the repackaged version of Apache HttpClient in your class. You should not do that: download your own HttpClient .jar, add it to the dependencies and use that one.
AppEngine also has some issues with HttpClient. There's an adapter available here that fixes most of the issues.
Thanks #Marcelo, you were right!
Here is the solution I found.
I added httpcore.jar and httpclient.jar to my build path and wrote the code below for the server (the client is the same) :
String xmlRequest = "xmlToSend";
CloseableHttpClient httpclient = HttpClients.custom().build();
//RequestConfig requestConfig = RequestConfig.custom()
// .setConnectionRequestTimeout(10000)
// .build();
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
Writer writer = new OutputStreamWriter(out);
writer.write(xmlToSend);
writer.flush();
writer.close();
HttpPost request = new HttpPost("https://www.externalWebService.com/path");
request.setEntity(new ByteArrayEntity(out.toByteArray()));
//request.setConfig(requestConfig);
CloseableHttpResponse response = httpclient.execute(request);
if(response.getStatusLine().getStatusCode() == 200){
// retrieve content with a BufferReader
// from response.getEntity().getContent()
...
}
The code works and is up to date.
Edit
Here is the rest of the solution when using a proxy. Mine only deals with NTCredentials but otherwise UsernamePasswordCredentials can be used instead.
HttpHost proxy = new HttpHost("addresse.proxy.com", port);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
new AuthScope(proxy),
new NTCredentials(System.getProperty("user.name") + ":" + password));
RequestConfig requestConfig = RequestConfig.custom()
.setProxy(proxy)
.build();
CloseableHttpClient httpclient = HttpClients.custom()
.setDefaultCredentialsProvider(credsProvider)
.setDefaultRequestConfig(requestConfig)
.build();
Thanks again for your help, I really appreciated!
I created application that runs on GAE and connects to specific page, automatically logins to this page and AFTER login I want to receive html and process it.
here is the problematic (writer.write part and connection.connect()) part of code:
this.username = URLEncoder.encode(username, "UTF-8");
this.password = URLEncoder.encode(password, "UTF-8");
this.login = "login";
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
OutputStreamWriter writer = new OutputStreamWriter(
connection.getOutputStream());
writer.write("str_login=" + login + "&str_user=" + username
+ "&str_pass=" + password);
writer.close();
connection.connect();
I am getting IOException (connection.connect()) while establishig connection. Problem is "application/x-www-form-urlencoded" data. When I pass wrong parameters to page (for example str_pasSSs, str_usernaAAme, or no parameters at all) I am not possible to login but I do get response with html of login page. So, it seems that Google App Engine does not support this kind of communication. Is it possible to login to this page in some other way that GAE supports?
In Wireshark, I saw that username and password are sent in plaintext as line-based text data (application/x-www-form-urlencoded). I know this is not safe, but it's the way it is.
The connection is already established implicitly when you call getOutputStream(). There is no need to call connection.connect() again.
Also, instead of closing the output writer, try flush() instead.
As a best practice, you should be closing in, out and conn in a finally block:
InputStream in = null;
OutputStream out = null;
HttpUrlConnection conn = null;
try {
...
} catch (IOException ioe) {
...
} finally {
if (in!=null) {try {in.close()} catch (IOException e) {}}
if (out!=null) {try {out.close()} catch (IOException e) {}}
if (conn!=null) {try {conn.close()} catch (IOException e) {}}
}
I am getting no where now. When I Google around to find a sample of "Exporting data to Google Spreadsheet from Google App Engine", I see a lot of Google Conversion API, Google Conversion API, Google Spreadsheet API and Google Docs API tutorials. But then they all deprecated from Google when I check at the Google site??? So, what is the most updated right now so I can make use of it?
=====
Okay now I use the Google Drive SDK via OAuth2 to create text file. But I have problem with this:
this is the error:
java.lang.NullPointerException at
java.net.URI$Parser.parse(URI.java:3004) at
java.net.URI.(URI.java:577) at
com.google.api.client.http.GenericUrl.(GenericUrl.java:100) at
com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:269)
at
com.google.api.services.drive.Drive$Files$Insert.executeUnparsed(Drive.java:309)
at
com.google.api.services.drive.Drive$Files$Insert.execute(Drive.java:331)
at
com.company.dashboard.service.DriveService.initDoc(DriveService.java:84)
this is the code:
private GoogleCredential buildGoogleCredential(Credential credential) {
try {
logger.warning(oauth2Service.getClientCredential().toString());
GoogleCredential googleCredential = new GoogleCredential.Builder()
.setClientSecrets(oauth2Service.getClientCredential())
.setTransport(new NetHttpTransport())
.setJsonFactory(new JacksonFactory()).build();
googleCredential.setAccessToken(credential.getAccessToken());
googleCredential.setRefreshToken(credential.getRefreshToken());
return googleCredential;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
private Drive buildService(GoogleCredential credential) {
return new Drive.Builder(new NetHttpTransport(), new JacksonFactory(), credential).build();
}
public void initDoc(HttpServletRequest req)
throws Exception {
User user = UserServiceFactory.getUserService().getCurrentUser();
Credential credential = oauth2Service.getStoredCredential(
user.getUserId(),
(CredentialStore)req.getSession().getServletContext().getAttribute(OAuth2Constant.GOOG_CREDENTIAL_STORE));
if (credential != null) {
logger.warning("Using access token: " + credential.getAccessToken());
try {
GoogleCredential googleCredential = buildGoogleCredential(credential);
Drive service = buildService(googleCredential);
if (service == null) {
logger.warning("very bad!");
}
File body = new File();
body.setTitle("My document");
body.setDescription("A test document");
body.setMimeType("text/plain");
java.io.File fileContent = new java.io.File("document.txt");
FileContent mediaContent = new FileContent("text/plain", fileContent);
service.files().insert(body, mediaContent).execute();
//File file = service.files().insert(body, mediaContent).execute();
//System.out.println("File ID: " + file.getId());
} catch (HttpResponseException e) {
if (e.getStatusCode() == 401) {
logger.warning(e.getMessage());
// Credentials have been revoked.
// TODO: Redirect the user to the authorization URL.
throw new UnsupportedOperationException();
}
} catch (IOException e) {
System.out.println("An error occurred: " + e);
}
}
oauth2Service.getClientCredential() returns (xxx = client id and client secret code, not shown in here)
{"web":{"client_id":"xxx.apps.googleusercontent.com","client_secret":"xxx"}}
This is my scope:
final static List<String> SCOPES = Arrays.asList("https://www.googleapis.com/auth/userinfo.profile",
"https://www.googleapis.com/auth/userinfo.email",
"https://www.googleapis.com/auth/drive",
"https://www.googleapis.com/auth/docs",
"https://www.googleapis.com/auth/drive.file");
final static String AUTH_RESOURCE_LOC = "/client_secrets.json";
final static String OATH_CALLBACK = "http://localhost:8888/oauth2callback";
This line
service.files().insert(body, mediaContent).execute();
throws NullPointerException. Any idea what had gone wrong???
P/S: Credential = com.google.api.client.auth.oauth2.Credential. I have my OAuth2 everything works good. I can retrieve user info without problem but not the Drive API. service is not null as you see I put it the log as "very bad" and it is not shown. No 401 exception is thrown means my Oauth2 is good with the scopes.
=======
DARN!!!! Finally solved the issue!!! My code was totally correct! Just I enabled the wrong API! It should be Drive API instead of Drive SDK API :/
You can generate a csv file from GAE and upload it using the Drive API with ?convert=true to have it automatically converted to a Google spreadsheet:
https://developers.google.com/drive/v2/reference/files/insert
Seems to be a simple thing using a WebClient object to retrieve content of a text file in a Silverlight web application.
It works in our intranet setup on a local server, but when deployed on a hosted "hostgator" server, WebClient gets a security exception. Following suggestions from numerous web blogs, I tried crossdomain access policy and crossdomain.xml with no result - no wonder, all in on the same domain.
private void LoadCSVFile(string csvFile)
{
try
{
WebClient client = new WebClient();
client.OpenReadCompleted += new OpenReadCompletedEventHandler(client_OpenReadCompleted);
client.OpenReadAsync(new Uri(csvFile, UriKind.Absolute));
}
catch (Exception ex)
{
string errorMessage = String.Format("Ex accessing csv: {0}\n{1}", csvFile,
}
}
//the csvFile is an absolute path: http://myserver.com/myapplication:port/sourcefiles/file.csv
//The sourcefiles is a virtual directory in the same website containing csv files
void client_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
if (e.Error == null)
{
try
{
using (System.IO.StreamReader reader = new System.IO.StreamReader(e.Result))
{
mainFileData = new List<string>();
string line;
}
}
catch (Exception ex)
{
string errorMessage = ex.InnerException;
}else
{
"here is the place for the exception!!!!!!! so, e.Error is not null"
}
}
The text of the exception reads: System.SecurityException: Security error at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult) etc.
Spend 2 days gooogling with no result.
Please help.
V
A couple of things to check:
(1) The format for the URL that you list (http://myserver.com/myapplication:port/sourcefiles/file.csv) looks odd. Shouldn't it be http://myserver.com:port/myapplication/sourcefiles/file.csv? Or is that just a typo?
(2) What's the port number that you're accessing? The Silverlight WebClient will only access ports 80 and 443.
(3) Can you get to the file using a direct URL? Sometimes web servers need to be told about a specific file type before they'll allow it to be served up.
I'm using the MailService feature of Google App Engine in my
application. It works fine in one application without any issues.
But the same code doesn't work in another app. I'm not able to figure
it out. Please help. Following is the piece of code that I use to
send mail.
public static void sendHTMLEmail(String from, String fromName, String
to, String toName, String subject, String body) {
_logger.info("entering ...");
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
_logger.info("got mail session ...");
String htmlBody = body;
try {
Message msg = new MimeMessage(session);
_logger.info("created mimemessage ...");
msg.setFrom(new InternetAddress(from,
fromName));
_logger.info("from is set ...");
msg.addRecipient(Message.RecipientType.TO, new InternetAddress(
to, toName));
_logger.info("recipient is set ...");
msg.setSubject(subject);
_logger.info("subject is set ...");
Multipart mp = new MimeMultipart();
MimeBodyPart htmlPart = new MimeBodyPart();
htmlPart.setContent(htmlBody, "text/html");
mp.addBodyPart(htmlPart);
_logger.info("body part added ...");
msg.setContent(mp);
_logger.info("content is set ...");
Transport.send(msg);
_logger.info("email sent successfully.");
} catch (AddressException e) {
e.printStackTrace();
} catch (MessagingException e) {
e.printStackTrace();
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
System.err.println(e.getMessage());
}
}
When I look at the log (on the server admin console), it prints the
statement "content is set ..." and after that there is nothing in the
log. The mail is not sent. At times I get the following error after
the above statement is printed (and the mail is not sent).
com.google.appengine.repackaged.com.google.common.base.internal.Finalizer
getInheritableThreadLocalsField: Couldn't access
Thread.inheritableThreadLocals. Reference finalizer threads will
inherit thread local values.
But the mail quota usage keeps increasing.
Remember, this works fine in one application, but not in other. I'm
using the same set of email addresses in both the apps (for from and
to).
I'm really stuck with this. Appreciate any help.
Thank you.
Velu
Have you tried logging the exceptions? I bet one of them is being thrown - your printStackTrace will go nowhere.