In my use case where I have a Spring Webflux microservice with Reactor Netty, I have the following dependencies:
org.springframework.boot.spring-boot-starter-webflux (2.0.1.RELEASE)
org.springframework.boot.spring-boot-starter-data-mongodb-reactive (2.0.1.RELEASE)
org.projectreactor.reactor-spring (1.0.1.RELEASE)
For a very specific case I need to retrieve some information from my Mongo database, and process this into query parameters send with my reactive WebClient. As the WebClient nor the UriComponentsBuilder accepts a Publisher (Mono / Flux) I used a #block() call to receive the results.
Since reactor-core (version 0.7.6.RELEASE) which has been included in the latest spring-boot-dependencies (version 2.0.1.RELEASE) it is not possible anymore to use: block()/blockFirst()/blockLast() are blocking, which is not supported in thread xxx, see -> https://github.com/reactor/reactor-netty/issues/312
My code snippet:
public Mono<FooBar> getFooBar(Foo foo) {
MultiValueMap<String, String> parameters = new LinkedMultiValueMap<>();
parameters.add("size", foo.getSize());
parameters.addAll("bars", barReactiveCrudRepository.findAllByIdentifierIn(foo.getBarIdentifiers()) // This obviously returns a Flux
.map(Bar::toString)
.collectList()
.block());
String url = UriComponentsBuilder.fromHttpUrl("https://base-url/")
.port(8081)
.path("/foo-bar")
.queryParams(parameters)
.build()
.toString();
return webClient.get()
.uri(url)
.retrieve()
.bodyToMono(FooBar.class);
}
This worked with spring-boot version 2.0.0.RELEASE, but since the upgrade to version 2.0.1.RELEASE and hence the upgrade from reactor-core to version 0.7.6.RELEASE it is not allowed anymore.
The only real solution I see is to include a block (non-reactive) repository / mongo client as well, but I'm not sure if that is encouraged. Any suggestions?
The WebClient does not accept a Publisher type for its request URL, but nothing prevents you from doing the following:
public Mono<FooBar> getFooBar(Foo foo) {
Mono<List<String>> bars = barReactiveCrudRepository
.findAllByIdentifierIn(foo.getBarIdentifiers())
.map(Bar::toString)
.collectList();
Mono<FooBar> foobar = bars.flatMap(b -> {
MultiValueMap<String, String> parameters = new LinkedMultiValueMap<>();
parameters.add("size", foo.getSize());
parameters.addAll("bars", b);
String url = UriComponentsBuilder.fromHttpUrl("https://base-url/")
.port(8081)
.path("/foo-bar")
.queryParams(parameters)
.build()
.toString();
return webClient.get()
.uri(url)
.retrieve()
.bodyToMono(FooBar.class);
});
return foobar;
}
If anything, this new reactor-core inspection saved you from crashing your whole application with this blocking call in the middle of a WebFlux handler.
Related
I tried to find out how to config. a server-side rest client (i.e. microservice A calls other microservice B using rest) to used a http cache.
The background is, that the binary entities transfered over the wire can be quite large. Overall performance can benefit from a cache on microservice A side which employs http caching headers and etags provided by microservice B.
I found a solution that seems to work, but I'm not sure it that is a proper solution, that work together with current requests, that can occur on microservice A at any time.
#Inject
/* package private */ ManagedExecutor executor;
//
// Instead of using a declarative rest client we create it ourselves, because we can then supply a server-side cache: See ctor()
//
private ServiceBApi serviceClientB;
#ConfigProperty(name="serviceB.url")
/* package private */ String serviceBUrl;
#ConfigProperty(name="cache-entries")
/* package private */ int cacheEntries;
#ConfigProperty(name="cache-entrysize")
/* package private */ int cacheEntrySize;
#PostConstruct
public void ctor()
{
// Create proxy ourselves, because we can then supply a server-side cache
final CacheConfig cc = CacheConfig.custom()
.setMaxCacheEntries(cacheEntries)
.setMaxObjectSize(cacheEntrySize)
.build();
final CloseableHttpClient httpClient = CachingHttpClientBuilder.create()
.setCacheConfig(cc)
.build();
final ResteasyClient client = new ResteasyClientBuilderImpl()
.httpEngine(new ApacheHttpClient43Engine(httpClient))
.executorService(executor)
.build();
final ResteasyWebTarget target = (ResteasyWebTarget) client.target(serviceBUrl);
this.serviceClientB = target.proxy(ServiceBApi.class);
}
#Override
public byte[] getDoc(final String id)
{
try (final Response response = serviceClientB.getDoc(id)) {
[...]
// Use normally and no need to handle conditional gets and caching headers and other HTTP protocol stuff here, because this does underlying impl.
[...]
}
}
My questions are:
Is my solution ok as server-side solution, i.e. can it handle concurrent requests?
Is there a declarative (quarkus) way (#RegisterRestClient. etc) to achieve the same?
--
Edit
To make things clear: I want service B to be able to control the caching based on the HTTP get request and the specific resource. Additionally I want to avoid the unnecessary transmission of the large documents service B provides.
--
Mik
Assuming that you have worked with the declarative way of using Quarkus' REST Client before, you would just inject the client in your serviceB-consuming class. The method, that will invoke Service B, should be annotated with #CacheResult. This will cache results depending on the incoming id. See also Quarkus Cache Guide.
Please note: As Quarkus and Vert.x are all about non-blocking operations, you should use the async support of the REST Client.
#Inject
#RestClient
ServiceBApi serviceB;
...
#Override
#CacheResult(cacheName = "service-b-cache")
public Uni<byte[]> getDoc(final String id) {
return serviceB.getDoc(id).map(...);
}
...
I have to develop a webapp for a CEF-Browser environment. There is no HTTP server available, everything will be served over file:// protocol.
When developing a Webapp nowadays one doesn't get round working with a framework like react/vue for frontend. The standard webpack build scripts of those build a bundle which only works served over HTTP.
Is it possible to configure webpacks build bundle to work on file:// or is there another way to use react or vue via file://?
I'm suggest read CEF wiki more carefully. You are especially interested in https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage.md#markdown-header-request-handling
In short:
You can register custom scheme handler to serve resources over http+custom fake domain.
You can pack resources in zip for example if you like, or leave them at file system as is (but in that case you can expect that some funny users can edit your files, and then report back unexisting errors back to you).
Important helpers already done (but you can write own when need.)
You can... many other things.
Main thing that "file" scheme are more restricted, and for example you can't do XHR requests. But for custom handler you can. Even if dynamic loader for some reason use XHR instead DOM-based loading it will work again same as on http without touching network.
cefclient itself also has usage of custom schemes. Check URL of Tests->Other... in menu. :)
PS: Sorry that my answer doesnt have direct answer for your question. But, custom resource handling in CEF is so common, that i'm just should say about.
fddima is right - you don't need to configure your webpack (although it would be theoretically possible). Instead you can use custom scheme handler in CEF. I made it work with angular at work.
I wrote blog post on how to serve web application via 'file' protocol in CEF.
What you want to add is your scheme handler and its factory:
using System;
using System.IO;
using CefSharp;
namespace MyProject.CustomProtocol
{
public class CustomProtocolSchemeHandler : ResourceHandler
{
// Specifies where you bundled app resides.
// Basically path to your index.html
private string frontendFolderPath;
public CustomProtocolSchemeHandler()
{
frontendFolderPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "./bundle/");
}
// Process request and craft response.
public override bool ProcessRequestAsync(IRequest request, ICallback callback)
{
var uri = new Uri(request.Url);
var fileName = uri.AbsolutePath;
var requestedFilePath = frontendFolderPath + fileName;
if (File.Exists(requestedFilePath))
{
byte[] bytes = File.ReadAllBytes(requestedFilePath);
Stream = new MemoryStream(bytes);
var fileExtension = Path.GetExtension(fileName);
MimeType = GetMimeType(fileExtension);
callback.Continue();
return true;
}
callback.Dispose();
return false;
}
}
public class CustomProtocolSchemeHandlerFactory : ISchemeHandlerFactory
{
public const string SchemeName = "customFileProtocol";
public IResourceHandler Create(IBrowser browser, IFrame frame, string schemeName, IRequest request)
{
return new CustomProtocolSchemeHandler();
}
}
}
And then register it before calling Cef.Initialize:
var settings = new CefSettings
{
BrowserSubprocessPath = GetCefExecutablePath()
};
settings.RegisterScheme(new CefCustomScheme
{
SchemeName = CustomProtocolSchemeHandlerFactory.SchemeName,
SchemeHandlerFactory = new CustomProtocolSchemeHandlerFactory()
});
I'm creating a REST client using Feign. I've got my calls working, but I want to add some timeout support, and I'm having a heck of a time figuring out how to do that.
Feign's documentation says "to use Hystrix with Feign, add the Hystrix module to your classpath. Then use the HystrixFeign builder." Ok, so now I've got this:
service = HystrixFeign.builder()
.decoder(new GsonDecoder())
.target(ProjectService.class, URL_TO_BE_MOVED_TO_PROPS);
Now all of my methods are returning HystrixCommands, which I can execute or queue, but I still can't see how to configure them.
The Hystrix wiki (https://github.com/Netflix/Hystrix/wiki/Configuration) says that configuration should be added into the HystrixCommand constructor like this:
public HystrixCommandInstance(int id) {
super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"))
.andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
.withExecutionTimeoutInMilliseconds(500)));
this.id = id;
But my commands are being built/return by Feign, so I don't have access to the constructors.
One more thing worth noting is that the Feign-Hystrix readme (https://github.com/Netflix/feign/tree/master/hystrix) says "To use Hystrix with Feign, add the Hystrix module to your classpath. Then, configure Feign to use the HystrixInvocationHandler," but a Google search for HystrixInvocationHandler points me toward a non-Netflix repo. Even if I used that, I don't see how to configure Feign to use it.
Please tell me I'm being dumb and that this is super simple, which will make me feel gladness that I'm past this issue, and shame for not being able to figure it out on my own.
TL;DR: I want to set timeouts on requests made by my Feign client. How do?
Turns out you can set Hystrix properties using an instance of com.netflix.config.ConfigurationManager (from com.netflix.archaius:archaius-core).
Feign uses method names as HystrixCommandKeys, so you can access their properties using those names:
ConfigurationManager.getConfigInstance().setProperty("hystrix.command." + methodName + ".execution.isolation.thread.timeoutInMilliseconds", 1500);
This is assuming you've used HystrixFeign to construct your client, which wraps each call in HystrixCommand objects.
To simplify, I created a loop of my methods so I could apply the timeout service-wide:
private void configureHystrix() {
Method[] methods = ProjectService.class.getMethods();
String methodName;
for(int i = 0; i < methods.length; i++) {
methodName = methods[i].getName();
ConfigurationManager.getConfigInstance().setProperty(String.format("hystrix.command.%s.execution.isolation.thread.timeoutInMilliseconds", methodName), config.getTimeoutInMillis());
}
}
After some debugging I managed to set Hystrix timeout as follows:
HystrixFeign.builder()
.setterFactory(getSetterFactory())
.target(...);
// copy-paste feign.hystrix.SetterFactory.Default, just add andCommandPropertiesDefaults
private SetterFactory getSetterFactory() {
return (target, method) -> {
String groupKey = target.name();
String commandKey = Feign.configKey(target.type(), method);
return HystrixCommand.Setter
.withGroupKey(HystrixCommandGroupKey.Factory.asKey(groupKey))
.andCommandKey(HystrixCommandKey.Factory.asKey(commandKey))
.andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
.withExecutionTimeoutInMilliseconds(15000));
};
}
tl;dr What is the best way to pass binary data (up to 1MBish) from a WPF application to a WebAPI service method?
I'm currently trying to pass binary data from a WPF application to a WebAPI web service, with variable results. Small files (< 100k) generally work fine, but any larger and the odds of success reduce.
A standard OpenFileDialog, and then File.ReadAllBytes pass the byte[] parameter into the client method in WPF. This always succeeds, and I then post the data to WebAPI via a PostAsync call and a ByteArrayContent parameter.
Is this the correct way to do this? I started off with a PostJSONAsync call, and passed the byte[] into that, but thought the ByteArrayContent seemed more appropriate, but neither work reliably.
Client Method in WPF
public static async Task<bool> UploadFirmwareMCU(int productTestId, byte[] mcuFirmware)
{
string url = string.Format("productTest/{0}/mcuFirmware", productTestId);
ByteArrayContent bytesContent = new ByteArrayContent(mcuFirmware);
HttpResponseMessage response = await GetClient().PostAsync(url, bytesContent);
....
}
WebAPI Method
[HttpPost]
[Route("api/productTest/{productTestId}/mcuFirmware")]
public async Task<bool> UploadMcuFirmware(int productTestId)
{
bool result = false;
try
{
Byte[] mcuFirmwareBytes = await Request.Content.ReadAsByteArrayAsync();
....
}
Web Config Settings
AFAIK these limits in web.config should be sufficient to allow 1MB files through to the service?
<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="1073741824" />
</requestFiltering>
</security>
<httpRuntime targetFramework="4.5" maxRequestLength="2097152"/>
I receive errors in WebAPI when calling ReadAsByteArrayAsync(). These vary, possibly due to the app pool in IIS Express having crashed / getting into a bad state, but they include the following (None of which have lead to any promising leads via google):
Specified argument was out of the range of valid values. Parameter name: offset
at System.Web.HttpInputStream.Seek(Int64 offset, SeekOrigin origin)\r\n
at System.Web.HttpInputStream.set_Position(Int64 value)\r\n at System.Web.Http.WebHost.SeekableBufferedRequestStream.SwapToSeekableStream()\r\n at System.Web.Http.WebHost.Seek
OR
Message = "An error occurred while communicating with the remote host. The error code is 0x800703E5."
InnerException = {"Overlapped I/O operation is in progress. (Exception from HRESULT: 0x800703E5)"}
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)\r\n
at System.Web.Hosting.IIS7WorkerRequest.ReadEntityCoreSync(Byte[] buffer, Int32 offset, Int32 size)\r\n
at System.Web.Hosting.IIS7WorkerRequ...
Initially I thought this was most likely down to IIS Express limitations (running on Windows 7 on my dev pc) but we've had the same issues on a staging server running Server 2012.
Any advice on how I might get this working would be great, or even just a basic example of uploading files to WebAPI from WPF would be great, as most of the code I've found out there relates to uploading files from multipart forms web pages.
Many thanks in advance for any help.
tl;dr It was a separate part of our code in the WebApi service that was causing it to go wrong, duh!
Ah, well, this is embarrassing.
It turns out our problem was down to a Request Logger class we'd registered in WebApiConfig.Register(HttpConfiguration config), and that I'd forgotten about.
It was reading the request content via async as StringContent, and then attempting to log it to the database in an ncarchar(max) field. This itself is probably OK, but I'm guessing all the weird problems started occurring when the LoggingHandler as well as the main WebApi controller, were both trying to access the Request content via async?
Removing the LoggingHandler fixed the problem immediately, and we're now able to upload files of up to 100MB without any problems. To fix it more permanently, I guess I rewrite of the LoggingHandler is required to set a limit on the maximum content size it tries to log / to ignore certain content types.
It's doubtful, but I hope this may be of use for someone one day!
public class LoggingHandler : DelegatingHandler
{
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
LogRequest(request);
return base.SendAsync(request, cancellationToken).ContinueWith(task =>
{
var response = task.Result;
// ToDo: Decide if/when we need to log responses
// LogResponse(response);
return response;
}, cancellationToken);
}
private void LogRequest(HttpRequestMessage request)
{
(request.Content ?? new StringContent("")).ReadAsStringAsync().ContinueWith(x =>
{
try
{
var callerId = CallerId(request);
var callerName = CallerName(request);
// Log request
LogEntry logEntry = new LogEntry
{
TimeStamp = DateTime.Now,
HttpVerb = request.Method.ToString(),
Uri = request.RequestUri.ToString(),
CorrelationId = request.GetCorrelationId(),
CallerId = callerId,
CallerName = callerName,
Controller = ControllerName(request),
Header = request.Headers.ToString(),
Body = x.Result
};
...........
I'm trying to implement pagination using App Engine's RPC and GWT (it's an app engine connected project).
How can I pass both the query results and the web-safe cursor object to the GWT client from the RPC?
I've seen examples using a servlet but I want to know how to do it without a servelt.
I've considered caching the cursor on the server using memcache but I'm not sure if that's appropriate or what should be used as the key (session identifier I would assume, but I'm not sure how those are handled on App Engine).
Links to example projects would be fantastic, I've been unable to find any.
OK, so the best way to do this is to store the cursor as a string on the client.
To do this you have to create a wrapper class that is transportable so you can pass back it to the client via RequestFactory that can hold the results list and the cursor string. To do that you create a normal POJO and then a proxy for it.
here's what the code looks like for the POJO:
public class OrganizationResultsWrapper {
public List<Organization> list;
public String webSafeCursorString;
public List<Organization> getList() {
return list;
}
public void setList(List<Organization> list) {
this.list = list;
}
public String getWebSafeCursorString() {
return this.webSafeCursorString;
}
public void setWebSafeCursorString(String webSafeCursorString) {
this.webSafeCursorString = webSafeCursorString;
}
}
for the proxy:
#ProxyFor(OrganizationResultsWrapper.class)
public interface OrganizationResultsWrapperProxy extends ValueProxy{
List<OrganizationProxy> getList();
void setList(List<OrganizationProxy> list);
String getWebSafeCursorString();
void setWebSafeCursorString(String webSafeCursorString);
}
set up your service and requestFactory to use the POJO and proxy respectively
// service class method
#ServiceMethod
public OrganizationResultsWrapper getOrganizations(String webSafeCursorString) {
return dao.getOrganizations(webSafeCursorString);
}
// request factory method
Request<OrganizationResultsWrapperProxy> getOrganizations(String webSafeCursorString);
Then make sure and run the RPC wizard so that your validation process runs otherwise you'll get a request context error on the server.
Here's the implementation in my data access class:
public OrganizationResultsWrapper getOrganizations(String webSafeCursorString) {
List<Organization> list = new ArrayList<Organization>();
OrganizationResultsWrapper resultsWrapper = new OrganizationResultsWrapper();
Query<Organization> query = ofy().load().type(Organization.class).limit(50);
if (webSafeCursorString != null) {
query = query.startAt(Cursor.fromWebSafeString(webSafeCursorString));
}
QueryResultIterator<Organization> iterator = query.iterator();
while (iterator.hasNext()) {
list.add(iterator.next());
}
resultsWrapper.setList(list);
resultsWrapper.setWebSafeCursorString(iterator.getCursor().toWebSafeString());
return resultsWrapper;
}
a second option would be to save the webSafeCursorString in the memcache, as you already mentioned.
my idea looks like this:
the client sends always request like this "getMyObjects(Object... myParams, int maxResults, String clientPaginationString)". the clientPaginationString is uniquely created like shown below
server receives request and looks into the memcache if there is a webSafeCursorString for the key clientPaginationString
if the server finds nothing, he creates the query and save the webSafeCursorString into memcache with the clientPaginationString as the key. -> returns the results
if the server finds the webSafeCursorString he restarts the query with it and returns the results
the problems are how to clean the memcache and how to find a unique clientPaginationString:
a unique clientPaginationString should be the current UserId + the params of the current query + timestemp. this should work just fine!
i really can't think of a easy way how to clean the memcache, however i think we do not have to clean it at all.
we could store all the webSafeCursorStrings and timestemps+params+userid in a WebSafeCursor-Class that contains a map and store all this in the memcache... and clean this Class ones in a while (timestamp older then...).
one improvement i can think of is to save the webSafeCursorString in the memcache with a key that is created on the server (userSessionId + servicename + servicemethodname + params). however, important is that the client sends an information if he is interested in a new query (memcache is overriden) or wants the next pagination results (gets webSafeCursorString from memcache). a reload of the page should work. a second tap in the browser would be a problem i think...
what would you say?