I am trying to migrate a custom dynamic partitioner from Flink 1.7 to Flink 1.9. The original partitioner implemented the selectChannels method within the StreamPartitioner interface like this:
// Original: working for Flink 1.7
//#Override
public int[] selectChannels(SerializationDelegate<StreamRecord<T>> streamRecordSerializationDelegate,
int numberOfOutputChannels) {
T value = streamRecordSerializationDelegate.getInstance().getValue();
if (value.f0.isBroadCastPartitioning()) {
// send to all channels
int[] channels = new int[numberOfOutputChannels];
for (int i = 0; i < numberOfOutputChannels; ++i) {
channels[i] = i;
}
return channels;
} else if (value.f0.getPartitionKey() == -1) {
// random partition
returnChannels[0] = random.nextInt(numberOfOutputChannels);
} else {
returnChannels[0] = partitioner.partition(value.f0.getPartitionKey(), numberOfOutputChannels);
}
return returnChannels;
}
I am not sure how to migrate this to Flink 1.9, since the StreamPartitioner interface has changed as illustrated below:
// New: required by Flink 1.9
#Override
public int selectChannel(SerializationDelegate<StreamRecord<T>> streamRecordSerializationDelegate) {
T value = streamRecordSerializationDelegate.getInstance().getValue();
if (value.f0.isBroadCastPartitioning()) {
/*
It is illegal to call this method for broadcast channel selectors and this method can remain not
implemented in that case (for example by throwing UnsupportedOperationException).
*/
} else if (value.f0.getPartitionKey() == -1) {
// random partition
returnChannels[0] = random.nextInt(numberOfChannels);
} else {
returnChannels[0] = partitioner.partition(value.f0.getPartitionKey(), numberOfChannels);
}
//return returnChannels;
return returnChannels[0];
}
Note that selectChannels has been replaced with selectChannel. So, it is no longer possible to return multiple output channels as originally done above for the case of broadcasted elements. As a matter of fact, selectChannel should not be invoked for this particular case. Any thoughts on how to tackle this?
With Flink 1.9, you cannot dynamically broadcast to all channels anymore. Your StreamPartitioner has to statically specify if it's a broadcast with isBroadcast. Then, selectChannel is never invoked.
Do you have a specific use case, where you'd need to dynamically switch?
Related
I'm following tutorial from the official Microsoft learning page (https://learn.microsoft.com/en-us/azure/quantum/tutorial-qdk-explore-entanglement?pivots=ide-azure-portal) about quantum entanglement.
Basically, I copied an example posted there and I am getting error:
QS5026 The variable "numOnesQ1" cannot be reassigned here. In conditional blocks that depend on a measurement result, the target QuantinuumProcessor only supports reassigning variables that were declared within the block.
I understand what it says but it's just a copy from the official Microsoft tutorial. Am I missing something simple like imports, wrong settings? If not, how can I in other way set variables declared outside conditional blocks that depend on a measurement result?
Here is my code:
namespace Quantum.QuantumDream {
open Microsoft.Quantum.Canon;
open Microsoft.Quantum.Intrinsic;
operation GetRandomResult() : Result {
use q = Qubit();
H(q);
return M(q);
}
#EntryPoint()
operation TestBellState(count : Int, initial : Result) : (Int, Int, Int, Int) {
mutable numOnesQ1 = 0;
mutable numOnesQ2 = 0;
// allocate the qubits
use (q1, q2) = (Qubit(), Qubit());
for test in 1..count {
SetQubitState(initial, q1);
SetQubitState(Zero, q2);
// measure each qubit
let resultQ1 = M(q1);
let resultQ2 = M(q2);
// Count the number of 'Ones':
if resultQ1 == One {
set numOnesQ1 += 1;
}
if resultQ2 == One {
set numOnesQ2 += 1;
}
}
// reset the qubits
SetQubitState(Zero, q1);
SetQubitState(Zero, q2);
// Return number of |0> states, number of |1> states
Message("q1:Zero, One q2:Zero, One");
return (count - numOnesQ1, numOnesQ1, count - numOnesQ2, numOnesQ2 );
}
operation SetQubitState(desired : Result, target : Qubit) : Unit {
if desired != M(target) {
X(target);
}
}
}
This tutorial code is only supposed to run on a local simulator (using %simulate magic commands in a Jupyter Notebook). From the error message, it looks like you've tried to run it on one of Quantinuum targets, which have some limitations on the kinds of things you can do in the code. To run equivalent code on Quantinuum, you'd need to define an operation for just the body of the loop (preparing a state and measuring it) and run it as a job - the cloud targets will take care of the loop themselves, running your code multiple times and returning to you a histogram of the results. For an example, you can see the QRNG sample in the samples gallery in Azure Portal.
I have a weird case that I need to create many processes from my main process.
These processes that I create will queue some messages from a web socket.
And in an interval, like every second or so, I will poll these small processes from my main process. The language I use is D and the messaging library is ZMQD ( which is just a wrapper for C zmq library ).
The minimal example I have for my main process :
Socket*[] socketList;
string sRecv( Socket* socket)
{
ubyte[256] buffer;
immutable size = socket.receive(buffer);
import std.algorithm: min;
return buffer[0 .. min(size,256)].idup.asString();
}
void startServer( string servername )
{
auto pid = spawnProcess(["/home/erdem/eclipse-workspace/WebSocketDenemesi/websocketdenemesi",
servername, "\n"]);
auto requester = new Socket(SocketType.req);
auto allName = "ipc:///tmp/" ~ servername;
requester.connect(allName);
socketList ~= requester;
}
void main() {
import std.array : split;
import std.algorithm : each;
startServer("iotabtc#depth");
startServer("iotabtc#aggTrade");
startServer("ethbtc#depth");
int counter = 30;
while(counter--) {
foreach ( requester; socketList)
{
requester.send("send");
}
foreach ( requester; socketList)
{
auto strList = sRecv(requester).split("\n");
strList.each!( str => writefln("Received [%d]reply [%s]", strList.length, str) );
}
sleep(1000.msecs);
}
foreach ( requester; socketList)
{
requester.send("done");
}
}
And the minimal example I have for my small processes :
WebSocket startSocket( string temp )
{
auto ws_url = URL(temp);
auto ws = connectWebSocket(ws_url);
if ( !ws.connected )
return null;
return ws;
}
void close( WebSocket ws )
{
int timeOut = 5;
while ( ws && ws.connected && timeOut-- )
{
vibe.core.concurrency.async( { ws.close(); return true;} );
sleep(5.msecs);
}
}
string sRecv(ref Socket socket)
{
ubyte[256] buffer;
immutable size = socket.tryReceive(buffer)[0];
import std.algorithm: min;
return size ? buffer[0 .. min(size,256)].idup.asString() : "";
}
void main( string[] args ) {
auto responder = Socket(SocketType.rep);
string seperatorChar = args[2];
string temp = "ipc:///tmp/" ~ args[1];
responder.bind(temp);
string socketName = "wss://stream.binance.com:9443/ws/" ~ args[1];
auto curSocket = startSocket(socketName);
string curString;
while (true) {
auto result = responder.sRecv();
if ( result == "send")
{
responder.send(curString);
curString = "";
}
else if ( result == "done" )
{
break;
}
else
{
if ( curSocket.dataAvailableForRead )
{
auto text = curSocket.receiveText();
if ( !curString.empty )
curString ~= seperatorChar;
curString ~= text;
}
}
sleep(100.msecs);
}
writeln( "Shutting down: ", args[1]);
curSocket.close();
}
This is the first time I am using this Messaging library. That is why I am using simple REQ/REP sockets. Is there a better way to achieve my requirement. Is there a better messaging pattern for example? For example is there a pattern in which my small processes are not blocked by responder.receive( buffer );.
If there is some, than I will not need to listen websocket from another thread.
Welcome to the ZeroMQ-based distributed-computing
Is there a better messaging pattern for example ?
This depends on how your processes need to communicate. In short, using REQ/REP in a blocking-mode is almost the worst option from the menu.
given your websocket just receives an async piece of information ( which is a common way, how Markets re-broadcast the flow of events ), the pure ws.recv() + PUSHer.send() + if PULLer.poll(): PULLer.recv() pipelined event-acquisition + PUSH/PULL propagation + conditional re-processing would best meet the real-world behaviour.
given your footprint of processing farm may grow beyond a single localhost, other transport-classes, for non-local nodes ~{ tipc:// | tcp:// | udp:// | pgm:// | epgm:// | norm:// | vmci:// } might get into the game, altogether with ipc://-links on your current localhost - ZeroMQ transparency in handling this mix is a cool benefit of moving into mastering the Zen-of-Zero.
given latency is critical on a massive scale of processing distribution, a PUB/SUB Scalable Formal Communication Archetype Pattern may become beneficial, with an option to use .setsockopt( zmq.CONFLATE, 1 ) for non-logging nodes, where just the most recent prices are relevant for taking any responsive XTO action of any sort of kind.
I am trying to save documents using pouchdb's bulkSave() function.
However, when these documents are saved it starts to sync with master database using sync gateway & in doing so the webapp slows down and when I try to navigate to different tabs no content is displayed on that tab.
Below is an example of how the documents are being created:
for (var i = 0; i <= instances; i++) {
if (i > 0) {
advTask.startDate = new Date(new Date(advTask.startDate).setHours(new Date(advTask.startDate).getHours() + offset));
}
if (advTask.estimatedDurationUnit == 'Minutes') {
advTask = $Date.getAdvTaskEndTimeIfMinutes(advTask);
} else if (advTask.estimatedDurationUnit == 'Hours') {
advTask = $Date.getAdvTaskEndTimeIfHours(advTask);
} else if (advTask.estimatedDurationUnit == 'Days') {
advTask = $Date.getAdvTaskEndTimeIfDays(advTask);
}
if(new Date(advTask.endDate).getTime() >= new Date($scope.advTask.endDate).getTime()) {
// here save the task array using bulkSave() function
$db.bulkSave(tasks).then(function (res) {
$db.sync();
});
break;
}
advTask.startDate = $Date.toGMT(advTask.startDate);
advTask.endDate = $Date.toGMT(advTask.endDate);
var adv = angular.copy(advTask);
tasks.push(adv); // here pushing the documents to an array
offset = advTask.every;
}
Thanks in advance!
bulkSave is not a core PouchDB API; are you using a plugin?
Also one piece of advice I'd give is that Couchbase Sync Gateway does not have 100% support for PouchDB and is known to be problematic in some cases.
Another piece of advice is that running PouchDB in a web worker can prevent your UI thread from getting overloaded, which would fix the problem of tabs not showing up.
Do you have a live test case to demonstrate?
Here is code that use this eval method in Dart platform.
This is done via reflection.
runtime/lib/mirrors_impl.dart
_getFieldSlow(unwrapped) {
// ..... Skipped
var atPosition = unwrapped.indexOf('#');
if (atPosition == -1) {
// Public symbol.
f = _eval('(x) => x.$unwrapped', null);
} else {
// Private symbol.
var withoutKey = unwrapped.substring(0, atPosition);
var privateKey = unwrapped.substring(atPosition);
f = _eval('(x) => x.$withoutKey', privateKey);
}
// ..... Skipped
}
static _eval(expression, privateKey)
native "Mirrors_evalInLibraryWithPrivateKey";
runtime/lib/mirrors.cc
DEFINE_NATIVE_ENTRY(Mirrors_evalInLibraryWithPrivateKey, 2) {
GET_NON_NULL_NATIVE_ARGUMENT(String, expression, arguments->NativeArgAt(0));
GET_NATIVE_ARGUMENT(String, private_key, arguments->NativeArgAt(1));
const GrowableObjectArray& libraries =
GrowableObjectArray::Handle(isolate->object_store()->libraries());
const int num_libraries = libraries.Length();
Library& each_library = Library::Handle();
Library& ctxt_library = Library::Handle();
String& library_key = String::Handle();
if (library_key.IsNull()) {
ctxt_library = Library::CoreLibrary();
} else {
for (int i = 0; i < num_libraries; i++) {
each_library ^= libraries.At(i);
library_key = each_library.private_key();
if (library_key.Equals(private_key)) {
ctxt_library = each_library.raw();
break;
}
}
}
ASSERT(!ctxt_library.IsNull());
return ctxt_library.Evaluate(expression);
runtime/vm/bootstrap_natives.h
V(Mirrors_evalInLibraryWithPrivateKey, 2) \
P.S.
I ask question here becuase I cannot ask it at Dart mail lists.
P.S.
As we can see it static private method in mirrors_impl.dart:
static _eval(expression, privateKey) native "Mirrors_evalInLibraryWithPrivateKey";
Does anyone want that this method should be public? (this is not a question but just a thought aloud).
According to the Dart FAQ a pure string eval like that is not likely to make it into the language, even though other dynamic features will likely be added:
So, for example, Dart isn’t likely to support evaluating a string as
code in the current context, but it may support loading that code
dynamically into a new isolate. Dart isn’t likely to support adding
fields to a value, but it may (through a mirror system) support adding
fields to a class, and you can effectively add methods using
noSuchMethod(). Using these features will have a runtime cost; it’s
important to us to minimize the cost for programs that don’t use them.
This area is still under development, so we welcome your thoughts on
what you need from runtime dynamism.
I have an array which is filled with platforms that are supposed to move.
var MovingPlatformArray:Array = new Array();
for (var c:int = numChildren - 1; c >= 0; c--){
var child3:DisplayObject = getChildAt(c);
if (child3.name == "movingplatform"){
MovingPlatformArray.push(child3);
}
}
this.addEventListener(Event.ENTER_FRAME,ctrl_birdie);
function ctrl_birdie(e:Event):void{
for(var c in MovingPlatformArray){
MovingPlatform[c].y += speed;
if(MovingPlatformArray[c].hitTestPoint(birdie.x,birdie.y,true)){
birdtelleryvertrager=0;
birdtellery = 0;
birdie.y-=14;
}
if(movingplatform.y <= 25){
speed = 2;
}
if(movingplatform.y >= 350){
speed = -2;
}
}
Right now I have 2 moving platforms in this array. But only one moves up and down. But they both register a touch with the birdie. Am I doing something wrong?
In your listener, you're only setting the position of one platform, which ever one "movingplatform" is a reference to. As all your stage instances of moving platforms are named "movingplatform", one lucky platform is getting referenced by name (the rest ignored), instead of what you intended, which is to use the references in your array and adjust each platform.
You probably meant for movingplatform to be a local variable in your event handler, declared something like this:
var movingplatform:DisplayObject = MovingPlatformArray[c] as DisplayObject;
I'd recommend using a for each loop in place of the for in, because I think it's a little cleaner, but this is a minor style thing:
for each (var platform:DisplayObject in MovingPlatformArray)
{
platform.y += speed;
... rest of your code ...
}
For the sake of clarity, I edited the loop variable to be platform instead of movingplatform, to avoid confusion of having a local variable shadow a stage instance (i.e. this.movingplatform). I wanted it to be clear that the stage instance name is not being used here, because the unintentional instance name reference in your code is the source of your problem in the first place.
As far as i'm concerned, you have two options. use a for each, as adam smith suggested or use a for-loop as it was intended to be used :)
for(var c:uint = 0; c < MovingPlatformArray.length; c++){...
and btw: should "MovingPlatform[c].y += speed;" not be "MovingPlatformArray[c].y += speed;"?
edit: looking at your code, i would also suggest you use MovingPlatformArray[c].hitTestObject(birdie) instead of MovingPlatformArray[c].hitTestPoint(birdie.x,birdie.y,true)
If I were you, I would bring the logic for the platform out, and store it in a class. (Ideally you would do this for the birdie object as well). I have created an example below. The movieclips on the stage should extend Platform rather than MovieClip so they invoke the methods at the bottom.
// Use vectors if you know all the items are going to be the same type
var platforms:Vector.<Platform> = new <Platform>[];
for (var c:int = numChildren - 1; c >= 0; c--){
var child:DisplayObject = getChildAt(c);
// You shouldn't check against names (as per the original post). Because
// names should be unique
if (child is Platform){
platforms.push(child);
// This could be random so each platform has a different range
// This means platform 1 could go from y 30 to y 400, platform 2
// could go from y 60 to y 200, etc
child.setRange(25, 400);
}
}
this.addEventListener(Event.ENTER_FRAME, gameLoop);
// Have an overall game loop
function gameLoop(e:Event):void {
// Loop over the platforms
platforms.forEach(function(item:Platform, i:int, a:Vector.<Platform>):void {
// Hit test function in the class means you only have to pass in one mc
// rather than the points and a boolean
if(item.hitTest(birdie)) {
birdtelleryvertrager=0;
birdtellery = 0;
birdie.y-=14;
}
// Removed the movement logic, this should be kept out of the game loop
// plus how much better does this read?
item.move();
});
}
Then in a class location somewhere, like in a folder game/activeObjects
// A class for the platform stored else where
package game.activeObjects
{
import flash.display.MovieClip;
/**
*
*/
public class Platform extends MovieClip {
private const SPEED:Number = 2;
private var _direction:int = 1;
private var _minimumHeight:Number = 25;
private var _maximumHeight:Number = 350;
public function Platform() {
}
public function setRange(minimumHeight:Number, maximumHeight:Number) {
_minimumHeight = minimumHeight;
_maximumHeight = maximumHeight;
}
public function move():void {
this.y += SPEED * _direction;
if(this.y <= _minimumHeight) {
_direction = 1;
} else if(this.y >= _maximumHeight) {
_direction = -1;
}
}
public function hitTest(mc:MovieClip):Boolean {
return hitTestPoint(mc.x,mc.y,true);
}
}
}