We use Axon and I am trying to make a distributed command bus for one of our services Deployed on Docker Swarm.
Anyone already have experiance/tips for setting this up?
We are using:
Axon
Docker Swarm
Spring-Boot
And I want to use JGroups icw DNSPING for the distributed setup.
Regards,
Johan.
Related
My goal is to create a streaming pipeline to read data from Apache Kafka, process the data, and write back to it.
Because of security reasons, I want to avoid Docker and use Podman.
I have set up a minimal cluster via a docker-compose.yml with a jobmanager, taskmanager and a Python SDK harness worker. The SDK harness worker seems to get stuck when i try to execute a pipeline.
Running the pipeline (reading a multi-line .txt file and writing it back in a file) it gets transferred to the jobmanager and taskmanager correctly, but then goes idle. When I look in the pythonsdk container, the logs show the following message repeatedly:
2022/12/04 16:13:02 Starting worker pool 1: python -m
apache_beam.runners.worker.worker_pool_main --service_port=50000
--container_executable=/opt/apache/beam/boot
Starting worker with command ['/opt/apache/beam/boot', '--id=1-1',
'--logging_endpoint=localhost:45087',
'--artifact_endpoint=localhost:35323',
'--provision_endpoint=localhost:36435',
'--control_endpoint=localhost:33237']
2022/12/04 16:16:31 Failed to obtain provisioning information: failed to
dial server at localhost:36435
caused by:
context deadline exceeded
Here is a link to a test pipeline that was created:
Example on github
Environment:
Debian 11;
Podman;
Python 3.2.9;
apache-beam==2.38.0; and
podman-compose
The setup of the cluster defined in:
docker-compose.yml
1x flink-jobmanager (flink version 1.14)
1x flink-taskmanager
1x Python Harness SDK
I chose to create a SDK container manually because I don't have Docker installed and Flink fails when it tries to create a container
over Docker.
I suspect that I have made a mistake in the network setup or there are some configurations missing for the harness worker, but I could not figure out the problem. Any thoughts?
Crossposted in user mailing list of beam.apache.org
I'm very new to docker and docker-compose things. I have tried to use Kiwi TCMS open source project which supposed to use with Docker.
My question is can I run the projects on docker on the same server, I'm supposed to configure my development and production sites on the same server (CentOS)?
I'm following the below link to install docker and configure the kiwitcms application for the first time, I read the basics about the docker and how its working and all.
https://kiwitcms.readthedocs.io/en/latest/installing_docker.html#
I want to use PostgreSQL as my Database, but the existing latest docker image has MariaDB. So after I pulled latest version of kiwitcms from docker hub using the following command,
docker pull kiwitcms/kiwi
Should I change to "docker-compose.yml" file db image value, and save it to a local new directory,
https://raw.githubusercontent.com/kiwitcms/Kiwi/master/docker-compose.yml
[![enter image description here][1]][1]
Also there can we edit use our own DB name, user name passwords right?
then execute the following command.
docker-compose up -d
I'm very new to this model and read most of the articles related to docker, but this leads me few of these doubts, hence asking for these clarifications.
Thanks,
Karthik.
The web app and the DB server are 2 different images (aka 2 different servers). For example how we use Postgres in testing see:
https://github.com/kiwitcms/Kiwi/blob/master/docker-compose.postgres
I want to run a Python code using Apache beam on Apache Flink. The command that the apache beam site for launching Python code on Apache Flink is as follows:
docker run --net=host apachebeam/flink1.9_job_server:latest --flink-master=localhost:8081
The following is a discussion of different methods of executing code using Apache Fail on Apache Flink. But I haven't seen an example of launching it.
https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html
I want this code to run without Docker. How is this code commanded?
You can spin up the flink job server directly using the beam source source. Note you'll need to install java.
1) Clone the beam source code:
git clone https://github.com/apache/beam.git
2) Start the job server
cd beam
./gradlew -p runners/flink/1.8/job-server runShadow -PflinkMasterUrl=localhost:8081
Some helpful tips:
This is not flink itself! You'll need to spin up flink separately.
The flink job service actually spins up a few services:
Expansion Service (port 8097) : This service allows you to use ExternalTransforms within your pipeline that exist within the java sdk. For example the transforms found within the python sdk apache_beam.io.external.* hit this expansion service.
Artifact Service (port 8098) : This is where the pipeline uploads your python artifacts (e.g. pickle files, etc) to be used by the flink taskmanager when it executes your python code. From what I recall you must share the artifact staging area (default to /tmp/beam-artifact-staging) between the flink taskworker and this artifact service.
Job Service (port 8099) : This is what you submit your pipeline to. It translates your pipeline into something for flink and submits it.
I'm on Macos Mojave, I'm learning docker and cocorico to use the marketplace.
I've installed cocorico from cocolabs, mongo and mysql with docker, but now the supervisor tell "mysql is exited".
I have some trouble with my db : http://localhost:9001/?message=Process%20mysql%20started
I need help to create db I think I'm noob in docker and symfony and there's no tutorial video or anything.
Thanks you so much
Could you upload your Dockerfile and docker-compose? I'll try to investigate from that.
I'm using Java but this isn't necessarily a Java question. Google's "java-compat" image is Debian (3.16.7-ckt20-1+deb8u3~bpo70+1 (2016-01-19)).
Here is my Dockerfile:
FROM gcr.io/google_appengine/java-compat
RUN apt-get -qqy update && apt-get qqy install curl xvfb x11vnc
RUN mkdir -p ~/.vnc
RUN x11vnc -storepasswd xxxxxxxx ~/.vnc/passwd
EXPOSE 5900
ADD . /app
And in the Admin Console I created a firewall rule to open up 5900. And lastly I am calling the vnc server itself in the "_ah/start" startup hook with this command:
x11vnc -forever -usepw -create
All seems to be setup correctly but I'm unable to connect with TightVNC. I use the public (ephemeral) IP address for the instance I find in the Admin Console followed by ::5900 (TightVNC requires two colons for some reason). I'm getting a message that the server refused the connection. And indeed when I try to telnet to port 5900 it's blocked.
Next I SSH into the container machine and when I test the port on the container with wget xxx.xxx.xxx.xxx:5900 I get a connection. So it seems to me the container is not accepting connections on port 5900. Am I getting this right? Is it possible to open up ports and route my VNC client into the docker container? Any help appreciated.
Why I can't use Compute Engine. Just to preempt some comments about using google's Compute Engine environment instead of Managed VMs. I make heavy use of the Datastore and Task Queues in my code. I don't think those can run (or run natively/efficiently) on Compute Engine. But I may pose that as a separate question.
Update: Per Paul in the comments... having learned some of the docker terminology: Can I publish a port on the container in Google's environment?
Out of curiosity - why are you trying to VNC into your instances? If it's just for management purposes, you can SSH into Managed VM instances.
That having been said - you can use the network/forwarded_ports config to route traffic from the VM to the application container:
network:
forwarded_ports:
- 5900
instance_tag: vnc
Put that in your app.yaml, and re-deploy your app. You'll also need to open the port in your firewall (if you intend on accessing this from the public internet):
gcloud compute firewall-rules create default-allow-vnc \
--allow tcp:5900 \
--target-tags vnc \
--description "Allow vnc traffic on port 5900"
Hope this helps!