When I run a Python file in Flink CLI using the following code:
python3 word_count.py
I got the error like this:
Traceback (most recent call last):
File "word_count.py", line 79, in <module>
word_count()
File "word_count.py", line 37, in word_count
t_config = TableConfig()
File "/usr/local/lib/python3.7/dist-packages/pyflink/table/table_config.py", line 49, in __init__
gateway = get_gateway()
File "/usr/local/lib/python3.7/dist-packages/pyflink/java_gateway.py", line 68, in get_gateway
callback_server_listening_address, callback_server_listening_port)
TypeError: 'JavaPackage' object is not callable
And I changed the way to run this Python file:
./bin/flink run --python3 /opt/flink/examples/python/table/batch/word_count.py
I got another error :
Could not build the program from JAR file.
Use the help option (-h or --help) to get help on the command.
Didn't figure out the first question but solved the second one.
The command within --python3 caused the error.
The official Pyflink command is
./bin/flink run -py word_count.py
Since there is only Python3 in my docker, I soft linked Python3 to Python using :
ln -s /usr/bin/python3 /usr/bin/python
And then the official Pyflink command works. You can see the completed job through Flink Web UI.
The default is to use the python interpreter on the machine to compile pyflink jobs, you can change it through python.client.executable.
Related
LE: I want to do this on macOS.
I am trying to run Apache Beam on a separate Flink cluster, as described here: https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
(1) Start a Flink cluster which exposes the Rest interface
(e.g. localhost:8081 by default).
Flink 1.13.1 starts successfully on localhost:8081.
(2) Start JobService with Flink Rest endpoint: docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081.
I am running this instead, which again starts successfully:
docker run --net=host apache/beam_flink1.13_job_server:latest --flink-master=localhost:8081
(3) Submit the pipeline as above.
I'm running the WordCount pipeline as:
python -m apache_beam.examples.wordcount --input /Users/stefan/datastore/input.txt
--output /Users/stefan/datastore/output.txt
--runner=PortableRunner --job_endpoint=localhost:8099
After 1 minute the pipeline crashes with a grpc.FutureTimeoutError. This is the full output:
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0
INFO:root:No image given, using default Python SDK image
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0
INFO:root:Python SDK container image set to "apache/beam_python3.8_sdk:2.31.0" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x137d441f0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x137d44280> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x137d449d0> ====================
Traceback (most recent call last):
File "/usr/local/Cellar/python#3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Cellar/python#3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 94, in <module>
run()
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 89, in run
output | 'Write' >> WriteToText(known_args.output)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 585, in __exit__
self.result = self.run()
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 564, in run
return self.runner.run_pipeline(self, self._options)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 438, in run_pipeline
job_service_handle = self.create_job_service(options)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 317, in create_job_service
return self.create_job_service_handle(server.start(), options)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/job_server.py", line 54, in start
grpc.channel_ready_future(channel).result(timeout=self._timeout)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result
self._block(timeout)
File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError
Any idea what's going on? Thank you for help. I also opened a bug regarding this issue here: https://issues.apache.org/jira/browse/BEAM-12657
The cluster is started from a docker container. You have to publish the ports so that the Python code executed on the host can access the port.
See https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/ for the --publish option used in the docker run cmd.
I'm creating a bot for Discord, and I just wrote this simple code:
import discord
TOKEN = "token"
client = discord.Client()
#client.event
async def on_ready():
print('Bot is ready.')
client.run(TOKEN)
and it produces the following error:
Traceback (most recent call last):
File "/Users/pcaires/Desktop/Programação/Python/Discord Bots/Main.py", line 1, in <module>
import discord
File "/Users/pcaires/Desktop/Programação/Python/Discord Bots/venv/lib/python3.7/site-packages/discord/__init__.py", line 20, in <module>
from .client import Client, AppInfo, ChannelPermissions
File "/Users/pcaires/Desktop/Programação/Python/Discord Bots/venv/lib/python3.7/site-packages/discord/client.py", line 38, in <module>
from .state import ConnectionState
File "/Users/pcaires/Desktop/Programação/Python/Discord Bots/venv/lib/python3.7/site-packages/discord/state.py", line 36, in <module>
from . import utils, compat
File "/Users/pcaires/Desktop/Programação/Python/Discord Bots/venv/lib/python3.7/site-packages/discord/compat.py", line 32
create_task = asyncio.async
^
SyntaxError: invalid syntax
I searched and searched in the internet, and most of the people say to use Python 3.7, and that's what I've been using. Also, I've been using PyCharm as my IDE for Python.
Where does the error come from?
The version of discord.py you are using does not support Python 3.7 (in which async becomes a reserved keyword), as explained in this issue.
This version of discord.py, which is the default branch on the GitHub repo, is sadly the one installed by Pip.
How to fix it
You can either:
downgrade your version of Python to 3.6.
install another version of discord.py, based on the rewrite branch which is under active development, for example with the command : python3 -m pip install --user -U https://github.com/Rapptz/discord.py/archive/rewrite.zip
You can manually edit the file and change that line from create_task = asyncio.async to create_task = getattr(asyncio, 'async')
See more info here: https://github.com/Rapptz/discord.py/issues/1249
Do NOT add asyncio in your requirements, it's already in Python (since 3.5).
It is only relevant for Python 3.3, which does not include asyncio in its stdlib.
As a quick fix you can change asyncio.async to asyncio.ensure_future in the installed offending module and run it. Obviously the right thing to do is get the module updated, but when that's not possible the above will get it running again.
fix it with
pip install --upgrade aiohttp
pip install --upgrade websockets
Adafruit LED backpack
Am I being stupid (probably yes)
I'm after python 3 versions of this module, every time I download it I get the python 2.7 versions
getting my module from
git clone https://github.com/adafruit/Adafruit_Python_LED_Backpack.git
I'm doing this on a raspberry pi
the python 2 version works fine but my program is written in python 3
new error I'm getting is:
Traceback (most recent call last): File
"/home/pi/Adafruit_Python_LED_Backpack/matrix8x16_test.py", line 37,
in
display.begin() File "/home/pi/Adafruit_Python_LED_Backpack/Adafruit_LED_Backpack/HT16K33.py",
line 53, in begin
self._device.writeList(HT16K33_SYSTEM_SETUP | HT16K33_OSCILLATOR, []) File
"/usr/local/lib/python3.4/dist-packages/Adafruit_GPIO/I2C.py", line
127, in writeList
self._bus.write_i2c_block_data(self._address, register, data) File
"/usr/local/lib/python3.4/dist-packages/Adafruit_PureIO/smbus.py",
line 274, in write_i2c_block_data
self._device.write(data) OSError: [Errno 5] Input/output error
this package seems to be compatible between both python 2 & 3.
after cloning the package, when you want to install it in python interpreter, execute this command:
sudo python -3 setup.py install
'-3' means running on python 3.
I am using libcloud 1.2.1 with pysphere 0.1.7. I am getting the below error when trying to connect to the ESXi host.
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/libcloud/compute/drivers/vsphere.py", line 152, in init
port=port, url=url)
File "/Library/Python/2.7/site-packages/libcloud/common/base.py", line 1177, in init
self.connection = self.connectionCls(args, *conn_kwargs)
TypeError: init() got an unexpected keyword argument 'retry_delay'
I am using the below commands to connect to the host.
from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver
vsphere = get_driver(Provider.VSPHERE)
driver = vsphere(host='',username='username',password='password')
I can do this using pysphere only but not with libcloud. Tried looking on the internet and changing the content of the libraries but in vain. Can anyone help ?
This is a bug in v1.5.0<, there is a committed fix. You can apply the patch manually using git am: https://patch-diff.githubusercontent.com/raw/apache/libcloud/pull/967.patch
Wait for v1.6.0, or install directly from GitHub trunk
pip install git+https://github.com/apache/libcloud.git#trunk#egg=apache-libcloud
I'm having issues getting started using GAE (1.7.6 on OS X 10.6.8 with Python 2.7) and compared to amazon services it is abysmal. However, I would like to at least get the sample app deployed. After working through the other issues I now have a small sample application ready to deploy. When I try to deploy it, I receive an SSL error as follows:
*** Running appcfg.py with the following flags:
--no_cookies --email=mathsboy#gmail.com --passin update
02:15 PM Host: appengine.google.com
02:15 PM Application: tew-helloworld; version: 1
Traceback (most recent call last):
File "google_appengine/appcfg.py", line 171, in <module>
run_file(__file__, globals())
File "google_appengine/appcfg.py", line 167, in run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4155, in <module>
main(sys.argv)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 4146, in main
result = AppCfgApp(argv).Run()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2334, in Run
self.action(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 3881, in __call__
return method()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appcfg.py", line 2892, in Update
updatecheck.CheckForUpdates()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/sdk_update_checker.py", line 258, in CheckForUpdates
runtime=runtime))
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 393, in Send
f = self.opener.open(req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 394, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 412, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1207, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 383, in do_open
url_error.reason.args[1])
fancy_urllib.InvalidCertificateException: Host appengine.google.com returned an invalid certificate (_ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed):
To learn more, see http://code.google.com/appengine/kb/general.html#rpcssl
If deploy fails you might need to 'rollback' manually.
The "Make Symlinks..." menu option can help with command-line work.
*** appcfg.py has finished with exit code 1 ***
There was some advice on this in another thread stating that you would need to remove cacerts.txt, though I am unable to locate this file on OS X. Also, it seems retarded that in order to have the code upload I'd have to delete a cacert text file. But maybe I'm an idiot.
Another problem I'm facing here, is that OS X doesn't work well with GAE due to the difficulty in managing python versions. I've got a total mess on my hands now after installing multiple versions of python, EPD, ActiveState, MacPorts. None of these worked in solving me earlier problems despite assurances they would. It's really pretty appalling that this is the state of google's application management. It just seems like GAE was built by a bunch of guys from mid-90s microsoft....anyone advice on the above issue would be greatly appreciated. Cheers
Fourth line from end in your error. Relevant instructions under the link.
http://code.google.com/appengine/kb/general.html#rpcssl
Make sure (from documentation):
To do this you must have the ssl Python module installed on your system.
Workaround: Use the command line argument --skip_sdk_update_check=yes when you run dev_appserver.py.
I had the same problem and noticed the GUI tool (GoogleAppEngineLauncher) uses --skip_sdk_update_check=yes every time you start an app. This is as of SDK 1.8.7.
The root cause seems to be a known bug in the GAE development server's module whiltelisting system. Even if you have the ssl module installed (as the documentation says is required), ssl relies on _ssl, which is not whitelisted. This is why you can execute import ssl in a regular python shell session with no problem, but get an error when you execute the same line in a GAE app.