Command:
hduser#bigtapp:/usr/local/giraph$ hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.7.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimplePageRankComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/input/tiny.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/output/SimplePageRank -w 1 -ca giraph.SplitMasterWorker=false
Result:
15/05/07 11:28:38 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one. 15/05/07
11:28:38 INFO utils.ConfigurationUtils: No edge output format
specified. Ensure your OutputFormat does not require one. 15/05/07
11:28:38 INFO utils.ConfigurationUtils: Setting custom argument
[giraph.SplitMasterWorker] to [false] in GiraphConfiguration 15/05/07
11:28:39 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable 15/05/07 11:28:39 INFO Configuration.deprecation:
mapreduce.job.counters.limit is deprecated. Instead, use
mapreduce.job.counters.max 15/05/07 11:28:39 INFO
Configuration.deprecation: mapred.job.map.memory.mb is deprecated.
Instead, use mapreduce.map.memory.mb 15/05/07 11:28:39 INFO
Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated.
Instead, use mapreduce.reduce.memory.mb 15/05/07 11:28:39 INFO
Configuration.deprecation: mapred.map.tasks.speculative.execution is
deprecated. Instead, use mapreduce.map.speculative 15/05/07 11:28:39
INFO Configuration.deprecation: mapreduce.user.classpath.first is
deprecated. Instead, use mapreduce.job.user.classpath.first 15/05/07
11:28:39 INFO Configuration.deprecation: mapred.map.max.attempts is
deprecated. Instead, use mapreduce.map.maxattempts 15/05/07 11:28:39
INFO job.GiraphJob: run: Since checkpointing is disabled (default), do
not allow any task retries (setting mapred.map.max.attempts = 1, old
value = 4) 15/05/07 11:28:39 INFO Configuration.deprecation:
mapred.job.tracker is deprecated. Instead, use
mapreduce.jobtracker.address 15/05/07 11:28:39 INFO client.RMProxy:
Connecting to ResourceManager at /0.0.0.0:8032 15/05/07 11:28:42 INFO
mapreduce.JobSubmitter: number of splits:1 15/05/07 11:28:42 INFO
mapreduce.JobSubmitter: Submitting tokens for job:
job_1430974889906_0002 15/05/07 11:28:42 INFO impl.YarnClientImpl:
Submitted application application_1430974889906_0002 15/05/07 11:28:42
INFO mapreduce.Job: The url to track the job:
http://bigtapp:8088/proxy/application_1430974889906_0002/ 15/05/07
11:28:42 INFO job.GiraphJob: Tracking URL:
http://bigtapp:8088/proxy/application_1430974889906_0002/ 15/05/07
11:28:42 INFO job.GiraphJob: Waiting for resources... Job will start
only when it gets all 2 mappers 15/05/07 11:28:55 INFO mapreduce.Job:
Running job: job_1430974889906_0002 15/05/07 11:28:55 INFO
mapreduce.Job: Job job_1430974889906_0002 running in uber mode : false
15/05/07 11:28:55 INFO mapreduce.Job: map 100% reduce 0% 15/05/07
11:28:55 INFO mapreduce.Job: Job job_1430974889906_0002 failed with
state FAILED due to: Task failed task_1430974889906_0002_m_000000 Job
failed as tasks failed. failedMaps:1 failedReduces:0
15/05/07 11:28:55 INFO mapreduce.Job: Counters: 8 Job Counters
Failed map tasks=1 Launched map tasks=1 Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=5785 Total
time spent by all reduces in occupied slots (ms)=0 Total time spent
by all map tasks (ms)=5785 Total vcore-seconds taken by all map
tasks=5785 Total megabyte-seconds taken by all map tasks=5923840
Related
I am trying to use adaptive schedule with flink 1.14 to run flink job based on available resources instead of waiting for required parallelism (scaling) but I don't see flink is getting recognize adaptive schedule.
Ex: flink run -m yarn-cluster -ynm jobName -p 128 -D jobmanager.scheduler=Adaptive -D cluster.declarative-resource-management.enabled=true -c className JarName
Reference : https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/elastic_scaling/
Caused by: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
at org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResource$8(DefaultScheduler.java:515)
... 37 more
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
... 35 more
Regards,
Madan
See the section of the docs describing Limitations of Elastic Scaling. In particular, this part, which explains that yarn is not supported:
Deployment is only supported as a standalone application deployment. Active resource providers (such as native Kubernetes, YARN) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications.
The only supported deployment options are Standalone in Application Mode (described on this page), Docker in Application Mode and Standalone Kubernetes Application Cluster.
DSE server Version : 6.8
Followed installation steps as per the datastax documentation (Tar file installation)
Startup command: bin/dse cassandra -s (Needs search featire , so enabled solr as well)
Error while executing start command:
WARN [main] 2022-03-01 19:05:59,855 DatabaseDescriptor.java:1531 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
INFO [main] 2022-03-01 19:05:59,857 DseDelegateSnitch.java:39 - Setting my workloads to [Cassandra, Search]
INFO [main] 2022-03-01 19:05:59,904 YamlConfigurationLoader.java:77 - Configuration location: file:/Users/rajamani/repositories/cassandra/dse-6.8.20/resources/cassandra/conf/cassandra.yaml
INFO [main] 2022-03-01 19:05:59,912 DseDelegateSnitch.java:41 - Initialized DseDelegateSnitch with workloads [Cassandra, Search], delegating to com.datastax.bdp.snitch.DseSimpleSnitch
INFO [main] 2022-03-01 19:06:00,049 YamlConfigurationLoader.java:77 - Configuration location: file:/Users/rajamani/repositories/cassandra/dse-6.8.20/resources/cassandra/conf/cassandra.yaml
INFO [main] 2022-03-01 19:06:01,154 AuthConfig.java:125 - System keyspaces filtering not enabled.
INFO [main] 2022-03-01 19:06:01,155 IAuditLogger.java:136 - Audit logging is disabled
WARN [main] 2022-03-01 19:06:01,215 DisabledTPCBackpressureController.java:20 - TPC backpressure is disabled. NOT RECOMMENDED.
INFO [main] 2022-03-01 19:06:01,216 TPC.java:137 - Created 9 NIO event loops (with I/O ratio set to 50).
INFO [main] 2022-03-01 19:06:01,239 TPC.java:144 - Created 1 TPC timers due to configured ratio of 5.
INFO [main] 2022-03-01 19:06:01,524 DseConfig.java:372 - CQL slow log is enabled
INFO [main] 2022-03-01 19:06:01,526 DseConfig.java:373 - CQL system info tables are not enabled
INFO [main] 2022-03-01 19:06:01,526 DseConfig.java:374 - Resource level latency tracking is not enabled
INFO [main] 2022-03-01 19:06:01,526 DseConfig.java:375 - Database summary stats are not enabled
INFO [main] 2022-03-01 19:06:01,526 DseConfig.java:376 - Cluster summary stats are not enabled
INFO [main] 2022-03-01 19:06:01,526 DseConfig.java:377 - Histogram data tables are not enabled
INFO [main] 2022-03-01 19:06:01,528 DseConfig.java:378 - User level latency tracking is not enabled
INFO [main] 2022-03-01 19:06:01,529 DseConfig.java:380 - Spark cluster info tables are not enabled
INFO [main] 2022-03-01 19:06:01,531 DseConfig.java:420 - Cql solr query paging is: off
INFO [main] 2022-03-01 19:06:01,535 DseUtil.java:324 - /proc/cpuinfo is not available, defaulting to 1 thread per CPU core...
INFO [main] 2022-03-01 19:06:01,536 DseConfig.java:424 - This instance appears to have 1 thread per CPU core and 10 total CPU threads.
INFO [main] 2022-03-01 19:06:01,538 DseConfig.java:441 - Server ID:F4-D4-88-66-17-8D
ERROR [main] 2022-03-01 19:06:02,024 DseModule.java:114 - Class not found: org/apache/lucene/uninverting/FieldCache. Exiting...
This particular class exists as part of solr-core.
does DSE-server does not have solr bundle ? (Even after placing the lib under solr lib, this particulat error occurs.
Can you please assist to resolve the issue ?
The error is most likely a symptom of another problem. For example, it's quite common to get "class not found" exceptions when using Java 11 with Cassandra. DataStax Enterprise 6.8 is compatible with Cassandra 3.11 which only supports Java 8.
For what it's worth, Java 11 support was only added to Cassandra 4.0 (CASSANDRA-16894). Older versions of Cassandra only work with Java 8.
Going back to your original question, we need a bit more information to investigate the issue but our ability to help you in a Q&A forum is limited. Please log a ticket with DataStax Support and one of our engineers will advise you on what diagnostic info is required and the next steps. Cheers!
For some broader context: I am wanting to move a react app that I have built and previously hosted through Firebase Hosting over to GAE to host it there so that I can connect to both my firebase and cloud postgresql databases (could not find a way to connect to cloud postgresql from my app hosted on Firebase).
Apologies in advance for the somewhat vague question, but I've run into several issues just working through the bookshelf app tutorial and have been able to work through some of the errors but have not been able to figure out the issue here:
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build eaf3e253-ee83-46d4-a640-fd7e68083a13 status: FAILURE.
Build error details: {"error":{"errorType":"BuildError","canonicalCode":"INVALID_ARGUMENT","errorId":"2BCB87EC","errorMessage":"INFO FTL version node-v0.17.0\n
INFO Beginning FTL build for node\n
INFO FTL arg passed: exposed_ports None\n
INFO FTL arg passed: cache_repository us.gcr.io/piva-primero/app-engine-tmp/build-cache/ttl-7d\n
INFO FTL arg passed: tar_base_image_path None\n
INFO FTL arg passed: export_cache_stats False\n
INFO FTL arg passed: builder_output_path \"\"\n
INFO FTL arg passed: name us.gcr.io/piva-primero/app-engine-tmp/app/ttl-2h:55db4aec-8e29-42ea-bea3-c457808a429c\n
INFO FTL arg passed: ttl 168\n
INFO FTL arg passed: global_cache False\n
INFO FTL arg passed: cache True\n
INFO FTL arg passed: upload True\n
INFO FTL arg passed: sh_c_prefix False\n
INFO FTL arg passed: fail_on_error True\n
INFO FTL arg passed: base us.gcr.io/gae-runtimes/nodejs10:nodejs10_20191019_10_16_3_RC00\n
INFO FTL arg passed: output_path None\n
INFO FTL arg passed: cache_key_version v0.17.0\n
INFO FTL arg passed: cache_salt \n
INFO FTL arg passed: directory /workspace\n
INFO FTL arg passed: entrypoint None\n
INFO FTL arg passed: additional_directory /.gaeconfig\n
INFO FTL arg passed: destination_path /srv\n
INFO FTL arg passed: verbosity NOTSET\n
INFO starting: full build\n
INFO starting: builder initialization\n
INFO Loading Docker credentials for repository 'us.gcr.io/gae-runtimes/nodejs10:nodejs10_20191019_10_16_3_RC00'\n
INFO Loading Docker credentials for repository 'us.gcr.io/piva-primero/app-engine-tmp/app/ttl-2h:55db4aec-8e29-42ea-bea3-c457808a429c'\n
INFO builder initialization took 0 seconds\n
INFO starting: build process for FTL image\n
INFO starting: rm_node_modules\n
INFO rm_node_modules rm -rf /workspace/node_modules\n
INFO `rm_node_modules` stdout:\n\n
INFO rm_node_modules took 0 seconds\n
INFO using descriptor:package-lock.json\n
INFO using descriptor:package.json\n
INFO starting: checking_cached_packages_json_layer\nDEBUG Checking cache for cache_key ddd19bc8f86cc8fedfd69dfce5aac6d21a6e2024dec42d6d3a96af6fc7a78dbd\n
INFO No cached base image found for entry: us.gcr.io/piva-primero/app-engine-tmp/build-cache/ttl-7d/node-cache:ddd19bc8f86cc8fedfd69dfce5aac6d21a6e2024dec42d6d3a96af6fc7a78dbd.\n
INFO Cache miss on local cache for us.gcr.io/piva-primero/app-engine-tmp/build-cache/ttl-7d/node-cache:ddd19bc8f86cc8fedfd69dfce5aac6d21a6e2024dec42d6d3a96af6fc7a78dbd\n
INFO No cached dependency layer for ddd19bc8f86cc8fedfd69dfce5aac6d21a6e2024dec42d6d3a96af6fc7a78dbd\n
INFO [CACHE][MISS] v0.17.0:NODE-\u003eddd19bc8f86cc8fedfd69dfce5aac6d21a6e2024dec42d6d3a96af6fc7a78dbd\n
INFO checking_cached_packages_json_layer took 0 seconds\n
INFO starting: building_packages_json_layer\n
INFO starting: npm_install\n
INFO npm_install npm install --production\n
INFO `npm_install` stdout:\n\n\u
003e grpc#1.7.3 install /workspace/node_modules/#google-cloud/video-intelligence/node_modules/grpc\n\u
003e node-pre-gyp install --fallback-to-build --library=static_library\n\n
make: Entering directory '/workspace/node_modules/#google-cloud/video-intelligence/node_modules/grpc/build'\n
make: Entering directory '/workspace/node_modules/#google-cloud/video-intelligence/node_modules/grpc/build'\n
CC(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o\n
CC(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o\n
grpc.target.mk:405: recipe for target 'Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o' failed\n
make: Leaving directory '/workspace/node_modules/#google-cloud/video-intelligence/node_modules/grpc/build'\n
CC(target) Release/obj.target/grpc/deps/grpc/src/core/lib/channel/channel_args.o\n
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/workspace/node_modules/#google-cloud/video-intelligence/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-glibc/grpc_node.node --modul.
I'm trying to run the tutorial code on my machine. I've been able to successfully deploy a different app in another project but have had some real trouble with this one. Any tips for resolving this error would be very helpful.
I'm trying to setup a cluster (Apache Flink 1.6.1) with Kubernetes and get following error when I run a job on it:
2018-10-09 14:29:43.212 [main] INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-10-09 14:29:43.214 [main] INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.flink.runtime.entrypoint.ClusterConfiguration.<init>(Ljava/lang/String;Ljava/util/Properties;[Ljava/lang/String;)V
at org.apache.flink.runtime.entrypoint.EntrypointClusterConfiguration.<init>(EntrypointClusterConfiguration.java:37)
at org.apache.flink.container.entrypoint.StandaloneJobClusterConfiguration.<init>(StandaloneJobClusterConfiguration.java:41)
at org.apache.flink.container.entrypoint.StandaloneJobClusterConfigurationParserFactory.createResult(StandaloneJobClusterConfigurationParserFactory.java:78)
at org.apache.flink.container.entrypoint.StandaloneJobClusterConfigurationParserFactory.createResult(StandaloneJobClusterConfigurationParserFactory.java:42)
at org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:55)
at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:153)
My job takes a configuration file (file.properties) as a parameter. This works fine in standalone mode but apparently the Kubernetes cluster cannot parse it
job-cluster-job.yaml:
args: ["job-cluster", "--job-classname", "com.test.Abcd", "-Djobmanager.rpc.address=flink-job-cluster",
"-Dparallelism.default=1", "-Dblob.server.port=6124", "-Dquery.server.ports=6125", "file.properties"]
How to fix this?
Update: The job was built for Apache 1.4.2 and this might be the issue, looking into it.
The job was built for 1.4.2, the class with the error (EntrypointClusterConfiguration.java) was added in 1.6.1 (https://github.com/apache/flink/commit/ab9bd87e521d19db7c7d783268a3532d2e876a5d#diff-d1169e00afa40576ea8e4f3c472cf858) it seems, so this caused the issue.
We updated the job's dependencies to point to new 1.6.1 release and the arguments are parsed correctly.
We are running Zeppelin on docker containers in a locked-down enterprise environment. When Zeppelin starts, it tries to connect to AWS, times-out after a while, but successfully starts. The log trace is below -
INFO [2018-09-03 14:26:25,131] ({main} Notebook.java[<init>]:128) - Notebook indexing finished: 0 indexed in 0s
INFO [2018-09-03 14:26:25,133] ({main} Helium.java[loadConf]:103) - Add helium local registry /opt/zeppelin-0.8.0/helium
INFO [2018-09-03 14:26:25,134] ({main} Helium.java[loadConf]:100) - Add helium online registry https://s3.amazonaws.com/helium-package/helium.json
WARN [2018-09-03 14:26:25,138] ({main} Helium.java[loadConf]:111) - /opt/zeppelin-0.8.0/conf/helium.json does not exists
ERROR [2018-09-03 14:28:32,864] ({main} HeliumOnlineRegistry.java[getAll]:80) - Connect to s3.amazonaws.com:443 [s3.amazonaws.com/54.231.81.59] failed: Connection timed out
INFO [2018-09-03 14:28:33,840] ({main} ContextHandler.java[doStart]:744) - Started o.e.j.w.WebAppContext#ef9296d{/,file:/opt/zeppelin-0.8.0/webapps/webapp/,AVAILABLE}{/opt/zeppelin-0.8.0/zeppelin-web-0.8.0.war}
INFO [2018-09-03 14:28:33,846] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector#1b1c538d{HTTP/1.1}{0.0.0.0:9991}
INFO [2018-09-03 14:28:33,847] ({main} Server.java[doStart]:379) - Started #145203ms
We have no use-case for Helium (as of now) and the delay in the zeppelin restart affects us. Is there a way we can disable this dependency on Helium?
Thanks!
There was PR3082 ([ZEPPELIN-3636] Add timeout for s3 amazon bucket endpoint) that allows not to wait to Amazon.
PR was merged to master, perhaps will be merged to branch-0.8.