I am trying to feed a Vespa index, but I face a NO_SPACE error:
Detail resultType=FATAL_ERRORexception='ReturnCode(NO_SPACE, Put operation rejected for document 'id:site-search:site::<urn:uuid:033689d2-5f23-4afb-815d-d6f5d6ecbcbd>' of type 'site': 'enumStoreLimitReached: { action: "add more content nodes", reason: "enum store address space used (0.92813) > limit (0.9)", enumStore: { used: 31890298144, dead: 0, limit: 34359738368}, attributeName: "text", subdb: "ready"}')' endpoint=vespa1:8080 ssl=false resultTimeLocally=1532685239428
My Vespa nodes are running through Docker with volumes on the machine.
Here are the volumes used:
-v /data/1/bench/vespa/sample-apps:/vespa-sample-apps -v /data/1/bench/vespa/logs:/opt/vespa/logs --volume /data/1/bench/vespa/var:/opt/vespa/var
So, if I'm right, the data should be stored in the /data/1 directory, and according to df:
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 65802860 0 65802860 0% /dev
tmpfs 13162916 108488 13054428 1% /run
/dev/mapper/VgRoot-root 78217900 3249540 70952028 5% /
/dev/mapper/VgRoot-boot 237861 36998 184275 17% /boot
/dev/mapper/VgRoot-srv 9775612 36888 9222424 1% /srv
/dev/mapper/VgData01-data 3688163424 48757376 3451987424 2% /data/1
There is still a lot of space left there. So I was wondering, if maybe Vespa was not looking at the right disk partition to check if it is using more than 90% of the disk space.
There is an upper limit to the number of unique values for attributes per node - please refer to https://docs.vespa.ai/documentation/performance/attribute-memory-usage.html#data
The only remedy is to add more content nodes to the Vespa application - this is auto-redistribute documents over more nodes and hence reduce number of unique values in the attribute
Related
I am trying to use flink for data-enrichment on multiple streams of data.
Here I have some data in account_stream and status_stream. I want to add that data to all other streams coming from multiple different sources. all the streams have one field common in their data: "account_id".
This is the approach i took.
account_stream.connect(status_stream)
.flat_map(EnrichmentFunction())
.filter(lambda x: x['name'] != "-" and x['date'] != "0000-00-00 00:00:00")
.key_by(lambda row: row['account_id'])
.connect(stream1)
.flat_map(function_2())
.filter(lambda x: x!="2")
.key_by(lambda row: row['account_id'])
.connect(stream2)
.flat_map(function_2())
.key_by(lambda row: row['account_id'])
.connect(stream3)
.flat_map(function_3())
.key_by(lambda row: row['account_id'])
.connect(stream4)
.flat_map(function_4())
.key_by(lambda row: row['account_id'])
.connect(stream5)
.flat_map(function_5())
.key_by(lambda row: row['account_id'])
.connect(stream6)
.flat_map(function_6())
.key_by(lambda row: row['account_id'])
.connect(stream7)
.flat_map(function_7())
.key_by(lambda row: row['account_id'])
.connect(stream_8)
.flat_map(function_8())
.map(lambda a: str(a),Types.STRING())
.add_sink(kafka_producer)
I am saving necessary data in state and appending that to all streams using flat_map function. And at the end adding one kafka sink to send all streams enriched with state.
Now once I execute this, I am getting this error:''java.io.IOException: Insufficient number of network buffers: required 17, but only 8 available. The total number of network buffers is currently set to 2048 of 32768 bytes each.''
I tried changing taskmanager.memory.network.fraction to 0.5 , taskmanager.memory.network.max to 15 gb and taskmanager.memory.process.size to 10 gb in flink config file. But it still gave same error. Do I have to do something other than just saving it to see the changes reflect in flink job? or problem is something else?
Also let me know if this approach is not efficient for the task and if there's something else I should try?
I am using single 32gb ram, 8-core server to run this in python with pyflink library, with kafka and elastic running on same server.
Thank you.
You can refer to the Set up TaskManager Memory page of the official documentation for how to configure the network memory for TaskManager. There are several things that need to be taken care of:
taskmanager.memory.network.fraction of total flink memory to be used as network memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used.
The size of network memory cannot exceeds the size of total process memory.
You can find the current max/min value of network memory at the beginning of TaskManager's log. Check it to see whether your configuration works or not.
If you can upgrade your Flink to 1.14, you can try the newest feature: Fine-Grained Resource Management. With this feature the network memory will be automatically configured as the amount each TaskManager requires. However, to use this feature you need to set SlotSharingGroups for each operator and config CPU and memory resources for them. For more details, please refer to the official documentation.
I have a requirement where I need to run query like below and fetch 2-3 attributes for all entities satisfying this query. The number of distinguishedName would be around 100 in a single query. As I see in the microsoft documentation, that distinguishedName is not indexed, I suspect that this might cause performance issues.
Does anybody know if this would indeed cause performance issues? Apart from the below ldap filter, I would obviously have to use SUBTREE scope.
(|(distinguishedName=<DN 1 goes here>)(distinguishedName=<DN 2 goes here>))
Edit 1:
I tried this in my test Active Directory which has ~6k entries.
Internal event: A client issued a search operation with the following options.
Starting node:
DC=example,DC=com
Filter:
( |
(distinguishedName=CN=user-1,CN=large-test,CN=Users,DC=example,DC=com)
(distinguishedName=CN=user-2,CN=large-test,CN=Users,DC=example,DC=com)
(distinguishedName=CN=user-3,CN=large-test,CN=Users,DC=example,DC=com)
(distinguishedName=CN=group1,CN=large-test,CN=Groups,DC=example,DC=com)
)
Search scope:
subtree
Attribute selection:
mail,objectClass
Server controls:
Visited entries:
4
Returned entries:
4
Used indexes:
idx_distinguishedName:4:N;idx_distinguishedName:1:N;idx_distinguishedName:1:N;idx_distinguishedName:1:N;
Pages referenced:
123
Pages read from disk:
0
From the results it looks like it only visited 4 entries that I searched for using some indexes. I checked with schema snap-in tool (just to be sure) and it doesn't show indexes on distinguishedName. Not sure how it's using these indexes though.
Microsoft Active Directory stores the group memberships at the entry level, so you could use this to fetch the email attribute.
E.g.
ldapsearch .... -b SEARCH_BASE "(|(memberOf=GROUP_DN_1)(memberOf=GROUP_DN_2)...)" mail
How do we use query configurations while using SQL client in Flink SQL?
The same fashion as mentioned in the link below for
https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/streaming/query_configuration.html
Want to use Idle state retention time.
Flink's SQL client can read a YAML configuration file on startup, and this file can include idle state retention specifications. For example, you might start the client via
sql-client.sh embedded -e sql-client-config.yaml
where the config file contains
execution:
planner: old # optional: either 'old' (default) or 'blink'
type: streaming # required: execution mode either 'batch' or 'streaming'
result-mode: table # required: either 'table' or 'changelog'
max-table-result-rows: 1000000 # optional: maximum number of maintained rows in
# 'table' mode (1000000 by default, smaller 1 means unlimited)
time-characteristic: event-time # optional: 'processing-time' or 'event-time' (default)
parallelism: 1 # optional: Flink's parallelism (1 by default)
periodic-watermarks-interval: 200 # optional: interval for periodic watermarks (200 ms by default)
min-idle-state-retention: 0 # optional: table program's minimum idle state time
max-idle-state-retention: 0 # optional: table program's maximum idle state time
See the docs for more details.
I have a 3 node SolrCloud setup (replication factor 3), running on Ubuntu 14.04 Solr 6.0 on SSDs. Much indexing taking place, only softCommits. After some time, indexing speed becomes really slow, but when i restart the solr service on the node that became slow, everything gets back to normal. Problem is that i need to guess which node becomes slow.
I have 5 collections, but only one collection (mostly used) is getting slow. Total data size is 144G including tlogs.
Said core/collection is 99G including tlogs, tlog is just 313M. Heap size is 16G, Total memory is 32G, data is stored on SSD. Every node is configured the same.
What appears to be strange is that i have literally hundreds or thousands of log lines per second on both slaves when this hits:
2016-09-16 10:00:30.476 INFO (qtp1190524793-46733) [c:mycollection s:shard1 r:core_node2 x:mycollection_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [mycollection_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&distrib.from=http://192.168.0.3:8983/solr/mycollection_shard1_replica3/&wt=javabin&version=2}{add=[ka2PZAqO_ (1545622027473256450)]} 0 0
2016-09-16 10:00:30.477 INFO (qtp1190524793-46767) [c:mycollection s:shard1 r:core_node2 x:mycollection_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [mycollection_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&distrib.from=http://192.168.0.3:8983/solr/mycollection_shard1_replica3/&wt=javabin&version=2}{add=[nlFpoYNt_ (1545622027474305024)]} 0 0
2016-09-16 10:00:30.477 INFO (qtp1190524793-46766) [c:mycollection s:shard1 r:core_node2 x:mycollection_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [mycollection_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&distrib.from=http://192.168.0.3:8983/solr/mycollection_shard1_replica3/&wt=javabin&version=2}{add=[tclMjXH6_ (1545622027474305025), 98OPJ3EJ_ (1545622027476402176)]} 0 0
2016-09-16 10:00:30.478 INFO (qtp1190524793-46668) [c:mycollection s:shard1 r:core_node2 x:mycollection_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [mycollection_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&distrib.from=http://192.168.0.3:8983/solr/mycollection_shard1_replica3/&wt=javabin&version=2}{add=[btceXK4M_ (1545622027475353600)]} 0 0
2016-09-16 10:00:30.479 INFO (qtp1190524793-46799) [c:mycollection s:shard1 r:core_node2 x:mycollection_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [mycollection_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&distrib.from=http://192.168.0.3:8983/solr/mycollection_shard1_replica3/&wt=javabin&version=2}{add=[3ndK3HzB_ (1545622027476402177), riCqrwPE_ (1545622027477450753)]} 0 1
2016-09-16 10:00:30.479 INFO (qtp1190524793-46820) [c:mycollection s:shard1 r:core_node2 x:mycollection_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [mycollection_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&distrib.from=http://192.168.0.3:8983/solr/mycollection_shard1_replica3/&wt=javabin&version=2}{add=[wr5k3mfk_ (1545622027477450752)]} 0 0
In this case 192.168.0.3 is the master.
My workflow is that i insert batches of 2500 docs with ~10 threads at the same time which works perfectly fine for most of the time but sometimes it becomes slow as described. Ocassionally there are updates / indexing calls from other sources, but it's less than a percent.
UPDATE
Complete config (output from Config API) is http://pastebin.com/GtUdGPLG
UPDATE 2
These are the command line args:
-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Dhost=192.168.0.1
-Djetty.home=/opt/solr/server
-Djetty.port=8983
-Dlog4j.configuration=file:/var/solr/log4j.properties
-Dsolr.install.dir=/opt/solr
-Dsolr.solr.home=/var/solr/data
-Duser.timezone=UTC
-DzkClientTimeout=15000
-DzkHost=192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181
-XX:+CMSParallelRemarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:+ParallelRefProcEnabled
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:ConcGCThreads=4
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs
-XX:ParallelGCThreads=4
-XX:PretenureSizeThreshold=64m
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90-Xloggc:/var/solr/logs/solr_gc.log
-Xms16G
-Xmx16G
-Xss256k
-verbose:gc
UPDATE 3
Happened again, these are some Sematext Graphs:
Sematext Dashboard for Master:
Sematext Dashboard for Secondary 1:
Sematext Dashboard for Secondary 2:
Sematext GC for Master:
Sematext GC for Secondary 1:
Sematext GC for Secondary 2:
UPDATE 4 (2018-01-10)
This is a quite old question, but i recently discovered that someone installed a cryptocoin miner on all of my solr machines using CVE-2017-12629 which i fixed with an upgrade to 6.6.2.
If you're not sure if your system is infiltrated check the processes for user solr using ps aux | grep solr. If you see two or more processes, especially a non-java process, you might be running a miner.
So you're seeing disk I/O hitting 100% during indexing with a high-write throughput application.
There are two major drivers of disk I/O with Solr indexing:
Flushing in-memory index segments to disk.
Merging disk segments into new larger segments.
If your indexer isn't directly calling commit as a part of the indexing process (and you should make sure it isn't), Solr will flush index segments to disk based on your current settings:
Every time your RAM buffer fills up ("ramBufferSizeMB":100.0)
Based on your 3 min hard commit policy ("maxTime":180000)
If your indexer isn't directly calling optimize as a part of the indexing process (and you should make sure it isn't), Solr will periodically merge index segments on disk based on your current settings (the default merge policy):
mergeFactor: 10, or roughly each time the number of on-disk index segments exceeds 10.
Based on the way you've described your indexing process:
2500 doc batches per thread x 10 parallel threads
... you could probably get away with a larger RAM buffer, to yield larger initial index segments (that are then flushed to disk less frequently).
However the fact that your indexing process
works perfectly fine for most of the time but sometimes it becomes slow
... makes me wonder if you're just seeing the effect of a large merge happening in the background, and cannibalizing system resources needed for fast indexing at that moment.
Ideas
You could experiment with a larger mergeFactor (e.g. 25). This will reduce the frequency of background index segment merges, but not the resource drain when they happen. (Also, be aware that more index segments often translates to worse query performance).
In the indexConfig, you can try overriding the default settings for the ConcurrentMergeScheduler to throttle the number of merges that can be running at one time (maxMergeCount), and/or throttle the number of threads that can be used for merges (maxThreadCount), based on the system resources you're willing to make available.
You could increase your ramBufferSizeMB. This will reduce the frequency of in-memory index segments being flushed to disk, also serving to slow down the merge cadence.
If you are not relying on Solr for durability, you'll want /var/solr/data pointing to a local SSD volume. If you're going over a network mount (this has been documented with Amazon's EBS), there is a significant write throughput penalty, up to 10x less than writing to ephemeral/local storage.
Do you have the CPU load of each core of the master and not only the combined CPU graph ? What I noticed is when I index with Solr when Xmx is too small (could be the case if you have 144GB data and Xmx=16GB), when the indexing progresses, merging will take more and more time.
During merging, typically one core=100% CPU and other cores do nothing.
Your master combined CPU graph looks like that: only 20% combined load during sequences.
So, check that the merge factor is a reasonable value (between 10 and 20 or something) and potentially raise Xmx.
That's the two things I would play with to start with.
Question: you don't have anything special with your analyzers (custom tokenizer, etc) ?
Is there a way to get a row count (key count) of a single column family in Cassandra? get_count can only be used to get the column count.
For instance, if I have a column family containing users and wanted to get the number of users. How could I do it? Each user is it's own row.
If you are working on a large data set and are okay with a pretty good approximation, I highly recommend using the command:
nodetool --host <hostname> cfstats
This will dump out a list for each column family looking like this:
Column Family: widgets
SSTable count: 11
Space used (live): 4295810363
Space used (total): 4295810363
Number of Keys (estimate): 9709824
Memtable Columns Count: 99008
Memtable Data Size: 150297312
Memtable Switch Count: 434
Read Count: 9716802
Read Latency: 0.036 ms.
Write Count: 9716806
Write Latency: 0.024 ms.
Pending Tasks: 0
Bloom Filter False Postives: 10428
Bloom Filter False Ratio: 1.00000
Bloom Filter Space Used: 18216448
Compacted row minimum size: 771
Compacted row maximum size: 263210
Compacted row mean size: 1634
The "Number of Keys (estimate)" row is a good guess across the cluster and the performance is a lot faster than explicit count approaches.
If you are using an order-preserving partitioner, you can do this with get_range_slice or get_key_range.
If you are not, you will need to store your user ids in a special row.
I found an excellent article on this here.. http://www.planetcassandra.org/blog/post/counting-keys-in-cassandra
select count(*) from cf limit 1000000
Above statement can be used if we have an approximate upper bound known before hand. I found this useful for my case.
[Edit: This answer is out of date as of Cassandra 0.8.1 -- please see the Counters entry in the Cassandra Wiki for the correct way to handle Counter Columns in Cassandra.]
I'm new to Cassandra, but I have messed around a lot with Google's App Engine. If no other solution presents itself, you may consider keeping a separate counter in a platform that supports atomic increment operations like memcached. I know that Cassandra is working on atomic counter increment/decrement functionality, but it's not yet ready for prime time.
I can only post one hyperlink because I'm new, so for progress on counter support see the link in my comment below.
Note that this thread suggests ZooKeeper, memcached, and redis as possible solutions. My personal preference would be memcached.
http://www.mail-archive.com/user#cassandra.apache.org/msg03965.html
There is always map/reduce but that probably goes without saying. If you have that with hive or pig, then you can do it for any table across the cluster though I am not sure tasktrackers know about cassandra locality and so it may have to stream the whole table across the network so you get task trackers on cassandra nodes but the data they receive may be from another cassandra node :(. I would love to hear if anyone knows for sure though.
NOTE: We are setting up map/reduce on cassandra mainly because if we want an index later, we can map/reduce one into cassandra.
I have been getting the counts like this after I convert the data into a hash in PHP.