I came to know that there is limit for Silk Test's SetText(). Can we exceed that limit and pass more characters (e.g. 100 000) using SetText()?
Related
does callout limitation depends on the number of times the execute method is invoked in a batch class?
I have read that it depends on the number of callouts per execute method, so we should use batch size of 1 if have to utilize the maximum of 100 callouts, but if we have 25000 records, and the batch size is 1, will it reach the maximum limit for callouts?
In batch job every start,execute and finish get fresh set of governor limits because they're separate transactions. So you get 100 callouts. You still must complete them under 120 seconds - but that's a different limit.
I'm not aware of any limit how many callouts you can make within 24 h so probably there isn't one. There's this limit though
The maximum number of asynchronous Apex method executions (batch Apex,
future methods, Queueable Apex, and scheduled Apex) per a 24-hour
period: 250,000 or the number of user licenses in your org multiplied
by 200, whichever is greater
You will have to balance it all out. If the callouts take 1 second (so you can do all 100) - feel free to set the batch size to 1 and really use 250K execute's. I can't imagine what functionality would require 100 webservice calls to process one record - but in theory you can.
If you need to process more than 250K records daily - well, increase the batch size but then you grand total of possible callouts goes down.
In my program you can book an item. This item has an id with 6 characters from 32 possible characters.
So my possibilities are 32^6. Every id must be unique.
func tryToAddItem {
if !db.contains(generateId()) {
addItem()
} else {
tryToAddItem()
}
}
For example 90% of my ids are used. So the probability that I call tryToAddItem 5 times is 0,9^5 * 100 = 59% isn't it?
So that is quite high. This are 5 database queries on a lot of datas.
When the probability is so high I want to implement a prefix „A-xxxxxx“.
What is a good condition for that? At which time do I will need a prefix?
In my example 90% ids were use. What is about the rest? Do I threw it away?
What is about database performance when I call tryToAddItem 5 times? I could imagine that this is not best practise.
For example 90% of my ids are used. So the probability that I call tryToAddItem 5 times is 0,9^5 * 100 = 59% isn't it?
Not quite. Let's represent the number of call you make with the random variable X, and let's call the probability of an id collision p. You want the probability that you make the call at most five times, or in general at most k times:
P(X≤k) = P(X=1) + P(X=2) + ... + P(X=k)
= (1-p) + (1-p)*p + (1-p)*p^2 +... + (1-p)*p^(k-1)
= (1-p)*(1 + p + p^2 + .. + p^(k-1))
If we expand this out all but two terms cancel and we get:
= 1- p^k
Which we want to be greater than some probability, x:
1 - p^k > x
Or with p in terms of k and x:
p < (1-x)^(1/k)
where you can adjust x and k for your specific needs.
If you want less than a 50% probability of 5 or more calls, then no more than (1-0.5)^(1/5) ≈ 87% of your ids should be taken.
First of all make sure there is an index on the id columns you are looking up. Then I would recommend thinking more in terms of setting a very low probability of a very bad event occurring. For example maybe making 20 calls slows down the database for too long, so we'd like to set the probability of this occurring to <0.1%. Using the formula above we find that no more than 70% of ids should be taken.
But you should also consider alternative solutions. Is remapping all ids to a larger space one time only a possibility?
Or if adding ids with prefixes is not a big deal then you could generate longer ids with prefixes for all new items going forward and not have to worry about collisions.
Thanks for response. I searched for alternatives and want show three possibilities.
First possibility: Create an UpcomingItemIdTable with 200 (more or less) valid itemIds. A task in the background can calculate them every minute (or what you need). So the action tryToAddItem will always get a valid itemId.
Second possibility
Is remapping all ids to a larger space one time only a possibility?
In my case yes. I think for other problems the answer will be: it depends.
Third possibility: Try to generate an itemId and when there is a collision try it again.
Possible collisions handling: Do some test before. Measure the time to generate itemIds when there are already 1000,10.000,100.000,1.000.000 etc. entries in the table. When the tryToAddItem method needs more than 100ms (or what you prefer) then increase your length from 6 to 7,8,9 characters.
Some thoughts
every request must be atomar
create an index on itemId
Disadvantages for long UUIDs in API: See https://zalando.github.io/restful-api-guidelines/#144
less usable, because...
-cannot be memorized and easily communicated by humans
-harder to use in debugging and logging analysis
-less convenient for consumer facing usage
-quite long: readable representation requires 36 characters and comes with higher memory and bandwidth consumption
-not ordered along their creation history and no indication of used id volume
-may be in conflict with additional backward compatibility support of legacy ids
[...]
TLDR: For my case every possibility is working. As so often it depends on the problem. Thanks for input.
I am using Random sorting. For Random sorting I am providing seed at run time which is random value from 1001 to 999999.
And the results I am getting are random but not properly distributed. Lets there are 5 results, then in 100 run, approx 20 times each item should come to top. But it is not happening,. One item is coming at top approx 80 times. How should I use random ordering so as the order is fairly distributed.
Also How Random sorting acutally works. How does seed play role? I mean using this seed, solr generates order. So how does it actually does it.? What algo it uses? How can I smartly change random seed?
It depends on number of documents?
I have to store some attributes in DynamoDB and confused if some of JSON attributes should be stored as String/Binary. I understand that storing it as binary will reduce the size of attribute.
I considered DDB limits as 1 Read/Write IOPS consumes 4KB.
My total data in item is less than 4KB even if I store it as String.
What things should I consider to choose binary vs String ?
Thanks.
Given that your item sizes are less than 4KB uncompressed, whether to encode attributes in byte or string depends on whether the attribute will be a partition / range key of the table and your typical read patterns.
A partition key has a max sz of 2048 bytes, or ~2Kb.
A sort key (if you specify one on the table) has a max sz of 1024 bytes, or ~1Kb.
If you foresee your string attribute exceeding the above maximums on any items, it would make sense to compress to binary first to keep your attribute sizes in congruence with DynamoDB requirements.
Depending on how many items are in your typical query and your tolerance for throttled queries, your RCU's may not satisfy a Query / Scan where you perform the read in a single request.
For instance,
If you have 1KB items and want to query 100 items in a single request, your RCU req will be as follows:
(100 * 1024 bytes = 100 KB) / 4 KB = 25 read capacity units
Converting some attributes to binary could reduce your RCU requirement in this case. Again it largely depends on your typical usage pattern.
See http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html#HowItWorks.ProvisionedThroughput.Reads
I want to use GUID's (uuid) for naming folders in a huge file store. Each storage item gets his own folder and guid.
The easiest way would be "x:\items\uuid\{uuid}..."
example: "x:\items\uuid\F3B16318-4236-4E45-92B3-3C2C3F31D44F..."
I see here one problem. What if you expect to get at least 10.000 items and probably a few 100.000 or more then 1 million. I don't want to put so many items (sub folders) in one folder.
I thought to solve this by splitting up the guid. Taking the 2 first chars to create sub folders at the first level and the take the next 2 chars and also create sub folders.
The above example would be --> "x:\items\uuid\F3\B1\6318-4236-4E45-92B3-3C2C3F31D44F..."
If the first 4 chars of guid's are really as random as expected then I get after a while 256 folder within 256 folders and I always end up with a reasonable amount of items within each of these folders
For example if you have 1 million items then you get --> 1 000 000 / 256 /256 = 15.25 items per folder
In the past I'v already tested the randomness of the first chars. (via vb.net app). Result: The items where spread quit evenly over the folders.
Also somebody else came to the same conclusion. see How evenly spread are the first four bytes of a Guid created in .NET?
Possible splits I thought of (1 million items as example)
C1 = character 1 of GUID, C2 = character 2, etc
C1\C2\Rest of GUID --> 16 * 16 * 3906 (almost 4000 are still al lot of folders)
C1\C2\C3\C4\Rest of Guid --> 16 * 16 * 16 * 16 * 15 ( unnecessary splitting up of folders)
C1C2\C3C4\Rest of Guid --> 256 * 256 * 15 (for me the best option ?)
C1C2C3\Rest of Guid --> 4096 * 244 (to many folders at first level??)
C1C2C3C4\Rest of Guid --> 65536 * 15 (to many folders at first level!)
My questions are:
Does anyone see drawbacks for this kind of implementation. (scheme: *C1C2\C3C4\Rest of Guid)
Is there some standard for splitting up Guids, or a general way of doing this.
What happens if you put a few 100 thousands of sub folders in one folder (I still prefer not to use any splitting if possible)
Thanks, Mumblic
This is fairly similar to the method git uses for sharding it's object database (although with SHA1 hashes instead of GUIDs...). As with any algorithm, there are pros and cons, but I don't think there are any significant cons in this case that would outweigh the definite pros. There's a little extra CPU overhead to calculate the directory structure, but in the long term, that overhead is probably significantly less than what is necessary to search through a single directory of a million files repeatedly.
Regarding how to do it, it depends a bit on what library you are using to generate the GUIDs - do you get them in a byte-array (or even a struct) format that then needs to be converted to a character representation in order to display it, or do you get them in an already formatted ASCII array? In the first case, you need to extract the appropriate bytes and format them yourself, in the second you just need to extract a substring.
As far as putting an extreme number of sub-folders (or even files) in one folder, the exact performance characteristics are highly dependent on the actual file system in use. Some perform better than others, but almost all will show significant performance degradation the more entries each directory has.