I am trying to load a 3776 * 2816 PNG, 24 bit image - 804KB onto my phone , the MOTO ROKR e6.It gives up with java.lang.OutOfMemoryError,Is their a general way to handle loading such high resolution images.The phone's internal memory is only 8MB, I think this has something to do with the error.
I have also, tried to split the image to 16 parts and load them, still there seems to be some limit on what it can handle.
Please advise.
So just some quick calculations:
24 bits = 3 bytes
space required (in bytes) = 3776 * 2816 * 3
= 31,899,648 bytes
= 31.9MB
That means once you've loaded the image (using ImageIO or JAI or whatever) you need 31.9MB to store the raw image data. As a result you can't load it on a device with only 8MB of memory (and I'm assuming no other kind of swap space).
You could load the raw file as bytes of data rather than an image -- the data is heavily compressed -- but I don't think that's what you're looking for.
Related
I have a BFF service that reaches a microservice, and the second one pulls a long list from the DB, formats it and return the list to the BFF.
When I'm trying to run it through AppEngine I receive the following error:
Exceeded hard memory limit of 256 MB with XXX MB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.
Where XXX is a different number each time, starting from 266 MB.
I tried to stop using pydantic (since it takes a lot of memory), to scale the instance to a huge machine, but the problem remains.
So I've copied the response (as I can run it locally) and copied it into the BFF (== skipping the whole microservice logic and store the response as a constant dictionary in the BFF).
And then, when the BFF has no logic besides loading a constant variable, I've received again the following error:
Exceeded hard memory limit of 256 MB with 919 MB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.
The file that contains the data is a 9 MB file, the response that we create is around 3 MB, but it seems that the AppEngine can't really handle loading this dictionary to its memory in the BFF as well.
As there is no memory-profiling tool for AppEngine I'm not really sure what DOES take the memory and how can I make it work, any ideas?
Thank you!
Apparently in python the dictionary size is also filled with metadata, and when the dictionary is big and has a complicated hierarchy - the size might raise exponentially.
That was the reason why a 9 MB data became a over 250 MB object in runtime.
MSDN says here: msdn maximums that max datafile size is "16 Terabytes" - not sure if their definition of terabyte is 1024^4 or 1000^4 - so valid max page number might be 2,147,483,648 (for 1024 basis) or 1,953,125,000 (for 1000 basis) or perhaps something else - does anyone know with certainty?
I have heard that this limit should be increasing with future releases - right now I'm using 2012.
Yes it is based on 1024 which is a kilobyte. Multiply that by 1024 and you get a megabyte. And so on.
Your also correct that newer versions have larger maximums.
The minimum unit required for storing any type of data in SQL Server is a Page, a page is 8 KB in size.i.e exactly 8192 Bytes , pages are stored in logical Extents.
Page Header
Yet not all of the 8192 Bytes is available for data storage, some of the space from 8192 Bytes is used to store information about the page itself. It is called Page Header and it is about 96 Bytes.
Row Set
This is another section on the page containing information about the rows on that page, it begins at the end of the page taking another 36 Bytes from the total page size 8192 Bytes.
Total Space Available for Data Storage
8912 Total space on a page
- 96 Space taken by the page header
- 36 Space taken by the Row set
----------------------------------------------
8060 Total Space Available for Data Storage
So if you are trying to calculate the amount of data you will be able to store in a database and especially when you are talking in Terabytes, dont forget to take Page header and row set into consideration.
Im trying to make an 2D online game (with Z positions), and currently im working with loading a map from a txt file. I have three different map files. One contains an int for each tile saying what kind of floor there is, one saying what kind of decoration there is, and one saying what might be covering the tile. The problem is that the current map (20, 20, 30) takes 200 ms to load, and I want it to be much much bigger. I have tried to find a good solution for this and have so far come up with some ideas.
Recently I'v thought about storing all tiles in separate files, one file per tile. I'm not sure if this is a good idea (it feels wrong somehow), but it would mean that I wouldn't have to store any unneccessary tiles as "-1" in a text file and I would be able to just pick the right tile from the folder easily during run time (read the file named mapXYZ). If the tile is empty I would just be able to catch the FileNotFoundException. Could anyone tell me a reason for this being a bad solution? Other solutions I'v thought about would be to split the map into smaller parts or reading the map during startup in a BackgroundWorker.
Try making a much larger map in the same format as your current one first - it may be that the 200ms is mostly just overhead of opening and initial processing of the file.
If I'm understanding your proposed solution (opening one file per X,Y or X,Y,Z coordinate of a single map), this is a bad idea for two reasons:
There will be significant overhead to opening so many files.
Catching a FileNotFoundException and eating it will be significantly slower - there is actually a lot of overhead with catching exceptions, so you shouldn't rely on them to perform application logic.
Are you loading the file from a remote server? If so, that's why it's taking so long. Instead you should embed the file into the game. I'm saying this because you probably take 2-3 bytes per tile, so the file's about 30kb and 200ms sounds like a reasonable download time for that size of file (including overhead etc, and depending on your internet connection).
Regarding how to lower the filesize - there are two easy techniques I can think of that will decrease the filesize a bit:
1) If you have mostly empty squares and only some significant ones, your map is what is often referred to as 'sparse'. When storing a sparse array of data you can use a simple compression technique (formally known as 'run-length encoding') where each time you come accross empty squares, you specify how many of them there are. So for example instead of {0,0,0,0,0,0,0,0,0,0,1,1,2,3,0,0,0,0,0,0,0,0,0,0,0,0,1} you could store {10 0's, 1, 1, 2, 3, 12 0's, 1}
2) To save space, I recommend that you store everything as binary data. The exact setup of the file mainly depends on how many possible tile types there are, but this is a better solution than storing the ascii characters corresponding to the base-10 representation of the numers, separated by delimiters.
Example Binary Format
File is organized into segments which are 3 or 4 bytes long, as explained below.
First segment indicates the version of the game for which the map was created. 3 bytes long.
Segments 2, 3, and 4 indicate the dimensions of the map (x, y, z). 3 bytes long each.
The remaining segments all indicate either a tile number and is 3 bytes long with an MSB of 0. The exception to this follows.
If one of the tile segments is an empty tile, it is 4 bytes long with an MSB of 1, and indicates the number of empty tiles including that tile that follow.
The reason I suggest the MSB flag is so that you can distinguish between segments which are for tiles, and segments which indicate the number of empty tiles which follow that segment. For those segments I increase the length to 4 bytes (you might want to make it 5) so that you can store larger numbers of empty tiles per segment.
Over the connections that most people in the USA have in their homes, what is the approximate length of time to send a list of 200,000 integers from a client's browser to an internet sever (say Google app engine)? Does it change much if the data is sent from an iPhone?
How does the length of time increase as the size of the integer list increases (say with a list of a million integers) ?
Context: I wasn't sure if I should write code to do some simple computations and sorting of such lists for the browser in javascript or for the server in python, so I wanted to explore this issue of how long it takes to send the output data from a browser to a server over the web in order to help me decide where (client's browser or app engine server) is the best place for such computations to be processed.
More Context:
Type of Integers: I am dealing with 2 lists of integers. One is a list of ids for the 200,000 objects whose integers look like {0,1,2,3,...,99,999}. The second list of 100,000 is just single digits {...,4,5,6,7,8,9,0,1,...} .
Type of Computations: From the browser a person will create her own custom index (or rankings) based changing the weights associated to about 10 variables referenced to the 100,000 objects. INDEX = w1*Var1 + w2*Var2 + ... wNVarN. So the computations refer to vector (array) multiplication to a scalar and addition of 2 vectors, as well as sorting the final INDEX variable vector of 100,000 values.
In a nutshell...
This is probably a bad idea,
in particular with/for mobile devices where, aside from the delay associated with transfer(s), limits and/or extra fees associated with monthly volumes exceeding various plans limits make this a lousy economical option...
A rough estimate (more info below) is that the one-way transmission takes between 0.7 and and 5 seconds.
There is a lot of variability in this estimate, due mainly to two factors
Network technology and plan
compression ratio which can be obtained for a 200k integers.
Since the network characteristics are more or less a given, the most significant improvement would come from the compression ratio. This in turn depends greatly on the statistic distribution of the 200,000 integers. For example, if most of them are smaller than say 65,000, it would be quite likely that the list would compress to about 25% of its original size (75% size reduction). The time estimates provided assumed only a 25 to 50% size reduction.
Another network consideration is the availability of binary mime extension (8 bits mime) which would avoid the 33% overhead of B64 for example.
Other considerations / idea:
This type of network usage for iPhone / mobile devices plans will not fare very well!!!
ATT will love you (maybe), your end-users will hate you at least the ones with plan limits, which many (most?) have.
Rather than sending one big list, you could split the list over 3 or 4 chunks, allowing the server-side sorting to take place [mostly] in parallel to the data transfer.
One gets better compression ratio for integers when they are [roughly] sorted, maybe you can have a first pass sorting of some kind client-side.
How do I figure? ...
1) Amount of data to transfer (one-way)
200,000 integers
= 800,000 bytes (assumes 4 bytes integers)
= 400,000 to 600,000 bytes compressed (you'll want to compress!)
= 533,000 to 800,000 bytes in B64 format for MIME encoding
2) Time to upload (varies greatly...)
Low-end home setup (ADSL) = 3 to 5 seconds
broadband (eg DOCSIS) = 0.7 to 1 second
iPhone = 0.7 to 5 seconds possibly worse;
possibly a bit better with high-end plan
3) Time to download (back from server, once list is sorted)
Assume same or slightly less than upload time.
With portable devices, the differential is more notable.
The question is unclear of what would have to be done with the resulting
(sorted) array; so I didn't worry to much about the "return trip".
==> Multiply by 2 (or 1.8) for a safe estimate of a round trip, or inquire
about specific network/technlogy.
By default, typically integers are stored in a 32-bit value, or 4 bytes. 200,000 integers would then be 800,000 bytes, or 781.25 kilobytes. It would depend on the client's upload speed, but at 640Kbps upload, that's about 10 seconds.
well that is 800000 bytes or 781.3 kb, or you could say the size of a normal jpeg photo. for broadband, that would be within seconds, and you could always consider compression (there are libraries for this)
the time increases linearly for data.
Since you're sending the data from JavaScript to the server, you'll be using a text representation. The size will depend a lot on the number of digits in each integer. Are talking about 200,000 two to three digit integers or six to eight integers? It also depends on if HTTP compression is enabled and if Safari on the iPhone supports it (I'm not sure).
The amount of time will be linear depending on the size. Typical upload speeds on an iPhone will vary a lot depending on if the user is on a business wifi, public wifi, home wifi, 3G, or Edge network.
If you're so dependent on performance perhaps this is more appropriate for a native app than an HTML app. Even if you don't do the calculations on the client, you can send/receive binary data and compress it which will reduce time.
I assume 100 bytes is too small and can slow down larger file transfers with all of the writes, but something like 1MB seems like it may be too much. Does anyone have any suggestions for an optimal chunk of bytes per write for sending data over a network?
To elaborate a bit more, I'm implementing something that sends data over a network connection and show the progress of that data being sent. I've noticed if I send large files at about 100 bytes each write, it is extremely slow but the progress bar works out very nicely. However, if I send at say 1M per write, it is much faster, but the progress bar doesn't work as nicely due to the larger chunks being sent.
No, there is no universal optimal byte size.
TCP packets are subject to fragmentation, and while it would be nice to assume that everything from here to your destination is true ethernet with huge packet sizes, the reality is even if you can get the packet sizes of all the individual networks one of your packets takes, each packet you send out may take a different path through the internet.
It's not a problem you can "solve" and there's no universal ideal size.
Feed the data to the OS and TCP/IP stack as quickly as you can, and it'll dynamically adapt the packet size to the network connections (you should see the code they use for this optimization - it's really, really interesting. At least on the better stacks.)
If you control all the networks and stacks being used and inbetween your clients/servers, though, then you can do some hand tuning. But generally even then you'd have to have a really good grasp of the network and the data your sending before I'd suggest you approach it.
-Adam
If you can, just let the IP stack handle it; most OSes have a lot of optimization already built in. Vista, for example, will dynamically alter various parameters to maximize throughput; second-guessing the algorithm is unlikely to be beneficial.
This is especially true in higher-order languages, far from the actual wire, like C#; there are enough layers between you and actual TCP/IP packets that I would expect your code to have relatively little impact on throughput.
At worst, test various message sizes in various situations for yourself; few solutions are one-size-fits-all.
If you are using TCP/IP over Ethernet, the maximum packet size is about 1500 bytes. If you try to send more than that at once, the data will be split up into multiple packets before being sent out on the wire. If the data in your application is already packetized, then you might want to choose a packet size of just under 1500 so that when you send a full packet, the underlying stack doesn't have to break it up. For example, if every send you do is 1600 bytes, the TCP stack will have to send out two packets for each send, with the second packet being mostly empty. This is rather inefficient.
Having said that, I don't know much of a visible impact on performance this will have.
Make a function named CalcChunkSize
Add some private variables to your class:
Private PreferredTransferDuration As Integer = 1800 ' milliseconds, the timespan the class will attempt to achieve for each chunk, to give responsive feedback on the progress bar.
Private ChunkSizeSampleInterval As Integer = 15 ' interval to update the chunk size, used in conjunction with AutoSetChunkSize.
Private ChunkSize As Integer = 16 * 1024 ' 16k by default
Private StartTime As DateTime
Private MaxRequestLength As Long = 4096 ' default, this is updated so that the transfer class knows how much the server will accept
Before every download of a chunk, check if its time to calculate new chunksize using the ChunkSizeSampleInterval
Dim currentIntervalMod As Integer = numIterations Mod Me.ChunkSizeSampleInterval
If currentIntervalMod = 0 Then
Me.StartTime = DateTime.Now
ElseIf currentIntervalMod = 1 Then
Me.CalcChunkSize()
End If
numIterations is set to 0 outside the download-loop and after every downloaded chunk set to numIterations += 1
Have the CalcChunkSize doing this:
Protected Sub CalcAndSetChunkSize()
' chunk size calculation is defined as follows
' * in the examples below, the preferred transfer time is 1500ms, taking one sample.
' *
' * Example 1 Example 2
' * Initial size = 16384 bytes (16k) 16384
' * Transfer time for 1 chunk = 800ms 2000 ms
' * Average throughput / ms = 16384b / 800ms = 20.48 b/ms 16384 / 2000 = 8.192 b/ms
' * How many bytes in 1500ms? = 20.48 * 1500 = 30720 bytes 8.192 * 1500 = 12228 bytes
' * New chunksize = 30720 bytes (speed up) 12228 bytes (slow down from original chunk size)
'
Dim transferTime As Double = DateTime.Now.Subtract(Me.StartTime).TotalMilliseconds
Dim averageBytesPerMilliSec As Double = Me.ChunkSize / transferTime
Dim preferredChunkSize As Double = averageBytesPerMilliSec * Me.PreferredTransferDuration
Me.ChunkSize = CInt(Math.Min(Me.MaxRequestLength, Math.Max(4 * 1024, preferredChunkSize)))
' set the chunk size so that it takes 1500ms per chunk (estimate), not less than 4Kb and not greater than 4mb // (note 4096Kb sometimes causes problems, probably due to the IIS max request size limit, choosing a slightly smaller max size of 4 million bytes seems to work nicely)
End Sub
Then just use the ChunkSize when requesting next chunk.
I found this in the "Sending files in chunks with MTOM web services and .Net 2.0" by Tim_mackey and have found it very useful myself to dynamically calculate most effective chunksize.
The source code in whole are here: http://www.codeproject.com/KB/XML/MTOMWebServices.aspx
And author here: http://www.codeproject.com/script/Membership/Profiles.aspx?mid=321767
I believe your problem is that you use blocking sockets and not non-blocking ones.
When you use blocking sockets and you send 1M of data the network stack can wait for all of the data to be placed in a buffer, if the buffers are full you'll be blocked and your progress bar will wait for the whole 1M to be accepted into the buffers, this may take a while and your progress bar will be jumpy.
If however you use non-blocking sockets, whatever buffer size you use will not block, and you will need to do the waiting yourself with select/poll/epoll/whatever-works-on-your-platform (select is the most portable though). In this way your progress bar will be updated fast and reflect the most accurate information.
Do note that at the sender the progress bar is partially broken any way since the kernel will buffer some data and you will reach 100% before the other side really received the data. The only way around this is if your protocol includes a reply on the amount of data received by the receiver.
As others said, second guessing the OS and the network is mostly futile, if you keep using blocking sockets pick a size that is large enough to include more data than a single packet so that you don't send too little data in a packet as this will reduce your throughput needlessly. I'd go with something like 4K to include at least two packets at a time.
The one thing I will add is that for a given ethernet connection it takes about as long to send a small packet as a large one. As other's have said: if you're just sending a stream of data let the system handle it. But if you're worried about individual short messages back and forth, a typical ethernet packet is about 1500 bytes- as long as you keep it under that you should be good.
You'd need to use Path MTU Discovery, or use a good default value (ie less than 1500 bytes).
One empirical test you can do, if you haven't already, is of course to use a sniffer (tcpdump, Wireshark etc.) and look at what packet sizes are achieved when using other software for up/downloading. That might give you a hint.
Here is the formula you need:
int optimalChunkSize = totalDataSize / progressBar1.Width;
Using this, each chunk you send will increment the progress bar by 1 pixel. A smaller chunk size than this is pointless, in terms of user feedback.