Not enough memory
While using the Huygens Software you may encounter the following error message that prevents your restoration from going on:
^The operating system did not allow a necessary increase of the size of the image memory pool from 1500 MB to 2300 MB for a 800 MB image with dimensions...(truncated) Failed to allocate heap memory. Not enough memory available.^
Sometimes the solution is something trivial: you may think you are using the 64 bit version of HuygensScripting in a Linux machine but you are actually invoking the 32 bit version, therefore you memory allocation is restricted. Reviewing your scripts with this idea in mind may provide a quick fix.
In any case this is usually not a problem in the application. As the message states, it is the operating system (OS) that did not provide the necessary memory when it was actually needed. Applications like Huygens can not access the memory directly, this is handled and provided by the OS. This problem appears many times with the current Mac OS X and many Windows systems. Details follow.
Deconvolution requirements
This memory problem can be very annoying when you try to restore a 300 MB image and you have for example 4 GB RAM installed. That's more than 13 times more! Why can't the program just do it?
You may think that a 300 MB image is not so large (it is a typical size for a Multi Channel 3D 16 bit image, see Bytes And Bits to estimate file sizes), but to deconvolve such an image many others have to be created for intermediate results, bricks, PSF, all of them in 32 bits, the double of your original 16 bit tiff image. The final restoration result will also be 32 bit if there is enough memory, but it can be even less if Huygens can't allocate the memory.
Huygens Deconvolution runs internally in 32 bit float mode to make use of a wider Dynamic Range, a benefit from deconvolution. In an ideal situation, without any memory restriction, an original 16 bit image of 300 MB would be transformed into a 600 MB 32-bit dataset before starting. Then you need the destination image, which is again 600 MB. And the PSF, as large as the original image. And probably some intermediate state images (for FFT and other calculations), about the same size...
600 × 4 = 2400 MB, more than the half of what you have installed! Maybe your operating system won't allow that allocation: the memory may be being used by other programs, perhaps the OS itself. Some operating systems (like Mac OS X) have shown to be very conservative, and even when the memory is available they won't provide a single program all that much, maybe to allow other programs or other users to still use the machine later. In such a system, just having more memory installed is not a solution: the OS will not serve it anyway!
(Tip: you may split the image and load each channel separately. An application like Huygens Essential will deconvolve them separately anyway, and having the other channels loaded in the memory while deconvolving just one may push things to the memory limits).
You may for example have 8 GB RAM installed in your Mac, but the current MacOS X (before Leopard) is still 32 bit in many aspects, and can not allocate more than 4 GB for a single program if it has a graphical interface. (We have an interfaceless 64 bit Huygens Scripting for MacOS X to circumvent this problem). Still the OS can handle different 32 bit programs running simultaneously until all the memory is used, but that doesn't help each independent program to allocate more than the theoretical maximum of 4 GB.
I say theoretical, because in practice is even less. For example, in a 8 GB RAM Mac G5, you typically have available for the Huygens about 2.5 GB or even less, which is not much at the end, as seen from the previous estimations... Despite of what you can find in some technical documentation , applications can get up to 4 GB of RAM only when demanded in small chunks. For this kind of image restoration we don't need small chunks but at least two large ones (for the original and the result), ideally more (for all the intermediate calculations)!!! The current Mac memory manager doesn't provide them for some reason we don't understand.
Moreover, once an application got a reasonable big chunk, it may not get another one later even after releasing the first one. Maybe the OS thinks "you already got your piece of the tart, you don't deserve more". One solution that we tried was to use an Embedded Memory Manager (EMM), to allocate the biggest memory chunk possible, and handle it internally without never releasing it. This mechanism is still optionally available in Huygens and maybe useful in some circumstances, but most of the problems we get are related with really big images. If the biggest memory chunk provided for the EMM is not large enough, it won't help that much.
More memory
The solution of course is to have more memory, in a real 64 bit OS which provides the programs as much memory as they require. Not every platform is like that, but Huygens has been a 64 bit unix application for a long time now.Windows XP 64 bit and Vista are also examples of 64 bit system that can handle large amounts of memory both in the RAM and in a virtual memory pool in the hard disk. Since version 3.1 Huygens will be also available for these systems. In Windows you can expand your virtual memory, but this is always slower than having physical RAM. The same holds true for the changing the virtual memory of Mac computers.
Save memory
When no more memory can be achieved by any means the alternative solution is to use the available memory in a smart way. Huygens will still do the internal calculations in 32 bit numeric format, but if the memory is not much it won't transform the original image to keep it using less memory, and if necessary also the final result will not be 32 bit despite the loss in Dynamic Range.When this is not enough, the 32 bit internal calculations will run in small chunks, not considering the image as a whole. This is called Brick Splitting. This can be problematic, specially when the Point Spread Function is very large (as in a Wide Field Microscope) and the contribution of every light source affects the whole of the image (the blur spreads over a large volume). If you do not consider all the contributions simultaneously the deconvolution result may not be so good. But if there is no memory, what other thing can be done? The Brick Splitting routine considers many variables to find the best compromise between available memory and deconvolution requirements. There is a limit for that, and if the bricks become so small that the deconvolution will be a disaster, it won't run. Moreover, if your Microscopic Parameters are not physically realistic, this routine can also fail (see Brick Splitting).
System limitations
Still we have seen problems when the OS reports that it has more memory available than what is going to be provided later. Just asking the OS how much free memory is available is not enough if later, when demanded, it will serve less than that!!! The provisions made based on that figure (like Brick Splitting) will fail.Moreover, a typical problem present in some operating systems is that they only allow memory allocation of contiguous memory chunks. The (reported) available memory may be large, but if there's not a contiguous piece of the size you are requesting, it won't be provided. Some versions of Windows are known for suffering from memory fragmentation, that reduce the size of the largest allocatable chunk. In such a case, the message "not enough memory available" actually means "not enough contiguous memory available". To circumvent this issue, the Huygens Embedded Memory Manager may help if the program is run in a freshly started operating system where no much fragmentation had already occurred.
Until these OS do not improve their memory management, there's not much that applications can do. Huygens does its best not to be fooled by these OS (usually this implies being pessimistic with the provided free memory figures, and calculate the brick splitting in a more conservative way) but when large amounts of memory are required we usually suggest using other platforms.
Exporting requirements
Sometimes the memory problem appears when saving your datasets. Why does just saving require extra memory usage? That depends on the File Format you want to use. A format like ICS will store the data directly without further memory demands, but other formats, like TIFF, require the dataset to be transformed, and for that the program needs to create a new destination image to store the transformation result (see Tiff Scaling). Maybe there's not enough memory to create a full third image, even if it is 16 bit!!!In such a case, store the results first in ICS or OME, restart the program, and try reloading and converting the image to the desired format.
More information
For 32 bit Windows systems, you can maybe expand the memory usage a little bit more. See this article.To expand your virtual memory in Windows, see this article.
For IRIX systems, see this article.
