Introduction
In this post, we will discuss the memory management in Intella and why setting optimal settings for every hardware and software configuration is not possible. The post explains how memory is used for various components in Intella and how it is more complicated than just moving the slider to the maximum.
There is also a video covering memory management:
Intella uses memory in three different processes:
- Case Manager
- Intella main process
- Crawlers
These processes automatically assign memory by Intella. However, the amount of memory used depends on the number and complexity of the artifacts and items processed, the amount of RAM available on the machine where Intella runs, and other factors.
Before Version 2.3
In earlier versions of Intella and Connect, users needed to edit the Intella.l4j.ini file located in the installation directory to modify the memory and crawler settings. However, the amount of memory used depends on several factors such as the number and complexity of artifacts and items processed and the available RAM on the machine where Intella runs. Hence, there is no optimal setting for every hardware and software configuration.
After Version 2.3
From version 2.3 onwards, Intella and Connect have added more memory controls in their user interfaces. With this update, users are able to modify all memory and crawler settings from within the user interface, eliminating the previous requirement of editing the Intella.l4j.ini file.
By modifying the settings, users can allocate more memory to the processes in Intella and adjust the crawler settings to match their hardware and software configurations. The changes made through the user interface are instant and do not require a restart of Intella.
Memory Setting for Case Manager
The Case Manager memory setting allocates memory to the Case Manager process only, which doesn't affect the processing of data. By default, this setting is fixed to 256MB, and it is typically unnecessary to change it. Hence, the user interface does not provide any controls to adjust this setting. However, if needed, users can manually change this setting. The only reason for changing this setting is for exporting and importing cases. In some rare cases, increasing the Case Manager memory may be needed to export a case to an ICF file. Nevertheless, it is usually unnecessary to change this setting.
The Case Manager process only remains alive for a few seconds. After selecting a case and clicking the 'Open' button, the process is terminated.
To change the case manager memory, edit the
Intella.l4j.ini file located in the c:\Program Files\Vound\Intella X.Y.Z folder. Change the line that starts with -Xmx. For example, to set the case manager memory to 5gb use this:
-Xmx5g
Memory Setting for Main Process
When a case is opened, the Case Manager starts the Main process that controls everything visible on Intella, except for indexing and exporting. The Main process typically requires the most memory out of all processes in Intella, and that's why a memory slider is added to the Case Editor window. This feature allows users to adjust the memory setting. Below is a table that shows the default memory allocation by Intella based on the amount of RAM in the system:
However, the amount of memory needed may vary based on various factors such as the number and size of the items being processed. Therefore, the slider in Intella allows users to manually adjust memory allocation for the Main process depending on their hardware configuration and processing requirements.
Memory Setting for Crawlers and Exporting Processes
The memory setting for Crawler processes is calculated automatically by Intella based on the amount of RAM available minus the memory allocated for the Main process and the number of crawlers to be used. By default, Intella calculates the number of crawlers based on the number of CPU cores in the system. However, the number of crawlers is capped at four as assigning more crawlers without other considerations can adversely affect performance.
It's worth noting that the crawler settings also control other processes, including exporting, PDF conversion in the Preview tab, TIFF to PDF conversion for Load File import, text extraction for OCR import, and Outlook and Notes validation.
While the job of the Crawler is to merely extract and collect information, they don't directly index the data. Instead, indexing takes place later in the post-processing steps, which happens in the Main process.
When do I optimize for better performance?
The amount of memory per crawler is capped at a maximum of 2GB per crawler when set automatically by Intella. Generally, it is unnecessary to change this setting. However, users can manually adjust the crawler memory setting if needed, such as to optimize processing requirements or hardware configurations.
Optimizing Memory and Crawler Settings for Better Performance
To get the best performance out of hardware resources when working with Intella, it's essential to optimize memory and crawler settings. However, it's worth noting that assigning all available memory to Intella processes manually can lead to an unstable system. It's recommended not to assign more than 75% of the total memory to Intella processes. In the last section of this post, an example is given where approximately 50% of the system's total memory is assigned to Intella processes.
It's also essential to consider the processes that will be performed on Intella when assigning memory. For instance, when indexing a data source, a significant portion of the memory needs to be allocated to the crawler processes, particularly when multiple crawlers are used on high-end machines. During the crawling phase, the Main process requires very little memory. However, for post-processing phases, the Main process requires considerably more memory.
When investigating an already indexed case, almost all available memory should be assigned to the Main process, proportionate to the case size. In other words, there is no need to assign 128GB memory to a 40GB case. Memory is assigned automatically to different processes in Intella based on hardware resources, but the user can manually adjust these memory and crawler settings to better suit their hardware specifications and the dataset being indexed.
The best memory and crawler settings to use are not straightforward and depend on the type of data being indexed. For instance, indexing a single large PST file may not show a significant difference in performance when allocating more memory and crawlers manually. In such cases, default memory settings and the number of crawlers may be enough to achieve the best performance. However, datasets containing a lot of loose files or disk images can benefit from an increased number of crawlers for better performance. Increasing the memory assigned to crawlers can also help to resolve out-of-memory errors or crawler crashes with specific large items.
In summary, when indexing, increasing the number of crawlers for datasets containing a large number of loose files can provide better performance. Additionally, increased memory assigned to crawlers can help with out-of-memory or crawler crash errors, particularly with larger items. When importing OCR text, increasing the number of crawlers can also increase performance. When investigating a case, almost all available memory should be assigned to the Main process, and assigning more crawler memory can improve performance when exporting to formats such as PDF or PST.
Steps to Optimize Performance
Intella's configuration settings depend on several factors, such as hardware specifications and data types. It's challenging to specify settings that would work optimally for all scenarios. Therefore, the instructions given below are recommendations that can be adjusted to suit specific use cases.
Changing the Main Process Memory
- Open the Case Editor window for the specific case.
- Adjust the slider to set the desired memory allocation for the Main process.
- The slider allows using up to half the memory in the system. This is for safeguarding against system instability due to insufficient memory assigned to Windows operations.
Changing the Number of Crawlers
For Intella/Connect version 2.3.x and up:
- Open the Case Editor window for the specific case.
- Click on the Advanced button to display advanced settings.
- Change the Crawler count option from Auto to Manual.
- Set the desired number of crawlers for the case.
For Intella/Connect version 2.2.x or lower:
- Close Intella.
- Manually edit the Intella.l4j.ini file located in the install directory.
- Look for the line '# -Dintella.crawlersCount=4.'
- Remove the leading '#' to enable the line of code.
- Change the crawlersCount to the desired value (e.g., 6).
- Save the file. Note that this only specifies an upper limit, and the actual number of crawlers used by Intella will depend on the evidence data.
Increasing the Memory for Each Crawler
For Intella/Connect version 2.3.x and up:
- Open the Case Editor window of the specific case.
- Click on the Advanced button to display advanced settings.
- Change the Service memory allocation option from Auto to Manual.
- Use the slider to set the desired amount of memory for each crawler.
Intella:
For Intella/Connect version 2.2.x or lower:
- Close Intella.
- Manually edit the Intella.l4j.ini file located in the install directory.
- Look for the line '# -Dintella.serviceMaxHeap=800M.'
- Remove the leading '#' to enable the line of code.
- Change the serviceMaxHeap to the desired value (e.g., 2g).
- Save the file. Note that Intella will use 2GB of memory for each crawler when indexing or exporting data in this case.
When modifying memory and crawler settings, make sure to allocate sufficient memory for all crawlers, the Main process, and other applications running on your computer system. Assigning too much memory to different Intella processes can lead to performance issues, such as reducing the number of crawlers, which in turn can make Intella indexing a single-thread process.
Intella can use memory that is not directly assigned to it. The Windows file system must have sufficient free memory available for caching evidence files and case index files. Assigning too much memory to different Intella processes can starve the operating system of memory.
For best performance, it's recommended to allocate memory based on hardware specifications and data types. For instance, when indexing a 500GB E01 disk image containing many heavy text documents using a 12-core machine with 128GB of RAM, allocating the main process memory to 16GB, setting the number of crawlers between 4-8, and assigning crawler memory (service memory allocation) to 8GB will provide the best performance.
Conclusion
In conclusion, managing memory settings in Intella requires careful consideration of hardware specifications, data types, and various processing requirements. While Intella automatically assigns memory to different processes, it's crucial to optimize memory and crawler settings to get the best performance outcomes. The instructions provided in this post are recommendations that can be adjusted to suit specific use cases. However, assigning too much memory to different Intella processes can lead to performance issues. Therefore, it's crucial to allocate memory based on hardware specifications and data types while factoring in the needs of other applications running on the system. Designating optimal memory and crawler settings is essential for achieving maximum performance when working with Intella.
Visit our forum post to see customer questions and answers related to this topic.
Updated April 2023